src/ObjectVisitor.h
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 188 bab29bcec7fe
child 209 5cbec15fadfe
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
     1 /* vim: set ts=4 sw=4 tw=99 et:
     2  *
     3  * Copyright (C) 2012 David Anderson
     4  *
     5  * This file is part of JITCraft.
     6  *
     7  * JITCraft is free software: you can redistribute it and/or modify it under
     8  * the terms of the GNU General Public License as published by the Free
     9  * Software Foundation, either version 3 of the License, or (at your option)
    10  * any later version.
    11  * 
    12  * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
    13  * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    14  * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
    15  *
    16  * You should have received a copy of the GNU General Public License along with
    17  * JITCraft. If not, see http://www.gnu.org/licenses/.
    18  */
    19 #ifndef _include_jc_object_visitor_h_
    20 #define _include_jc_object_visitor_h_
    21 
    22 #include "Objects.h"
    23 #include "Types.h"
    24 #include "FixedArray.h"
    25 #include "Code.h"
    26 #include "Reference.h"
    27 #include "Modules.h"
    28 #include "Functions.h"
    29 #include "Array.h"
    30 #include "Strings.h"
    31 #include "Environments.h"
    32 #include "Structures.h"
    33 #include "Box.h"
    34 
    35 namespace ke {
    36 
    37 // An ObjectVisitor is used for traversing object references in the heap. Since
    38 // this pattern must be lightning fast, we avoid virtual dispatch by forcing
    39 // visitors to compose themselves with the visiting logic.
    40 template <typename T>
    41 class ObjectVisitor : public T
    42 {
    43   private:
    44     void visitField(Object *obj, size_t offset) {
    45         this->visit(reinterpret_cast<Object **>(Address(obj) + offset));
    46     }
    47 
    48     size_t iterateType(Object *obj) {
    49         visitField(obj, Type::offsetOfContained());
    50         visitField(obj, Type::offsetOfName());
    51         return sizeof(Type);
    52     }
    53 
    54     size_t iterateStructType(Object *obj) {
    55         visitField(obj, Type::offsetOfContained());
    56         visitField(obj, Type::offsetOfName());
    57         visitField(obj, StructType::offsetOfFields());
    58         visitField(obj, StructType::offsetOfNewMap());
    59         return sizeof(StructType);
    60     }
    61 
    62     size_t iterateFunctionType(Object *obj) {
    63         visitField(obj, Type::offsetOfContained());
    64         visitField(obj, Type::offsetOfName());
    65         visitField(obj, FunctionType::offsetOfReturnType());
    66         visitField(obj, FunctionType::offsetOfParameters());      
    67         visitField(obj, FunctionType::offsetOfDefaults());
    68         return sizeof(FunctionType);
    69     }
    70 
    71     size_t iterateFixedArray(FixedArray *array) {
    72         unsigned length = array->length();
    73         Object **ptr = reinterpret_cast<Object **>(array + 1);
    74         for (unsigned i = 0; i < length; i++)
    75             this->visit(ptr + i);
    76         return array->objSize();
    77     }
    78 
    79     size_t iterateByteArray(ByteArray *array) {
    80         return array->objSize();
    81     }
    82 
    83     size_t iterateCode(Object *obj) {
    84         visitField(obj, Code::offsetOfModule());
    85         visitField(obj, Code::offsetOfBytecode());
    86         visitField(obj, Code::offsetOfLocals());
    87         visitField(obj, Code::offsetOfObjects());
    88         visitField(obj, Code::offsetOfSafepoints());
    89         visitField(obj, Code::offsetOfSourceMap());
    90         visitField(obj, Code::offsetOfStrings());
    91         return sizeof(Code);
    92     }
    93 
    94     size_t iterateReference(Object *obj) {
    95         visitField(obj, Reference::offsetOfBase());
    96         return sizeof(Reference);
    97     }
    98 
    99     size_t iterateModule(Object *obj) {
   100         visitField(obj, Module::offsetOfFile());
   101         visitField(obj, Module::offsetOfGlobals());
   102         visitField(obj, Module::offsetOfCode());
   103         return sizeof(Module);
   104     }
   105 
   106     size_t iterateFunction(Object *obj) {
   107         visitField(obj, Function::offsetOfCode());
   108         visitField(obj, Function::offsetOfType());
   109         visitField(obj, Function::offsetOfName());
   110         return sizeof(Function);
   111     }
   112 
   113     size_t iterateDescriptor(Object *obj) {
   114         visitField(obj, Descriptor::offsetOfName());
   115         visitField(obj, Descriptor::offsetOfType());
   116         return sizeof(Descriptor);
   117     }
   118 
   119     size_t iterateNative(Object *obj) {
   120         visitField(obj, Native::offsetOfType());
   121         return sizeof(Native);
   122     }
   123 
   124     size_t iterateString(Object *obj) {
   125         return String::cast(obj)->objSize();
   126     }
   127 
   128     size_t iterateArray(Array *array, ArrayMap *map) {
   129         if (!map->type()->fixedLength() || map->type()->contained()->isTraceable())
   130             visitField(array, Array::offsetOfElements());
   131         return array->objSize(map);
   132     }
   133 
   134     size_t iterateBox(Box *obj) {
   135         visitField(obj, Box::offsetOfType());
   136         if (obj->type()->isTraceable())
   137             visitField(obj, Box::offsetOfObj());
   138         return sizeof(Box);
   139     }
   140 
   141     size_t iterateArrayMap(ArrayMap *map) {
   142         visitField(map, ArrayMap::offsetOfType());
   143         return sizeof(ArrayMap);
   144     }
   145 
   146     size_t iterateEnvironmentSlotDescriptorArray(EnvironmentSlotDescriptorArray *env) {
   147         unsigned length = env->length();
   148         EnvironmentSlotDescriptor *desc = env->slots();
   149         for (unsigned i = 0; i < length; i++)
   150             this->visit(desc->type.addr());
   151         return env->objSize();
   152     }
   153 
   154     size_t iterateEnvironmentMap(EnvironmentMap *map) {
   155         visitField(map, EnvironmentMap::offsetOfSlots());
   156         return map->objSize();
   157     }
   158 
   159     size_t iterateStructMap(StructMap *map) {
   160         visitField(map, StructMap::offsetOfType());
   161         visitField(map, StructMap::offsetOfComposed());
   162         return map->objSize();
   163     }
   164 
   165     size_t iterateEnvironment(Environment *env) {
   166         EnvironmentMap *map = env->map();
   167         const unsigned *offsets = map->traceOffsets();
   168         unsigned noffsets = map->traceOffsetCount();
   169         for (unsigned i = 0; i < noffsets; i++)
   170             visitField(env, Environment::offsetOfSlotOffset(offsets[i]));
   171         return env->objSize();
   172     }
   173 
   174     size_t iterateStruct(Struct *obj) {
   175         StructMap *map = obj->map();
   176         const unsigned *offsets = map->traceOffsets();
   177         unsigned noffsets = map->traceOffsetCount();
   178         for (unsigned i = 0; i < noffsets; i++)
   179             visitField(obj, Struct::offsetOfSlotOffset(offsets[i]));
   180         return obj->objSize();
   181     }
   182 
   183   private:
   184     size_t iterateObject(Object *obj) {
   185         visitField(obj, Object::offsetOfMap());
   186         return iterate(obj, obj->map());
   187     }
   188 
   189     size_t iterateObject(Object *obj, Map *map) {
   190         switch (map->kind()) {
   191           case MapKind_Type:
   192             return iterateType(obj);
   193 
   194           case MapKind_FunctionType:
   195             return iterateFunctionType(obj);
   196 
   197           case MapKind_StructType:
   198             return iterateStructType(obj);
   199 
   200           case MapKind_FixedArray:
   201             return iterateFixedArray(FixedArray::cast(obj));
   202 
   203           case MapKind_ByteArray:
   204             return iterateByteArray(ByteArray::cast(obj));
   205 
   206           case MapKind_Code:
   207             return iterateCode(obj);
   208 
   209           case MapKind_Reference:
   210             return iterateReference(obj);
   211 
   212           case MapKind_Module:
   213             return iterateModule(obj);
   214 
   215           case MapKind_Function:
   216             return iterateFunction(obj);
   217 
   218           case MapKind_Descriptor:
   219             return iterateDescriptor(obj);
   220 
   221           case MapKind_String:
   222             return iterateString(obj);
   223 
   224           case MapKind_Array:
   225             return Align(iterateArray(Array::cast(obj), ArrayMap::cast(map)), kObjectAlignment);
   226 
   227           case MapKind_Box:
   228             return iterateBox(Box::cast(obj));
   229 
   230           case MapKind_Struct:
   231             return iterateStruct(Struct::cast(obj));
   232 
   233           case MapKind_EnvironmentSlotDescriptorArray:
   234             return iterateEnvironmentSlotDescriptorArray(EnvironmentSlotDescriptorArray::cast(obj));
   235 
   236           case MapKind_Environment:
   237             return iterateEnvironment(Environment::cast(obj));
   238 
   239           case MapKind_Native:
   240             return iterateNative(obj);
   241 
   242           case MapKind_MetaArray:
   243             return iterateArrayMap(ArrayMap::cast(obj));
   244 
   245           case MapKind_MetaEnvironment:
   246             return iterateEnvironmentMap(EnvironmentMap::cast(obj));
   247 
   248           case MapKind_MetaStruct:
   249             return iterateStructMap(StructMap::cast(obj));
   250 
   251           default:
   252             assert(map->kind() == MapKind_Meta);
   253             return sizeof(Map);
   254         }
   255     }
   256 
   257   public:
   258     size_t iterate(Object *obj) {
   259         return Align(iterateObject(obj), kObjectAlignment);
   260     }
   261 
   262     size_t iterate(Object *obj, Map *map) {
   263         return Align(iterateObject(obj, map), kObjectAlignment);
   264     }
   265 };
   266 
   267 static inline size_t
   268 SizeOfObjectRaw(Object *obj, Map *map)
   269 {
   270     switch (map->kind()) {
   271       case MapKind_Type:
   272         return sizeof(Type);
   273 
   274       case MapKind_FunctionType:
   275         return sizeof(FunctionType);
   276 
   277       case MapKind_StructType:
   278         return sizeof(StructType);
   279 
   280       case MapKind_FixedArray:
   281         return FixedArray::cast(obj)->objSize();
   282 
   283       case MapKind_ByteArray:
   284         return ByteArray::cast(obj)->objSize();
   285 
   286       case MapKind_Code:
   287         return sizeof(Code);
   288 
   289       case MapKind_Reference:
   290         return sizeof(Reference);
   291 
   292       case MapKind_Module:
   293         return sizeof(Module);
   294 
   295       case MapKind_Descriptor:
   296         return sizeof(Descriptor);
   297 
   298       case MapKind_Function:
   299         return sizeof(Function);
   300 
   301       case MapKind_String:
   302         return String::cast(obj)->objSize();
   303 
   304       case MapKind_Native:
   305         return sizeof(Native);
   306 
   307       case MapKind_Array:
   308         return Align(Array::cast(obj)->objSize(ArrayMap::cast(map)), kObjectAlignment);
   309 
   310       case MapKind_Box:
   311         return sizeof(Box);
   312 
   313       case MapKind_Struct:
   314         return Struct::cast(obj)->objSize();
   315 
   316       case MapKind_EnvironmentSlotDescriptorArray:
   317         return EnvironmentSlotDescriptorArray::cast(obj)->objSize();
   318 
   319       case MapKind_Environment:
   320         return Environment::cast(obj)->objSize();
   321 
   322       case MapKind_MetaArray:
   323         return sizeof(ArrayMap);
   324 
   325       case MapKind_MetaEnvironment:
   326         return EnvironmentMap::cast(obj)->objSize();
   327 
   328       case MapKind_MetaStruct:
   329         return StructMap::cast(obj)->objSize();
   330 
   331       case MapKind_FreeRegion:
   332         return *reinterpret_cast<size_t *>(reinterpret_cast<Address>(obj + 1));
   333 
   334       default:
   335         assert(map->kind() == MapKind_Meta);
   336         return sizeof(Map);
   337     }
   338 }
   339 
   340 static inline size_t
   341 SizeOfObject(Object *obj, Map *map)
   342 {
   343     return Align(SizeOfObjectRaw(obj, map), kObjectAlignment);
   344 }
   345 
   346 class ObjectSizeEvaluator
   347 {
   348   public:
   349     size_t operator()(Object *obj, Map *map) {
   350         return SizeOfObject(obj, map);
   351     }
   352 };
   353 
   354 }
   355 
   356 #endif // _include_jc_object_visitor_h_