Massive SemA/BC refactoring to support better type systems. Read on for more.
The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code
This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.
Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).
This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*
Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.
(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.
(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).
In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.
(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.
As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.
This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:
HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))
As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>
This simplifies the pipeline, and JITs should be able to melt the added
work away.
HIR is not used at a statement level, and HIR does not have any concept
of control-flow.
(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.
In addition, some minor refactorings have taken place:
(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.
Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
1 /* vim: set ts=4 sw=4 tw=99 et:
3 * Copyright (C) 2012 David Anderson
5 * This file is part of JITCraft.
7 * JITCraft is free software: you can redistribute it and/or modify it under
8 * the terms of the GNU General Public License as published by the Free
9 * Software Foundation, either version 3 of the License, or (at your option)
12 * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
13 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
14 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
16 * You should have received a copy of the GNU General Public License along with
17 * JITCraft. If not, see http://www.gnu.org/licenses/.
19 #ifndef _include_jc_object_visitor_h_
20 #define _include_jc_object_visitor_h_
24 #include "FixedArray.h"
26 #include "Reference.h"
28 #include "Functions.h"
31 #include "Environments.h"
32 #include "Structures.h"
37 // An ObjectVisitor is used for traversing object references in the heap. Since
38 // this pattern must be lightning fast, we avoid virtual dispatch by forcing
39 // visitors to compose themselves with the visiting logic.
41 class ObjectVisitor : public T
44 void visitField(Object *obj, size_t offset) {
45 this->visit(reinterpret_cast<Object **>(Address(obj) + offset));
48 size_t iterateType(Object *obj) {
49 visitField(obj, Type::offsetOfContained());
50 visitField(obj, Type::offsetOfName());
54 size_t iterateStructType(Object *obj) {
55 visitField(obj, Type::offsetOfContained());
56 visitField(obj, Type::offsetOfName());
57 visitField(obj, StructType::offsetOfFields());
58 visitField(obj, StructType::offsetOfNewMap());
59 return sizeof(StructType);
62 size_t iterateFunctionType(Object *obj) {
63 visitField(obj, Type::offsetOfContained());
64 visitField(obj, Type::offsetOfName());
65 visitField(obj, FunctionType::offsetOfReturnType());
66 visitField(obj, FunctionType::offsetOfParameters());
67 visitField(obj, FunctionType::offsetOfDefaults());
68 return sizeof(FunctionType);
71 size_t iterateFixedArray(FixedArray *array) {
72 unsigned length = array->length();
73 Object **ptr = reinterpret_cast<Object **>(array + 1);
74 for (unsigned i = 0; i < length; i++)
76 return array->objSize();
79 size_t iterateByteArray(ByteArray *array) {
80 return array->objSize();
83 size_t iterateCode(Object *obj) {
84 visitField(obj, Code::offsetOfModule());
85 visitField(obj, Code::offsetOfBytecode());
86 visitField(obj, Code::offsetOfLocals());
87 visitField(obj, Code::offsetOfObjects());
88 visitField(obj, Code::offsetOfSafepoints());
89 visitField(obj, Code::offsetOfSourceMap());
90 visitField(obj, Code::offsetOfStrings());
94 size_t iterateReference(Object *obj) {
95 visitField(obj, Reference::offsetOfBase());
96 return sizeof(Reference);
99 size_t iterateModule(Object *obj) {
100 visitField(obj, Module::offsetOfFile());
101 visitField(obj, Module::offsetOfGlobals());
102 visitField(obj, Module::offsetOfCode());
103 return sizeof(Module);
106 size_t iterateFunction(Object *obj) {
107 visitField(obj, Function::offsetOfCode());
108 visitField(obj, Function::offsetOfType());
109 visitField(obj, Function::offsetOfName());
110 return sizeof(Function);
113 size_t iterateDescriptor(Object *obj) {
114 visitField(obj, Descriptor::offsetOfName());
115 visitField(obj, Descriptor::offsetOfType());
116 return sizeof(Descriptor);
119 size_t iterateNative(Object *obj) {
120 visitField(obj, Native::offsetOfType());
121 return sizeof(Native);
124 size_t iterateString(Object *obj) {
125 return String::cast(obj)->objSize();
128 size_t iterateArray(Array *array, ArrayMap *map) {
129 if (!map->type()->fixedLength() || map->type()->contained()->isTraceable())
130 visitField(array, Array::offsetOfElements());
131 return array->objSize(map);
134 size_t iterateBox(Box *obj) {
135 visitField(obj, Box::offsetOfType());
136 if (obj->type()->isTraceable())
137 visitField(obj, Box::offsetOfObj());
141 size_t iterateArrayMap(ArrayMap *map) {
142 visitField(map, ArrayMap::offsetOfType());
143 return sizeof(ArrayMap);
146 size_t iterateEnvironmentSlotDescriptorArray(EnvironmentSlotDescriptorArray *env) {
147 unsigned length = env->length();
148 EnvironmentSlotDescriptor *desc = env->slots();
149 for (unsigned i = 0; i < length; i++)
150 this->visit(desc->type.addr());
151 return env->objSize();
154 size_t iterateEnvironmentMap(EnvironmentMap *map) {
155 visitField(map, EnvironmentMap::offsetOfSlots());
156 return map->objSize();
159 size_t iterateStructMap(StructMap *map) {
160 visitField(map, StructMap::offsetOfType());
161 visitField(map, StructMap::offsetOfComposed());
162 return map->objSize();
165 size_t iterateEnvironment(Environment *env) {
166 EnvironmentMap *map = env->map();
167 const unsigned *offsets = map->traceOffsets();
168 unsigned noffsets = map->traceOffsetCount();
169 for (unsigned i = 0; i < noffsets; i++)
170 visitField(env, Environment::offsetOfSlotOffset(offsets[i]));
171 return env->objSize();
174 size_t iterateStruct(Struct *obj) {
175 StructMap *map = obj->map();
176 const unsigned *offsets = map->traceOffsets();
177 unsigned noffsets = map->traceOffsetCount();
178 for (unsigned i = 0; i < noffsets; i++)
179 visitField(obj, Struct::offsetOfSlotOffset(offsets[i]));
180 return obj->objSize();
184 size_t iterateObject(Object *obj) {
185 visitField(obj, Object::offsetOfMap());
186 return iterate(obj, obj->map());
189 size_t iterateObject(Object *obj, Map *map) {
190 switch (map->kind()) {
192 return iterateType(obj);
194 case MapKind_FunctionType:
195 return iterateFunctionType(obj);
197 case MapKind_StructType:
198 return iterateStructType(obj);
200 case MapKind_FixedArray:
201 return iterateFixedArray(FixedArray::cast(obj));
203 case MapKind_ByteArray:
204 return iterateByteArray(ByteArray::cast(obj));
207 return iterateCode(obj);
209 case MapKind_Reference:
210 return iterateReference(obj);
213 return iterateModule(obj);
215 case MapKind_Function:
216 return iterateFunction(obj);
218 case MapKind_Descriptor:
219 return iterateDescriptor(obj);
222 return iterateString(obj);
225 return Align(iterateArray(Array::cast(obj), ArrayMap::cast(map)), kObjectAlignment);
228 return iterateBox(Box::cast(obj));
231 return iterateStruct(Struct::cast(obj));
233 case MapKind_EnvironmentSlotDescriptorArray:
234 return iterateEnvironmentSlotDescriptorArray(EnvironmentSlotDescriptorArray::cast(obj));
236 case MapKind_Environment:
237 return iterateEnvironment(Environment::cast(obj));
240 return iterateNative(obj);
242 case MapKind_MetaArray:
243 return iterateArrayMap(ArrayMap::cast(obj));
245 case MapKind_MetaEnvironment:
246 return iterateEnvironmentMap(EnvironmentMap::cast(obj));
248 case MapKind_MetaStruct:
249 return iterateStructMap(StructMap::cast(obj));
252 assert(map->kind() == MapKind_Meta);
258 size_t iterate(Object *obj) {
259 return Align(iterateObject(obj), kObjectAlignment);
262 size_t iterate(Object *obj, Map *map) {
263 return Align(iterateObject(obj, map), kObjectAlignment);
268 SizeOfObjectRaw(Object *obj, Map *map)
270 switch (map->kind()) {
274 case MapKind_FunctionType:
275 return sizeof(FunctionType);
277 case MapKind_StructType:
278 return sizeof(StructType);
280 case MapKind_FixedArray:
281 return FixedArray::cast(obj)->objSize();
283 case MapKind_ByteArray:
284 return ByteArray::cast(obj)->objSize();
289 case MapKind_Reference:
290 return sizeof(Reference);
293 return sizeof(Module);
295 case MapKind_Descriptor:
296 return sizeof(Descriptor);
298 case MapKind_Function:
299 return sizeof(Function);
302 return String::cast(obj)->objSize();
305 return sizeof(Native);
308 return Align(Array::cast(obj)->objSize(ArrayMap::cast(map)), kObjectAlignment);
314 return Struct::cast(obj)->objSize();
316 case MapKind_EnvironmentSlotDescriptorArray:
317 return EnvironmentSlotDescriptorArray::cast(obj)->objSize();
319 case MapKind_Environment:
320 return Environment::cast(obj)->objSize();
322 case MapKind_MetaArray:
323 return sizeof(ArrayMap);
325 case MapKind_MetaEnvironment:
326 return EnvironmentMap::cast(obj)->objSize();
328 case MapKind_MetaStruct:
329 return StructMap::cast(obj)->objSize();
331 case MapKind_FreeRegion:
332 return *reinterpret_cast<size_t *>(reinterpret_cast<Address>(obj + 1));
335 assert(map->kind() == MapKind_Meta);
341 SizeOfObject(Object *obj, Map *map)
343 return Align(SizeOfObjectRaw(obj, map), kObjectAlignment);
346 class ObjectSizeEvaluator
349 size_t operator()(Object *obj, Map *map) {
350 return SizeOfObject(obj, map);
356 #endif // _include_jc_object_visitor_h_