Massive SemA/BC refactoring to support better type systems. Read on for more.
The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code
This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.
Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).
This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*
Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.
(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.
(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).
In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.
(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.
As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.
This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:
HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))
As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>
This simplifies the pipeline, and JITs should be able to melt the added
work away.
HIR is not used at a statement level, and HIR does not have any concept
of control-flow.
(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.
In addition, some minor refactorings have taken place:
(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.
Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
1 /* vim: set ts=4 sw=4 tw=99 et:
3 * Copyright (C) 2012 David Anderson
5 * This file is part of JITCraft.
7 * JITCraft is free software: you can redistribute it and/or modify it under
8 * the terms of the GNU General Public License as published by the Free
9 * Software Foundation, either version 3 of the License, or (at your option)
12 * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
13 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
14 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
16 * You should have received a copy of the GNU General Public License along with
17 * JITCraft. If not, see http://www.gnu.org/licenses/.
19 #ifndef _include_bytecode_emitter_h_
20 #define _include_bytecode_emitter_h_
24 #include "../Opcodes.h"
25 #include "../Handles.h"
26 #include "../CompactBuffer.h"
28 #include "../HashTable.h"
29 #include "../VirtualObjectVisitor.h"
33 // An overarching note on overflows: Though we should use CheckedInt, we rely
34 // on a source-length cap covering all possible overflow checks. As long as
35 // the source length is capped to INT_MAX, the number of AST nodes will be
36 // strictly beneath that cap, and so to will most things generated by the
37 // AST. For example, local and argument counts, number of functions, fields
38 // in a struct, et cetera. Not everything can be covered by this cap: the
39 // stack depth, number of bytecodes for example. In these cases we explicitly
40 // test for overflow of INT_MAX and in other cases we assert.
42 // In the future, we should of course use a checked int wrapper.
56 OopRecord(unsigned length, unsigned offset)
76 Entry(String *string, unsigned index)
85 typedef Entry Payload;
87 static uint32 hash(String *string) {
88 return HashPointer(string);
91 static bool matches(String *key, Entry &e) {
92 return key == e.string;
96 typedef HashTable<Policy, ZoneAllocatorPolicy> Set;
101 void accept(VirtualObjectVisitor *visitor);
103 bool add(Handle<String> name, unsigned *indexp);
104 size_t stringCount() const {
105 return strings_.length();
107 String *stringAt(size_t i) const {
108 return strings_[i].string;
113 Vector<Entry> strings_;
117 // The bytecode emitter assists in building Code objects, by emitting opcodes
118 // and tracking stack information. Since the SemA creates temporary pool/root
119 // scopes, we cannot use ScopedRoot or PoolObjects. Instead, BCE gets a
120 // special marking function.
121 class BytecodeEmitter : public AutoRooted
124 PoolAllocator &pool_;
127 bytecode *buffer_pos_;
128 bytecode *buffer_end_;
129 bytecode inline_buffer_[512];
131 unsigned maxStackDepth_;
132 Vector<bool> oopMap_;
133 Vector<OopRecord *> oopRecords_;
134 CompactBufferWriter sourcemap_;
135 unsigned lastCodePosition_;
137 StringTable strings_;
138 Vector<Symbol *> locals_;
139 Vector<Object *> objects_;
142 // The maximum size of a code object must be such that the distance between
143 // two instructions fits into a signed, 31-bit integer. This ensures that
144 // all code-bounded buffers (like the number of object literals or
145 // safepoints) does not need to be bounds checked. It also means we can add
146 // any two offsets and be guaranteed that they don't overflow.
147 static const size_t MAX_LENGTH = INT_MAX / 2;
150 bool growBuffer(size_t bytes);
152 // Checking the return value is not necessary, since the inline buffer is
153 // used as a ballast.
154 bool ensure(size_t bytes) {
155 if (buffer_pos_ + bytes > buffer_end_)
156 return growBuffer(bytes);
160 void writeUint8(unsigned value) {
161 assert(value <= UCHAR_MAX);
162 *buffer_pos_++ = value;
164 void writeFloat(float value) {
165 *(float *)buffer_pos_ = value;
166 buffer_pos_ += sizeof(float);
168 void writeInt32(int value) {
169 *(int *)buffer_pos_ = value;
170 buffer_pos_ += sizeof(int);
172 void writeUint32(unsigned value) {
173 *(unsigned *)buffer_pos_ = value;
174 buffer_pos_ += sizeof(unsigned);
176 void writeOp(Opcode op) {
181 for (int i = 0; i < OpcodeInfo[op].nuses; i++)
183 for (int i = 0; i < OpcodeInfo[op].ndefs; i++)
184 oopMap_.append(false);
185 if (oopMap_.length() > maxStackDepth_)
186 maxStackDepth_ = oopMap_.length();
188 void writeObject(Object *obj) {
189 writeUint32(objects_.length());
190 ScopedRoot<Object> root(obj);
191 if (!objects_.append(root))
195 int position() const {
196 assert(buffer_pos_ - buffer_ < INT_MAX / 2);
197 return int(buffer_pos_ - buffer_);
201 BytecodeEmitter(PoolAllocator &pool);
204 void accept(VirtualObjectVisitor *visitor);
206 bytecode *buffer() const {
209 unsigned length() const {
210 return unsigned(buffer_pos_ - buffer_);
213 return oom_ || sourcemap_.oom();
215 unsigned stackDepth() const {
216 return oopMap_.length();
218 unsigned maxStackDepth() const {
219 return maxStackDepth_;
222 void drop(unsigned amt) {
223 assert(amt <= stackDepth());
224 for (unsigned i = 0; i < amt; i++)
228 void note_position(const SourcePosition &pos);
229 void setFirstLine(unsigned line) {
230 assert(!sourcemap_.length());
235 // A label represents a local control-flow target in the bytecode. A label
236 // may be in three states:
238 // (1) Empty - the label is not used yet.
239 // (2) Pending - the label has incoming jumps, but the actual location of
240 // is not yet known. The list of jumps is threaded through
241 // the four bytes reserved for each emitted jump target. In
242 // this case, the offset field is the location of the most
243 // recently emitted incoming jump.
244 // (3) Bound - the label's position has been pinned.
246 // A jump can be emitted to a label in any state. Binding a label
247 // immediately backpatches all pending incoming jumps. A label must be
248 // bound if it is used.
254 void setOffset(int offset) {
256 assert(offset_ == offset);
260 static const int INVALID_OFFSET = -1;
264 : offset_(INVALID_OFFSET),
270 assert(bound_ || offset_ == INVALID_OFFSET);
279 void bind(int offset) {
284 int link(int offset) {
293 int patchJumpAt(int at, int to) {
294 assert(at >= 0 && at < position());
295 int old = *(int *)(buffer_ + at + 1);
296 *(int *)(buffer_ + at + 1) = (to - at);
300 void singleByteOp(Opcode op) {
301 assert(OpcodeInfo[op].length == 1);
302 ensure(OpcodeInfo[op].length);
306 // Mark a stack slot as having an object pointer.
307 void markOop(unsigned depth, bool oop = true) {
310 oopMap_[oopMap_.length() - depth] = oop;
312 bool getOop(unsigned depth) {
315 return oopMap_[oopMap_.length() - depth];
318 // Mark the current bytecode position as either re-entrant (meaning, it
319 // can leave the interpreter or call to another function), or causing GC.
320 // This will record which stack slots have objects and associate that
321 // information with the current pc.
325 void int_(int value) {
334 void getlocal(VariableSymbol *var);
335 void setlocal(VariableSymbol *var);
337 void getarg(Variable *var) {
338 ensure(OP_GETARG_LENGTH);
340 writeUint32(var->slot());
342 if (var->type()->isTraceable())
345 void setarg(Variable *var) {
346 ensure(OP_SETARG_LENGTH);
348 writeUint32(var->slot());
350 if (var->type()->isTraceable())
355 void j(Opcode op, Label *label) {
356 // Since jumps can skip around the stream, do not try to do anything
357 // in a potentially truncated state.
361 assert(op == OP_JT || op == OP_JF || op == OP_JUMP);
362 ensure(OP_JUMP_LENGTH);
364 if (label->bound()) {
365 // We can immediately emit this jump.
366 int distance = label->offset() - position();
368 writeInt32(distance);
370 // We can't emit this jump yet. Add a placeholder.
371 int old = label->link(position());
376 void bind(Label *label) {
380 assert(!label->bound());
382 int offset = label->offset();
383 while (offset != Label::INVALID_OFFSET)
384 offset = patchJumpAt(offset, position());
386 label->bind(position());
388 void jump(Label *label) {
392 void binary(Opcode op) {
395 void unary(Opcode op) {
400 singleByteOp(OP_POP);
403 singleByteOp(OP_STOP);
405 void call(Handle<FunctionType> fun, unsigned argc) {
408 if (fun->isNative()) {
409 ensure(OP_CALLNATIVE_LENGTH);
410 writeOp(OP_CALLNATIVE);
414 ensure(OP_CALL_LENGTH);
418 if (fun->returnType()->isVoid())
420 else if (fun->returnType()->isTraceable())
425 void getelem(Type *type) {
426 singleByteOp(OP_GETELEM);
428 if (type->isTraceable())
431 void setelem(Type *type) {
432 singleByteOp(OP_SETELEM);
435 void getfield(unsigned index, Type *type) {
436 ensure(OP_GETFIELD_LENGTH);
437 writeOp(OP_GETFIELD);
440 if (type->isTraceable())
443 void setfield(unsigned index, Type *type) {
444 ensure(OP_SETFIELD_LENGTH);
445 writeOp(OP_SETFIELD);
448 if (type->isTraceable())
453 void getglobal(Variable *var) {
454 ensure(OP_GETGLOBAL_F_LENGTH);
455 if (var->type()->isPrimitive()) {
456 switch (var->type()->primitive()) {
457 case PrimitiveType_Float:
458 writeOp(OP_GETGLOBAL_F);
461 assert(var->type()->primitive() == PrimitiveType_Int32);
462 writeOp(OP_GETGLOBAL_I);
466 writeOp(OP_GETGLOBAL_O);
469 writeUint32(var->slot());
471 void setglobal(Variable *var) {
472 ensure(OP_SETGLOBAL_F_LENGTH);
473 if (var->type()->isPrimitive()) {
474 switch (var->type()->primitive()) {
475 case PrimitiveType_Float:
476 writeOp(OP_SETGLOBAL_F);
479 assert(var->type()->primitive() == PrimitiveType_Int32);
480 writeOp(OP_SETGLOBAL_I);
484 writeOp(OP_SETGLOBAL_O);
486 writeUint32(var->slot());
490 void float_(float value) {
491 ensure(OP_FLOAT_LENGTH);
496 singleByteOp(OP_CVT_I2F);
498 void newarray(Opcode op, Handle<ArrayMap> map, Heap::Tenure tenure) {
499 assert(op == OP_NEWFIXED ||
504 ensure(OpcodeInfo[op].length);
509 if (op == OP_NEWSIZED)
510 drop(map->type()->levels());
514 void newstruct(Handle<Map> map, Heap::Tenure tenure) {
516 ensure(OP_NEWSTRUCT_LENGTH);
517 writeOp(OP_NEWSTRUCT);
522 void savelifo(unsigned slot) {
523 ensure(OP_SAVELIFO_LENGTH);
524 writeOp(OP_SAVELIFO);
527 void unwindlifo(unsigned slot) {
528 ensure(OP_UNWINDLIFO_LENGTH);
529 writeOp(OP_UNWINDLIFO);
533 void newref(Variable *var) {
535 if (var->location() == Variable::STACK) {
536 ensure(OP_VARREF_LIFO_LENGTH);
537 writeOp(OP_VARREF_LIFO);
539 assert(var->location() == Variable::PARAMETER);
540 ensure(OP_ARGREF_LIFO_LENGTH);
541 writeOp(OP_ARGREF_LIFO);
543 writeUint32(var->slot());
548 void slotref_lifo() {
550 singleByteOp(OP_SLOTREF_LIFO);
553 void getref(Type *type) {
554 if (type->pod() == PrimitiveType_Int32) {
555 singleByteOp(OP_GETREF_I);
557 assert(type->pod() == PrimitiveType_Float);
558 singleByteOp(OP_GETREF_F);
561 void setref(Type *type) {
562 if (type->pod() == PrimitiveType_Int32) {
563 singleByteOp(OP_SETREF_I);
565 assert(type->pod() == PrimitiveType_Float);
566 singleByteOp(OP_SETREF_F);
569 void object(Handle<Object> obj) {
570 ensure(OP_OBJECT_LENGTH);
576 singleByteOp(OP_COPYARRAY);
580 singleByteOp(OP_COPYSTRUCT);
584 singleByteOp(OP_SIZEOF);
588 bool oop = getOop(1);
589 singleByteOp(OP_DUP);
593 bool oop2 = getOop(2);
594 bool oop1 = getOop(1);
596 singleByteOp(OP_DUP2);
602 bool oop2 = getOop(2);
603 bool oop1 = getOop(1);
605 singleByteOp(OP_SWAP);
611 bool oop3 = getOop(3);
612 bool oop2 = getOop(2);
613 bool oop1 = getOop(1);
615 singleByteOp(OP_ROLL3);
622 bool oop2 = getOop(2);
623 bool oop1 = getOop(1);
625 singleByteOp(OP_PICK2);
632 bool oop3 = getOop(3);
633 bool oop2 = getOop(2);
634 bool oop1 = getOop(1);
636 singleByteOp(OP_PICK3);
645 singleByteOp(OP_RETURNVOID);
648 singleByteOp(OP_BITNOT);
651 singleByteOp(OP_NOT);
654 singleByteOp(OP_NOT_F);
657 singleByteOp(OP_NEG);
660 singleByteOp(OP_NEG_F);
663 singleByteOp(OP_BITS_I2F);
666 singleByteOp(OP_BITS_F2I);
669 singleByteOp(OP_TRUE);
672 singleByteOp(OP_FALSE);
675 unsigned addString(Handle<String> name) {
677 if (!strings_.add(name, &index)) {
684 void getglobal(Symbol *sym);
685 void setglobal(Symbol *sym);
689 singleByteOp(OP_PUBLIC);
692 void tableswitch(int low, int high, Label *def, Label **labels) {
693 unsigned tablesize = TableSwitchEntries(low, high);
694 assert(tablesize <= MAX_TABLESWITCH_ENTRIES);
696 ensure(OP_TABLESWITCH_LENGTH);
697 writeOp(OP_TABLESWITCH);
702 // When labels are patched, we assume each position is a jump. Rather
703 // than hack position offsets to make labels work without actual
704 // incoming jumps, we emit a jump inline.
705 for (unsigned i = 0; i < tablesize; i++)
709 bool allocate(VariableSymbol *sym);
715 void get(Variable *var) {
716 if (var->location() == Variable::STACK) {
719 assert(var->location() == Variable::PARAMETER);
723 void set(Variable *var) {
724 assert(!var->type()->isReference());
725 if (var->location() == Variable::STACK) {
728 assert(var->location() == Variable::PARAMETER);
737 #endif // _include_bytecode_emitter_h_