src/compiler/BytecodeEmitter.cpp
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 177 src/BytecodeEmitter.cpp@56f7e5b0243c
child 220 5df4a112967d
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
     1 /* vim: set ts=4 sw=4 tw=99 et:
     2  *
     3  * Copyright (C) 2012 David Anderson
     4  *
     5  * This file is part of JITCraft.
     6  *
     7  * JITCraft is free software: you can redistribute it and/or modify it under
     8  * the terms of the GNU General Public License as published by the Free
     9  * Software Foundation, either version 3 of the License, or (at your option)
    10  * any later version.
    11  * 
    12  * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
    13  * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    14  * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
    15  *
    16  * You should have received a copy of the GNU General Public License along with
    17  * JITCraft. If not, see http://www.gnu.org/licenses/.
    18  */
    19 #include <stdlib.h>
    20 #include <string.h>
    21 #include "../Utility.h"
    22 #include "../Types.h"
    23 #include "../Zone.h"
    24 #include "../Strings.h"
    25 #include "../Zone.h"
    26 #include "BytecodeEmitter.h"
    27 #include "Symbols.h"
    28 #include "Scopes.h"
    29 #include "../Spaces-inl.h"
    30 #include "../Heap-inl.h"
    31 
    32 using namespace ke;
    33 
    34 StringTable::StringTable()
    35   : set_(ZoneAllocatorPolicy()),
    36     initialized_(false)
    37 {
    38 }
    39 
    40 bool
    41 StringTable::initialize()
    42 {
    43     assert(!initialized_);
    44     initialized_ = true;
    45     if (!set_.init(16))
    46         return false;
    47     return true;
    48 }
    49 
    50 bool
    51 StringTable::add(Handle<String> name, unsigned *indexp)
    52 {
    53     if (!initialized_ && !initialize())
    54         return false;
    55 
    56     Set::Insert p = set_.findForAdd(*name);
    57     if (p.found()) {
    58         *indexp = p->index;
    59         return true;
    60     }
    61 
    62     if (strings_.length() > INT_MAX) {
    63         ZONE()->reportAllocationOverflow();
    64         return false;
    65     }
    66 
    67     Entry e(name, strings_.length());
    68     if (!strings_.append(e))
    69         return false;
    70 
    71     if (!set_.add(p, e))
    72         return false;
    73 
    74     *indexp = e.index;
    75 
    76     return true;
    77 }
    78 
    79 void
    80 StringTable::accept(VirtualObjectVisitor *visitor)
    81 {
    82     for (size_t i = 0; i < strings_.length(); i++)
    83         visitor->visit(reinterpret_cast<Object **>(&strings_[i].string));
    84 }
    85 
    86 BytecodeEmitter::BytecodeEmitter(PoolAllocator &pool)
    87   : AutoRooted(ZONE()),
    88     pool_(pool),
    89     buffer_(inline_buffer_),
    90     buffer_pos_(buffer_),
    91     buffer_end_(buffer_pos_ + ARRAY_LENGTH(inline_buffer_)),
    92     oom_(false),
    93     maxStackDepth_(0)
    94 {
    95     lastCodePosition_ = 0;
    96     firstLine_ = 1;
    97 }
    98 
    99 BytecodeEmitter::~BytecodeEmitter()
   100 {
   101     for (size_t i = 0; i < oopRecords_.length(); i++)
   102         delete oopRecords_[i];
   103 }
   104 
   105 void
   106 BytecodeEmitter::accept(VirtualObjectVisitor *visitor)
   107 {
   108     for (size_t i = 0; i < objects_.length(); i++)
   109         visitor->visit(&objects_[i]);
   110 
   111     strings_.accept(visitor);
   112 }
   113 
   114 bool
   115 BytecodeEmitter::growBuffer(size_t nbytes)
   116 {
   117     if (oom_) {
   118         buffer_pos_ = buffer_;
   119         return false;
   120     }
   121 
   122     size_t newSize = size_t((buffer_end_ - buffer_) * 2);
   123     if (newSize >= MAX_LENGTH) {
   124         oom_ = true;
   125         buffer_pos_ = buffer_;
   126         ZONE()->reportOutOfMemory();
   127         return false;
   128     }
   129 
   130     bytecode *newBuffer;
   131     if (buffer_ == inline_buffer_)
   132         newBuffer = (bytecode *)malloc(newSize);
   133     else
   134         newBuffer = (bytecode *)realloc(buffer_, newSize);
   135 
   136     // If we can't allocate a buffer, reset the buffer to the start and set
   137     // a flag to indicate out-of-memory.
   138     if (!newBuffer) {
   139         oom_ = true;
   140         buffer_pos_ = buffer_;
   141         ZONE()->reportOutOfMemory();
   142         return false;
   143     }
   144 
   145     if (buffer_ == inline_buffer_)
   146         memcpy(newBuffer, inline_buffer_, (buffer_pos_ - inline_buffer_));
   147 
   148     buffer_pos_ = newBuffer + size_t(buffer_pos_ - buffer_);
   149     buffer_ = newBuffer;
   150     buffer_end_ = buffer_ + newSize;
   151     assert(buffer_pos_ + nbytes < buffer_end_);
   152 
   153     return true;
   154 }
   155 
   156 void
   157 BytecodeEmitter::recordOops()
   158 {
   159     OopRecord *record = new OopRecord(oopMap_.length(), position());
   160 
   161     if (!record->slots.ensure(record->depth)) {
   162         oom_ = true;
   163         return;
   164     }
   165 
   166     bool hadOops = false;
   167     for (size_t i = 0; i < oopMap_.length(); i++) {
   168         bool bit = oopMap_[i];
   169         record->slots.infallibleAppend(bit);
   170         hadOops |= bit;
   171     }
   172 
   173     if (!hadOops) {
   174         delete record;
   175         return;
   176     }
   177 
   178     oom_ &= oopRecords_.append(record);
   179 }
   180 
   181 void
   182 BytecodeEmitter::note_position(const SourcePosition &pos)
   183 {
   184     assert(pos.line > 0);
   185     assert(unsigned(pos.line) >= firstLine_);
   186 
   187     unsigned lineDelta = unsigned(pos.line) - firstLine_;
   188     unsigned pcDelta = unsigned(position()) - lastCodePosition_;
   189 
   190     sourcemap_.writeUint32(pcDelta);
   191     sourcemap_.writeUint32(lineDelta);
   192     sourcemap_.writeUint32(pos.col);
   193 
   194     lastCodePosition_ = unsigned(position());
   195 }
   196 
   197 void
   198 BytecodeEmitter::getglobal(Symbol *sym)
   199 {
   200     ensure(OP_GETGLOBAL_LENGTH);
   201     writeOp(OP_GETGLOBAL);
   202     writeUint32(addString(sym->name()));
   203     if (sym->type()->isTraceable())
   204         markOop(1);
   205 }
   206 void
   207 BytecodeEmitter::setglobal(Symbol *sym)
   208 {
   209     ensure(OP_SETGLOBAL_LENGTH);
   210     writeOp(OP_SETGLOBAL);
   211     writeUint32(addString(sym->name()));
   212     if (sym->type()->isTraceable())
   213         markOop(1);
   214 }
   215 
   216 void
   217 BytecodeEmitter::getlocal(VariableSymbol *var)
   218 {
   219     ensure(OP_GETLOCAL_LENGTH);
   220     writeOp(OP_GETLOCAL);
   221     writeUint32(var->slot());
   222     if (var->type()->isTraceable())
   223         markOop(1);
   224 }
   225 
   226 void
   227 BytecodeEmitter::setlocal(VariableSymbol *var)
   228 {
   229     ensure(OP_SETLOCAL_LENGTH);
   230     writeOp(OP_SETLOCAL);
   231     writeUint32(var->slot());
   232     if (var->type()->isTraceable())
   233         markOop(1);
   234 }
   235 
   236 bool
   237 BytecodeEmitter::allocate(VariableSymbol *sym)
   238 {
   239     assert(sym->isVariable());
   240     assert(sym->scope()->kind() == Scope::BLOCK);
   241 
   242     switch (sym->scope()->kind()) {
   243       case Scope::BLOCK:
   244       {
   245         // See overall comment on overflows.
   246         assert(locals_.length() < UINT_MAX);
   247         sym->allocate(VariableSymbol::Local, locals_.length());
   248         if (!locals_.append(sym))
   249             return false;
   250         break;
   251       }
   252 
   253       default:
   254         assert(false);
   255         return false;
   256     }
   257 
   258     return true;
   259 }
   260 
   261 Code *
   262 BytecodeEmitter::finish()
   263 {
   264     stop();
   265 
   266     Zone *zone = ZONE();
   267     if (oom() || zone->hasPendingException())
   268         return NULL;
   269 
   270     Local<ByteArray> bytes(zone, ByteArray::New(zone, length(), Heap::Tenure_Old));
   271     if (!bytes)
   272         return NULL;
   273     memcpy(bytes->raw(), buffer(), length());
   274 
   275     Local<FixedArray> objects(zone, FixedArray::New(zone, objects_.length(), Heap::Tenure_Old));
   276     if (!objects)
   277         return NULL;
   278 
   279     for (size_t i = 0; i < objects_.length(); i++)
   280         objects->set(zone, i, objects_[i]);
   281 
   282     // If any stack slots hold gc pointers at potential failure points in the
   283     // bytecode, record them now. We do this by building a list of tuples;
   284     // each tuple contains the pc offset, and a bitset containing which stack
   285     // slots must be scanned by the gc.
   286     Local<ByteArray> safepoints(zone);
   287     if (oopRecords_.length()) {
   288         size_t bytesPerSafepoint = Code::BytesPerSafepoint(maxStackDepth());
   289         size_t gcmapSize = bytesPerSafepoint * oopRecords_.length();
   290         safepoints = ByteArray::New(zone, gcmapSize, Heap::Tenure_Old);
   291         if (!safepoints)
   292             return NULL;
   293 
   294         Address cursor = safepoints->raw();
   295         for (size_t i = 0; i < oopRecords_.length(); i++) {
   296             OopRecord *record = oopRecords_[i];
   297             assert(record->depth <= maxStackDepth());
   298 
   299             *reinterpret_cast<unsigned *>(cursor) = record->offset;
   300 
   301             unsigned char *bits = reinterpret_cast<unsigned char *>(cursor + sizeof(unsigned));
   302             for (unsigned i = 0; i < record->depth; i++)
   303                 bits[i / 8] |= (1 << (i % 8));
   304 
   305             cursor += bytesPerSafepoint;
   306         }
   307 
   308         assert(cursor == safepoints->raw() + gcmapSize);
   309     }
   310 
   311     Local<ByteArray> sourcemap(zone);
   312     if (sourcemap_.length()) {
   313         sourcemap = ByteArray::New(zone, sourcemap_.length(), Heap::Tenure_Old);
   314         if (!sourcemap)
   315             return NULL;
   316 
   317         memcpy(sourcemap->raw(), sourcemap_.buffer(), sourcemap_.length());
   318     }
   319 
   320     Local<FixedArray> strings(zone, FixedArray::New(zone, strings_.stringCount(), Heap::Tenure_Old));
   321     if (!strings)
   322         return NULL;
   323     for (size_t i = 0; i < strings_.stringCount(); i++)
   324         strings->set(zone, i, strings_.stringAt(i));
   325 
   326     Local<FixedArray> locals(zone, FixedArray::New(zone, locals_.length(), Heap::Tenure_Old));
   327     if (!locals)
   328         return NULL;
   329     for (size_t i = 0; i < locals_.length(); i++)
   330         locals->set(zone, i, locals_[i]->type());
   331 
   332     Local<Code> code(zone);
   333 
   334     code = Code::New(zone,
   335                      bytes,
   336                      locals,
   337                      objects,
   338                      strings,
   339                      maxStackDepth(),
   340                      safepoints,
   341                      sourcemap);
   342 
   343     return code;
   344 }