src/compiler/BytecodeEmitter.cpp
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 177 src/BytecodeEmitter.cpp@56f7e5b0243c
child 220 5df4a112967d
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
dvander@47
     1
/* vim: set ts=4 sw=4 tw=99 et:
dvander@47
     2
 *
dvander@0
     3
 * Copyright (C) 2012 David Anderson
dvander@0
     4
 *
dvander@0
     5
 * This file is part of JITCraft.
dvander@0
     6
 *
dvander@0
     7
 * JITCraft is free software: you can redistribute it and/or modify it under
dvander@0
     8
 * the terms of the GNU General Public License as published by the Free
dvander@0
     9
 * Software Foundation, either version 3 of the License, or (at your option)
dvander@0
    10
 * any later version.
dvander@0
    11
 * 
dvander@0
    12
 * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
dvander@0
    13
 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
dvander@0
    14
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
dvander@0
    15
 *
dvander@0
    16
 * You should have received a copy of the GNU General Public License along with
dvander@0
    17
 * JITCraft. If not, see http://www.gnu.org/licenses/.
dvander@0
    18
 */
dvander@0
    19
#include <stdlib.h>
dvander@177
    20
#include <string.h>
dvander@195
    21
#include "../Utility.h"
dvander@195
    22
#include "../Types.h"
dvander@195
    23
#include "../Zone.h"
dvander@195
    24
#include "../Strings.h"
dvander@195
    25
#include "../Zone.h"
dvander@0
    26
#include "BytecodeEmitter.h"
dvander@195
    27
#include "Symbols.h"
dvander@195
    28
#include "Scopes.h"
dvander@195
    29
#include "../Spaces-inl.h"
dvander@195
    30
#include "../Heap-inl.h"
dvander@0
    31
dvander@59
    32
using namespace ke;
dvander@0
    33
dvander@195
    34
StringTable::StringTable()
dvander@195
    35
  : set_(ZoneAllocatorPolicy()),
dvander@195
    36
    initialized_(false)
dvander@195
    37
{
dvander@195
    38
}
dvander@195
    39
dvander@195
    40
bool
dvander@195
    41
StringTable::initialize()
dvander@195
    42
{
dvander@195
    43
    assert(!initialized_);
dvander@195
    44
    initialized_ = true;
dvander@195
    45
    if (!set_.init(16))
dvander@195
    46
        return false;
dvander@195
    47
    return true;
dvander@195
    48
}
dvander@195
    49
dvander@195
    50
bool
dvander@195
    51
StringTable::add(Handle<String> name, unsigned *indexp)
dvander@195
    52
{
dvander@195
    53
    if (!initialized_ && !initialize())
dvander@195
    54
        return false;
dvander@195
    55
dvander@195
    56
    Set::Insert p = set_.findForAdd(*name);
dvander@195
    57
    if (p.found()) {
dvander@195
    58
        *indexp = p->index;
dvander@195
    59
        return true;
dvander@195
    60
    }
dvander@195
    61
dvander@195
    62
    if (strings_.length() > INT_MAX) {
dvander@195
    63
        ZONE()->reportAllocationOverflow();
dvander@195
    64
        return false;
dvander@195
    65
    }
dvander@195
    66
dvander@195
    67
    Entry e(name, strings_.length());
dvander@195
    68
    if (!strings_.append(e))
dvander@195
    69
        return false;
dvander@195
    70
dvander@195
    71
    if (!set_.add(p, e))
dvander@195
    72
        return false;
dvander@195
    73
dvander@195
    74
    *indexp = e.index;
dvander@195
    75
dvander@195
    76
    return true;
dvander@195
    77
}
dvander@195
    78
dvander@195
    79
void
dvander@195
    80
StringTable::accept(VirtualObjectVisitor *visitor)
dvander@195
    81
{
dvander@195
    82
    for (size_t i = 0; i < strings_.length(); i++)
dvander@195
    83
        visitor->visit(reinterpret_cast<Object **>(&strings_[i].string));
dvander@195
    84
}
dvander@195
    85
dvander@47
    86
BytecodeEmitter::BytecodeEmitter(PoolAllocator &pool)
dvander@195
    87
  : AutoRooted(ZONE()),
dvander@195
    88
    pool_(pool),
dvander@47
    89
    buffer_(inline_buffer_),
dvander@0
    90
    buffer_pos_(buffer_),
dvander@0
    91
    buffer_end_(buffer_pos_ + ARRAY_LENGTH(inline_buffer_)),
dvander@0
    92
    oom_(false),
dvander@47
    93
    maxStackDepth_(0)
dvander@0
    94
{
dvander@70
    95
    lastCodePosition_ = 0;
dvander@70
    96
    firstLine_ = 1;
dvander@0
    97
}
dvander@0
    98
dvander@195
    99
BytecodeEmitter::~BytecodeEmitter()
dvander@195
   100
{
dvander@195
   101
    for (size_t i = 0; i < oopRecords_.length(); i++)
dvander@195
   102
        delete oopRecords_[i];
dvander@195
   103
}
dvander@195
   104
dvander@195
   105
void
dvander@195
   106
BytecodeEmitter::accept(VirtualObjectVisitor *visitor)
dvander@195
   107
{
dvander@195
   108
    for (size_t i = 0; i < objects_.length(); i++)
dvander@195
   109
        visitor->visit(&objects_[i]);
dvander@195
   110
dvander@195
   111
    strings_.accept(visitor);
dvander@195
   112
}
dvander@195
   113
dvander@0
   114
bool
dvander@0
   115
BytecodeEmitter::growBuffer(size_t nbytes)
dvander@0
   116
{
dvander@0
   117
    if (oom_) {
dvander@0
   118
        buffer_pos_ = buffer_;
dvander@0
   119
        return false;
dvander@0
   120
    }
dvander@0
   121
dvander@0
   122
    size_t newSize = size_t((buffer_end_ - buffer_) * 2);
dvander@0
   123
    if (newSize >= MAX_LENGTH) {
dvander@0
   124
        oom_ = true;
dvander@0
   125
        buffer_pos_ = buffer_;
dvander@69
   126
        ZONE()->reportOutOfMemory();
dvander@0
   127
        return false;
dvander@0
   128
    }
dvander@0
   129
dvander@0
   130
    bytecode *newBuffer;
dvander@0
   131
    if (buffer_ == inline_buffer_)
dvander@177
   132
        newBuffer = (bytecode *)malloc(newSize);
dvander@0
   133
    else
dvander@0
   134
        newBuffer = (bytecode *)realloc(buffer_, newSize);
dvander@0
   135
dvander@0
   136
    // If we can't allocate a buffer, reset the buffer to the start and set
dvander@0
   137
    // a flag to indicate out-of-memory.
dvander@0
   138
    if (!newBuffer) {
dvander@0
   139
        oom_ = true;
dvander@0
   140
        buffer_pos_ = buffer_;
dvander@69
   141
        ZONE()->reportOutOfMemory();
dvander@0
   142
        return false;
dvander@0
   143
    }
dvander@0
   144
dvander@177
   145
    if (buffer_ == inline_buffer_)
dvander@177
   146
        memcpy(newBuffer, inline_buffer_, (buffer_pos_ - inline_buffer_));
dvander@177
   147
dvander@0
   148
    buffer_pos_ = newBuffer + size_t(buffer_pos_ - buffer_);
dvander@0
   149
    buffer_ = newBuffer;
dvander@0
   150
    buffer_end_ = buffer_ + newSize;
dvander@177
   151
    assert(buffer_pos_ + nbytes < buffer_end_);
dvander@177
   152
dvander@0
   153
    return true;
dvander@0
   154
}
dvander@0
   155
dvander@47
   156
void
dvander@47
   157
BytecodeEmitter::recordOops()
dvander@47
   158
{
dvander@195
   159
    OopRecord *record = new OopRecord(oopMap_.length(), position());
dvander@47
   160
dvander@195
   161
    if (!record->slots.ensure(record->depth)) {
dvander@47
   162
        oom_ = true;
dvander@47
   163
        return;
dvander@47
   164
    }
dvander@47
   165
dvander@47
   166
    bool hadOops = false;
dvander@58
   167
    for (size_t i = 0; i < oopMap_.length(); i++) {
dvander@47
   168
        bool bit = oopMap_[i];
dvander@195
   169
        record->slots.infallibleAppend(bit);
dvander@47
   170
        hadOops |= bit;
dvander@47
   171
    }
dvander@47
   172
dvander@195
   173
    if (!hadOops) {
dvander@195
   174
        delete record;
dvander@47
   175
        return;
dvander@195
   176
    }
dvander@47
   177
dvander@58
   178
    oom_ &= oopRecords_.append(record);
dvander@47
   179
}
dvander@47
   180
dvander@70
   181
void
dvander@70
   182
BytecodeEmitter::note_position(const SourcePosition &pos)
dvander@70
   183
{
dvander@70
   184
    assert(pos.line > 0);
dvander@70
   185
    assert(unsigned(pos.line) >= firstLine_);
dvander@70
   186
dvander@70
   187
    unsigned lineDelta = unsigned(pos.line) - firstLine_;
dvander@70
   188
    unsigned pcDelta = unsigned(position()) - lastCodePosition_;
dvander@70
   189
dvander@70
   190
    sourcemap_.writeUint32(pcDelta);
dvander@70
   191
    sourcemap_.writeUint32(lineDelta);
dvander@70
   192
    sourcemap_.writeUint32(pos.col);
dvander@70
   193
dvander@70
   194
    lastCodePosition_ = unsigned(position());
dvander@70
   195
}
dvander@195
   196
dvander@195
   197
void
dvander@195
   198
BytecodeEmitter::getglobal(Symbol *sym)
dvander@195
   199
{
dvander@195
   200
    ensure(OP_GETGLOBAL_LENGTH);
dvander@195
   201
    writeOp(OP_GETGLOBAL);
dvander@195
   202
    writeUint32(addString(sym->name()));
dvander@195
   203
    if (sym->type()->isTraceable())
dvander@195
   204
        markOop(1);
dvander@195
   205
}
dvander@195
   206
void
dvander@195
   207
BytecodeEmitter::setglobal(Symbol *sym)
dvander@195
   208
{
dvander@195
   209
    ensure(OP_SETGLOBAL_LENGTH);
dvander@195
   210
    writeOp(OP_SETGLOBAL);
dvander@195
   211
    writeUint32(addString(sym->name()));
dvander@195
   212
    if (sym->type()->isTraceable())
dvander@195
   213
        markOop(1);
dvander@195
   214
}
dvander@195
   215
dvander@195
   216
void
dvander@195
   217
BytecodeEmitter::getlocal(VariableSymbol *var)
dvander@195
   218
{
dvander@195
   219
    ensure(OP_GETLOCAL_LENGTH);
dvander@195
   220
    writeOp(OP_GETLOCAL);
dvander@195
   221
    writeUint32(var->slot());
dvander@195
   222
    if (var->type()->isTraceable())
dvander@195
   223
        markOop(1);
dvander@195
   224
}
dvander@195
   225
dvander@195
   226
void
dvander@195
   227
BytecodeEmitter::setlocal(VariableSymbol *var)
dvander@195
   228
{
dvander@195
   229
    ensure(OP_SETLOCAL_LENGTH);
dvander@195
   230
    writeOp(OP_SETLOCAL);
dvander@195
   231
    writeUint32(var->slot());
dvander@195
   232
    if (var->type()->isTraceable())
dvander@195
   233
        markOop(1);
dvander@195
   234
}
dvander@195
   235
dvander@195
   236
bool
dvander@195
   237
BytecodeEmitter::allocate(VariableSymbol *sym)
dvander@195
   238
{
dvander@195
   239
    assert(sym->isVariable());
dvander@195
   240
    assert(sym->scope()->kind() == Scope::BLOCK);
dvander@195
   241
dvander@195
   242
    switch (sym->scope()->kind()) {
dvander@195
   243
      case Scope::BLOCK:
dvander@195
   244
      {
dvander@195
   245
        // See overall comment on overflows.
dvander@195
   246
        assert(locals_.length() < UINT_MAX);
dvander@195
   247
        sym->allocate(VariableSymbol::Local, locals_.length());
dvander@195
   248
        if (!locals_.append(sym))
dvander@195
   249
            return false;
dvander@195
   250
        break;
dvander@195
   251
      }
dvander@195
   252
dvander@195
   253
      default:
dvander@195
   254
        assert(false);
dvander@195
   255
        return false;
dvander@195
   256
    }
dvander@195
   257
dvander@195
   258
    return true;
dvander@195
   259
}
dvander@195
   260
dvander@195
   261
Code *
dvander@195
   262
BytecodeEmitter::finish()
dvander@195
   263
{
dvander@195
   264
    stop();
dvander@195
   265
dvander@195
   266
    Zone *zone = ZONE();
dvander@195
   267
    if (oom() || zone->hasPendingException())
dvander@195
   268
        return NULL;
dvander@195
   269
dvander@195
   270
    Local<ByteArray> bytes(zone, ByteArray::New(zone, length(), Heap::Tenure_Old));
dvander@195
   271
    if (!bytes)
dvander@195
   272
        return NULL;
dvander@195
   273
    memcpy(bytes->raw(), buffer(), length());
dvander@195
   274
dvander@195
   275
    Local<FixedArray> objects(zone, FixedArray::New(zone, objects_.length(), Heap::Tenure_Old));
dvander@195
   276
    if (!objects)
dvander@195
   277
        return NULL;
dvander@195
   278
dvander@195
   279
    for (size_t i = 0; i < objects_.length(); i++)
dvander@195
   280
        objects->set(zone, i, objects_[i]);
dvander@195
   281
dvander@195
   282
    // If any stack slots hold gc pointers at potential failure points in the
dvander@195
   283
    // bytecode, record them now. We do this by building a list of tuples;
dvander@195
   284
    // each tuple contains the pc offset, and a bitset containing which stack
dvander@195
   285
    // slots must be scanned by the gc.
dvander@195
   286
    Local<ByteArray> safepoints(zone);
dvander@195
   287
    if (oopRecords_.length()) {
dvander@195
   288
        size_t bytesPerSafepoint = Code::BytesPerSafepoint(maxStackDepth());
dvander@195
   289
        size_t gcmapSize = bytesPerSafepoint * oopRecords_.length();
dvander@195
   290
        safepoints = ByteArray::New(zone, gcmapSize, Heap::Tenure_Old);
dvander@195
   291
        if (!safepoints)
dvander@195
   292
            return NULL;
dvander@195
   293
dvander@195
   294
        Address cursor = safepoints->raw();
dvander@195
   295
        for (size_t i = 0; i < oopRecords_.length(); i++) {
dvander@195
   296
            OopRecord *record = oopRecords_[i];
dvander@195
   297
            assert(record->depth <= maxStackDepth());
dvander@195
   298
dvander@195
   299
            *reinterpret_cast<unsigned *>(cursor) = record->offset;
dvander@195
   300
dvander@195
   301
            unsigned char *bits = reinterpret_cast<unsigned char *>(cursor + sizeof(unsigned));
dvander@195
   302
            for (unsigned i = 0; i < record->depth; i++)
dvander@195
   303
                bits[i / 8] |= (1 << (i % 8));
dvander@195
   304
dvander@195
   305
            cursor += bytesPerSafepoint;
dvander@195
   306
        }
dvander@195
   307
dvander@195
   308
        assert(cursor == safepoints->raw() + gcmapSize);
dvander@195
   309
    }
dvander@195
   310
dvander@195
   311
    Local<ByteArray> sourcemap(zone);
dvander@195
   312
    if (sourcemap_.length()) {
dvander@195
   313
        sourcemap = ByteArray::New(zone, sourcemap_.length(), Heap::Tenure_Old);
dvander@195
   314
        if (!sourcemap)
dvander@195
   315
            return NULL;
dvander@195
   316
dvander@195
   317
        memcpy(sourcemap->raw(), sourcemap_.buffer(), sourcemap_.length());
dvander@195
   318
    }
dvander@195
   319
dvander@195
   320
    Local<FixedArray> strings(zone, FixedArray::New(zone, strings_.stringCount(), Heap::Tenure_Old));
dvander@195
   321
    if (!strings)
dvander@195
   322
        return NULL;
dvander@195
   323
    for (size_t i = 0; i < strings_.stringCount(); i++)
dvander@195
   324
        strings->set(zone, i, strings_.stringAt(i));
dvander@195
   325
dvander@195
   326
    Local<FixedArray> locals(zone, FixedArray::New(zone, locals_.length(), Heap::Tenure_Old));
dvander@195
   327
    if (!locals)
dvander@195
   328
        return NULL;
dvander@195
   329
    for (size_t i = 0; i < locals_.length(); i++)
dvander@195
   330
        locals->set(zone, i, locals_[i]->type());
dvander@195
   331
dvander@195
   332
    Local<Code> code(zone);
dvander@195
   333
dvander@195
   334
    code = Code::New(zone,
dvander@195
   335
                     bytes,
dvander@195
   336
                     locals,
dvander@195
   337
                     objects,
dvander@195
   338
                     strings,
dvander@195
   339
                     maxStackDepth(),
dvander@195
   340
                     safepoints,
dvander@195
   341
                     sourcemap);
dvander@195
   342
dvander@195
   343
    return code;
dvander@195
   344
}