src/Opcodes.cpp
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 194 b029d63565a2
child 230 fb3cd08bc7fb
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
dvander@47
     1
/* vim: set ts=4 sw=4 tw=99 et:
dvander@47
     2
 *
dvander@0
     3
 * Copyright (C) 2012 David Anderson
dvander@0
     4
 *
dvander@0
     5
 * This file is part of JITCraft.
dvander@0
     6
 *
dvander@0
     7
 * JITCraft is free software: you can redistribute it and/or modify it under
dvander@0
     8
 * the terms of the GNU General Public License as published by the Free
dvander@0
     9
 * Software Foundation, either version 3 of the License, or (at your option)
dvander@0
    10
 * any later version.
dvander@0
    11
 * 
dvander@0
    12
 * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
dvander@0
    13
 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
dvander@0
    14
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
dvander@0
    15
 *
dvander@0
    16
 * You should have received a copy of the GNU General Public License along with
dvander@0
    17
 * JITCraft. If not, see http://www.gnu.org/licenses/.
dvander@0
    18
 */
dvander@54
    19
#include <stdlib.h>
dvander@0
    20
#include "Opcodes.h"
dvander@0
    21
#include "Utility.h"
dvander@0
    22
#include "Code.h"
dvander@12
    23
#include "Zone.h"
dvander@12
    24
#include "Array.h"
dvander@54
    25
#include "Strings.h"
dvander@70
    26
#include "CompactBuffer.h"
dvander@147
    27
#include "Structures.h"
dvander@12
    28
#include "Heap-inl.h"
dvander@0
    29
dvander@59
    30
using namespace ke;
dvander@0
    31
dvander@59
    32
OpInfo ke::OpcodeInfo[] =
dvander@0
    33
{
dvander@195
    34
#define OPDEF(name, text, length, uses, defs) \
dvander@0
    35
    {text, length, uses, defs},
dvander@0
    36
#include "Opcodes.tbl"
dvander@0
    37
#undef OPDEF
dvander@0
    38
};
dvander@0
    39
dvander@12
    40
Code *
dvander@34
    41
Code::New(Zone *zone, Handle<ByteArray> bytecode, Handle<FixedArray> locals,
dvander@195
    42
          Handle<FixedArray> objects, Handle<FixedArray> strings, unsigned maxStackDepth,
dvander@195
    43
          Handle<ByteArray> safepoints, Handle<ByteArray> sourcemap)
dvander@12
    44
{
dvander@34
    45
    Local<Code> code(zone, Code::cast(zone->allocate(MapKind_Code, sizeof(Code), Heap::Tenure_Old)));
dvander@34
    46
    if (!code)
dvander@12
    47
        return NULL;
dvander@12
    48
dvander@13
    49
    code->bytecode_ = bytecode;
dvander@13
    50
    code->locals_ = locals;
dvander@21
    51
    code->objects_ = objects;
dvander@13
    52
    code->maxStackDepth_ = maxStackDepth;
dvander@195
    53
    code->module_ = NULL;
dvander@47
    54
    code->safepoints_ = safepoints;
dvander@70
    55
    code->sourcemap_ = sourcemap;
dvander@195
    56
    code->lineNumber_ = 0;
dvander@195
    57
    code->strings_ = strings;
dvander@12
    58
    return code;
dvander@12
    59
}
dvander@12
    60
dvander@47
    61
static int
dvander@47
    62
CompareSafepoints(const void *a, const void *b)
dvander@47
    63
{
dvander@47
    64
    const unsigned *sa = reinterpret_cast<const unsigned *>(a);
dvander@47
    65
    const unsigned *sb = reinterpret_cast<const unsigned *>(b);
dvander@47
    66
    if (*sa > *sb)
dvander@47
    67
        return 1;
dvander@47
    68
    if (*sa == *sb)
dvander@47
    69
        return 0;
dvander@47
    70
    return -1;
dvander@47
    71
}
dvander@47
    72
dvander@47
    73
uint8 *
dvander@47
    74
Code::safepointAt(unsigned pcOffset)
dvander@47
    75
{
dvander@47
    76
    // Only specific opcodes may have safepoints. Asking for a safepoint at
dvander@47
    77
    // any other position indicates a failure to mark that position as a
dvander@47
    78
    // safepoint, which is a bug.
danderson@93
    79
#ifndef NDEBUG
dvander@47
    80
    Opcode op = Opcode(*(start() + pcOffset));
dvander@185
    81
    assert(op == OP_CALL ||
dvander@185
    82
           op == OP_NEWFIXED ||
dvander@185
    83
           op == OP_NEWEMPTY ||
dvander@185
    84
           op == OP_NEWSIZED ||
dvander@185
    85
           op == OP_VARREF_LIFO ||
dvander@185
    86
           op == OP_ARGREF_LIFO ||
dvander@185
    87
           op == OP_SLOTREF_LIFO ||
dvander@195
    88
           op == OP_COPYARRAY ||
dvander@195
    89
           op == OP_CALLNATIVE ||
dvander@195
    90
           op == OP_PUBLIC);
danderson@93
    91
#endif
dvander@47
    92
dvander@47
    93
    if (!safepoints_)
dvander@47
    94
        return NULL;
dvander@47
    95
dvander@47
    96
    size_t bytesPerSafepoint = BytesPerSafepoint(maxStackDepth());
dvander@47
    97
    assert(!(safepoints_->length() % bytesPerSafepoint));
dvander@47
    98
dvander@47
    99
    size_t count = safepoints_->length() / bytesPerSafepoint;
dvander@47
   100
dvander@47
   101
    void *result = bsearch(&pcOffset,
dvander@47
   102
                           safepoints_->raw(),
dvander@47
   103
                           count,
dvander@47
   104
                           bytesPerSafepoint,
dvander@47
   105
                           CompareSafepoints);
dvander@47
   106
dvander@47
   107
    if (!result)
dvander@47
   108
        return NULL;
dvander@47
   109
dvander@47
   110
    assert(*reinterpret_cast<unsigned *>(result) == pcOffset);
dvander@47
   111
dvander@47
   112
    return Address(result) + sizeof(unsigned);
dvander@47
   113
}
dvander@47
   114
dvander@70
   115
SourcePosition
dvander@70
   116
Code::pcToSource(unsigned pc)
dvander@70
   117
{
dvander@70
   118
    assert(pc < bytecode_->length());
dvander@70
   119
dvander@70
   120
    CompactBufferReader reader(sourcemap_->raw(), sourcemap_->raw() + sourcemap_->length());
dvander@70
   121
dvander@70
   122
    unsigned line = 1;
dvander@70
   123
    unsigned col = 0;
dvander@70
   124
    unsigned curPc = 0;
dvander@70
   125
    while (!reader.done()) {
dvander@70
   126
        curPc += reader.readUint32();
dvander@70
   127
        if (curPc > pc)
dvander@70
   128
            break;
dvander@70
   129
dvander@70
   130
        line = reader.readUint32();
dvander@70
   131
        col = reader.readUint32();
dvander@70
   132
    }
dvander@70
   133
dvander@70
   134
    return SourcePosition(lineNumber_ + line, col);
dvander@70
   135
}
dvander@70
   136
dvander@0
   137
// All jumps must be the same length.
dvander@0
   138
STATIC_ASSERT(OP_JT_LENGTH == OP_JUMP_LENGTH);
dvander@0
   139
STATIC_ASSERT(OP_JF_LENGTH == OP_JUMP_LENGTH);
dvander@0
   140
dvander@0
   141
void
dvander@59
   142
ke::Disassemble(Code *code, FILE *fp)
dvander@0
   143
{
dvander@0
   144
    bytecode *pc = code->start();
dvander@0
   145
    for (;;) {
dvander@0
   146
        Opcode op = Opcode(*pc);
dvander@0
   147
dvander@0
   148
        fprintf(fp, "%-4d %s ", int(pc - code->start()), OpcodeInfo[op].name);
dvander@0
   149
dvander@0
   150
        switch (op) {
dvander@0
   151
          case OP_NOP:
dvander@0
   152
          case OP_RETURN:
dvander@72
   153
          case OP_RETURNVOID:
dvander@0
   154
          case OP_ADD:
dvander@0
   155
          case OP_SUB:
dvander@0
   156
          case OP_MUL:
dvander@0
   157
          case OP_DIV:
dvander@4
   158
          case OP_ADD_F:
dvander@4
   159
          case OP_SUB_F:
dvander@4
   160
          case OP_MUL_F:
dvander@4
   161
          case OP_DIV_F:
dvander@4
   162
          case OP_CVT_I2F:
dvander@0
   163
          case OP_POP:
dvander@157
   164
          case OP_GETELEM:
dvander@157
   165
          case OP_SETELEM:
dvander@8
   166
          case OP_SHL:
dvander@8
   167
          case OP_SHR:
dvander@8
   168
          case OP_USHR:
dvander@8
   169
          case OP_BITAND:
dvander@8
   170
          case OP_BITOR:
dvander@8
   171
          case OP_BITXOR:
dvander@14
   172
          case OP_GETREF_I:
dvander@14
   173
          case OP_SETREF_I:
dvander@14
   174
          case OP_GETREF_F:
dvander@14
   175
          case OP_SETREF_F:
dvander@16
   176
          case OP_SLOTREF_LIFO:
dvander@152
   177
          case OP_COPYARRAY:
dvander@152
   178
          case OP_COPYSTRUCT:
dvander@22
   179
          case OP_DUP:
dvander@22
   180
          case OP_DUP2:
dvander@23
   181
          case OP_SWAP:
dvander@23
   182
          case OP_ROLL3:
dvander@23
   183
          case OP_PICK2:
dvander@23
   184
          case OP_PICK3:
dvander@73
   185
          case OP_CALL:
dvander@74
   186
          case OP_LT:
dvander@74
   187
          case OP_LE:
dvander@74
   188
          case OP_GE:
dvander@74
   189
          case OP_GT:
dvander@74
   190
          case OP_EQ:
dvander@74
   191
          case OP_NE:
dvander@74
   192
          case OP_LT_F:
dvander@74
   193
          case OP_LE_F:
dvander@74
   194
          case OP_GE_F:
dvander@74
   195
          case OP_GT_F:
dvander@74
   196
          case OP_EQ_F:
dvander@74
   197
          case OP_NE_F:
dvander@78
   198
          case OP_BITNOT:
dvander@79
   199
          case OP_NOT:
dvander@79
   200
          case OP_NOT_F:
dvander@80
   201
          case OP_NEG:
dvander@80
   202
          case OP_NEG_F:
dvander@82
   203
          case OP_MOD:
dvander@142
   204
          case OP_BITS_I2F:
dvander@142
   205
          case OP_BITS_F2I:
dvander@143
   206
          case OP_TRUE:
dvander@143
   207
          case OP_FALSE:
dvander@194
   208
          case OP_SIZEOF:
dvander@195
   209
          case OP_PUBLIC:
dvander@0
   210
            break;
dvander@0
   211
dvander@0
   212
          case OP_STOP:
dvander@0
   213
            fprintf(fp, "\n");
dvander@47
   214
            break;
dvander@0
   215
dvander@0
   216
          case OP_INT:
dvander@0
   217
            fprintf(fp, "%d", READ_INT32(pc));
dvander@0
   218
            break;
dvander@0
   219
dvander@4
   220
          case OP_FLOAT:
dvander@4
   221
            fprintf(fp, "%f", READ_FLOAT(pc));
dvander@4
   222
            break;
dvander@4
   223
dvander@0
   224
          case OP_JT:
dvander@0
   225
          case OP_JF:
dvander@0
   226
          case OP_JUMP:
dvander@5
   227
            fprintf(fp, "%u (%+d)", unsigned((pc + READ_INT32(pc)) - code->start()), READ_INT32(pc));
dvander@0
   228
            break;
dvander@0
   229
dvander@185
   230
          case OP_NEWFIXED:
dvander@185
   231
          case OP_NEWEMPTY:
dvander@185
   232
          case OP_NEWSIZED:
dvander@0
   233
          {
dvander@85
   234
            Heap::Tenure tenure = (Heap::Tenure)pc[1];
dvander@154
   235
            //ArrayMap *map = ArrayMap::cast(code->map(READ_UINT32(pc + 1)));
dvander@85
   236
            if (tenure == Heap::Tenure_Stack)
dvander@85
   237
                fprintf(fp, " ; lifo");
dvander@85
   238
            else if (tenure == Heap::Tenure_Old)
dvander@85
   239
                fprintf(fp, " ; pretenured");
dvander@182
   240
            else if (tenure == Heap::Tenure_Default)
dvander@182
   241
                fprintf(fp, " ; newborn");
dvander@12
   242
            break;
dvander@0
   243
          }
dvander@0
   244
dvander@147
   245
          case OP_NEWSTRUCT:
dvander@147
   246
          {
dvander@147
   247
            Heap::Tenure tenure = (Heap::Tenure)pc[1];
dvander@147
   248
            StructMap *map = StructMap::cast(code->map(READ_UINT32(pc + 1)));
dvander@147
   249
            fprintf(fp, "%s", map->type()->name()->chars());
dvander@147
   250
            if (tenure == Heap::Tenure_Stack)
dvander@147
   251
                fprintf(fp, " ; lifo");
dvander@147
   252
            else if (tenure == Heap::Tenure_Old)
dvander@147
   253
                fprintf(fp, " ; pretenured");
dvander@182
   254
            else if (tenure == Heap::Tenure_Default)
dvander@182
   255
                fprintf(fp, " ; newborn");
dvander@147
   256
            break;
dvander@147
   257
          }
dvander@147
   258
dvander@0
   259
          case OP_GETLOCAL:
dvander@0
   260
          case OP_SETLOCAL:
dvander@0
   261
          case OP_GETARG:
dvander@0
   262
          case OP_SETARG:
dvander@12
   263
          case OP_SAVELIFO:
dvander@12
   264
          case OP_UNWINDLIFO:
dvander@14
   265
          case OP_VARREF_LIFO:
dvander@14
   266
          case OP_ARGREF_LIFO:
dvander@72
   267
          case OP_OBJECT:
dvander@148
   268
          case OP_GETFIELD:
dvander@148
   269
          case OP_SETFIELD:
dvander@0
   270
            fprintf(fp, "%u", READ_UINT32(pc));
dvander@0
   271
            break;
dvander@0
   272
dvander@68
   273
          case OP_CALLNATIVE:
dvander@195
   274
          {
dvander@195
   275
            unsigned argc = READ_UINT32(pc);
dvander@195
   276
            fprintf(fp, " ; %u args", argc);
dvander@68
   277
            break;
dvander@195
   278
          }
dvander@68
   279
dvander@172
   280
          case OP_TABLESWITCH:
dvander@172
   281
          {
dvander@172
   282
            int low = READ_TABLESWITCH_LOW(pc);
dvander@172
   283
            int high = READ_TABLESWITCH_HIGH(pc);
dvander@172
   284
            fprintf(fp, "%d, %d\n", low, high);
dvander@172
   285
            break;
dvander@172
   286
          }
dvander@172
   287
dvander@195
   288
          case OP_GETGLOBAL:
dvander@195
   289
          case OP_SETGLOBAL:
dvander@195
   290
          {
dvander@195
   291
            unsigned index = READ_UINT32(pc);
dvander@195
   292
            fprintf(fp, "%s", String::cast(code->strings()->at(index))->chars());
dvander@195
   293
            break;
dvander@195
   294
          }
dvander@195
   295
dvander@0
   296
          default:
dvander@0
   297
            // Unknown opcode!
dvander@0
   298
            assert(false);
dvander@0
   299
            return;
dvander@0
   300
        }
dvander@0
   301
dvander@47
   302
        if (op == OP_STOP)
dvander@47
   303
            break;
dvander@47
   304
dvander@0
   305
        fprintf(fp, "\n");
dvander@0
   306
        pc += OpcodeInfo[op].length;
dvander@12
   307
        assert(pc < code->start() + code->length());
dvander@0
   308
    }
dvander@47
   309
dvander@47
   310
    fprintf(fp, "\nSafepoints:\n");
dvander@47
   311
    ByteArray *safepoints = code->safepoints();
dvander@47
   312
    if (safepoints) {
dvander@47
   313
        size_t nsafepoints = safepoints->length() / Code::BytesPerSafepoint(code->maxStackDepth());
dvander@47
   314
        for (size_t i = 0; i < nsafepoints; i++) {
dvander@47
   315
            Address address = safepoints->raw() + Code::BytesPerSafepoint(code->maxStackDepth()) * i;
dvander@47
   316
            fprintf(fp, "%4d: ", *reinterpret_cast<unsigned *>(address));
dvander@47
   317
            unsigned char *bits = (unsigned char *)(address + sizeof(unsigned));
dvander@47
   318
            for (unsigned j = 0; j < code->maxStackDepth(); j++) {
dvander@47
   319
                if (bits[j / 8] & (1 << (j % 8)))
dvander@47
   320
                    fprintf(fp, "%d ", j);
dvander@47
   321
            }
dvander@47
   322
            fprintf(fp, "\n");
dvander@47
   323
        }
dvander@47
   324
    }
dvander@0
   325
}