src/Opcodes.cpp
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 194 b029d63565a2
child 230 fb3cd08bc7fb
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
     1 /* vim: set ts=4 sw=4 tw=99 et:
     2  *
     3  * Copyright (C) 2012 David Anderson
     4  *
     5  * This file is part of JITCraft.
     6  *
     7  * JITCraft is free software: you can redistribute it and/or modify it under
     8  * the terms of the GNU General Public License as published by the Free
     9  * Software Foundation, either version 3 of the License, or (at your option)
    10  * any later version.
    11  * 
    12  * Foobar is distributed in the hope that it will be useful, but WITHOUT ANY
    13  * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    14  * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
    15  *
    16  * You should have received a copy of the GNU General Public License along with
    17  * JITCraft. If not, see http://www.gnu.org/licenses/.
    18  */
    19 #include <stdlib.h>
    20 #include "Opcodes.h"
    21 #include "Utility.h"
    22 #include "Code.h"
    23 #include "Zone.h"
    24 #include "Array.h"
    25 #include "Strings.h"
    26 #include "CompactBuffer.h"
    27 #include "Structures.h"
    28 #include "Heap-inl.h"
    29 
    30 using namespace ke;
    31 
    32 OpInfo ke::OpcodeInfo[] =
    33 {
    34 #define OPDEF(name, text, length, uses, defs) \
    35     {text, length, uses, defs},
    36 #include "Opcodes.tbl"
    37 #undef OPDEF
    38 };
    39 
    40 Code *
    41 Code::New(Zone *zone, Handle<ByteArray> bytecode, Handle<FixedArray> locals,
    42           Handle<FixedArray> objects, Handle<FixedArray> strings, unsigned maxStackDepth,
    43           Handle<ByteArray> safepoints, Handle<ByteArray> sourcemap)
    44 {
    45     Local<Code> code(zone, Code::cast(zone->allocate(MapKind_Code, sizeof(Code), Heap::Tenure_Old)));
    46     if (!code)
    47         return NULL;
    48 
    49     code->bytecode_ = bytecode;
    50     code->locals_ = locals;
    51     code->objects_ = objects;
    52     code->maxStackDepth_ = maxStackDepth;
    53     code->module_ = NULL;
    54     code->safepoints_ = safepoints;
    55     code->sourcemap_ = sourcemap;
    56     code->lineNumber_ = 0;
    57     code->strings_ = strings;
    58     return code;
    59 }
    60 
    61 static int
    62 CompareSafepoints(const void *a, const void *b)
    63 {
    64     const unsigned *sa = reinterpret_cast<const unsigned *>(a);
    65     const unsigned *sb = reinterpret_cast<const unsigned *>(b);
    66     if (*sa > *sb)
    67         return 1;
    68     if (*sa == *sb)
    69         return 0;
    70     return -1;
    71 }
    72 
    73 uint8 *
    74 Code::safepointAt(unsigned pcOffset)
    75 {
    76     // Only specific opcodes may have safepoints. Asking for a safepoint at
    77     // any other position indicates a failure to mark that position as a
    78     // safepoint, which is a bug.
    79 #ifndef NDEBUG
    80     Opcode op = Opcode(*(start() + pcOffset));
    81     assert(op == OP_CALL ||
    82            op == OP_NEWFIXED ||
    83            op == OP_NEWEMPTY ||
    84            op == OP_NEWSIZED ||
    85            op == OP_VARREF_LIFO ||
    86            op == OP_ARGREF_LIFO ||
    87            op == OP_SLOTREF_LIFO ||
    88            op == OP_COPYARRAY ||
    89            op == OP_CALLNATIVE ||
    90            op == OP_PUBLIC);
    91 #endif
    92 
    93     if (!safepoints_)
    94         return NULL;
    95 
    96     size_t bytesPerSafepoint = BytesPerSafepoint(maxStackDepth());
    97     assert(!(safepoints_->length() % bytesPerSafepoint));
    98 
    99     size_t count = safepoints_->length() / bytesPerSafepoint;
   100 
   101     void *result = bsearch(&pcOffset,
   102                            safepoints_->raw(),
   103                            count,
   104                            bytesPerSafepoint,
   105                            CompareSafepoints);
   106 
   107     if (!result)
   108         return NULL;
   109 
   110     assert(*reinterpret_cast<unsigned *>(result) == pcOffset);
   111 
   112     return Address(result) + sizeof(unsigned);
   113 }
   114 
   115 SourcePosition
   116 Code::pcToSource(unsigned pc)
   117 {
   118     assert(pc < bytecode_->length());
   119 
   120     CompactBufferReader reader(sourcemap_->raw(), sourcemap_->raw() + sourcemap_->length());
   121 
   122     unsigned line = 1;
   123     unsigned col = 0;
   124     unsigned curPc = 0;
   125     while (!reader.done()) {
   126         curPc += reader.readUint32();
   127         if (curPc > pc)
   128             break;
   129 
   130         line = reader.readUint32();
   131         col = reader.readUint32();
   132     }
   133 
   134     return SourcePosition(lineNumber_ + line, col);
   135 }
   136 
   137 // All jumps must be the same length.
   138 STATIC_ASSERT(OP_JT_LENGTH == OP_JUMP_LENGTH);
   139 STATIC_ASSERT(OP_JF_LENGTH == OP_JUMP_LENGTH);
   140 
   141 void
   142 ke::Disassemble(Code *code, FILE *fp)
   143 {
   144     bytecode *pc = code->start();
   145     for (;;) {
   146         Opcode op = Opcode(*pc);
   147 
   148         fprintf(fp, "%-4d %s ", int(pc - code->start()), OpcodeInfo[op].name);
   149 
   150         switch (op) {
   151           case OP_NOP:
   152           case OP_RETURN:
   153           case OP_RETURNVOID:
   154           case OP_ADD:
   155           case OP_SUB:
   156           case OP_MUL:
   157           case OP_DIV:
   158           case OP_ADD_F:
   159           case OP_SUB_F:
   160           case OP_MUL_F:
   161           case OP_DIV_F:
   162           case OP_CVT_I2F:
   163           case OP_POP:
   164           case OP_GETELEM:
   165           case OP_SETELEM:
   166           case OP_SHL:
   167           case OP_SHR:
   168           case OP_USHR:
   169           case OP_BITAND:
   170           case OP_BITOR:
   171           case OP_BITXOR:
   172           case OP_GETREF_I:
   173           case OP_SETREF_I:
   174           case OP_GETREF_F:
   175           case OP_SETREF_F:
   176           case OP_SLOTREF_LIFO:
   177           case OP_COPYARRAY:
   178           case OP_COPYSTRUCT:
   179           case OP_DUP:
   180           case OP_DUP2:
   181           case OP_SWAP:
   182           case OP_ROLL3:
   183           case OP_PICK2:
   184           case OP_PICK3:
   185           case OP_CALL:
   186           case OP_LT:
   187           case OP_LE:
   188           case OP_GE:
   189           case OP_GT:
   190           case OP_EQ:
   191           case OP_NE:
   192           case OP_LT_F:
   193           case OP_LE_F:
   194           case OP_GE_F:
   195           case OP_GT_F:
   196           case OP_EQ_F:
   197           case OP_NE_F:
   198           case OP_BITNOT:
   199           case OP_NOT:
   200           case OP_NOT_F:
   201           case OP_NEG:
   202           case OP_NEG_F:
   203           case OP_MOD:
   204           case OP_BITS_I2F:
   205           case OP_BITS_F2I:
   206           case OP_TRUE:
   207           case OP_FALSE:
   208           case OP_SIZEOF:
   209           case OP_PUBLIC:
   210             break;
   211 
   212           case OP_STOP:
   213             fprintf(fp, "\n");
   214             break;
   215 
   216           case OP_INT:
   217             fprintf(fp, "%d", READ_INT32(pc));
   218             break;
   219 
   220           case OP_FLOAT:
   221             fprintf(fp, "%f", READ_FLOAT(pc));
   222             break;
   223 
   224           case OP_JT:
   225           case OP_JF:
   226           case OP_JUMP:
   227             fprintf(fp, "%u (%+d)", unsigned((pc + READ_INT32(pc)) - code->start()), READ_INT32(pc));
   228             break;
   229 
   230           case OP_NEWFIXED:
   231           case OP_NEWEMPTY:
   232           case OP_NEWSIZED:
   233           {
   234             Heap::Tenure tenure = (Heap::Tenure)pc[1];
   235             //ArrayMap *map = ArrayMap::cast(code->map(READ_UINT32(pc + 1)));
   236             if (tenure == Heap::Tenure_Stack)
   237                 fprintf(fp, " ; lifo");
   238             else if (tenure == Heap::Tenure_Old)
   239                 fprintf(fp, " ; pretenured");
   240             else if (tenure == Heap::Tenure_Default)
   241                 fprintf(fp, " ; newborn");
   242             break;
   243           }
   244 
   245           case OP_NEWSTRUCT:
   246           {
   247             Heap::Tenure tenure = (Heap::Tenure)pc[1];
   248             StructMap *map = StructMap::cast(code->map(READ_UINT32(pc + 1)));
   249             fprintf(fp, "%s", map->type()->name()->chars());
   250             if (tenure == Heap::Tenure_Stack)
   251                 fprintf(fp, " ; lifo");
   252             else if (tenure == Heap::Tenure_Old)
   253                 fprintf(fp, " ; pretenured");
   254             else if (tenure == Heap::Tenure_Default)
   255                 fprintf(fp, " ; newborn");
   256             break;
   257           }
   258 
   259           case OP_GETLOCAL:
   260           case OP_SETLOCAL:
   261           case OP_GETARG:
   262           case OP_SETARG:
   263           case OP_SAVELIFO:
   264           case OP_UNWINDLIFO:
   265           case OP_VARREF_LIFO:
   266           case OP_ARGREF_LIFO:
   267           case OP_OBJECT:
   268           case OP_GETFIELD:
   269           case OP_SETFIELD:
   270             fprintf(fp, "%u", READ_UINT32(pc));
   271             break;
   272 
   273           case OP_CALLNATIVE:
   274           {
   275             unsigned argc = READ_UINT32(pc);
   276             fprintf(fp, " ; %u args", argc);
   277             break;
   278           }
   279 
   280           case OP_TABLESWITCH:
   281           {
   282             int low = READ_TABLESWITCH_LOW(pc);
   283             int high = READ_TABLESWITCH_HIGH(pc);
   284             fprintf(fp, "%d, %d\n", low, high);
   285             break;
   286           }
   287 
   288           case OP_GETGLOBAL:
   289           case OP_SETGLOBAL:
   290           {
   291             unsigned index = READ_UINT32(pc);
   292             fprintf(fp, "%s", String::cast(code->strings()->at(index))->chars());
   293             break;
   294           }
   295 
   296           default:
   297             // Unknown opcode!
   298             assert(false);
   299             return;
   300         }
   301 
   302         if (op == OP_STOP)
   303             break;
   304 
   305         fprintf(fp, "\n");
   306         pc += OpcodeInfo[op].length;
   307         assert(pc < code->start() + code->length());
   308     }
   309 
   310     fprintf(fp, "\nSafepoints:\n");
   311     ByteArray *safepoints = code->safepoints();
   312     if (safepoints) {
   313         size_t nsafepoints = safepoints->length() / Code::BytesPerSafepoint(code->maxStackDepth());
   314         for (size_t i = 0; i < nsafepoints; i++) {
   315             Address address = safepoints->raw() + Code::BytesPerSafepoint(code->maxStackDepth()) * i;
   316             fprintf(fp, "%4d: ", *reinterpret_cast<unsigned *>(address));
   317             unsigned char *bits = (unsigned char *)(address + sizeof(unsigned));
   318             for (unsigned j = 0; j < code->maxStackDepth(); j++) {
   319                 if (bits[j / 8] & (1 << (j % 8)))
   320                     fprintf(fp, "%d ", j);
   321             }
   322             fprintf(fp, "\n");
   323         }
   324     }
   325 }