src/Utility.h
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 115 133bf2d87aa9
child 251 ca99a81745fe
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
     1 /* vim: set ts=4 sw=4 tw=99 et:
     2  *
     3  * Copyright (C) 2012 David Anderson
     4  *
     5  * This file is part of SourcePawn.
     6  *
     7  * SourcePawn is free software: you can redistribute it and/or modify it under
     8  * the terms of the GNU General Public License as published by the Free
     9  * Software Foundation, either version 3 of the License, or (at your option)
    10  * any later version.
    11  * 
    12  * SourcePawn is distributed in the hope that it will be useful, but WITHOUT ANY
    13  * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
    14  * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
    15  *
    16  * You should have received a copy of the GNU General Public License along with
    17  * SourcePawn. If not, see http://www.gnu.org/licenses/.
    18  */
    19 #ifndef _include_jitcraft_utility_h_
    20 #define _include_jitcraft_utility_h_
    21 
    22 #include <assert.h>
    23 #include <stddef.h>
    24 #include "../include/keima_util.h"
    25 #ifdef _MSC_VER
    26 # include <intrin.h>
    27 #endif
    28 
    29 #ifdef _MSC_VER
    30 # pragma warning(disable:4355)
    31 #endif
    32 
    33 namespace ke {
    34 
    35 static const size_t kMallocAlignment = sizeof(void *) * 2;
    36 
    37 typedef uint8_t uint8;
    38 typedef int32_t int32;
    39 typedef uint32_t uint32;
    40 typedef int64_t int64;
    41 typedef uint64_t uint64;
    42 
    43 typedef uint8 * Address;
    44 
    45 static const size_t kKB = 1024;
    46 static const size_t kMB = 1024 * kKB;
    47 static const size_t kGB = 1024 * kMB;
    48 
    49 template <typename T>
    50 class AutoVec
    51 {
    52     T *t_;
    53 
    54   public:
    55     AutoVec()
    56       : t_(NULL)
    57     {
    58     }
    59     AutoVec(T *t)
    60       : t_(t)
    61     {
    62     }
    63     ~AutoVec() {
    64         delete [] t_;
    65     }
    66     operator T *() {
    67         return t_;
    68     }
    69     void operator =(T *t) {
    70         if (t_)
    71             delete [] t_;
    72         t_ = t;
    73     }
    74 };
    75 
    76 // Bob Jenkin's one-at-a-time hash function[1].
    77 //
    78 // [1] http://burtleburtle.net/bob/hash/doobs.html
    79 class CharacterStreamHasher
    80 {
    81     uint32 hash;
    82 
    83   public:
    84     CharacterStreamHasher()
    85       : hash(0)
    86     { }
    87 
    88     void add(char c) {
    89         hash += c;
    90         hash += (hash << 10);
    91         hash ^= (hash >> 6);
    92     }
    93 
    94     void add(const char *s, size_t length) {
    95         for (size_t i = 0; i < length; i++)
    96             add(s[i]);
    97     }
    98 
    99     uint32 result() {
   100         hash += (hash << 3);
   101         hash ^= (hash >> 11);
   102         hash += (hash << 15);
   103         return hash;
   104     }
   105 };
   106 
   107 static inline uint32
   108 HashCharSequence(const char *s, size_t length)
   109 {
   110     CharacterStreamHasher hasher;
   111     hasher.add(s, length);
   112     return hasher.result();
   113 }
   114 
   115 // From http://burtleburtle.net/bob/hash/integer.html
   116 static inline uint32
   117 HashInt32(int32 a)
   118 {
   119     a = (a ^ 61) ^ (a >> 16);
   120     a = a + (a << 3);
   121     a = a ^ (a >> 4);
   122     a = a * 0x27d4eb2d;
   123     a = a ^ (a >> 15);
   124     return a;
   125 }
   126 
   127 // From http://www.cris.com/~Ttwang/tech/inthash.htm
   128 static inline uint32
   129 HashInt64(int64 key)
   130 {
   131   key = (~key) + (key << 18); // key = (key << 18) - key - 1;
   132   key = key ^ (uint64(key) >> 31);
   133   key = key * 21; // key = (key + (key << 2)) + (key << 4);
   134   key = key ^ (uint64(key) >> 11);
   135   key = key + (key << 6);
   136   key = key ^ (uint64(key) >> 22);
   137   return uint32(key);
   138 }
   139 
   140 static inline uint32
   141 HashPointer(void *p)
   142 {
   143 #if defined(KE_32BIT)
   144     return HashInt32(reinterpret_cast<int32>(p));
   145 #elif defined(KE_64BIT)
   146     return HashInt64(reinterpret_cast<int64>(p));
   147 #endif
   148 }
   149 
   150 static inline size_t
   151 Log2(size_t number)
   152 {
   153     assert(number != 0);
   154 
   155 #ifdef _MSC_VER
   156     unsigned long rval;
   157 # ifdef _M_IX86
   158     _BitScanReverse(&rval, number);
   159 # elif _M_X64
   160     _BitScanReverse64(&rval, number);
   161 # endif
   162     return rval;
   163 #else
   164 	size_t bit;
   165 	asm("bsr %1, %0\n"
   166 		: "=r" (bit)
   167 		: "rm" (number));
   168 	return bit;
   169 #endif
   170 }
   171 
   172 static inline size_t
   173 FindRightmostBit(size_t number)
   174 {
   175     assert(number != 0);
   176 
   177 #ifdef _MSC_VER
   178     unsigned long rval;
   179 # ifdef _M_IX86
   180     _BitScanForward(&rval, number);
   181 # elif _M_X64
   182     _BitScanForward64(&rval, number);
   183 # endif
   184     return rval;
   185 #else
   186 	size_t bit;
   187 	asm("bsf %1, %0\n"
   188 		: "=r" (bit)
   189 		: "rm" (number));
   190 	return bit;
   191 #endif
   192 }
   193 
   194 static inline bool
   195 IsPowerOfTwo(size_t value)
   196 {
   197     if (value == 0)
   198         return false;
   199     return !(value & (value - 1));
   200 }
   201 
   202 static inline size_t
   203 Align(size_t count, size_t alignment)
   204 {
   205     assert(IsPowerOfTwo(alignment));
   206     return count + (alignment - (count % alignment)) % alignment;
   207 }
   208 
   209 static inline bool
   210 IsUint32AddSafe(unsigned a, unsigned b)
   211 {
   212     if (!a || !b)
   213         return true;
   214     size_t log2_a = Log2(a);
   215     size_t log2_b = Log2(b);
   216     return (log2_a < sizeof(unsigned) * 8) &&
   217            (log2_b < sizeof(unsigned) * 8);
   218 }
   219 
   220 static inline bool
   221 IsUintPtrAddSafe(size_t a, size_t b)
   222 {
   223     if (!a || !b)
   224         return true;
   225     size_t log2_a = Log2(a);
   226     size_t log2_b = Log2(b);
   227     return (log2_a < sizeof(size_t) * 8) &&
   228            (log2_b < sizeof(size_t) * 8);
   229 }
   230 
   231 static inline bool
   232 IsUint32MultiplySafe(unsigned a, unsigned b)
   233 {
   234     if (a <= 1 || b <= 1)
   235         return true;
   236 
   237     size_t log2_a = Log2(a);
   238     size_t log2_b = Log2(b);
   239     return log2_a + log2_b <= sizeof(unsigned) * 8;
   240 }
   241 
   242 static inline bool
   243 IsUintPtrMultiplySafe(size_t a, size_t b)
   244 {
   245     if (a <= 1 || b <= 1)
   246         return true;
   247 
   248     size_t log2_a = Log2(a);
   249     size_t log2_b = Log2(b);
   250     return log2_a + log2_b <= sizeof(size_t) * 8;
   251 }
   252 
   253 #define ARRAY_LENGTH(array) (sizeof(array) / sizeof(array[0]))
   254 #define STATIC_ASSERT(cond) extern int static_assert_f(int a[(cond) ? 1 : -1])
   255 
   256 #define IS_ALIGNED(addr, alignment)	(!(uintptr_t(addr) & ((alignment) - 1)))
   257 
   258 template <typename T>
   259 static inline bool
   260 IsAligned(T addr, size_t alignment)
   261 {
   262     assert(IsPowerOfTwo(alignment));
   263     return !(uintptr_t(addr) & (alignment - 1));
   264 }
   265 
   266 static inline Address
   267 AlignedBase(Address addr, size_t alignment)
   268 {
   269     assert(IsPowerOfTwo(alignment));
   270     return Address(uintptr_t(addr) & ~(alignment - 1));
   271 }
   272 
   273 template <typename T> static inline T
   274 Min(const T &t1, const T &t2)
   275 {
   276     return t1 < t2 ? t1 : t2;
   277 }
   278 
   279 #define OFFSETOF(Class, Member) reinterpret_cast<size_t>(&((Class *)NULL)->Member)
   280 
   281 }
   282 
   283 #endif // _include_jitcraft_utility_h_