src/Utility.h
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 115 133bf2d87aa9
child 251 ca99a81745fe
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
dvander@49
     1
/* vim: set ts=4 sw=4 tw=99 et:
dvander@49
     2
 *
dvander@28
     3
 * Copyright (C) 2012 David Anderson
dvander@28
     4
 *
dvander@115
     5
 * This file is part of SourcePawn.
dvander@28
     6
 *
dvander@115
     7
 * SourcePawn is free software: you can redistribute it and/or modify it under
dvander@28
     8
 * the terms of the GNU General Public License as published by the Free
dvander@28
     9
 * Software Foundation, either version 3 of the License, or (at your option)
dvander@28
    10
 * any later version.
dvander@28
    11
 * 
dvander@115
    12
 * SourcePawn is distributed in the hope that it will be useful, but WITHOUT ANY
dvander@28
    13
 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
dvander@28
    14
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
dvander@28
    15
 *
dvander@28
    16
 * You should have received a copy of the GNU General Public License along with
dvander@115
    17
 * SourcePawn. If not, see http://www.gnu.org/licenses/.
dvander@28
    18
 */
dvander@28
    19
#ifndef _include_jitcraft_utility_h_
dvander@28
    20
#define _include_jitcraft_utility_h_
dvander@28
    21
dvander@28
    22
#include <assert.h>
danderson@92
    23
#include <stddef.h>
dvander@105
    24
#include "../include/keima_util.h"
dvander@28
    25
#ifdef _MSC_VER
dvander@28
    26
# include <intrin.h>
dvander@28
    27
#endif
dvander@28
    28
dvander@28
    29
#ifdef _MSC_VER
dvander@28
    30
# pragma warning(disable:4355)
dvander@28
    31
#endif
dvander@28
    32
dvander@59
    33
namespace ke {
dvander@28
    34
dvander@28
    35
static const size_t kMallocAlignment = sizeof(void *) * 2;
dvander@28
    36
dvander@28
    37
typedef uint8_t uint8;
dvander@28
    38
typedef int32_t int32;
dvander@28
    39
typedef uint32_t uint32;
dvander@195
    40
typedef int64_t int64;
dvander@195
    41
typedef uint64_t uint64;
dvander@28
    42
dvander@28
    43
typedef uint8 * Address;
dvander@28
    44
dvander@28
    45
static const size_t kKB = 1024;
dvander@28
    46
static const size_t kMB = 1024 * kKB;
dvander@28
    47
static const size_t kGB = 1024 * kMB;
dvander@28
    48
dvander@28
    49
template <typename T>
dvander@28
    50
class AutoVec
dvander@28
    51
{
dvander@28
    52
    T *t_;
dvander@28
    53
dvander@28
    54
  public:
dvander@28
    55
    AutoVec()
dvander@28
    56
      : t_(NULL)
dvander@28
    57
    {
dvander@28
    58
    }
dvander@28
    59
    AutoVec(T *t)
dvander@28
    60
      : t_(t)
dvander@28
    61
    {
dvander@28
    62
    }
dvander@28
    63
    ~AutoVec() {
dvander@28
    64
        delete [] t_;
dvander@28
    65
    }
dvander@28
    66
    operator T *() {
dvander@28
    67
        return t_;
dvander@28
    68
    }
dvander@28
    69
    void operator =(T *t) {
dvander@28
    70
        if (t_)
dvander@28
    71
            delete [] t_;
dvander@28
    72
        t_ = t;
dvander@28
    73
    }
dvander@28
    74
};
dvander@28
    75
dvander@54
    76
// Bob Jenkin's one-at-a-time hash function[1].
dvander@54
    77
//
dvander@54
    78
// [1] http://burtleburtle.net/bob/hash/doobs.html
dvander@54
    79
class CharacterStreamHasher
dvander@54
    80
{
dvander@54
    81
    uint32 hash;
dvander@54
    82
dvander@54
    83
  public:
dvander@54
    84
    CharacterStreamHasher()
dvander@54
    85
      : hash(0)
dvander@54
    86
    { }
dvander@54
    87
dvander@54
    88
    void add(char c) {
dvander@54
    89
        hash += c;
dvander@54
    90
        hash += (hash << 10);
dvander@54
    91
        hash ^= (hash >> 6);
dvander@54
    92
    }
dvander@54
    93
dvander@54
    94
    void add(const char *s, size_t length) {
dvander@54
    95
        for (size_t i = 0; i < length; i++)
dvander@54
    96
            add(s[i]);
dvander@54
    97
    }
dvander@54
    98
dvander@54
    99
    uint32 result() {
dvander@54
   100
        hash += (hash << 3);
dvander@54
   101
        hash ^= (hash >> 11);
dvander@54
   102
        hash += (hash << 15);
dvander@54
   103
        return hash;
dvander@54
   104
    }
dvander@54
   105
};
dvander@54
   106
dvander@54
   107
static inline uint32
dvander@54
   108
HashCharSequence(const char *s, size_t length)
dvander@54
   109
{
dvander@54
   110
    CharacterStreamHasher hasher;
dvander@54
   111
    hasher.add(s, length);
dvander@54
   112
    return hasher.result();
dvander@54
   113
}
dvander@54
   114
dvander@195
   115
// From http://burtleburtle.net/bob/hash/integer.html
dvander@195
   116
static inline uint32
dvander@195
   117
HashInt32(int32 a)
dvander@195
   118
{
dvander@195
   119
    a = (a ^ 61) ^ (a >> 16);
dvander@195
   120
    a = a + (a << 3);
dvander@195
   121
    a = a ^ (a >> 4);
dvander@195
   122
    a = a * 0x27d4eb2d;
dvander@195
   123
    a = a ^ (a >> 15);
dvander@195
   124
    return a;
dvander@195
   125
}
dvander@195
   126
dvander@195
   127
// From http://www.cris.com/~Ttwang/tech/inthash.htm
dvander@195
   128
static inline uint32
dvander@195
   129
HashInt64(int64 key)
dvander@195
   130
{
dvander@195
   131
  key = (~key) + (key << 18); // key = (key << 18) - key - 1;
dvander@195
   132
  key = key ^ (uint64(key) >> 31);
dvander@195
   133
  key = key * 21; // key = (key + (key << 2)) + (key << 4);
dvander@195
   134
  key = key ^ (uint64(key) >> 11);
dvander@195
   135
  key = key + (key << 6);
dvander@195
   136
  key = key ^ (uint64(key) >> 22);
dvander@195
   137
  return uint32(key);
dvander@195
   138
}
dvander@195
   139
dvander@195
   140
static inline uint32
dvander@195
   141
HashPointer(void *p)
dvander@195
   142
{
dvander@195
   143
#if defined(KE_32BIT)
dvander@195
   144
    return HashInt32(reinterpret_cast<int32>(p));
dvander@195
   145
#elif defined(KE_64BIT)
dvander@195
   146
    return HashInt64(reinterpret_cast<int64>(p));
dvander@195
   147
#endif
dvander@195
   148
}
dvander@195
   149
dvander@28
   150
static inline size_t
dvander@28
   151
Log2(size_t number)
dvander@28
   152
{
dvander@28
   153
    assert(number != 0);
dvander@28
   154
dvander@53
   155
#ifdef _MSC_VER
dvander@28
   156
    unsigned long rval;
dvander@86
   157
# ifdef _M_IX86
dvander@28
   158
    _BitScanReverse(&rval, number);
dvander@86
   159
# elif _M_X64
dvander@86
   160
    _BitScanReverse64(&rval, number);
dvander@86
   161
# endif
dvander@28
   162
    return rval;
dvander@28
   163
#else
dvander@28
   164
	size_t bit;
dvander@28
   165
	asm("bsr %1, %0\n"
dvander@28
   166
		: "=r" (bit)
dvander@28
   167
		: "rm" (number));
dvander@28
   168
	return bit;
dvander@28
   169
#endif
dvander@28
   170
}
dvander@28
   171
dvander@53
   172
static inline size_t
dvander@53
   173
FindRightmostBit(size_t number)
dvander@53
   174
{
dvander@53
   175
    assert(number != 0);
dvander@53
   176
dvander@53
   177
#ifdef _MSC_VER
dvander@53
   178
    unsigned long rval;
dvander@86
   179
# ifdef _M_IX86
dvander@53
   180
    _BitScanForward(&rval, number);
dvander@86
   181
# elif _M_X64
dvander@86
   182
    _BitScanForward64(&rval, number);
dvander@86
   183
# endif
dvander@53
   184
    return rval;
dvander@53
   185
#else
dvander@53
   186
	size_t bit;
dvander@53
   187
	asm("bsf %1, %0\n"
dvander@53
   188
		: "=r" (bit)
dvander@53
   189
		: "rm" (number));
dvander@53
   190
	return bit;
dvander@53
   191
#endif
dvander@53
   192
}
dvander@53
   193
dvander@28
   194
static inline bool
dvander@28
   195
IsPowerOfTwo(size_t value)
dvander@28
   196
{
dvander@28
   197
    if (value == 0)
dvander@28
   198
        return false;
dvander@28
   199
    return !(value & (value - 1));
dvander@28
   200
}
dvander@28
   201
dvander@28
   202
static inline size_t
dvander@28
   203
Align(size_t count, size_t alignment)
dvander@28
   204
{
dvander@28
   205
    assert(IsPowerOfTwo(alignment));
dvander@28
   206
    return count + (alignment - (count % alignment)) % alignment;
dvander@28
   207
}
dvander@28
   208
dvander@28
   209
static inline bool
dvander@28
   210
IsUint32AddSafe(unsigned a, unsigned b)
dvander@28
   211
{
dvander@55
   212
    if (!a || !b)
dvander@55
   213
        return true;
dvander@86
   214
    size_t log2_a = Log2(a);
dvander@86
   215
    size_t log2_b = Log2(b);
dvander@28
   216
    return (log2_a < sizeof(unsigned) * 8) &&
dvander@28
   217
           (log2_b < sizeof(unsigned) * 8);
dvander@28
   218
}
dvander@28
   219
dvander@28
   220
static inline bool
dvander@55
   221
IsUintPtrAddSafe(size_t a, size_t b)
dvander@55
   222
{
dvander@55
   223
    if (!a || !b)
dvander@55
   224
        return true;
dvander@55
   225
    size_t log2_a = Log2(a);
dvander@55
   226
    size_t log2_b = Log2(b);
dvander@55
   227
    return (log2_a < sizeof(size_t) * 8) &&
dvander@55
   228
           (log2_b < sizeof(size_t) * 8);
dvander@55
   229
}
dvander@55
   230
dvander@55
   231
static inline bool
dvander@28
   232
IsUint32MultiplySafe(unsigned a, unsigned b)
dvander@28
   233
{
dvander@28
   234
    if (a <= 1 || b <= 1)
dvander@28
   235
        return true;
dvander@28
   236
dvander@86
   237
    size_t log2_a = Log2(a);
dvander@86
   238
    size_t log2_b = Log2(b);
dvander@28
   239
    return log2_a + log2_b <= sizeof(unsigned) * 8;
dvander@28
   240
}
dvander@28
   241
dvander@55
   242
static inline bool
dvander@55
   243
IsUintPtrMultiplySafe(size_t a, size_t b)
dvander@55
   244
{
dvander@55
   245
    if (a <= 1 || b <= 1)
dvander@55
   246
        return true;
dvander@55
   247
dvander@55
   248
    size_t log2_a = Log2(a);
dvander@55
   249
    size_t log2_b = Log2(b);
dvander@55
   250
    return log2_a + log2_b <= sizeof(size_t) * 8;
dvander@55
   251
}
dvander@55
   252
dvander@28
   253
#define ARRAY_LENGTH(array) (sizeof(array) / sizeof(array[0]))
dvander@28
   254
#define STATIC_ASSERT(cond) extern int static_assert_f(int a[(cond) ? 1 : -1])
dvander@28
   255
dvander@49
   256
#define IS_ALIGNED(addr, alignment)	(!(uintptr_t(addr) & ((alignment) - 1)))
dvander@49
   257
dvander@11
   258
template <typename T>
dvander@11
   259
static inline bool
dvander@11
   260
IsAligned(T addr, size_t alignment)
dvander@11
   261
{
dvander@11
   262
    assert(IsPowerOfTwo(alignment));
dvander@11
   263
    return !(uintptr_t(addr) & (alignment - 1));
dvander@28
   264
}
dvander@28
   265
dvander@11
   266
static inline Address
dvander@11
   267
AlignedBase(Address addr, size_t alignment)
dvander@11
   268
{
dvander@11
   269
    assert(IsPowerOfTwo(alignment));
dvander@11
   270
    return Address(uintptr_t(addr) & ~(alignment - 1));
dvander@28
   271
}
dvander@28
   272
dvander@28
   273
template <typename T> static inline T
dvander@28
   274
Min(const T &t1, const T &t2)
dvander@28
   275
{
dvander@28
   276
    return t1 < t2 ? t1 : t2;
dvander@28
   277
}
dvander@28
   278
danderson@91
   279
#define OFFSETOF(Class, Member) reinterpret_cast<size_t>(&((Class *)NULL)->Member)
danderson@91
   280
dvander@28
   281
}
dvander@28
   282
dvander@28
   283
#endif // _include_jitcraft_utility_h_