src/HashTable.h
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 132 d8379f5d5a1d
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
dvander@132
     1
/* vim: set ts=4 sw=4 tw=99 et:
dvander@132
     2
 *
dvander@54
     3
 * Copyright (C) 2012 David Anderson
dvander@54
     4
 *
dvander@132
     5
 * This file is part of SourcePawn.
dvander@54
     6
 *
dvander@132
     7
 * SourcePawn is free software: you can redistribute it and/or modify it under
dvander@54
     8
 * the terms of the GNU General Public License as published by the Free
dvander@54
     9
 * Software Foundation, either version 3 of the License, or (at your option)
dvander@54
    10
 * any later version.
dvander@54
    11
 * 
dvander@132
    12
 * SourcePawn is distributed in the hope that it will be useful, but WITHOUT ANY
dvander@54
    13
 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
dvander@54
    14
 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
dvander@54
    15
 *
dvander@54
    16
 * You should have received a copy of the GNU General Public License along with
dvander@132
    17
 * SourcePawn. If not, see http://www.gnu.org/licenses/.
dvander@54
    18
 */
dvander@0
    19
#ifndef _INCLUDE_KEIMA_HASHTABLE_H_
dvander@0
    20
#define _INCLUDE_KEIMA_HASHTABLE_H_
dvander@0
    21
dvander@0
    22
#include <new>
dvander@0
    23
#include <limits.h>
dvander@54
    24
#include "Utility.h"
dvander@0
    25
dvander@59
    26
namespace ke {
dvander@0
    27
dvander@0
    28
template <typename T>
dvander@0
    29
class HashTableEntry
dvander@0
    30
{
dvander@0
    31
    uint32 hash_;
dvander@0
    32
    T t_;
dvander@0
    33
dvander@0
    34
  public:
dvander@0
    35
    static const uint32 kFreeHash = 0;
dvander@0
    36
    static const uint32 kRemovedHash = 1;
dvander@0
    37
dvander@0
    38
  public:
dvander@0
    39
    void setHash(uint32 hash) {
dvander@0
    40
        hash_ = hash;
dvander@0
    41
    }
dvander@0
    42
    uint32 hash() const {
dvander@0
    43
        return hash_;
dvander@0
    44
    }
dvander@0
    45
    void setRemoved() {
dvander@0
    46
        hash_ = kRemovedHash;
dvander@54
    47
        t_ = T();
dvander@0
    48
    }
dvander@0
    49
    void construct() {
dvander@54
    50
        hash_ = kFreeHash;
dvander@0
    51
        new (&t_) T();
dvander@0
    52
    }
dvander@0
    53
    void destruct() {
dvander@0
    54
        t_.~T();
dvander@0
    55
    }
dvander@0
    56
    bool removed() const {
dvander@0
    57
        return hash_ == kRemovedHash;
dvander@0
    58
    }
dvander@0
    59
    bool free() const {
dvander@0
    60
        return hash_ == kFreeHash;
dvander@0
    61
    }
dvander@0
    62
    bool isLive() const {
dvander@0
    63
        return hash_ > kRemovedHash;
dvander@0
    64
    }
dvander@0
    65
    T &payload() {
dvander@0
    66
        return t_;
dvander@0
    67
    }
dvander@0
    68
    bool sameHash(uint32 hash) const {
dvander@0
    69
        return hash_ == hash;
dvander@0
    70
    }
dvander@0
    71
};
dvander@0
    72
dvander@0
    73
template <typename HashPolicy, typename AllocPolicy>
dvander@0
    74
class HashTable : public AllocPolicy
dvander@0
    75
{
dvander@54
    76
    friend class iterator;
dvander@54
    77
dvander@0
    78
    typedef typename HashPolicy::Payload Payload;
dvander@0
    79
    typedef HashTableEntry<Payload> Entry;
dvander@0
    80
dvander@0
    81
    uint32 capacity_;
dvander@0
    82
    uint32 nelements_;
dvander@0
    83
    uint32 ndeleted_;
dvander@0
    84
    Entry *table_;
dvander@54
    85
    uint32 minCapacity_;
dvander@0
    86
dvander@0
    87
  private:
dvander@0
    88
    static const uint32 kMinCapacity = 16;
dvander@0
    89
    static const uint32 kMaxCapacity = INT_MAX / sizeof(Entry);
dvander@0
    90
dvander@0
    91
    template <typename Key>
dvander@0
    92
    uint32 computeHash(const Key &key) {
dvander@0
    93
        // Multiply by golden ratio.
dvander@0
    94
        uint32 hash = HashPolicy::hash(key) * 0x9E3779B9;
dvander@0
    95
        if (hash == Entry::kFreeHash || hash == Entry::kRemovedHash)
dvander@0
    96
            hash += 2;
dvander@0
    97
        return hash;
dvander@0
    98
    }
dvander@0
    99
dvander@0
   100
    Entry *createTable(uint32 capacity) {
dvander@54
   101
        assert(capacity <= kMaxCapacity);
dvander@0
   102
dvander@0
   103
        Entry *table = (Entry *)this->malloc(capacity * sizeof(Entry));
dvander@0
   104
        if (!table)
dvander@0
   105
            return NULL;
dvander@0
   106
dvander@0
   107
        for (size_t i = 0; i < capacity; i++)
dvander@0
   108
            table[i].construct();
dvander@0
   109
dvander@0
   110
        return table;
dvander@0
   111
    }
dvander@0
   112
dvander@0
   113
  public:
dvander@0
   114
    class Result
dvander@0
   115
    {
dvander@0
   116
        friend class HashTable;
dvander@0
   117
dvander@0
   118
        Entry *entry_;
dvander@0
   119
dvander@0
   120
        Entry &entry() {
dvander@0
   121
            return *entry_;
dvander@0
   122
        }
dvander@0
   123
dvander@0
   124
      public:
dvander@0
   125
        Result(Entry *entry)
dvander@0
   126
          : entry_(entry)
dvander@0
   127
        { }
dvander@0
   128
dvander@0
   129
        Payload * operator ->() {
dvander@195
   130
            return &entry_->payload();
dvander@0
   131
        }
dvander@0
   132
        Payload & operator *() {
dvander@0
   133
            return entry_->payload();
dvander@0
   134
        }
dvander@0
   135
dvander@0
   136
        bool found() const {
dvander@0
   137
            return entry_->isLive();
dvander@0
   138
        }
dvander@0
   139
    };
dvander@0
   140
dvander@0
   141
    class Insert : public Result
dvander@0
   142
    {
dvander@0
   143
        uint32 hash_;
dvander@0
   144
dvander@0
   145
      public:
dvander@0
   146
        Insert(Entry *entry, uint32 hash)
dvander@0
   147
          : Result(entry),
dvander@0
   148
            hash_(hash)
dvander@0
   149
        {
dvander@0
   150
        }
dvander@0
   151
dvander@0
   152
        uint32 hash() const {
dvander@0
   153
            return hash_;
dvander@0
   154
        }
dvander@0
   155
    };
dvander@0
   156
dvander@0
   157
  private:
dvander@0
   158
    class Probulator {
dvander@0
   159
        uint32 hash_;
dvander@0
   160
        uint32 capacity_;
dvander@0
   161
dvander@0
   162
      public:
dvander@0
   163
        Probulator(uint32 hash, uint32 capacity)
dvander@0
   164
          : hash_(hash),
dvander@0
   165
            capacity_(capacity)
dvander@0
   166
        {
dvander@54
   167
            assert(IsPowerOfTwo(capacity_)); 
dvander@0
   168
        }
dvander@0
   169
dvander@0
   170
        uint32 entry() const {
dvander@0
   171
            return hash_ & (capacity_ - 1);
dvander@0
   172
        }
dvander@0
   173
        uint32 next() {
dvander@0
   174
            hash_++;
dvander@0
   175
            return entry();
dvander@0
   176
        }
dvander@0
   177
    };
dvander@0
   178
dvander@0
   179
    bool underloaded() const {
dvander@0
   180
        // Check if the table is underloaded: < 25% entries used.
dvander@0
   181
        return (capacity_ > kMinCapacity) && (nelements_ + ndeleted_ < capacity_ / 4);
dvander@0
   182
    }
dvander@0
   183
    bool overloaded() const {
dvander@0
   184
        // Grow if the table is overloaded: > 75% entries used.
dvander@0
   185
        return (nelements_ + ndeleted_) > ((capacity_ / 2) + (capacity_ / 4));
dvander@0
   186
    }
dvander@0
   187
dvander@0
   188
    bool shrink() {
dvander@54
   189
        if ((capacity_ >> 1) < minCapacity_)
dvander@54
   190
            return true;
dvander@0
   191
        return changeCapacity(capacity_ >> 1);
dvander@0
   192
    }
dvander@0
   193
    
dvander@0
   194
    bool grow() {
dvander@0
   195
        if (capacity_ >= kMaxCapacity) {
dvander@0
   196
            this->reportAllocationOverflow();
dvander@0
   197
            return false;
dvander@0
   198
        }
dvander@0
   199
        return changeCapacity(capacity_ << 1);
dvander@0
   200
    }
dvander@0
   201
dvander@0
   202
    bool changeCapacity(uint32 newCapacity) {
dvander@54
   203
        assert(newCapacity <= kMaxCapacity);
dvander@0
   204
dvander@0
   205
        Entry *newTable = createTable(newCapacity);
dvander@0
   206
        if (!newTable)
dvander@0
   207
            return false;
dvander@0
   208
dvander@0
   209
        Entry *oldTable = table_;
dvander@0
   210
        uint32 oldCapacity = capacity_;
dvander@0
   211
dvander@0
   212
        table_ = newTable;
dvander@0
   213
        capacity_ = newCapacity;
dvander@0
   214
        ndeleted_ = 0;
dvander@0
   215
dvander@0
   216
        for (uint32 i = 0; i < oldCapacity; i++) {
dvander@0
   217
            Entry &oldEntry = oldTable[i];
dvander@0
   218
            if (oldEntry.isLive()) {
dvander@0
   219
                Insert p = insertUnique(oldEntry.hash());
dvander@0
   220
                p.entry().setHash(p.hash());
dvander@0
   221
                p.entry().payload() = oldEntry.payload();
dvander@0
   222
            }
dvander@0
   223
            oldEntry.destruct();
dvander@0
   224
        }
dvander@54
   225
        this->free(oldTable);
dvander@0
   226
dvander@0
   227
        return true;
dvander@0
   228
    }
dvander@0
   229
dvander@0
   230
    // For use when the key is known to be unique.
dvander@0
   231
    Insert insertUnique(uint32 hash) {
dvander@0
   232
        Probulator probulator(hash, capacity_);
dvander@0
   233
dvander@0
   234
        Entry *e = &table_[probulator.entry()];
dvander@0
   235
        for (;;) {
dvander@0
   236
            if (e->free() || e->removed())
dvander@0
   237
                break;
dvander@0
   238
            e = &table_[probulator.next()];
dvander@0
   239
        }
dvander@0
   240
dvander@0
   241
        return Insert(e, hash);
dvander@0
   242
    }
dvander@0
   243
dvander@0
   244
    template <typename Key>
dvander@0
   245
    Result lookup(const Key &key) {
dvander@0
   246
        uint32 hash = computeHash(key);
dvander@0
   247
        Probulator probulator(hash, capacity_);
dvander@0
   248
dvander@0
   249
        Entry *e = &table_[probulator.entry()];
dvander@0
   250
        for (;;) {
dvander@0
   251
            if (e->free())
dvander@0
   252
                break;
dvander@0
   253
            if (e->isLive() &&
dvander@0
   254
                e->sameHash(hash) &&
dvander@0
   255
                HashPolicy::matches(key, e->payload()))
dvander@0
   256
            {
dvander@0
   257
                return Result(e);
dvander@0
   258
            }
dvander@0
   259
            e = &table_[probulator.next()];
dvander@0
   260
        }
dvander@0
   261
dvander@0
   262
        return Result(e);
dvander@0
   263
    }
dvander@0
   264
dvander@0
   265
    template <typename Key>
dvander@0
   266
    Insert lookupForAdd(const Key &key) {
dvander@0
   267
        uint32 hash = computeHash(key);
dvander@0
   268
        Probulator probulator(hash, capacity_);
dvander@0
   269
dvander@0
   270
        Entry *e = &table_[probulator.entry()];
dvander@0
   271
        for (;;) {
dvander@0
   272
            if (!e->isLive())
dvander@0
   273
                break;
dvander@0
   274
            if (e->sameHash(hash) && HashPolicy::matches(key, e->payload()))
dvander@0
   275
                break;
dvander@0
   276
            e = &table_[probulator.next()];
dvander@0
   277
        }
dvander@0
   278
dvander@0
   279
        return Insert(e, hash);
dvander@0
   280
    }
dvander@0
   281
dvander@0
   282
    bool add(Insert &i) {
dvander@54
   283
        assert(!i.found());
dvander@0
   284
dvander@0
   285
        // If the entry is deleted, just re-use the slot.
dvander@0
   286
        if (i.entry().removed()) {
dvander@0
   287
            ndeleted_--;
dvander@0
   288
            return true;
dvander@0
   289
        }
dvander@0
   290
dvander@0
   291
        // Otherwise, see if we're at max capacity.
dvander@0
   292
        if (nelements_ == kMaxCapacity) {
dvander@0
   293
            this->reportAllocationOverflow();
dvander@0
   294
            return false;
dvander@0
   295
        }
dvander@0
   296
dvander@0
   297
        // Check if the table is over or underloaded. The table is always at
dvander@0
   298
        // least 25% free, so this check is enough to guarantee one free slot.
dvander@0
   299
        // (Without one free slot, insertion search could infinite loop.)
dvander@0
   300
        uint32 oldCapacity = capacity_;
dvander@0
   301
        if (!checkDensity())
dvander@0
   302
            return false;
dvander@0
   303
dvander@0
   304
        // If the table changed size, we need to find a new insertion point.
dvander@0
   305
        // Note that a removed entry is impossible: either we caught it above,
dvander@0
   306
        // or we just resized and no entries are removed.
dvander@0
   307
        if (capacity_ != oldCapacity)
dvander@0
   308
            i = insertUnique(i.hash());
dvander@0
   309
dvander@0
   310
        i.entry().setHash(i.hash());
dvander@0
   311
        nelements_++;
dvander@0
   312
dvander@0
   313
        return true;
dvander@0
   314
    }
dvander@0
   315
dvander@54
   316
    void remove(Entry &e) {
dvander@62
   317
        assert(e.isLive());
dvander@54
   318
        e.setRemoved();
dvander@54
   319
        ndeleted_++;
dvander@54
   320
        nelements_--;
dvander@54
   321
    }
dvander@54
   322
dvander@0
   323
  public:
dvander@0
   324
    HashTable(AllocPolicy ap)
dvander@0
   325
      : AllocPolicy(ap),
dvander@0
   326
        capacity_(0),
dvander@0
   327
        nelements_(0),
dvander@0
   328
        ndeleted_(0),
dvander@54
   329
        table_(NULL),
dvander@54
   330
        minCapacity_(kMinCapacity)
dvander@0
   331
    {
dvander@0
   332
    }
dvander@0
   333
dvander@0
   334
    ~HashTable()
dvander@0
   335
    {
dvander@0
   336
        for (uint32 i = 0; i < capacity_; i++)
dvander@0
   337
            table_[i].destruct();
dvander@0
   338
        this->free(table_);
dvander@0
   339
    }
dvander@0
   340
dvander@0
   341
    bool init(uint32 capacity = 0) {
dvander@132
   342
        if (capacity < kMinCapacity) {
dvander@0
   343
            capacity = kMinCapacity;
dvander@132
   344
        } else if (capacity > kMaxCapacity) {
dvander@132
   345
            this->reportAllocationOverflow();
dvander@132
   346
            return false;
dvander@132
   347
        }
dvander@0
   348
dvander@54
   349
        minCapacity_ = capacity;
dvander@54
   350
dvander@54
   351
        assert(IsPowerOfTwo(capacity));
dvander@0
   352
        capacity_ = capacity;
dvander@0
   353
dvander@0
   354
        table_ = createTable(capacity_);
dvander@0
   355
        if (!table_)
dvander@0
   356
            return false;
dvander@0
   357
dvander@0
   358
        return true;
dvander@0
   359
    }
dvander@0
   360
dvander@0
   361
    template <typename Key>
dvander@0
   362
    Result find(const Key &key) {
dvander@0
   363
        return lookup(key);
dvander@0
   364
    }
dvander@0
   365
dvander@0
   366
    template <typename Key>
dvander@0
   367
    Insert findForAdd(const Key &key) {
dvander@0
   368
        return lookupForAdd(key);
dvander@0
   369
    }
dvander@0
   370
dvander@0
   371
    bool add(Insert &i, const Payload &payload) {
dvander@0
   372
        if (!add(i))
dvander@0
   373
            return false;
dvander@0
   374
        *i = payload;
dvander@0
   375
        return true;
dvander@0
   376
    }
dvander@54
   377
dvander@54
   378
    bool checkDensity() {
dvander@54
   379
        if (underloaded())
dvander@54
   380
            return shrink(); 
dvander@54
   381
        if (overloaded())
dvander@54
   382
            return grow();
dvander@54
   383
        return true;
dvander@54
   384
    }
dvander@54
   385
dvander@54
   386
  public:
dvander@54
   387
    class iterator
dvander@54
   388
    {
dvander@54
   389
        HashTable *table_;
dvander@54
   390
        Entry *i_;
dvander@54
   391
        Entry *end_;
dvander@54
   392
dvander@54
   393
      public:
dvander@54
   394
        iterator(HashTable *table)
dvander@54
   395
          : table_(table),
dvander@54
   396
            i_(table->table_),
dvander@54
   397
            end_(table->table_ + table->capacity_)
dvander@54
   398
        {
dvander@54
   399
            while (i_ < end_ && !i_->isLive())
dvander@54
   400
                i_++;
dvander@54
   401
        }
dvander@54
   402
dvander@54
   403
        bool empty() const {
dvander@54
   404
            return i_ == end_;
dvander@54
   405
        }
dvander@54
   406
dvander@54
   407
        void erase() {
dvander@54
   408
            assert(!empty());
dvander@54
   409
            table_->remove(*i_);
dvander@54
   410
        }
dvander@54
   411
dvander@54
   412
        const Payload &operator *() const {
dvander@54
   413
            return i_->payload();
dvander@54
   414
        }
dvander@54
   415
dvander@54
   416
        void next() {
dvander@62
   417
            do {
dvander@54
   418
                i_++;
dvander@62
   419
            } while (i_ < end_ && !i_->isLive());
dvander@54
   420
        }
dvander@54
   421
    };
dvander@0
   422
};
dvander@0
   423
dvander@195
   424
dvander@195
   425
dvander@0
   426
}
dvander@0
   427
dvander@0
   428
#endif // _INCLUDE_KEIMA_HASHTABLE_H_