Massive SemA/BC refactoring to support better type systems. Read on for more.
The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code
This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.
Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).
This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*
Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.
(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.
(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).
In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.
(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.
As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.
This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:
HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))
As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>
This simplifies the pipeline, and JITs should be able to melt the added
work away.
HIR is not used at a statement level, and HIR does not have any concept
of control-flow.
(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.
In addition, some minor refactorings have taken place:
(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.
Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
1 /* vim: set ts=4 sw=4 tw=99 et:
3 * Copyright (C) 2012 David Anderson
5 * This file is part of SourcePawn.
7 * SourcePawn is free software: you can redistribute it and/or modify it under
8 * the terms of the GNU General Public License as published by the Free
9 * Software Foundation, either version 3 of the License, or (at your option)
12 * SourcePawn is distributed in the hope that it will be useful, but WITHOUT ANY
13 * WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
14 * FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
16 * You should have received a copy of the GNU General Public License along with
17 * SourcePawn. If not, see http://www.gnu.org/licenses/.
19 #include "../Strings.h"
21 #include "../TypeManager.h"
22 #include "NameBinding.h"
24 #include "../Spaces-inl.h"
25 #include "../Heap-inl.h"
29 // Create symbol tables and symbols for every scope and declaration. For type
30 // declarations and methods, we also pre-allocate a Type object, so that
31 // circular dependencies between types will work.
32 class NamePopulator : public AstVisitor
35 // This class is analagous to a Scope, however, it allows us to create
36 // scopes lazily. If a scope is requested, and the current environment
37 // has no scope, then a scope is created. If no scope is ever requested,
38 // a scope does not need to be created.
42 SymbolEnv(SymbolEnv **prevp, Scope *scope = NULL)
56 void setScope(Scope *scope) {
60 SymbolEnv *prev() const {
79 FunctionScope(FunctionScope **prevp)
89 ReturnStatus returns() const {
92 void returns(ReturnStatus status) {
97 FunctionScope **prevp_;
103 NamePopulator(Zone *zone, CompileContext &cc, TranslationUnit *unit)
114 ImportScope *importScope = ImportScope::New(pool_);
118 // At the import level, we declare system types.
119 if (!declareSystemTypes(importScope))
122 GlobalScope *globalScope = GlobalScope::New(pool_, importScope);
126 unit_->setImportScope(importScope);
127 unit_->setGlobalScope(globalScope);
129 SymbolEnv env(&env_, globalScope);
131 for (size_t i = 0; i < unit_->tree()->statements()->length(); i++) {
132 Statement *stmt = unit_->tree()->statements()->at(i);
139 bool declareSystemType(ImportScope *scope, const char *name, Handle<Type> type) {
140 Local<String> tag(zone_, zone_->makeSymbol(name));
144 TypeSymbol *sym = new (pool_) TypeSymbol(scope, tag, type);
145 return scope->addSymbol(sym);
148 bool declareSystemType(ImportScope *scope, const char *name, PrimitiveType prim) {
149 Local<Type> primitive(zone_, zone_->types()->getPrimitive(prim));
151 return declareSystemType(scope, name, primitive);
154 bool declareSystemTypes(ImportScope *scope) {
155 if (!declareSystemType(scope, "float", PrimitiveType_Float))
157 if (!declareSystemType(scope, "_", PrimitiveType_Int32))
159 if (!declareSystemType(scope, "int", PrimitiveType_Int32))
161 if (!declareSystemType(scope, "bool", PrimitiveType_Bool))
163 if (!declareSystemType(scope, "char", PrimitiveType_Char))
166 Local<Type> voidType(zone_, zone_->types()->getVoid());
167 if (!declareSystemType(scope, "void", voidType))
173 void visitVariableDeclaration(VariableDeclaration *node) {
174 Scope *scope = env_->scope();
175 if (!scope || scope->kind() != Scope::GLOBAL) {
176 if ((scope = getOrCreateBlockScope()) == NULL)
180 if (scope->kind() == Scope::GLOBAL) {
181 // The symbol must be added now, since it could be an export.
182 VariableSymbol *sym = new (pool_) VariableSymbol(scope, node->name(), node->pos());
183 node->setSymbol(sym);
185 if (!registerName(scope, sym))
188 // We ensure the scope will exist at the name binding phase, but
189 // we don't actually register the variable symbol yet. Otherwise,
190 // when binding the initializer, we could encounter:
193 // And we don't necessarily want this to bind.
196 node->initialization()->accept(this);
198 void visitEnumStatement(EnumStatement *node) {
201 void visitFunctionStatement(FunctionStatement *node) {
202 // We create function types ahead of time, not for any particular
203 // reason. It makes things consistent with FunctionTypeStatement,
204 // which actually does produce a type.
205 Local<FunctionType> type(zone_, FunctionType::New(zone_, node->token()));
209 assert(env_->scope());
211 // Function statements are always named, so add the name to the outer scope.
212 FunctionSymbol *sym = new (pool_) FunctionSymbol(env_->scope(), node->name(), node->pos(), type);
213 if (!registerName(env_->scope(), sym))
216 if (node->token() == TOK_PUBLIC ||
217 node->token() == TOK_FORWARD ||
218 node->token() == TOK_NATIVE)
220 // We're in the global scope, for sure.
221 assert(env_->scope()->kind() == Scope::GLOBAL);
222 GlobalScope *globals = (GlobalScope *)env_->scope();
223 if (!globals->addPublic(sym))
227 node->setSymbol(sym);
229 // Scope for the function's arguments.
230 SymbolEnv argEnv(&env_);
231 if (!registerArguments(node->signature()))
234 // Scope for function body.
235 SymbolEnv localEnv(&env_);
236 FunctionScope fun(&fun_);
238 node->body()->accept(this);
240 // Function statements have quirky semantics around their return
241 // type. For reasonable compatibility with SP1, we use the following
242 // heuristics for when no explicit type is declared.
243 if (!node->signature().returnType()) {
244 Local<Type> type(zone_);
245 if (node->token() == TOK_FORWARD || node->token() == TOK_NATIVE) {
246 // If the function type is a native or forward, we always
247 // assume it returns an int32.
248 type = zone_->types()->getPrimitive(PrimitiveType_Int32);
249 } else if (fun.returns() != ValueReturn) {
250 // This function never returns a value, so mark it as
252 type = zone_->types()->getVoid();
254 // No need to NULL check here, we'll catch OOM later.
255 node->sym()->type()->setReturnType(type);
258 node->setScopes(argEnv.scope(), localEnv.scope());
260 void visitStructureStatement(StructureStatement *node) {
263 void visiFunctionTypeStatementt(FunctionTypeStatement *node) {
267 void visitNameProxy(NameProxy *name) {
269 void visitAssignment(Assignment *node) {
270 node->lvalue()->accept(this);
271 node->expression()->accept(this);
273 void visitBinaryExpression(BinaryExpression *node) {
274 node->left()->accept(this);
275 node->right()->accept(this);
277 void visitReturnStatement(ReturnStatement *node) {
278 if (node->expression())
279 fun_->returns(ValueReturn);
281 fun_->returns(VoidReturn);
283 void visitForStatement(ForStatement *node) {
284 SymbolEnv env(&env_);
286 if (node->initialization())
287 node->initialization()->accept(this);
288 if (node->condition())
289 node->condition()->accept(this);
291 node->update()->accept(this);
292 node->body()->accept(this);
294 node->setScope(env.scope());
296 void visitBlockStatement(BlockStatement *node) {
297 SymbolEnv env(&env_);
299 for (size_t i = 0; i < node->statements()->length(); i++)
300 node->statements()->at(i)->accept(this);
302 node->setScope(env.scope());
304 void visitIntegerLiteral(IntegerLiteral *node) {
306 void visitExpressionStatement(ExpressionStatement *node) {
307 node->expression()->accept(this);
309 void visitCallExpression(CallExpression *node) {
310 node->callee()->accept(this);
311 for (size_t i = 0; i < node->arguments()->length(); i++)
312 node->arguments()->at(i)->accept(this);
314 void visitIfStatement(IfStatement *node) {
315 node->ifTrue()->accept(this);
317 node->ifFalse()->accept(this);
319 void visitIndexExpression(IndexExpression *node) {
322 void visitFloatLiteral(FloatLiteral *node) {
324 void visitWhileStatement(WhileStatement *node) {
325 node->condition()->accept(this);
326 node->body()->accept(this);
328 void visitBreakStatement(BreakStatement *node) {
330 void visitContinueStatement(ContinueStatement *node) {
332 void visitStringLiteral(StringLiteral *node) {
334 void visitIncDecExpression(IncDecExpression *node) {
337 void visitUnaryExpression(UnaryExpression *node) {
340 void visitTernaryExpression(TernaryExpression *node) {
343 void visitBooleanLiteral(BooleanLiteral *node) {
345 void visitFieldExpression(FieldExpression *node) {
347 void visitSwitchStatement(SwitchStatement *node) {
348 if (node->defaultCase())
349 node->defaultCase()->accept(this);
351 for (size_t i = 0; i < node->cases()->length(); i++)
352 node->cases()->at(i)->statement()->accept(this);
354 void visitArrayLiteral(ArrayLiteral *node) {
356 void visitImportStatement(ImportStatement *node) {
360 Scope *getOrCreateBlockScope() {
362 assert(env_->scope()->kind() == Scope::BLOCK);
363 return env_->scope();
366 LocalScope *scope = LocalScope::New(pool_);
370 env_->setScope(scope);
374 bool registerName(Scope *scope, Symbol *sym) {
378 if (Symbol *other = scope->localLookup(sym->name())) {
379 // Report, but allow errors to continue.
380 cc_.reportError(sym->pos(), Message_RedeclaredName,
381 sym->name()->chars(),
387 return scope->addSymbol(sym);
390 bool registerArguments(const FunctionSignature &sig) {
391 if (!sig.parameters()->length())
394 ArgumentScope *argScope = ArgumentScope::New(pool_);
397 env_->setScope(argScope);
399 for (size_t i = 0; i < sig.parameters()->length(); i++) {
400 Parameter *param = sig.parameters()->at(i);
401 VariableSymbol *sym = new (pool_) VariableSymbol(argScope, param->name(), param->pos());
402 if (!registerName(env_->scope(), sym))
404 param->setSymbol(sym);
412 PoolAllocator &pool_;
414 TranslationUnit *unit_;
420 ke::PopulateNamesAndTypes(Zone *zone, CompileContext &cc, TranslationUnit *unit)
422 NamePopulator populator(zone, cc, unit);
423 return populator.analyze();
426 class NameBinder : public AstVisitor
432 AutoLinkScope(AutoLinkScope **prevp, Scope *scope)
440 scope_->setParent(prev_->scope());
445 assert(*prevp_ == this);
449 Scope *scope() const {
454 AutoLinkScope **prevp_;
455 AutoLinkScope *prev_;
460 NameBinder(Zone *zone, CompileContext &cc, TranslationUnit *unit)
470 // Start with the import scope, and collect all imported names.
471 AutoLinkScope importScope(&link_, unit_->importScope());
473 for (size_t i = 0; i < unit_->tree()->imports()->length(); i++)
474 unit_->tree()->imports()->at(i)->accept(this);
476 // Next, enter the global scope and walk all statements.
477 AutoLinkScope globalScope(&link_, unit_->globalScope());
478 for (size_t i = 0; i < unit_->tree()->statements()->length(); i++) {
479 Statement *stmt = unit_->tree()->statements()->at(i);
486 void visitVariableDeclaration(VariableDeclaration *node) {
487 // Bind the initializer before registering the declaration, so that we
488 // can error on bogus initializers (new x = x).
489 node->initialization()->accept(this);
491 Local<Type> type(zone_, bindType(node->type()));
495 // If the node already has a symbol (meaning it was a global), then
496 // we don't have to do anything more.
498 assert(node->sym()->scope()->kind() == Scope::GLOBAL);
502 VariableSymbol *sym = new (pool_) VariableSymbol(link_->scope(), node->name(), node->pos(), type);
503 node->setSymbol(sym);
505 if (Symbol *other = link_->scope()->localLookup(sym->name())) {
506 // Report, but allow errors to continue.
507 cc_.reportError(sym->pos(), Message_RedeclaredName,
508 sym->name()->chars(),
514 if (link_->scope()->addSymbol(sym))
517 void visitEnumStatement(EnumStatement *node) {
520 void visitFunctionStatement(FunctionStatement *node) {
521 AutoLinkScope argScope(&link_, node->argScope());
522 AutoLinkScope varScope(&link_, node->varScope());
524 Local<FunctionType> type(zone_, node->sym()->type());
525 if (!fillFunctionType(type, node->signature()))
529 node->body()->accept(this);
531 void visitStructureStatement(StructureStatement *node) {
534 void visitFunctionTypeStatement(FunctionTypeStatement *node) {
538 void visitNameProxy(NameProxy *proxy) {
539 Scope *scope = link_->scope();
540 Symbol *sym = scope->lookup(proxy->name());
542 cc_.reportError(proxy->pos(), Message_IdentifierNotFound, proxy->name()->chars());
545 void visitAssignment(Assignment *node) {
546 node->lvalue()->accept(this);
547 node->expression()->accept(this);
549 void visitBinaryExpression(BinaryExpression *node) {
550 node->left()->accept(this);
551 node->right()->accept(this);
553 void visitReturnStatement(ReturnStatement *node) {
554 node->expression()->accept(this);
556 void visitForStatement(ForStatement *node) {
557 AutoLinkScope scope(&link_, node->scope());
559 if (node->initialization())
560 node->initialization()->accept(this);
561 if (node->condition())
562 node->condition()->accept(this);
564 node->update()->accept(this);
565 node->body()->accept(this);
567 void visitBlockStatement(BlockStatement *node) {
568 AutoLinkScope scope(&link_, node->scope());
569 for (size_t i = 0; i < node->statements()->length(); i++)
570 node->statements()->at(i)->accept(this);
572 void visitIntegerLiteral(IntegerLiteral *node) {
574 void visitExpressionStatement(ExpressionStatement *node) {
575 node->expression()->accept(this);
577 void visitCallExpression(CallExpression *node) {
578 node->callee()->accept(this);
579 for (size_t i = 0; i < node->arguments()->length(); i++)
580 node->arguments()->at(i)->accept(this);
582 void visitIfStatement(IfStatement *node) {
583 node->condition()->accept(this);
584 node->ifTrue()->accept(this);
585 node->ifFalse()->accept(this);
587 void visitIndexExpression(IndexExpression *node) {
588 node->left()->accept(this);
589 node->right()->accept(this);
591 void visitFloatLiteral(FloatLiteral *node) {
593 void visitWhileStatement(WhileStatement *node) {
594 node->condition()->accept(this);
595 node->body()->accept(this);
597 void visitBreakStatement(BreakStatement *node) {
599 void visitContinueStatement(ContinueStatement *node) {
601 void visitStringLiteral(StringLiteral *node) {
603 void visitIncDecExpression(IncDecExpression *node) {
604 node->expression()->accept(this);
606 void visitUnaryExpression(UnaryExpression *node) {
608 node->tag()->accept(this);
609 node->expression()->accept(this);
611 void visitTernaryExpression(TernaryExpression *node) {
612 node->condition()->accept(this);
613 node->left()->accept(this);
614 node->right()->accept(this);
616 void visitBooleanLiteral(BooleanLiteral *node) {
618 void visitFieldExpression(FieldExpression *node) {
619 node->left()->accept(this);
621 void visitSwitchStatement(SwitchStatement *node) {
622 node->expression()->accept(this);
623 node->defaultCase()->accept(this);
624 for (size_t i = 0; i < node->cases()->length(); i++) {
625 // We don't test case expressions because they are literals.
626 Case *c = node->cases()->at(i);
627 c->statement()->accept(this);
630 void visitArrayLiteral(ArrayLiteral *node) {
631 for (size_t i = 0; i < node->expressions()->length(); i++)
632 node->expressions()->at(i)->accept(this);
634 void visitImportStatement(ImportStatement *node) {
635 if (node->suffix() == TOK_STAR) {
636 // Import every name.
639 GlobalScope *other = node->source()->globalScope();
640 Symbol *sym = other->findPublic(node->name());
642 cc_.reportError(node->pos(), Message_CannotResolveImport, node->name()->chars());
645 link_->scope()->addSymbol(sym);
650 bool fillFunctionType(Handle<FunctionType> fun, const FunctionSignature &sig) {
651 Local<Type> type(zone_);
652 if (!fun->returnType()) {
653 if ((type = bindType(sig.returnType())) == NULL)
655 fun->setReturnType(type);
658 Local<FixedArray> params(zone_, FixedArray::New(zone_, sig.parameters()->length(), Heap::Tenure_Old));
662 for (size_t i = 0; i < sig.parameters()->length(); i++) {
663 Parameter *param = sig.parameters()->at(i);
664 if ((type = bindType(param->type())) == NULL)
666 params->set(zone_, i, type);
669 fun->setParameterTypes(params);
674 Type *bindType(Expression *expr) {
676 // If no type was declared, we return int32.
677 return zone_->types()->getPrimitive(PrimitiveType_Int32);
682 NameProxy *proxy = expr->asNameProxy();
685 Symbol *sym = proxy->sym();
689 if (!sym->isType()) {
690 cc_.reportError(expr->pos(), Message_IdentifierIsNotAType, sym->name()->chars());
699 PoolAllocator &pool_;
701 TranslationUnit *unit_;
702 AutoLinkScope *link_;
706 ke::BindNamesAndTypes(Zone *zone, CompileContext &cc, TranslationUnit *unit)
708 NameBinder binder(zone, cc, unit);
709 return binder.bind();