AMBuildScript
author David Anderson <dvander@alliedmods.net>
Sat Nov 17 18:41:01 2012 -0800 (2012-11-17)
changeset 195 c1ba166f3fc4
parent 106 8534dbfb8c1a
child 251 ca99a81745fe
permissions -rw-r--r--
Massive SemA/BC refactoring to support better type systems. Read on for more.

The previous compiler had a simple pipeline, best outlined as:
(1) Parsing and name binding -> AST
(2) Storage allocation -> AST
(3) Semantic Analysis (SemA) -> AST
(4) Bytecode Compilation (BC) -> Code

This pipeline, unfortunately, had two problems. First, name binding cannot be performed in one pass if we wish to have multi-file support in the form of modules. Name binding must be two-pass.

Second, because SemA was only capable of annotating AST nodes with a single type, most coercion work had to be duplicated in the BC phase, as coercion was not explicit in the SemA output. This made introducing new type rules extremely difficult, and all but precluded concepts like operator overloading (or overloading at all).

This patch rewrites most of Keima's backend. Of interest are the following changes:
(1) CompileContext has been refactored around future multi-file support.
(2) Parsing no longer performs any name binding.
(3) The grammar for type-and-name has been changed from:
(Label | Identifier)? Identifier
to:
Type? Identifier
where:
Type ::= OldType | NewType
OldType ::= Label
NewType ::= Identifier (. NewType)*


Although we do not implement the full production for Type yet, this
distinction is important. A type identifier is now affixed to the AST
as an Expression (right now, always a NameProxy), so it can fully
participate in name binding.

(4) Immediately after parsing, the AST goes through a NamePopulation phase.
This phase creates Scope objects for every scope which declares a name,
and if those names are declaring types or functions, a Symbol is created
and entered into the scope. Symbols entirely replace the old BoundName
classes.

(5) After NamePopulation comes NameBinding. This phase performs three steps:
(a) Links Scope objects together, to form a tree.
(b) Binds any free name to an existing name in the scope chain.
(c) Creates and registers any Symbols for names which have not yet been
added to the scope (for example, local variables).

In accordance, all "allocation" and "binding" concepts have been removed
from Scopes, which are now lightweight container classes.

(6) SemA now generates bytecode for each statement in the AST. Expression
compilation is performed via a separate mechanism called HIR (High-level
Intermediate Representation). To analyze an expression, SemA will walk
the AST, and produce a HIR object for each node. HIR is typed, and SemA
may insert coercion nodes, or even expand an AST node into multiple HIR
nodes. The result of evaluating an expression in SemA is therefore the
the root of a HIR tree, which SemA then sends to the HIRTranslator, which
performs bytecode generation.

As before, SemA still performs full semantic analysis. However, not it
also produces each function's bytecode.

This split allows us to decompose potentially complex semantics into
a more fine-grained, AST-like structure, which can have very simple
code-generation logic. For example, (1.0 + 5) might look like:

HAdd(HFloat(1.0), HConvert(HInteger(5), <Float>))

As part of this decomposition, many opcodes have been removed. Rather than
using typed jumps, we now expand jumps into longer tests. For example:
jge.f <label>
becomes:
ge.f
cvt.f2b
jt <label>

This simplifies the pipeline, and JITs should be able to melt the added
work away.

HIR is not used at a statement level, and HIR does not have any concept
of control-flow.

(7) The old BytecodeCompiler has been removed, as its work is now split
between SemA and HIRTranslator.


In addition, some minor refactorings have taken place:

(1) Opcodes.tbl no longer hardcodes numbers (aaah).
(2) Type has been split into smaller, typed structs.
(3) BytecodeEmitter no longer relies on Pools or RootScopes.
(4) SemA is now responsible for variable/storage allocation.
(5) Native table are now global, rather than per-module. As such, native
declarations now result in a Native object (which still requires a
CALLNATIVE opcode), which is an index into the global table. Natives
must be bound globally. This is in preparation for module support.
(6) Publics are no longer tracked, but rather registered via a global
callback upon module load. This callback can be set by embedders. This
is in preparation for multi-file support.
(7) The BytecodeEmitter now performs Symbol allocations itself, and it
also generates Code objects itself, which removes a good deal of
complexity.

Finally, a test harness has been added. This harness is a python script which finds *.test files, recursively, in the tests folder. For each file it runs the corresponding .sp. The contents of the .test file must match stdout+stderr.
[email protected]
     1
# vim: set ts=2 sw=2 tw=99 noet ft=python:
[email protected]
     2
# 
[email protected]
     3
# Copyright (C) 2004-2012 David Anderson
[email protected]
     4
# 
[email protected]
     5
# This file is part of SourcePawn.
[email protected]
     6
# 
[email protected]
     7
# SourcePawn is free software: you can redistribute it and/or modify it under
[email protected]
     8
# the terms of the GNU General Public License as published by the Free
[email protected]
     9
# Software Foundation, either version 3 of the License, or (at your option)
[email protected]
    10
# any later version.
[email protected]
    11
# 
[email protected]
    12
# SourcePawn is distributed in the hope that it will be useful, but WITHOUT ANY
[email protected]
    13
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
[email protected]
    14
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
[email protected]
    15
# 
[email protected]
    16
# You should have received a copy of the GNU General Public License along with
[email protected]
    17
# SourcePawn. If not, see http://www.gnu.org/licenses/.
[email protected]
    18
#
[email protected]
    19
[email protected]
    20
import os
[email protected]
    21
import sys
[email protected]
    22
from ambuild.command import SymlinkCommand
[email protected]
    23
[email protected]
    24
class KE:
[email protected]
    25
	def __init__(self):
[email protected]
    26
		self.compiler = Cpp.Compiler()
[email protected]
    27
[email protected]
    28
		if AMBuild.mode == 'config':
[email protected]
    29
			#Detect compilers
[email protected]
    30
			self.compiler.DetectAll(AMBuild)
[email protected]
    31
[email protected]
    32
			#Set up defines
[email protected]
    33
			cxx = self.compiler.cxx
[email protected]
    34
			if isinstance(cxx, Cpp.CompatGCC):
[email protected]
    35
				if isinstance(cxx, Cpp.GCC):
[email protected]
    36
					self.vendor = 'gcc'
[email protected]
    37
				elif isinstance(cxx, Cpp.Clang):
[email protected]
    38
					self.vendor = 'clang'
[email protected]
    39
				self.compiler.AddToListVar('CFLAGS', '-pipe')
[email protected]
    40
				self.compiler.AddToListVar('CFLAGS', '-fno-strict-aliasing')
[email protected]
    41
				if (self.vendor == 'gcc' and cxx.majorVersion >= 4) or self.vendor == 'clang':
[email protected]
    42
					self.compiler.AddToListVar('CFLAGS', '-fvisibility=hidden')
[email protected]
    43
					self.compiler.AddToListVar('CXXFLAGS', '-fvisibility-inlines-hidden')
[email protected]
    44
				self.compiler.AddToListVar('CFLAGS', '-Wall')
[email protected]
    45
				self.compiler.AddToListVar('CFLAGS', '-Werror')
[email protected]
    46
				self.compiler.AddToListVar('CFLAGS', '-Wno-switch')
[email protected]
    47
[email protected]
    48
				if AMBuild.options.arch == 'x86':
[email protected]
    49
					self.compiler.AddToListVar('CFLAGS', '-m32')
[email protected]
    50
					self.compiler.AddToListVar('POSTLINKFLAGS', '-m32')
[email protected]
    51
				elif AMBuild.options.arch == 'x64':
[email protected]
    52
					self.compiler.AddToListVar('CFLAGS', '-m64')
[email protected]
    53
					self.compiler.AddToListVar('POSTLINKFLAGS', '-m64')
[email protected]
    54
				elif AMBuild.options.arch:
[email protected]
    55
					raise Exception('Unknown architecture: {0}'.format(AMBuild.options.arch))
[email protected]
    56
[email protected]
    57
				# Disable some stuff we don't use, that gives us better binary
[email protected]
    58
        # compatibility on Linux.
[email protected]
    59
				self.compiler.AddToListVar('CXXFLAGS', '-fno-exceptions')
[email protected]
    60
				self.compiler.AddToListVar('CXXFLAGS', '-fno-rtti')
[email protected]
    61
				self.compiler.AddToListVar('CXXFLAGS', '-fno-threadsafe-statics')
[email protected]
    62
[email protected]
    63
				# We don't really care about these.
[email protected]
    64
				self.compiler.AddToListVar('CXXFLAGS', '-Wno-non-virtual-dtor')
[email protected]
    65
				self.compiler.AddToListVar('CXXFLAGS', '-Wno-overloaded-virtual')
[email protected]
    66
[email protected]
    67
				# The compiler is too aggressive about what what should be a valid offsetof.
[email protected]
    68
				self.compiler.AddToListVar('CXXFLAGS', '-Wno-invalid-offsetof')
[email protected]
    69
[email protected]
    70
				# This would otherwise forbid ((x = y) == NULL) in clang apparently.
[email protected]
    71
				if self.vendor == 'clang':
[email protected]
    72
					self.compiler.AddToListVar('CXXFLAGS', '-Wno-null-arithmetic')
[email protected]
    73
[email protected]
    74
				if (self.vendor == 'gcc' and cxx.majorVersion >= 4 and cxx.minorVersion >= 7) or \
[email protected]
    75
						(self.vendor == 'clang' and cxx.majorVersion >= 3):
[email protected]
    76
					self.compiler.AddToListVar('CXXFLAGS', '-Wno-delete-non-virtual-dtor')
[email protected]
    77
[email protected]
    78
				self.compiler.AddToListVar('POSTLINKFLAGS', '-lm')
[email protected]
    79
			elif isinstance(cxx, Cpp.MSVC):
[email protected]
    80
				self.vendor = 'msvc'
[email protected]
    81
				if AMBuild.options.debug == '1':
[email protected]
    82
					self.compiler.AddToListVar('CFLAGS', '/MTd')
[email protected]
    83
					self.compiler.AddToListVar('POSTLINKFLAGS', '/NODEFAULTLIB:libcmt')
[email protected]
    84
				else:
[email protected]
    85
					self.compiler.AddToListVar('CFLAGS', '/MT')
[email protected]
    86
				self.compiler.AddToListVar('CDEFINES', '_CRT_SECURE_NO_DEPRECATE')
[email protected]
    87
				self.compiler.AddToListVar('CDEFINES', '_CRT_SECURE_NO_WARNINGS')
[email protected]
    88
				self.compiler.AddToListVar('CDEFINES', '_CRT_NONSTDC_NO_DEPRECATE')
[email protected]
    89
				self.compiler.AddToListVar('CXXFLAGS', '/EHsc')
[email protected]
    90
				self.compiler.AddToListVar('CXXFLAGS', '/GR-')
[email protected]
    91
				self.compiler.AddToListVar('CFLAGS', '/W3')
[email protected]
    92
				self.compiler.AddToListVar('CFLAGS', '/nologo')
[email protected]
    93
				self.compiler.AddToListVar('CFLAGS', '/Zi')
[email protected]
    94
				self.compiler.AddToListVar('CXXFLAGS', '/TP')
[email protected]
    95
				self.compiler.AddToListVar('POSTLINKFLAGS', '/DEBUG')
[email protected]
    96
[email protected]
    97
				self.compiler.AddToListVar('POSTLINKFLAGS', 'kernel32.lib')
[email protected]
    98
				self.compiler.AddToListVar('POSTLINKFLAGS', 'user32.lib')
[email protected]
    99
				self.compiler.AddToListVar('POSTLINKFLAGS', 'gdi32.lib')
[email protected]
   100
				self.compiler.AddToListVar('POSTLINKFLAGS', 'winspool.lib')
[email protected]
   101
				self.compiler.AddToListVar('POSTLINKFLAGS', 'comdlg32.lib')
[email protected]
   102
				self.compiler.AddToListVar('POSTLINKFLAGS', 'advapi32.lib')
[email protected]
   103
				self.compiler.AddToListVar('POSTLINKFLAGS', 'shell32.lib')
[email protected]
   104
				self.compiler.AddToListVar('POSTLINKFLAGS', 'ole32.lib')
[email protected]
   105
				self.compiler.AddToListVar('POSTLINKFLAGS', 'oleaut32.lib')
[email protected]
   106
				self.compiler.AddToListVar('POSTLINKFLAGS', 'uuid.lib')
[email protected]
   107
				self.compiler.AddToListVar('POSTLINKFLAGS', 'odbc32.lib')
[email protected]
   108
				self.compiler.AddToListVar('POSTLINKFLAGS', 'odbccp32.lib')
[email protected]
   109
[email protected]
   110
			#Optimization
[email protected]
   111
			if AMBuild.options.opt == '1':
[email protected]
   112
				self.compiler.AddToListVar('CDEFINES', 'NDEBUG')
[email protected]
   113
				if self.vendor == 'gcc' or self.vendor == 'clang':
[email protected]
   114
					self.compiler.AddToListVar('CFLAGS', '-O3')
[email protected]
   115
				elif self.vendor == 'msvc':
[email protected]
   116
					self.compiler.AddToListVar('CFLAGS', '/Ox')
[email protected]
   117
					self.compiler.AddToListVar('POSTLINKFLAGS', '/OPT:ICF')
[email protected]
   118
					self.compiler.AddToListVar('POSTLINKFLAGS', '/OPT:REF')
[email protected]
   119
[email protected]
   120
			#Debugging
[email protected]
   121
			if AMBuild.options.debug == '1':
[email protected]
   122
				if self.vendor == 'gcc' or self.vendor == 'clang':
[email protected]
   123
					self.compiler.AddToListVar('CFLAGS', '-g3')
[email protected]
   124
				elif self.vendor == 'msvc':
[email protected]
   125
					self.compiler.AddToListVar('CFLAGS', '/Od')
[email protected]
   126
					self.compiler.AddToListVar('CFLAGS', '/RTC1')
[email protected]
   127
[email protected]
   128
			#Platform-specifics
[email protected]
   129
			if AMBuild.target['platform'] == 'linux':
[email protected]
   130
				self.compiler.AddToListVar('CDEFINES', '_LINUX')
[email protected]
   131
				if self.vendor == 'gcc':
[email protected]
   132
					self.compiler.AddToListVar('POSTLINKFLAGS', '-static-libgcc')
[email protected]
   133
				if self.vendor == 'clang':
[email protected]
   134
					self.compiler.AddToListVar('POSTLINKFLAGS', '-lgcc_eh')
[email protected]
   135
			elif AMBuild.target['platform'] == 'darwin':
[email protected]
   136
				self.compiler.AddToListVar('POSTLINKFLAGS', '-mmacosx-version-min=10.5')
[email protected]
   137
				if AMBuild.options.arch == 'x86':
[email protected]
   138
					self.compiler.AddToListVar('POSTLINKFLAGS', ['-arch', 'i386'])
[email protected]
   139
				elif AMBuild.options.arch == 'x64':
[email protected]
   140
					self.compiler.AddToListVar('POSTLINKFLAGS', ['-arch', 'x86_64'])
[email protected]
   141
				self.compiler.AddToListVar('POSTLINKFLAGS', '-lstdc++')
[email protected]
   142
[email protected]
   143
				# For OS X dylib versioning
[email protected]
   144
				import re
d[email protected]
   145
				productFile = open(os.path.join(AMBuild.sourceFolder, 'product.version'), 'r')
[email protected]
   146
				productContents = productFile.read()
[email protected]
   147
				productFile.close()
[email protected]
   148
				m = re.match('(\d+)\.(\d+)\.(\d+).*', productContents)
[email protected]
   149
				if m == None:
[email protected]
   150
					self.version = '1.0.0'
[email protected]
   151
				else:
[email protected]
   152
					major, minor, release = m.groups()
[email protected]
   153
					self.version = '{0}.{1}.{2}'.format(major, minor, release)
[email protected]
   154
				AMBuild.cache.CacheVariable('version', self.version)
[email protected]
   155
[email protected]
   156
			#Finish up
[email protected]
   157
			self.compiler.ToConfig(AMBuild, 'compiler')
[email protected]
   158
			AMBuild.cache.CacheVariable('vendor', self.vendor)
[email protected]
   159
			self.targetMap = { }
[email protected]
   160
			AMBuild.cache.CacheVariable('targetMap', self.targetMap)
[email protected]
   161
		else:
[email protected]
   162
			self.compiler.FromConfig(AMBuild, 'compiler')
[email protected]
   163
			self.targetMap = AMBuild.cache['targetMap']
[email protected]
   164
[email protected]
   165
	def DefaultCompiler(self):
[email protected]
   166
		return self.compiler.Clone()
[email protected]
   167
[email protected]
   168
	def JobMatters(self, jobname):
[email protected]
   169
		file = sys._getframe().f_code.co_filename
[email protected]
   170
		if AMBuild.mode == 'config':
[email protected]
   171
			self.targetMap[jobname] = file
[email protected]
   172
			return True
[email protected]
   173
		if len(AMBuild.args) == 0:
[email protected]
   174
			return True
[email protected]
   175
		if not jobname in AMBuild.args:
[email protected]
   176
			return False
[email protected]
   177
[email protected]
   178
ke = KE()
[email protected]
   179
globals = {
[email protected]
   180
	'KE': ke
[email protected]
   181
}
[email protected]
   182
[email protected]
   183
FileList = [
[email protected]
   184
		['src', 'AMBuild.library'],
[email protected]
   185
		['src', 'AMBuild.shell'],
[email protected]
   186
		['AMBuild.dist']
[email protected]
   187
	]
[email protected]
   188
[email protected]
   189
for parts in FileList:
[email protected]
   190
	AMBuild.Include(os.path.join(*parts), globals)
[email protected]
   191