Geblang Internals

This chapter describes how Geblang is implemented, written for readers who want a structural walkthrough of a scripting language built in Go. It covers the full pipeline from source text to execution, and the design choices made at each layer.

All source paths below are relative to the repository root.

The Pipeline

A Geblang program goes through these stages before any code executes:

source text
    |
    v
Lexer (internal/lexer)         -- produces a stream of tokens
    |
    v
Parser (internal/parser)       -- produces an AST
    |
    v
Semantic Analyzer (internal/semantic)  -- validates declarations
    |
    v
  two possible execution paths:

  [Bytecode Compiler]           -- preferred path
  (internal/bytecode/compiler)
    |
    v
  [VM]
  (internal/bytecode/vm)

  [Evaluator]                   -- compatibility / fallback path
  (internal/evaluator)

The bytecode path is tried first. If compilation succeeds the VM runs the resulting chunk. If compilation fails and strict-VM mode is not set, the evaluator runs the AST directly. The --trace-exec flag prints which engine ran a given script.

---

Tokens: internal/token

token.Token is the atom the lexer produces and the parser consumes. Each token has a Type (an integer constant), a Literal string, a Raw string, and source Line/Column fields.

Token types are simple integer constants declared in token/token.go: Ident, Int, Decimal, Float, String, Assign, Plus, Eq, And, Or, LParen, LBracket, LBrace, Dot, Arrow, NullCoalesce, Range, EOF, and so on. The full list is about 80 types covering every syntactic atom the language uses.

---

Lexer: internal/lexer

lexer.Lexer turns a source string into a sequence of tokens. Its state is minimal:

type Lexer struct {
    input        string
    position     int       // current byte offset
    readPosition int       // next byte offset
    ch           rune      // current character
    line         int
    column       int
    pendingDocs  []string  // accumulated doc comments
}

NextToken() is the only interface the parser calls. It dispatches on the current character using a large switch, emitting one token per call. Multi-byte sequences like ==, !=, <=, **=, .., ..<, ?., and ?? are handled by peeking one character ahead.

Identifiers and keywords share the same token type initially; the lexer checks a keyword table before returning the token, upgrading Ident to the keyword type where appropriate. String literals handle escape sequences and Unicode in the lexer. Numeric literals support decimal underscores (1_000_000), hex (0xFF), octal (0o77), and binary (0b1010) prefixes.

Doc comments (/// or /** ... */) are accumulated in pendingDocs and attached to the next non-comment token so the parser can associate them with the declaration they annotate.

---

AST: internal/ast

The AST defines the tree of nodes the parser builds.

Core interfaces

type Node interface {
    TokenLiteral() string
    String() string
}

type Statement interface {
    Node
    statementNode()
}

type Expression interface {
    Node
    expressionNode()
}

ast.Program holds []Statement and is the root returned by every parse call.

Statements and expressions

There are around 40 statement types (LetStatement, FunctionStatement, ClassStatement, ReturnStatement, ForStatement, IfStatement, TryCatchStatement, DeferStatement, ImportStatement, etc.) and a similar number of expression types (Identifier, IntegerLiteral, StringLiteral, CallExpression, SelectorExpression, IndexExpression, InfixExpression, PrefixExpression, FunctionLiteral, ListLiteral, DictLiteral, and more).

TypeRef

TypeRef is a recursive type annotation node used everywhere a type appears:

type TypeRef struct {
    Token     token.Token
    Name      string      // base name, e.g. "list", "string", "T"
    Nullable  bool        // ?string
    Arguments []*TypeRef  // generic arguments: list<string> -> Arguments[0].Name="string"
    ListAlias bool        // T[] shorthand -- Name holds the element type
    Left      *TypeRef    // left side of a union/intersection operator
    Operator  string      // "|" or "&"
    Right     *TypeRef    // right side
}

list<int> produces {Name:"list", Arguments:[{Name:"int"}]}. int[] produces {Name:"int", ListAlias:true}. string|int produces {Operator:"|", Left:{Name:"string"}, Right:{Name:"int"}}.

---

Parser: internal/parser

The parser is a hand-written Pratt (top-down operator precedence) parser. Its main entry point is parser.New(lexer).ParseProgram(), which returns *ast.Program.

Pratt parsing

Each token type optionally has two parse functions registered against it:

• a prefix function, called when the token appears at the start of an expression (literal values, prefix operators, (grouped expr))
• an infix function, called when the token appears between two expressions (binary operators, [index], (call), .member)

parseExpression(precedence) calls the prefix function for the current token, then repeatedly calls infix functions while the next token has higher precedence than precedence. This naturally handles operator precedence and associativity without a grammar table.

Precedence levels

lowest
assign          =  +=  -=  ...
ternary         ?
nullCoalesce    ??
logicalOr       ||
logicalAnd      &&
bitOr           |
bitXor          ^
bitAnd          &
equality        ==  !=  is  instanceof
compare         <  <=  >  >=  ..  ..<  as
shift           <<  >>
sum             +  -
product         *  /  //  %
power           **
prefix          !  -  ~  ++  --
postfix         ++  --
call            .  ?.  (  [

Type reference parsing

parseTypeRefFromCurrent() handles the full type syntax: nullable prefix ?, generic arguments <T, U>, union | and intersection & operators, and the list shorthand []. It converts int[] into a TypeRef with ListAlias=true and Name="int".

Statement parsing

Top-level statements are parsed with parseStatement(). When a keyword is seen (let, func, class, if, for, return, etc.) the corresponding dedicated parse function is called. Otherwise the statement is treated as an expression statement. The parser requires semicolons to terminate most statements, following C-style syntax.

---

Semantic Analyzer: internal/semantic

semantic.Analyzer performs a lightweight pre-execution pass over the AST. It does not do full type inference; its job is to catch structural errors that neither the parser nor runtime is well-placed to handle.

type Analyzer struct {
    diagnostics []Diagnostic
    scopes      []map[string]typeInfo
    functions   map[string][]methodInfo
    classes     map[string]classInfo
    interfaces  map[string]interfaceInfo
    aliases     map[string]typeInfo
}

Analyze(program) does four things in order:

1. collectTypeDeclarations: walks all statements to register functions, classes and interfaces so that forward references work.
2. validateTopLevelOverloads: checks that overloaded functions at the top level have distinct signatures.
3. validateClassOverloads: same check for methods and constructors.
4. validateInterfaceImplementations: verifies that classes marked implements Foo actually provide every method the interface declares.

Errors are collected as []Diagnostic rather than halting immediately, so the caller receives all problems at once.

---

Runtime Values: internal/runtime

Every value a Geblang program can hold implements the Value interface:

type Value interface {
    TypeName() string
    Inspect() string
}

TypeName() returns the runtime type string ("int", "string", "list", "func", etc.). Inspect() returns a human-readable representation.

Primitive types

SmallInt is the int64-backed representation used wherever the value fits in a signed 64-bit register; the bytecode VM hot path operates on SmallInt without ever touching runtime.Value (see VMValue below). Int is the arbitrary-precision fallback for overflow and for literals that do not fit in int64.

Decimal uses math/big.Rat so that 0.1 + 0.2 produces 0.3 exactly. Literal 3.14 parses as Decimal, not Float; 3.14f is a Float.

Collection types

Dict entries are keyed internally by the string form of the key value, but the original key value is preserved alongside each entry so the runtime can return the original type.

Callable types

Function is the evaluator's callable value. It holds the AST parameter list, body, captured environment, and optional Native hook for Go-implemented functions. OverloadedFunction wraps a slice of Function values that share a name; dispatch selects the overload whose parameter count and types match the call arguments.

BytecodeFunction and BytecodeClosure are the VM equivalents. They store an index into the chunk's Functions table and (for closures) a slice of captured upvalues.

Object types

Instance represents a class instance. It holds a pointer to its class descriptor and a map of field names to values.

Module holds an exported name-to-value map and is what import statements bind.

Special runtime values

Generator wraps either a pre-computed value slice or a next callback function, and provides Next()/(value, bool, error) iteration. Generator coroutines in the VM communicate through a Go channel pair.

DateTimeInstant, DateTimeDuration, DateTimeZone, URLValue, HTTPHeaders, HTTPCookie, TemplateValue, and TemplateEngine are typed opaque wrappers for domain-specific native values. They carry type names like "datetime.Instant" so the runtime can dispatch methods on them correctly.

---

Bytecode Format: internal/bytecode/bytecode.go

Chunk

A Chunk is the compiled form of one source file or module:

type Chunk struct {
    SourceHash   [32]byte      // SHA-256 of the source bytes
    Compiler     string        // version string, for cache invalidation
    Constants    []Value       // literal pool
    Instructions []Instruction // flat instruction array
    Functions    []FunctionInfo
    Classes      []ClassInfo
    Interfaces   []InterfaceInfo
    Exports      []ExportInfo
}

All functions, including the top-level script, share the same flat Instructions array. Each FunctionInfo holds an Entry offset into that array, so function calls are just an IP jump.

Instruction

type Instruction struct {
    Op       Op       // opcode byte
    Operands []int64  // variable-length operand list
    Line     int
    Column   int
}

Operands are 64-bit signed integers. Most opcodes take zero or one operand (a constant pool index, a local slot, a jump target). Multi-operand instructions are rare.

Opcodes

The Op type is a byte. There are around 110 opcodes. Representative examples:

FunctionInfo

type FunctionInfo struct {
    Name             string
    TypeParameters   []string   // generic parameter names
    Entry            int64      // offset into Instructions
    ParamNames       []string
    ParamSlots       []int64    // local slot index for each parameter
    ParamTypes       []string   // type strings for runtime checks
    ReturnType       string
    DefaultConstants []int64    // constant pool indices for default values
    UpvalueCount     int64
    Variadic         bool
    Async            bool
    IsGenerator      bool
    Decorators       []DecoratorMetadata
}

Serialization

bytecode.Encode(chunk) serializes a Chunk to bytes. The format starts with the magic bytes "GEBBC", a 2-byte version number, the SHA-256 source hash, and then length-prefixed sections for constants, instructions, functions, classes, interfaces, and exports. bytecode.Decode(bytes) is the inverse.

The source hash lets the runtime skip recompilation: if the cached .gbc file's hash matches the source file, the cached chunk is loaded directly.

---

Bytecode Compiler: internal/bytecode/compiler.go

bytecode.Compile(program, source, version) is the entry point. It creates a Compiler, walks the AST, and returns a Chunk.

Compiler state

type Compiler struct {
    chunk         Chunk
    loops         []loopContext      // break/continue target stack
    globals       map[string]int64   // name to constant pool index
    globalTypes   map[string]string  // name to type string
    scopes        []map[string]binding // lexical scope stack
    locals        int64              // next local slot
    funcs         map[string][]int64 // name to overload function indices
    classes       map[string]int64   // name to class index
    interfaces    map[string]int64
    enums         map[string]int64
    typeAliases   map[string]*ast.TypeRef
    inFunc        int                // nesting depth for functions
    classStack    []int64            // for `this` inside methods
    finalizers    []finalizerContext // defer/finally cleanup stack
    expectedTypes []string           // declared types of let bindings in scope
    returnTypes   []string           // expected return types of enclosing functions
    reflectFuncs  map[string]DecoratorTarget // decorator metadata
    ...
}

Compilation pass

The compiler does a single forward pass over program.Statements. At the end it patches all forward-reference jump targets by back-filling jump instruction operands.

For each statement kind the compiler emits a sequence of instructions. A function declaration, for example:

1. Appends a new FunctionInfo entry to chunk.Functions.
2. Saves the current emit position and emits an OpJump placeholder to skip over the function body.
3. Records the function entry point, then compiles the body.
4. Emits OpReturn at the end.
5. Patches the OpJump operand to point past the function body.
6. Emits OpDefineGlobal (or OpDefineLocal) to bind the function value in the current scope.

Local variable scopes are managed with a stack of maps (scopes). Each let binding allocates the next locals slot, recorded in the innermost scope map. Reads and writes emit OpGetLocal/OpSetLocal with the slot number. At scope exit the slot count is not reclaimed immediately (the VM allocates a fixed local array per call frame).

Closures are compiled as functions with a non-zero UpvalueCount. The compiler identifies captured variables and emits OpMakeClosure with a list of upvalue indices.

Type strings (for OpTypeAssert, ParamTypes, ReturnType) are produced by bytecodeTypeNameForParam, which flattens a TypeRef tree into a string like "list<int>", "?string", or "dict<string,list<float>>". Nested generic types are serialized with brackets, and the VM parses them back with bracket-depth-aware helpers.

---

VM: internal/bytecode/vm.go

VM state

type VM struct {
    chunk             Chunk
    stdout            io.Writer
    stack             []runtime.VMValue // operand stack (tagged union)
    globals           []runtime.VMValue // indexed by global slot
    locals            []runtime.VMValue // current frame's locals (alias into frames)
    frames            []callFrame       // call stack
    defers            [][]deferredAction
    exceptionHandlers []exceptionHandler
    pendingThrow      *runtime.Error
    moduleLoader      ModuleLoader
    statefulNative    StatefulNativeCaller
    classIndex        map[string]int    // class name to Classes index
    natives           *native.Registry
    syncMode          bool              // run async funcs synchronously
    ...
}

The stack / locals / globals slices hold runtime.VMValue (internal/runtime/vmvalue.go), a 32-byte tagged union with an inline int64 payload and an interface-typed boxed fallback:

type VMValue struct {
    Kind  VMKind        // 1-byte tag (Null, Bool, SmallInt, Float, Boxed, ...)
    I64   int64         // SmallInt payload / bool 0|1 / Float64bits
    Boxed runtime.Value // catch-all for List, Dict, Class, Instance, Int, Decimal, ...
}

This representation removes the two runtime.SmallInt heap allocations per integer arithmetic step that a runtime.Value interface previously imposed. The fast paths for OpAddInt, OpSubInt, OpLessInt, the fused jump-compare opcodes, and the load-op-store opcodes all operate on the Kind == VMKindSmallInt case inline. Any non-primitive value takes the VMKindBoxed fallback and routes through the same runtime.Value interface used by the evaluator, preserving identity through ToValue() / VMValueFromValue() at the boundary.

Call frame

type callFrame struct {
    returnIP     int
    locals       []runtime.Value
    typeBindings map[string]string  // generic type parameter bindings
    generator    chan vmGeneratorItem
    functionName string
    callLine     int
    ...
}

Each function call pushes a new callFrame. returnIP is the instruction index to resume after the call returns. typeBindings holds generic type parameter names resolved to concrete type strings for the duration of the call, populated by OpSetTypeBindings.

Run loop

vm.Run() is a simple fetch-decode-execute loop:

ip := 0
for ip < len(vm.chunk.Instructions) {
    instr := vm.chunk.Instructions[ip]
    ip++
    switch instr.Op {
    case OpConstant:
        vm.push(vm.chunk.Constants[instr.Operands[0]])
    case OpAdd:
        right := vm.pop()
        left := vm.pop()
        vm.push(vm.add(left, right))
    // ... ~165 cases
    }
}

The stack grows upward; push appends to the slice and pop removes from the end. All arithmetic, comparisons, collections, control flow, function calls, and error handling are implemented as opcode handlers in this loop.

A handful of opcodes operate on VMValue directly without round-tripping through runtime.Value:

• Integer fast paths: OpAddInt, OpSubInt, OpMulInt, OpModInt, OpLessInt, OpGreaterInt, OpEqualInt, OpIncLocalInt, OpDecLocalInt. Each checks Kind == VMKindSmallInt on its operands and performs the arithmetic on the inline I64 field. Overflow falls back to runtime.Int (big.Int) promotion.
• Fused compare-and-branch: OpJumpIfNotLessInt, OpJumpIfNotEqualInt, etc. A single opcode pops two SmallInts and jumps when the underlying boolean condition is false, removing the intermediate Bool push/pop.
• Fused load-op-store: OpAddLocalIntLocal, OpAddGlobalIntConst, etc. Cover the a = a + b / a = a + N pattern with a single instruction.

The compiler emits these specialised opcodes whenever both operands are statically typed as int; otherwise the generic dispatch path runs and routes through runtime.Value.

Function calls

OpCall pops N arguments and the callee value, then dispatches:

• If the callee is a runtime.Function (evaluator-style), it calls back into the evaluator via StatefulNativeCaller.
• If the callee is a FunctionInfo index, it pushes a new callFrame with the function's locals pre-allocated, sets ip to Entry, and continues the loop.
• For native functions, startFunction calls the registered Go function directly.
• For closures, upvalues are injected into the new frame's locals before entering the body.

Type checking at call sites is done in startFunction using matchValueToTypeStr, which walks the string-encoded type recursively. Generic type parameters in typeBindings are skipped during element-level checks.

Locals snapshot

On entering a callee, the VM snapshots the current frame's locals so the caller and callee can share the same backing storage without one clobbering the other:

frame.locals = vm.snapshotLocals()   // internal/bytecode/vm.go

The snapshot is a per-call copy() of the parent frame's locals slice into a fresh buffer (reused from vm.localsFree when available). For deep recursion this copy is measurable in CPU profiles; runtime.duffcopy typically shows up at single-digit percent on BenchmarkRecursiveFib, and is one of the larger remaining costs on recursion-heavy workloads.

Generators

A generator function runs in a separate goroutine. When the VM compiles a generator call it creates a Go channel pair. The generator goroutine runs the function body, sending each yielded value on the channel. The outer VM receives values through OpIterNext, which reads from the channel, and signals completion through OpIterClose.

Exception handling

OpPushExceptionHandler pushes an exceptionHandler onto a stack, recording the instruction offset of the handler code. When an error is thrown (either by OpThrow or a runtime operation), the VM unwinds to the nearest handler and jumps to its offset. OpPopExceptionHandler removes the handler when the protected block exits normally. Uncaught throws propagate as a Go error from Run().

Deferred calls

defer statements push deferredAction values onto a per-scope slice. When a function returns or a try block exits, the VM pops and executes all deferred actions in reverse order.

Module loading

OpImportModule calls ModuleLoader.LoadModule(canonical, alias). The loader compiles (or loads from cache) the target module's chunk, runs it in a fresh VM, and returns the exported runtime.Module. Modules are cached after first load.

---

Evaluator: internal/evaluator

The evaluator is a tree-walking interpreter that executes the AST directly without compilation. It is the older and more complete execution path; new language features sometimes appear in the evaluator before the compiler and VM support them.

The main types are Evaluator (holds stdlib registrations, import caches, and the evaluator configuration) and Session (wraps an Evaluator and an Environment for one REPL session or script run).

evalExpression(expr, env) and evalStatements(stmts, env) are the core recursive functions. They return a runtime.Value and an error.

Control-flow signals (return, break, continue, throw, exit) are communicated as a signal struct rather than a Go error, avoiding the need to unwrap errors at each level:

type signal struct {
    kind     string        // "return", "break", "continue", "throw"
    value    runtime.Value // return or thrown value
    thrown   runtime.Value
    exited   bool
    exitCode int
}

The evaluator handles all the same language features as the VM but through direct Go function calls rather than opcode dispatch.

Builtin modules

The evaluator implements stateful standard library modules (HTTP client, database, filesystem, Redis, templates, etc.) as Go structs with method maps. Each module is registered in e.builtins under its canonical name and provides a map[string]builtinFunction of callable entries. Module functions receive []runtime.Value arguments and return (runtime.Value, error).

---

Native Registry: internal/native

native.Registry holds pure, stateless functions shared between the evaluator and the VM. These are functions whose behavior depends only on their arguments and produces no side effects beyond their return value: math, string manipulation, encoding, parsing, cryptography, and similar utilities.

Registering a native function associates a module name, a function name, and a Go func([]runtime.Value) (runtime.Value, error) implementation. The registry is initialized once and passed to both the evaluator and the VM.

The evaluator calls registry.Call(module, name, args) directly. The VM emits OpNativeCall with the module and name embedded as constants, and resolves the call through the same registry at runtime.

---

Module System: internal/modules

modules.Resolver locates .gb source files for import statements.

type Resolver struct {
    ModulePaths   []string  // user-specified search paths
    StdlibPaths   []string  // stdlib installation paths
    DisableStdlib bool
    Manifests     map[string]*Manifest
}

Resolution order:

1. Check if the import name matches a builtin native module (registered in the evaluator or native registry).
2. Look for <name>.gb relative to the importing file's directory.
3. Search each ModulePaths entry.
4. Search each StdlibPaths entry.

A Manifest (geblang.yaml) can declare a module name, version, source entrypoint, and additional module paths. The resolver loads manifests to support multi-file packages.

---

Source Stdlib: stdlib/

Not all standard library code is written in Go. Several modules are written in Geblang itself and distributed as source files in the stdlib/ directory:

These modules are installed alongside the geblang binary and are found via StdlibPaths. They are compiled and cached like any other user module.

---

CLI: cmd/geblang

The geblang binary entry point is cmd/geblang/main.go. It parses command- line arguments and delegates to one of several execution modes:

• Script mode: geblang script.gb reads and parses the file, runs the semantic analyzer, then tries the bytecode path and falls back to the evaluator.
• Module mode: geblang -m moduleName generates a thin wrapper script that imports the named module and calls its main() function.
• REPL mode: geblang with no arguments starts an interactive session. The REPL (repl.go) maintains a persistent evaluator.Session. Each input line is parsed and evaluated; expression results are printed if non-void.
• Check mode: geblang check script.gb runs parse and semantic analysis only, reporting errors without executing.
• Format mode: geblang fmt script.gb runs the formatter (internal/formatter).
• Build mode: geblang build compiles the application and its dependencies into a self-contained binary (build.go, internal/bundle).
• LSP mode: geblang lsp starts the Language Server Protocol server (lsp.go, internal/lsp).
• DAP mode: geblang dap starts the Debug Adapter Protocol server (dap.go, internal/dap).

Execution mode selection

runScript tries the bytecode path by calling loadOrCompileBytecode, which either decodes a cached .gbc file (if the source hash matches) or calls bytecode.Compile. If compilation succeeds, a VM is constructed and Run() is called. If compilation fails (for example because a feature is not yet implemented in the compiler), the function falls back to runEvaluator unless --vm-only was passed.

The --trace-exec flag writes a one-line note to stderr indicating which path (vm or evaluator) handled the script and, on the evaluator path, the compilation error that triggered the fallback.

Bytecode module loader

The VM calls into a ModuleLoader implementation (bytecodeModuleLoader in main.go) for each OpImportModule instruction. The loader:

1. Resolves the module name to a file path via modules.Resolver.
2. Reads and parses the source file.
3. Compiles it to a Chunk (or loads from the .gbc cache).
4. Creates a fresh VM, runs the chunk, and collects the exported values.
5. Caches the resulting runtime.Module so subsequent imports of the same module in the same process return the cached value.

For modules that use stateful native functions (HTTP, database, etc.) the loader holds a shared evaluator.Evaluator instance and routes StatefulNativeCaller calls through it.

---

Adding a New Feature

The typical path for adding a language feature is:

1. Add any new token types to internal/token/token.go.
2. Update internal/lexer/lexer.go to emit the new tokens.
3. Add AST node types to internal/ast/ast.go.
4. Update internal/parser/parser.go to parse the new syntax into AST nodes.
5. Update internal/semantic/analyzer.go if the feature involves declarations that need structural validation.
6. Implement the feature in internal/evaluator/evaluator.go.
7. Add opcode(s) to internal/bytecode/bytecode.go.
8. Emit the opcodes in internal/bytecode/compiler.go.
9. Handle the opcodes in internal/bytecode/vm.go.
10. Add parity tests in internal/bytecode/parity_test.go to verify that the evaluator and VM produce the same result.
11. Update documentation in docs/user/ and add examples in examples/.

Features typically land in the evaluator first (steps 1-6) and get compiler/VM support (steps 7-10) as a follow-up. Until both paths support a feature, programs using it run via the evaluator fallback.