Runtimes, JITs &
Modern Backends

Decks 03–08 took source text to machine code. This part asks the next question: how does that code actually run, and what does the runtime do for it? We trace the execution spectrum, then the engines and services that make modern dynamic languages fast.

There is no single way to “run” a program. The same source can be compiled ahead-of-time to native code, interpreted directly, or compiled just-in-time while it runs. Most real systems are hybrids.

Each strategy optimises a different point on the startup–peak–footprint–portability surface. A serverless function and a long-running server want opposite ends of it.

The simplest interpreter walks the AST directly. It is easy to write but slow: every node is a virtual dispatch, pointers chase all over the heap, and there is no locality. Real engines first lower the AST to a linear bytecode and interpret that.

// Stack bytecode interpreter — the classic
// fetch/decode/execute switch dispatch loop
for (;;) {
  Op op = *ip++;              // fetch
  switch (op) {               // decode + dispatch
    case OP_CONST: push(*ip++);          break;
    case OP_ADD:   { int b=pop(),a=pop();
                     push(a+b); }        break;
    case OP_JMP:   ip += (int8_t)*ip;    break;
    case OP_RET:   return pop();
    /* ... one case per opcode ... */
  }
}

In a bytecode interpreter the useful work of an opcode (an add, a load) is tiny; the dispatch — fetch, decode, branch to the handler — dominates. Faster interpreters are mostly about cheaper dispatch and better branch prediction.

The standard speedup: a GNU C labels-as-values table maps each opcode to its handler's address, and every handler ends by jumping directly to the next op's handler — no central switch. CPython adopted this; it gave a 15–20% interpreter win.

// Direct-threaded dispatch (GCC/Clang extension)
static void *table[] = {
  [OP_CONST]=&&do_const, [OP_ADD]=&&do_add,
  [OP_JMP]=&&do_jmp,     [OP_RET]=&&do_ret,
};
#define NEXT() goto *table[*ip++]

  NEXT();                       // start dispatch
do_const: push(*ip++);  NEXT();
do_add:  { int b=pop(),a=pop();
           push(a+b); }         NEXT();
do_jmp:   ip += (int8_t)*ip;    NEXT();
do_ret:   return pop();

A JIT compiles bytecode (or a trace) to native machine code at runtime, then jumps into it. Because it runs during execution, it can specialise to the program's actual behaviour — the inputs, the types, the hot paths it has observed. The idea dates to McCarthy's LISP (1960) and Hansen's adaptive FORTRAN.

No single compiler wins on both startup and peak, so modern VMs run a tiered pipeline: start interpreting for instant launch, then promote hot code through progressively more optimising compilers as profiles accumulate. Counters (invocation + back-edge) trigger each promotion.

V8's modern shape is Ignition → Sparkplug → Maglev → TurboFan. HotSpot uses interpreter → C1 → C2; .NET uses the interpreter-free Quick JIT (Tier-0) → optimising RyuJIT (Tier-1), plus ReadyToRun AOT.

Tiering trades a brief slow start for a far higher ceiling. Plotting throughput over time shows the classic warmup curve: each tier steps performance up as hot code is recompiled — with a temporary dip whenever a deopt throws optimised code away.

In a dynamic language a.x or a + b could mean anything — the lookup is potentially a full hash search every time. The inline cache (Deutsch & Schiffman, Smalltalk-80; refined in Self) caches the result of the last lookup at the call site and reuses it while the type stays the same.

An IC degrades as it sees more types: monomorphic (1) → polymorphic (2–4) → megamorphic (many). The optimising JIT reads these states as type feedback: a monomorphic site can be inlined to a single guarded field load; a megamorphic one cannot.

Inline caches need a cheap, stable answer to “what type is this object?” — but dynamic objects have no declared class. The trick from Self: synthesise one. Objects with the same set of properties in the same order share a hidden class (V8 calls it a Map; JSC a Structure; SpiderMonkey a Shape).

function Point(x, y) {
  this.x = x;   // transition: {} -> M1 (has x)
  this.y = y;   // transition: M1 -> M2 (has x,y)
}
let a = new Point(1, 2);   // hidden class M2
let b = new Point(3, 4);   // SAME hidden class M2  ✓

// M2 records: x @ offset 0, y @ offset 1
// IC at "p.x" specialises to: load slot 0 if map==M2

b.z = 9;   // b transitions M2 -> M3; a stays M2
           // now the .x site is polymorphic — slower

The optimising JIT bets: “x is always a small integer”, “this call is always Array.map”. It compiles fast code that assumes the bet, protected by a cheap guard. If a guard ever fails, it must deoptimise — bail out of the optimised code mid-execution, without losing the running computation.

// What the optimiser speculates on:
function sum(arr) {
  let s = 0;
  for (let i = 0; i < arr.length; i++)
    s += arr[i];   // guard: arr is a packed
                   // Smi (small-int) array
  return s;
}
// Hot, monomorphic -> TurboFan inlines a
// tight int loop, no bounds re-check, no boxing.
//
// Later: sum([1, 2, "x"])  -> guard FAILS
//   -> DEOPT: rebuild interpreter frame,
//      resume at the "+=" bytecode, mark
//      the array element type as polymorphic.

Compiled code rarely manages memory alone. The runtime provides allocation and reclamation — either manual (malloc/free, RAII, ownership) or automatic (a garbage collector). The compiler must cooperate: it emits the metadata the collector needs.

All tracing collectors find the live set the same way; they differ in how they reclaim and compact — trading throughput, pause time, space overhead and fragmentation.

A typical generational layout: a small young generation (an Eden plus two survivor spaces for copying), and a large old generation for tenured survivors. New objects bump-allocate into Eden; minor GCs copy survivors and age them; long-lived objects are promoted.

A naive collector is stop-the-world: it pauses every thread while it traces. For interactive and server workloads those pauses are unacceptable, so modern collectors do most of their work concurrently with the running program — the headline goal is bounded, sub-millisecond pauses.

Exceptions need the runtime to abandon the current computation and transfer control to a handler further up the stack — unwinding intervening frames and running their cleanup. How that is implemented determines whether the non-throwing path costs anything.

Deck 08 emitted an object file with relocations. At load time the dynamic linker (ld.so) finishes the job: it maps shared libraries, resolves cross-module symbols, and patches addresses — ideally lazily, only on first use.

; A call to an external function via the PLT
    call    printf@PLT
; printf@PLT (the stub):
printf_plt:
    jmp     *printf@GOT(%rip)   ; 1st call: GOT
                                ; still points back
                                ; into the resolver
    push    $reloc_index        ; which symbol
    jmp     _dl_runtime_resolve ; ld.so resolves,
                                ; rewrites the GOT,
                                ; then tail-calls printf
; 2nd call onward: jmp *GOT goes straight to printf

Writing a production backend (instruction selection, scheduling, register allocation, every ISA) is enormous. The field's answer: reusable backend infrastructure. A new language emits a shared IR and inherits decades of optimisation and dozens of targets — the N×M argument from deck 06, realised.

LLVM optimises hard but compiles slowly. A wave of newer infrastructure targets different points: fast codegen, language-agnostic JITs, and extensible, multi-level IRs.

WebAssembly (Wasm, 2017) is a compact, portable bytecode for a stack machine — designed as a compile target, not a language to write by hand. It is fast to validate, fast to compile, sandboxed by construction, and deterministic. C/C++/Rust/Go/Swift all emit it.

;; A Wasm function (text format) — stack machine
(func $sum (param $n i32) (result i32)
  (local $acc i32) (local $i i32)
  (loop $L
    (local.set $acc
      (i32.add (local.get $acc) (local.get $i)))
    (local.set $i
      (i32.add (local.get $i) (i32.const 1)))
    (br_if $L                       ;; structured
      (i32.lt_s (local.get $i)      ;; control flow:
                (local.get $n))))   ;; no raw jumps
  (local.get $acc))

A Wasm module is itself executed by an engine that re-uses everything in this deck: an interpreter, a baseline JIT, an optimising JIT — or an AOT compile. Many source languages converge on Wasm, and many engines run it: Wasm is increasingly the portable waypoint McCarthy & Strachey called UNCOL in 1958.

The runtime story keeps moving. A few directions visible from here — the common thread is co-design: compilers, runtimes, and hardware increasingly evolve together rather than over a fixed interface.