Compilers: From Source Code to Silicon

The code you wrote is not what runs#

Here is a fact that sounds wrong the first time you hear it: the processor in your machine has never executed a single line of the code you wrote. Not the Python, not the JavaScript, not the C. When you type x = 2 + 3 * y, you are writing for a human audience — and for one very particular translator — but not for the CPU. The CPU cannot read it.

A processor understands exactly one language: machine code, a stream of binary instructions defined by its specific architecture. An instruction might mean "load the value at this address into register 1" or "multiply register 1 by register 2." That is the entire universe of things a CPU can do. Words like if, while, and print do not exist down there. Neither do variable names or arithmetic expressions. They are conveniences invented for us, and something has to strip them away before the silicon can act.

That something is a compiler. Its job is translation — and, crucially, it does not translate in one leap. It works in stages, each handing a cleaner, more structured representation to the next. Understanding those stages is understanding how a wall of human-readable text becomes the physical dance of voltages that computes your result.

From text to tokens: the lexer#

The first stage confronts the ugliest truth about source code: to a program, it is just a flat string of characters. x = 2 + 3 * y is not yet an assignment or an expression — it is the letter x, a space, an equals sign, another space, and so on. Spaces, tabs, and newlines carry no more meaning than the shape of the letters.

The lexer (or scanner) makes the first cut. It groups raw characters into tokens — the smallest meaningful units, like words in a sentence. Our line becomes a stream:

IDENT(x)  ASSIGN(=)  NUMBER(2)  PLUS(+)  NUMBER(3)  STAR(*)  IDENT(y)

Whitespace vanishes; each token now has a kind. This is not so different from how a finite automaton recognizes patterns character by character — in fact, lexers are classically built exactly that way, as small state machines that switch states as each character arrives. (The article on finite-automata walks through that machinery.) The output is still linear, a list, but it is a list the next stage can reason about.

From tokens to structure: the parser#

A list of tokens still says nothing about structure. Does 2 + 3 * y mean "add 2 and 3, then multiply by y," or "multiply 3 by y, then add 2"? The tokens alone cannot tell you. The answer comes from a grammar — a formal set of rules describing which sequences of tokens are legal and how they nest. The parser applies that grammar to build an abstract syntax tree (AST): a tree that captures the program's structure explicitly, discarding incidental details like parentheses and spacing.

The AST for our line looks like this:

      =
     / \
    x   +
       / \
      2   *
         / \
        3   y

Notice what the shape encodes. The * node sits below the + node, and that depth is not decorative — it means "evaluate me first." The grammar assigns multiplication a higher precedence than addition, so the parser buries 3 * y deeper in the tree. When the tree is later walked bottom-up, the product is computed before the sum, and 2 + 3 * y correctly means 2 + (3 * y). Precedence, associativity, and nesting all become pure geometry.

After parsing comes semantic analysis: checking that the tree makes sense. Is y actually declared? Do the types line up — are you adding a number to a number, or accidentally to a string? This is where a compiler catches the errors an interpreter might only stumble into at runtime. Only once the tree is validated does translation proper begin.

From tree to instructions: code generation#

Now the compiler walks the validated AST and emits instructions. For a real CPU it emits machine code directly; many modern compilers first emit an intermediate representation (IR) — a simpler, architecture-neutral language — optimize that, and only then lower it to the target machine. Either way, our tiny tree might become something like:

LOAD  R1, 3
LOAD  R2, y
MUL   R1, R1, R2   ; 3 * y  (done first — it was deepest)
LOAD  R2, 2
ADD   R1, R2, R1   ; 2 + (3*y)
STORE x,  R1

Between the tree and this output sits optimization: the compiler may fold constants, drop dead code, or reuse registers to make the result faster or smaller — while preserving exactly what the program computes. The numbers here, ultimately, are binary; the assembly-like mnemonics are just a readable stand-in for the bit patterns that will flow through the logic gates. If you want to see where those instructions physically bottom out, the piece on logic-gates shows arithmetic emerging from nothing but switches.

Compilers versus interpreters: a false binary#

People often sort languages into two bins: "compiled" languages like C, and "interpreted" languages like Python. It is a tidy story, and it is wrong. The distinction is not a property of the language — it is a property of the implementation.

A compiler translates the whole program ahead of time, producing an artifact that runs on its own. That means a slower startup (you pay the translation cost once, up front) but fast execution afterward, and errors caught before the program ever runs. An interpreter takes a different bargain: it walks the program statement by statement, translating and executing each as it goes. There is no separate build step, which makes it flexible and easy to debug, but it re-does translation work every run, so it is typically slower.

The neat binary collapses the moment you look at real systems. "Interpreted" Python does not execute your source text directly — it first compiles your code to bytecode, a compact instruction set run by a virtual machine. Java does the same. And JIT (just-in-time) compilers, found in JavaScript engines and the JVM, watch a program as it runs, notice which paths execute most often, and compile those hot paths to native machine code on the fly — interpreting at first, then compiling for speed. A single language can be interpreted, bytecode-compiled, and JIT-compiled all at once. The stages we walked through — lexing, parsing, code generation — happen in nearly every one of them. What differs is only when, and into what, the translation lands. The way floating-point results can vary across those layers is its own rabbit hole, taken up in floating-point.