Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Thorn

Thorn is a fast, deterministic meta-build generator written in C99 that embeds the Pith scripting language as its specification frontend.

Thorn reads a build.thorn file describing your project’s targets, sources, flags, and dependencies, and emits:

  • Ninja files (build.ninja), optimized for tools like ninja or samurai (samu) with automatic dependency generation (deps = gcc).
  • Makefiles (Makefile), strictly POSIX-compatible pattern rules with -MMD -MP automatic header dependency tracking.

In addition to forward generation, Thorn features a reverse decompiler capable of reading existing build.ninja files or Makefiles and reconstructing idiomatic, clean build.thorn specifications that regenerate byte-identical outputs.

Key Features

  • Embedded Pith Runtime: Specification files are written in Pith, evaluated directly via Pith’s C embed API with zero AOT compile step or intermediate plugins.
  • Dual Backends: Emit fast, minimal Ninja files, portable Makefiles, or both simultaneously.
  • Custom Build Edges: Declare custom build commands with $in / $out token substitutions.
  • Roundtrip Decompilation: Ingest legacy or hand-written build files back into build.thorn.
  • Strict & Deterministic: Whitespace rejection for flags and identifiers, ordered dependency edges, and byte-for-byte reproducible build artifacts.

Getting Started

Installation

Pre-built Nightly Binaries

Pre-compiled nightly binaries for Linux (x86_64) are automatically published via GitHub Actions. You can download the latest tarball directly from the Releases page.

Each release includes:

  • thorn: The standalone executable binary.
  • libthorn_core.a: The pre-built core static library.
  • include/: Public C header interfaces.

To install manually:

tar -xzf thorn-x86_64-linux-nightly.tar.gz
sudo install -m 755 thorn /usr/local/bin/thorn

Building From Source

Thorn requires a C99 compiler (cc), GNU make, and vendored pith submodules.

# Clone the repository with submodules
git clone --recurse-submodules https://github.com/abit-foggy/thorn.git
cd thorn

# Build Thorn (outputs to out/thorn)
make

# Run the acceptance and verification suite
make check

Creating Your First Project

A Thorn project is configured with a build.thorn file at the root of the project directory.

Create build.thorn:

engine.project("myapp")
engine.backend("ninja")

engine.add_target("myapp", engine.exe)
engine.add_source("myapp", "src/main.c")
engine.add_source("myapp", "src/util.c")
engine.add_source("myapp", "src/arch/boot.S")
engine.add_include("myapp", "src/include")
engine.add_cflag("myapp", "-O2")
engine.add_cflag("myapp", "-Wall")

Generate build files:

thorn

This generates build.ninja in the current directory. You can then build your application using Ninja or Samurai:

samu
# or
ninja

To generate a Makefile instead, specify --engine make on the command line or configure engine.backend("make") in build.thorn:

thorn --engine make
make

Command-Line Interface

Thorn provides a single unified binary that acts as both a meta-build generator and a reverse decompiler.

Usage Syntax

USAGE:
    thorn [spec]
    thorn build [-f spec] [--out-dir dir] [--engine ninja|make|both]
                [--compiler cc] [--ar ar]
    thorn decompile <build.ninja|Makefile> [-o out.thorn]
    thorn help | version

Commands & Subcommands

Default / build Command

Evaluates the specification file (build.thorn or a custom spec) and emits the selected backend files.

# Evaluate build.thorn in current directory
thorn

# Evaluate custom spec file positionally
thorn myproject.thorn

# Explicit build command with output directory and engine override
thorn build -f custom.thorn --out-dir out/artifacts --engine ninja

Options:

  • [spec], -f, --file <spec>
    Path to the specification file. Defaults to build.thorn if omitted.
  • -o, --out-dir <dir>
    Destination directory for generated build files (build.ninja, Makefile) and build artifacts. Defaults to current working directory (.).
  • -e, --engine <ninja|make|both>
    Override the target build engine(s) to emit:
    • ninja: Emit only build.ninja.
    • make: Emit only Makefile.
    • both: Emit both backends simultaneously.
      Note: Can also be defined inside the specification via engine.backend("...").
  • --compiler <cc>
    Specify the C compiler executable (e.g. gcc, clang, cc, tcc). Precedence: --compiler flag > CC environment variable > system default cc. Thorn validates that the compiler exists in PATH before writing rules.
  • --ar <ar>
    Specify the archiver executable (e.g. ar, llvm-ar) used for static library archives. Precedence: --ar flag > AR environment variable > system default ar.

decompile Command

Parses an existing Ninja file or Makefile and synthesizes an idiomatic, equivalent build.thorn specification.

# Decompile build.ninja and print to standard output
thorn decompile build.ninja

# Decompile Makefile directly to a specification file
thorn decompile Makefile -o build.thorn

Options:

  • -o, --out <path>
    Output destination file for the recovered specification. If omitted, the recovered spec is written to standard output.

Informational Commands

  • thorn version / -v, --version: Print version information (thorn 0.3.0).
  • thorn help / -h, --help: Display usage summary and help text.

Specification Reference

Thorn specifications are written in Pith script (build.thorn or .thorn). The Thorn host engine registers the engine namespace with typed functions and constants.

Target Types

Target types configured via engine.add_target:

  • engine.exe — Executable binary.
  • engine.static_lib — Static library archive (.a).
  • engine.shared_lib — Dynamic/shared library (.so).

Project & Backend Configuration

engine.project(name: str)

Defines the name of the project.

engine.project("myproject")

engine.backend(backend: str)

Selects the build file backend(s) to emit. Accepted values:

  • "ninja": Emit build.ninja.
  • "make": Emit Makefile.
  • "both": Emit both build.ninja and Makefile.
engine.backend("ninja")

Note: A backend must be selected either via engine.backend() in the spec or via --engine on the command line.

Target Definition

engine.add_target(name: str, target_type: int)

Declares a new target in the build graph. Target type is one of engine.exe, engine.static_lib, or engine.shared_lib.

engine.add_target("myapp", engine.exe)
engine.add_target("mylib", engine.static_lib)

engine.add_source(target: str, source_path: str)

Associates a source file with a target. Thorn automatically classifies the source kind based on file extension:

  • .c: C source compiled with the C compiler rule (cc).
  • .s: Raw assembly assembled via the assembly rule (as).
  • .S: Assembly with C preprocessor directives, compiled via the preprocessed assembly rule (as_cpp).
engine.add_source("myapp", "src/main.c")
engine.add_source("myapp", "src/arch/bootstub.S")
engine.add_source("myapp", "src/arch/trampoline.s")

engine.add_include(target: str, include_path: str)

Appends an include directory (-I<path>) for compiling sources belonging to this target.

engine.add_include("myapp", "src/include")

engine.add_cflag(target: str, flag: str)

Appends a C compiler flag used when compiling C sources (.c) of this target.

engine.add_cflag("myapp", "-Wall")
engine.add_cflag("myapp", "-O2")

engine.add_asflag(target: str, flag: str)

Appends an assembler/preprocessor flag used when compiling assembly sources (.s, .S) of this target.

engine.add_asflag("myapp", "-Wa,--noexecstack")
engine.add_asflag("myapp", "-D__ASSEMBLY__")

engine.add_ldflag(target: str, flag: str)

Appends a linker flag or linked library when creating the target artifact.

engine.add_ldflag("myapp", "-lpthread")
engine.add_ldflag("myapp", "-lm")

engine.add_order_dep(target: str, dependency_path: str)

Adds an order-only prerequisite on the target (e.g. generated headers, assets, or pre-requisite build steps).

engine.add_order_dep("myapp", "generated/config.h")

Custom Commands

engine.add_command(output: str, command: str, input: str)

Defines an explicit build edge where running command generates output from input.

Variables supported inside the command string:

  • $in / $<: The input prerequisite.
  • $out / $@: The output file.
  • $$: An escaped dollar sign literal ($).
engine.add_command("generated/version.h", "sh scripts/gen_version.sh $in $out", "VERSION")

When emitting Ninja, the rule is emitted as cmd = <command> with $in and $out. When emitting Makefile, $out and $in are translated to $@ and $< respectively.

Package Configuration

engine.pkg_config(target: str, package_name: str)

Queries system pkg-config at build-generation time and automatically appends --cflags to compiler flags and --libs to linker flags for the target.

engine.pkg_config("myapp", "libssl")

Emission

engine.emit()

Dispatches graph emission to the chosen backend(s). When running thorn, this call is automatically appended as an epilogue to build.thorn if not explicitly called.

engine.emit()

Backends

Thorn supports two target build engines: Ninja and Make. Both backends are designed for minimal overhead, strict dependency tracking, and reproducible output.

Ninja Backend (build.ninja)

The Ninja backend produces clean, minimal Ninja syntax suitable for execution with either ninja or samurai (samu).

Key Properties

  • Compiler Rules:
    • rule cc: Compiles C files (.c) with automatic header dependency extraction (deps = gcc, depfile = $out.d). 
      rule cc
  depfile = $out.d
  deps = gcc
  command = $cc -MD -MF $out.d $cflags $includes -c $in -o $out
    • rule as: Assembles raw assembly files (.s). 
      rule as
  command = $as $asflags -c $in -o $out
    • rule as_cpp: Compiles assembly with C preprocessor directives (.S). 
      rule as_cpp
  depfile = $out.d
  deps = gcc
  command = $cc -MD -MF $out.d $asflags $cflags $includes -c $in -o $out
  • Archiving & Linking:
    • rule ar: Packages static archives ($ar rcs $out $in).
    • Executable and dynamic linking rules invoke $cc with target $ldflags.
  • Automatic Target Extensions:
    • Executable: <target>
    • Static library: <target>.a
    • Shared library: <target>.so
  • Deterministic Emission: Rules and build edges are topologically sorted in graph order to guarantee byte-for-byte reproducibility across runs.

Make Backend (Makefile)

The Make backend generates portable, standard POSIX-compatible Makefiles.

Key Properties

  • Header Dependencies: Uses -MMD -MP flags during compilation and includes generated .d files: 
    -include $(OBJS:.o=.d)
  • Pattern Rules:
    • %.o: %.c: Compiles C files with $(CC) $(CFLAGS) $(INCLUDES) -c $< -o $@.
    • %.o: %.s: Assembles raw assembly with $(AS) $(ASFLAGS) -c $< -o $@.
    • %.o: %.S: Preprocesses and compiles assembly with $(CC) $(ASFLAGS) $(CFLAGS) $(INCLUDES) -c $< -o $@.
  • Rule Ordering: Emits an all: rule as the very first default target before any explicit targets or custom commands, ensuring that running make with no arguments builds all primary targets.
  • Token Translation: In custom commands (engine.add_command), $in is mapped to Make’s automatic variable $< and $out is mapped to $@.
  • Phony Targets: Declares .PHONY: all clean and provides an automatic clean target removing all build products.

Engine Selection

You can select the engine in build.thorn:

engine.backend("ninja") # or "make", or "both"

Or override it from the CLI:

thorn --engine both

Decompilation

Thorn includes a built-in reverse decompiler capable of reading an existing build.ninja or Makefile and synthesizing an equivalent, idiomatic build.thorn specification script.

CLI Usage

# Decompile build.ninja and print specification to stdout
thorn decompile build.ninja

# Decompile Makefile directly to a specification file
thorn decompile Makefile -o build.thorn

Reverse Engineering Capabilities

The decompiler performs heuristic AST analysis on backend rules and targets:

  • Target Recovery: Detects targets, outputs, and classifies whether they are executables (engine.exe), static archives (engine.static_lib), or shared objects (engine.shared_lib).
  • Source Mapping: Maps compiled object files back to their .c, .s, and .S source files.
  • Flags & Includes: Extracts -I include flags into engine.add_include(), compiler options into engine.add_cflag(), and assembly flags into engine.add_asflag().
  • Linker Flags: Extracts library links (-l, -L) into engine.add_ldflag().
  • Custom Commands: Recovers custom build edges into engine.add_command(), transforming Make automatic variables ($@, $<) back into Thorn’s portable tokens ($out, $in).
  • Backend Recognition: Automatically emits engine.backend("ninja") or engine.backend("make") based on the ingested file format.

Hookable Decompiler API (Pith & C FFI)

In addition to CLI usage, Thorn exposes a standalone, hookable decompiler API (decompile.*) designed for advanced build-system transformations and automated migrations (such as thornk for Linux Kbuild translation):

Configuration & Filtering

  • decompile.reset(): Clear the current decompiler graph state.
  • decompile.set_project(name: str): Set the project name for the synthesized spec.
  • decompile.set_compiler(cc: str): Override the target compiler.
  • decompile.set_ar(ar: str): Override the target archiver.
  • decompile.ignore_target(pattern: str): Filter out unwanted targets (e.g. tests, firmware blobs).
  • decompile.keep_target(pattern: str): Whitelist specific targets.

Flag Manipulation

  • decompile.strip_cflag(pattern: str): Remove specific compiler flags (e.g. unsupported GCC internal flags).
  • decompile.inject_cflag(target_pattern: str, flag: str): Ingest custom compiler flags (e.g. -nostdinc, -D__KERNEL__).
  • decompile.inject_include(target_pattern: str, include_dir: str): Inject include paths into matching targets.
  • decompile.remap_target(old_name: str, new_name: str): Rename targets in the synthesized graph.

Extraction & Synthesis

  • decompile.parse_file(path: str): Ingest a Ninja or Makefile from disk.
  • decompile.target_count() -> int: Returns the number of recovered targets.
  • decompile.to_thorn() -> str: Returns the synthesized build.thorn specification as a string.
  • decompile.emit_thorn(path: str): Writes the synthesized build.thorn directly to disk.

Byte-Identical Roundtripping

Thorn’s forward emitter and reverse decompiler are verified against each other:

  1. A project emits build.ninja via Thorn.
  2. thorn decompile build.ninja -o build_recovered.thorn generates a recovered spec.
  3. Running thorn build_recovered.thorn regenerates a byte-identical build.ninja.

This ensures deterministic, lossless migration from legacy Ninja or Make setups into Thorn.

Architecture

Thorn is structured around a single-binary embedding model: one tool binary serves every project without per-project compilation or AOT plugin extraction.

Component Overview

Thorn’s core is implemented in clean, modular C99 with separated compilation units and public headers in src/include/:

1. CLI Driver & Embedding (src/main.c, src/engine.c)

  • src/main.c: The command-line entry point. Handles option parsing (supporting subcommands build, decompile, version, help, and positional spec paths), toolchain binary verification in PATH, Pith runtime context initialization, host function registration under the engine namespace, and execution orchestration.
  • src/engine.c: The host runtime API bridge. Implements the C host functions (c_engine_*) exposed to the embedded Pith evaluation environment, providing typed wrappers between Pith’s ARC/value system and Thorn’s internal build graph.

2. Build Graph & Invariants (src/graph.c, src/include/graph.h)

  • src/graph.c: In-memory Directed Acyclic Graph (DAG) state manager. Manages projects, targets (Target), custom commands (Command), flags, includes, and order dependencies.
  • Enforces strict deterministic invariants:
    • Whitespace Rejection: Rejects spaces or newlines in target names, source paths, flags, and include directories.
    • Source Classification: Classifies sources into C (SRC_C), raw assembly (SRC_ASM_RAW), and preprocessed assembly (SRC_ASM_CPP).
    • Duplicate Elimination: Detects and eliminates duplicate sources, includes, and flags.
    • Dynamic Package Queries: Resolves system pkg-config flags at generation time without deferred shell subshells.

3. Backend Emitters (src/emit.c, src/ninja.c, src/make.c)

  • src/emit.c: Generation coordinator and filesystem driver. Validates the graph, ensures destination directories exist, and dispatches generation to the active backend(s).
  • src/ninja.c: High-performance Ninja backend generator (build.ninja). Emits canonical rule definitions (cc, as, as_cpp, ar, custom commands), dependency tracking with deps = gcc and depfile = $out.d, and topologically ordered build edges for byte-for-byte reproducibility.
  • src/make.c: Portable POSIX / GNU Makefile generator (Makefile). Emits pattern rules (%.o: %.c, %.o: %.s, %.o: %.S), header dependency tracking (-MMD -MP), automatic default all: rules, token translation ($in -> $<, $out -> $@), and automatic clean targets.

4. Reverse Decompiler Engine (src/decompile.c, src/parser.c, src/lex.c, src/spec.c)

  • src/decompile.c: Reverse engineering orchestrator. Drives parsing of existing Ninja or Make syntax and exposes a rich, hookable FFI API (decompile.*) usable by standalone scripts and external tools (such as thornk for Linux Kbuild conversion).
  • src/lex.c: Lexical scanner and tokenizer for Ninja and Make syntax.
  • src/parser.c: AST parser extracting variables, rule definitions, and build edges from backend files.
  • src/spec.c: Specification synthesizer; translates extracted build graphs into idiomatic, formatted build.thorn Pith specifications.

5. Modular Header Architecture (src/include/)

Internal subsystem boundaries are strictly enforced via modular C headers:

  • engine.h: Target model, source kinds, and host function prototypes.
  • graph.h: Graph allocation, source classification, and validation.
  • emit.h: Backend generation entry points.
  • ninja.h: Ninja backend emitter declarations.
  • make.h: Makefile backend emitter declarations.
  • parser.h: Ninja and Makefile AST structures and parse APIs.
  • lex.h: Token definitions and lexical scanner state.
  • spec.h: Spec synthesis and formatting declarations.

Runtime Embedding of Pith

Thorn embeds the Pith compiler frontend directly:

  1. Host Function Registration: During startup, main.c registers the engine functions using pith_register_ns_fn() under the engine namespace. Each function specifies its return and parameter classes according to Pith’s C ABI (e.g. v for void, w for 32-bit int, p for ARC string pointer).
  2. Epilogue Evaluation: When evaluating build.thorn, Thorn wraps or appends a call to engine.emit() after executing the project statements.
  3. Execution Fallback & Link Objects:
    • In environments permitting JIT (libtcc in-memory execution), Pith executes the byte instructions directly against the loaded host symbols in Thorn’s process memory.
    • On SELinux-hardened or W^X systems where runtime memory execution is restricted, Pith falls back to compiling a temporary child executable via QBE/TCC.
    • To support this fallback child, Thorn registers out/artifacts/engine.o (or libthorn_core.a) using pith_register_link_object(). The child links against this object and builds the identical graph, emitting the backend files seamlessly.

Self-Hosting

Thorn builds itself using its own generated backend. During tests/verify.sh:

  1. out/thorn is built from C source using the bootstrapping Makefile.
  2. out/thorn runs on its own build.thorn, emitting out/artifacts/build.ninja.
  3. samu builds out/artifacts/thorn from build.ninja.
  4. The resulting Stage 2 binary runs, builds projects, and decompiles backends deterministically.

Limits & Constraints

Thorn is intentionally designed to be small, predictable, and deterministic. It enforces explicit design boundaries:

Identifier & Path Rules

  • No Whitespace: Thorn strictly rejects whitespace (spaces, tabs, newlines) in target names, source paths, include directories, compiler flags, and custom command outputs. Any whitespace triggers a clear diagnostic error: 
    thorn: error: target name 'my app' contains whitespace
  • Portability: All source paths within build.thorn should be relative to the project root directory.

System & Graph Limits

The graph model allocates fixed limits to ensure fast, deterministic operation without unbounded heap fragmentation:

  • Maximum targets per project: 64
  • Maximum sources per target: 256
  • Maximum includes per target: 64
  • Maximum cflags per target: 64
  • Maximum ldflags per target: 64
  • Maximum custom commands: 128
  • Maximum order dependencies: 64

Exceeding these limits produces a descriptive diagnostic error at configuration time.

Design Philosophy

  • No Implicit Globbing: Sources must be explicitly enumerated in build.thorn to prevent non-deterministic build ordering and missing dependencies.
  • Strict Dependencies: All order dependencies must be explicitly declared using engine.add_order_dep().
  • Minimal Dependencies: The Thorn binary relies only on standard POSIX C99 and the embedded Pith frontend/runtime.