Benchmarks

Every figure here comes from deterministic evaluator counters: thunks created and bytes allocated. These counts depend only on the expression, so they reproduce exactly on any machine with the same Nix version. CPU time is reported in the hub’s BENCHMARKS.md but gates nothing, because it moved between 0.73× and 1.87× across repeated runs of identical inputs on one machine.

Each benchmark first checks that both sides produce byte-identical output. A change that is faster but gives a different result fails before its cost is compared.

gen’s module system against nixpkgs

These rows evaluate the same configuration shapes through gen’s module system and through the nixpkgs module system, measured with Nix 2.34.8 at the largest size of each workload. A ratio below 1.00 means gen does less work.

workload	what it models	n	thunks, gen ÷ nixpkgs	allocation, gen ÷ nixpkgs
deepSubmodule	nested submodules	1600	0.52	0.41
registry	attrsOf (submodule …) registries	2000	0.70	0.56
lazyRegistry	the same, read lazily	2000	0.70	0.56
scalar	wide, flat typed option sets	8000	0.88	0.72
schemaHosts	a host registry with kinds	1600	0.89	0.79
wideFreeform	unknown keys absorbed by a freeform type	8000	1.10	0.81

Every workload measured at two sizes grows linearly: counters grow between 3.95× and 4.00× when the input grows 4×. CI fails if any workload grows faster than 5.5× for a 4× step, which catches quadratic behaviour.

Reuse

workload	what it measures	thunks, reused ÷ full	allocation, reused ÷ full
overrideWarm	six edits re-evaluated against one warm base, against re-evaluating each from scratch	0.24	0.24
classShare	rebuilding nodes from a shared class core, against merging each in full	0.17	0.17

Both produce byte-identical output to the full path. CI fails if either ratio rises above 0.30.

On a real three-host NixOS fleet, an edit to one host skips 66.7% of the fleet’s composition work and gives a result identical to a full rebuild. The measurement lives in hola at d643a8d, a separate public repository, where a gate fails if the saving drops below 60%.

Against flake-parts and adios-flake

One small flake, with three packages, a dev shell and a formatter for three systems, written three ways: with flake-parts, with adios-flake, and with gen’s gen.lib. All three use the same pinned nixpkgs, 5 runs per cell.

output	flake-parts	adios-flake	gen
packages.x86_64-linux, function calls	90,979	84,009	81,708
packages.x86_64-linux, primitive calls	61,934	61,910	57,207

gen makes the fewest function calls, primitive calls and thunks of the three on every output measured. All 15 outputs (five outputs × three systems) have identical derivation paths under all three frameworks, so each builds exactly the same result.

Most of this flake’s cost is importing nixpkgs, which all three pay equally, so the differences are small. CPU time at this size is within noise, and gen is the slowest of the three on the dev shell output.

Running the benchmarks

From the root of the gen repository:

nix run ./ci#perf-bench      # module system against nixpkgs, and the reuse workloads
nix run ./ci#flake-compare   # flake-parts, adios-flake and gen