Benchmarks¶

This page shows what specsolve costs to build a model and hand it to a solver, beside four other ways of writing the same model, and how far to trust each number.

Each model is built and handed to a solver. run() and optimize() are never called, so no number includes solve time. Every rival model is hand-written in bench/models/ beside ours, and CI solves each against ours, so nothing here is fast because it built a different model.

dispatch

Meets a demand from a fleet of units: bounds and one balance. It is the floor.

transport

A network of buses and lines: mapping tables, three joins per row.

fleet

Twelve declarations rather than one large one: cost per declaration, not per row.

Reading the charts¶

Each panel is one model through one solver, and each line is one library's cost against the size of the model.

• specsolve is the thick violet line. The other libraries are thinner, and the legend hides any of them.
• The grey line is the matrix floor. It is a hand-written matrix handed straight to the solver's bulk API: gurobipy-matrix on Gurobi, highspy-matrix on HiGHS. Nobody writes a model this way. It is the limit a modelling library can approach, so its distance from specsolve is what specsolve's modelling layer costs.
• The pale violet line is specsolve on polars' in-memory engine. The specsolve line leaves the engine to specsolve, which builds a small model in memory and a large one on polars' default engine. The how-to for small models says where the two part.
• The line is the median of the rounds, and the band is the middle half of them. The band runs from the first quartile to the third. Two lines whose bands overlap are two numbers this run cannot tell apart.
• Grown by length adds snapshots; grown by width adds entities. Both reach the same sizes, so switching holds the size and changes only the shape that got there. transport is the model where this matters: its bus-by-generator join grows with width and never with length. No case on this run reached a width rung, so the width view is empty for the reason below.
• Log–log gives every size equal room, and the slope is the scaling order. A line that rises one decade per decade grows in step with the model. Linear–linear draws the gap at the largest size at its real size and puts the small sizes at the origin. Both axes switch together, because a linear axis over a logarithmic one draws linear growth as a curve.
• A dashed line is a projection, not a measurement. It continues a library past the size its budget refused, at the growth rate of its last measured step and never slower than linear. The axes are scaled to the measurements, so a steep projection leaves the top of the panel, and the tooltip gives its value.
• The table divides by specsolve. The specsolve column is its own cost. Every other cell is that library's cost divided by specsolve's at the same size: green where specsolve is faster, blue where the other library is, grey within ten per cent. The matrix floor's column is not shaded, because a floor is meant to be faster. Hover a cell for the library's own number.
• >30 s and >16 GB are sizes the harness refused. It projected the measurement past its time or memory budget and skipped it. Hover one for the projection. An em dash is a cell with no number for another reason: gurobipy has no HiGHS.

Why the median¶

Every published number is the median of nine rounds. Every measurement gets the same nine, pinned rather than calibrated by duration, so no cell is a best-of-nine beside a neighbour's best-of-forty. The median beats the fastest round because a cell whose nine rounds all ran slow has no clean round to pick. It beats the mean because one slow round moves a mean and leaves a median where it was. On this run (#1934) 15 published cells have a mean above 1.10x their median, the worst at 3.00x: dispatch/xs on pyomo, on the highs sink. In 11 of them the first round is the slowest of the nine, at 1.6 to 18.9 times the median of the other eight, so the median does not move. The other four are matrix-floor and gurobipy-loop cells at xs and s whose rounds fall into two groups, the slow one about twice the fast one: dispatch/xs on gurobipy-loop takes either 23 to 26 ms or 48 to 49 ms.

The median flipped two cells on this run, both against specsolve: dispatch/xs and dispatch/s on the highs sink, against linopy. On each, specsolve's fastest round is the faster one, 28.0 ms against 28.4 ms and 40.7 ms against 44.7 ms, and its median is the slower, 39.9 ms against 29.1 ms and 56.3 ms against 46.4 ms.

How to reproduce it¶

uv run --locked bench/reproduce.py

bench/reproduce.py.lock freezes every version, git commits included: two of the five libraries install from git and one of those is a branch. --locked refuses to start if the resolution has drifted.

Everything the tables are drawn from is in bench/results: one file per sink, case and ladder. Each carries the machine, the versions, the commit and every round of every measurement. A ladder the box could not finish leaves no file behind, and the other ladder of that case keeps its own. pixi run table prints the directory as one long CSV and commits nothing; the JSON stays the archive because it keeps the rounds. pixi run refresh re-takes the numbers, writes the tables into their fences and the chart rows into benchmarks.json.

A script that solves one model¶

The tables above are what a process pays once it is running. A script that imports a library and solves one model also pays the interpreter, the import, and whatever the library loads the first time it builds. linopy's arrays are xarray's, and xarray imports dask.array on the first array it makes, so linopy's first model costs more than its import alone. This table times a new process from launch to exit at xs, the rung where starting is a share of the cost. A run that did not take it leaves the table out.

First model against every model after it¶

Marginal cost per model¶

Build only, repeated in one process. first is the first recorded round and steady the best of the rounds after it, so the pair is what a rolling horizon pays for its second window against its first. The harness warms up before it records, so neither column carries the one-time import cost: the median gap between them is +11.8 ms on specsolve and +1.7 ms on linopy and +5.1 ms on pyomo and +1.2 ms on gurobipy-loop and +9.4 ms on gurobipy-matrix and +2.4 ms on highspy-matrix.

Read down a column, not across the row. The build is not the same work in every library — one that defers materialising its coefficients to its writer spends almost nothing here and pays it at the seam — so these columns carry no ratios. The tables above measure to a common artifact and are where a comparison belongs.

The same size, reached by widening¶

This table renders from highs rungs of storage and transport, and no run has published it: both cases are killed on both sinks before they write a file, for the reason below. bench.report drops a fragment it cannot render rather than blanking the fence, so the fence stays empty until pixi run ladder brings those rungs back.

Not measured yet¶

Listed so that a claim with no table under it is visible as one.

• Sparsity, which is the engine's whole premise. Every published cell is 100% live: the where in dispatch removes nothing, and transport and fleet carry no mask at all. A dense coordinate product is the shape an array engine is built for. The published ladder therefore compares the two lanes only where the relational one has the least to win. nodal and sector are the sparse cases, at 25% and 8.3% of their product. nodal has a linopy formulation now, so pixi run density sweeps it at four densities; no published run has taken it.
• Solve time. Every number stops at the hand-off. The simplex is the solver's work whoever filled the model.
• The LP-file round trip. The tables price writing a file, never reading one back.
• Sizes past l. xl and 2xl exist in the harness and no run publishes them.
• storage and transport are absent from every table. The memory budget projects the next rung off a w10 or w100 cell that took under a gigabyte. Where the projection comes in under 16 GB the run starts a rung that does not fit, and bench/memory-watchdog.sh kills the case to save the box. Four cases died that way on this run: transport/w100 on highs at 25.5 GB and on gurobi at 26.4 GB, and storage/w1000 on highs at 26.1 GB and on gurobi at 27.8 GB. That run measured both ladders of a case in one process, and a killed process writes no file, so each case lost every rung it had already measured, its rows in the marginal table above included. Each ladder now runs in a process and a file of its own, so a kill on the width rungs keeps the size rungs. The last highs numbers past w10 are in #1285, taken on a machine that could hold them. There specsolve took 0.11 s and 0.59 GB at transport/w100, against linopy's 53.53 s and 14.26 GB.
• Anything about expressiveness. Four models say nothing about a fifth.

Method¶

Each measurement runs in a process of its own. Peak memory is ru_maxrss rather than a tracker. Import is excluded from the timing and teardown is included. A run refuses to start on a machine that is already working. The rest is in bench/README.md: every flag, every default switched off and what it costs.

Peak carries an allocator cost that only the polars arms pay. polars ships its own jemalloc settings, so a peak measured through it holds pages freed and not yet returned. An arm on the system allocator never enters jemalloc. Our peak moves 12–27% with the decay clock on and off, where linopy's does not move at three digits (#896). It runs against us and is left in.

memray never times anything. Its tracker slows an allocation-heavy engine several-fold and overcounts reserved arenas. Peak RSS is the metric; memray is for attribution.