Tarik Al Hadethi Download CV

The benchmark I nearly published

My first result said one inference library was 21 times slower than another. It was reproducible to within a few percent, and it was measuring the wrong thing. The check that caught it took thirty seconds and I nearly skipped it.

There are three realistic ways to run an ONNX model from Rust: tract, which is pure Rust and what my screenshot app uses; candle, also pure Rust, from HuggingFace; and ort, a binding to Microsoft’s C++ onnxruntime. Nobody has published a decent comparison of them on a real model, so I ran one.

The workload is a genuine PaddleOCR recognition model, 21 MB and 481 nodes, at a realistic input size, on an i7-12700K with 20 threads. Release build, median of 30 runs.

Correctness before speed

The first thing measured was not time. A faster runtime that computes something else is not a faster runtime, so all of them got byte-identical input and their outputs were compared:

tract   [1, 40, 18710]   checksum 39.8805
ort     [1, 40, 18710]   checksum 39.8815
candle  did not run

Agreement to 0.0025%, which is the right size for a difference caused by different kernel orders and vectorisation. Only now do timings mean anything.

That third line is its own finding. candle refused the model outright: unsupported auto_pad SAME_UPPER. Three nodes out of 481 use that padding mode, 0.6% of the graph, and the whole model is unusable. It handles a model with explicit padding perfectly, so this is operator coverage rather than a design limit, but it is worth knowing before you pick it for anything exported by a tool you do not control.

The number that was too good

Then the timings, and they were lopsided:

tract   125.4 ms
ort       6.0 ms

Twenty-one times. Reproducible, tight variance, same output. Everything about it looked solid, and it is exactly the sort of headline that gets shared.

It was also wrong, and the reason is that a benchmark is a claim about your own setup before it is a claim about anything else. So before believing it I measured how much CPU time each process burned against how much wall-clock time it took:

tract   wall 5.27s   cpu 5.05s   ratio 1.0x
ort     wall 0.38s   cpu 1.61s   ratio 4.2x

tract was using exactly one core. I had put a twenty-thread machine in front of it and it had used a twentieth of it.

The flag nobody mentions

tract can use more than one core. The executor is a feature called multithread-mm, it is off by default, and nothing tells you it exists: not the README, not the API when you build a plan, not any warning at runtime.

tract executormedian
single (the default)125.4 ms
4 threads54.3 ms
8 threads40.3 ms
20 threads36.0 ms

A 3.5x speed-up for a two-line change. Anyone benchmarking tract out of the box, which is to say anyone benchmarking tract, is measuring one core.

The honest comparison is therefore 36 ms against 6 ms. Microsoft is still six times faster, and it gets there on about four threads while tract needs twenty, so per unit of CPU work the gap is roughly five times. That is the price of being pure Rust with no C++ toolchain in the build. tract buys something back: 47 MB of memory against 66 MB, which is the real trade on offer.

The part that surprised me

My app already runs several text crops at once on its own threads. So does turning on tract’s threading help, or does it fight what is already there? Not guessable, so measured, with 40 crops at realistic widths:

tract pool sizetotal for 40 crops
off626 ms
22888 ms
41476 ms
20438 ms

A pool of two is 4.6 times worse than no threading at all.

The reason is that the setting is not threads-per-inference, as the name suggests to a reader. It is the size of one shared pool that every concurrent inference dispatches into. Twenty threads queueing their matrix multiplies onto two workers serialises precisely the work that was already running in parallel.

A second theory that died

I had a good explanation for that, and it was wrong too.

tract reads its executor from behind a global Mutex, on every call. With twenty threads hammering it, that is an obvious contention point, and tract even offers a thread-local way to bypass it. Tidy story.

Measured: 2884 ms against 2888 ms. Identical. The lock is not the problem, the shared pool is.

A benchmark is a claim about your own setup before it is a claim about anybody’s software.

Had I stopped at the first number I would have published something true and thoroughly misleading: a library measured on one core, presented as though it had been given the whole machine. It would have been reproducible. Anyone rerunning it would have got the same figure and the same wrong impression.

What separates the two versions is not skill. It is one thirty-second check on whether the thing I was measuring was in the state I assumed.

Written while building Clarus.  ·  More writing