There are three realistic ways to run an ONNX model from Rust:
tract, which is pure Rust and what my
screenshot app uses; candle, also
pure Rust, from HuggingFace; and ort, a binding to Microsoft’s C++
onnxruntime. Nobody has published a decent comparison of them on a real model, so I
ran one.
The workload is a genuine PaddleOCR recognition model, 21 MB and 481 nodes, at a realistic input size, on an i7-12700K with 20 threads. Release build, median of 30 runs.
Correctness before speed
The first thing measured was not time. A faster runtime that computes something else is not a faster runtime, so all of them got byte-identical input and their outputs were compared:
tract [1, 40, 18710] checksum 39.8805
ort [1, 40, 18710] checksum 39.8815
candle did not run
Agreement to 0.0025%, which is the right size for a difference caused by different kernel orders and vectorisation. Only now do timings mean anything.
That third line is its own finding. candle refused the model outright:
unsupported auto_pad SAME_UPPER. Three nodes out of 481
use that padding mode, 0.6% of the graph, and the whole model is unusable. It handles
a model with explicit padding perfectly, so this is operator coverage rather than a
design limit, but it is worth knowing before you pick it for anything exported by a
tool you do not control.
The number that was too good
Then the timings, and they were lopsided:
tract 125.4 ms
ort 6.0 ms
Twenty-one times. Reproducible, tight variance, same output. Everything about it looked solid, and it is exactly the sort of headline that gets shared.
It was also wrong, and the reason is that a benchmark is a claim about your own setup before it is a claim about anything else. So before believing it I measured how much CPU time each process burned against how much wall-clock time it took:
tract wall 5.27s cpu 5.05s ratio 1.0x
ort wall 0.38s cpu 1.61s ratio 4.2x
tract was using exactly one core. I had put a twenty-thread machine in front of it and it had used a twentieth of it.
The flag nobody mentions
tract can use more than one core. The executor is a feature called
multithread-mm, it is off by default, and nothing tells you it exists: not
the README, not the API when you build a plan, not any warning at runtime.
| tract executor | median |
|---|---|
| single (the default) | 125.4 ms |
| 4 threads | 54.3 ms |
| 8 threads | 40.3 ms |
| 20 threads | 36.0 ms |
A 3.5x speed-up for a two-line change. Anyone benchmarking tract out of the box, which is to say anyone benchmarking tract, is measuring one core.
The honest comparison is therefore 36 ms against 6 ms. Microsoft is still six times faster, and it gets there on about four threads while tract needs twenty, so per unit of CPU work the gap is roughly five times. That is the price of being pure Rust with no C++ toolchain in the build. tract buys something back: 47 MB of memory against 66 MB, which is the real trade on offer.
The part that surprised me
My app already runs several text crops at once on its own threads. So does turning on tract’s threading help, or does it fight what is already there? Not guessable, so measured, with 40 crops at realistic widths:
| tract pool size | total for 40 crops |
|---|---|
| off | 626 ms |
| 2 | 2888 ms |
| 4 | 1476 ms |
| 20 | 438 ms |
A pool of two is 4.6 times worse than no threading at all.
The reason is that the setting is not threads-per-inference, as the name suggests to a reader. It is the size of one shared pool that every concurrent inference dispatches into. Twenty threads queueing their matrix multiplies onto two workers serialises precisely the work that was already running in parallel.
A second theory that died
I had a good explanation for that, and it was wrong too.
tract reads its executor from behind a global Mutex, on every call.
With twenty threads hammering it, that is an obvious contention point, and tract even
offers a thread-local way to bypass it. Tidy story.
Measured: 2884 ms against 2888 ms. Identical. The lock is not the problem, the shared pool is.
A benchmark is a claim about your own setup before it is a claim about anybody’s software.
Had I stopped at the first number I would have published something true and thoroughly misleading: a library measured on one core, presented as though it had been given the whole machine. It would have been reproducible. Anyone rerunning it would have got the same figure and the same wrong impression.
What separates the two versions is not skill. It is one thirty-second check on whether the thing I was measuring was in the state I assumed.