Experiment ·
The runtime configuration cost more power than the model
On Jetson AGX Thor, the same intrusion detector drew 54.1 W or 24.3 W depending on one threading setting. The model never changed.
We ran a defensive intrusion detector on a Jetson AGX Thor and measured it the way an operator would care about: board power, junction temperature, and detection latency, on the device, under live traffic.
With the default ONNX Runtime threading, the board drew 54.1 W. With the detector pinned to a single thread, it drew 24.3 W. The work was the same. The junction ran about 15 °C cooler, and p95 detector latency came in at 0.0237 ms. We repeated it across three sessions and the gap held every time.
What it means
The runtime’s defaults are tuned for throughput on machines with power to spare. On an edge box with a fixed power and thermal budget, those same defaults spin up threads that fight each other for a workload that never needed them. Nothing about the model was wrong. The thing that cost 31 W was a setting nobody benchmarks.
What most teams get wrong
Model benchmarks stop at the model. Latency per inference on a desktop GPU gets quoted as if it carries to the target, and the runtime configuration is treated as plumbing. In the field it is the other way round: the model is fixed, and the runtime is where the power, the heat, and the tail latency actually come from.
What we do about it
Runtime configuration is a first-class variable in every evaluation the lab runs. We measure board power and junction temperature on the target hardware with the exact threading, batching, and provider settings that will ship, and we ship the configuration with the model. A model without its runtime configuration is not a deployable artifact.
One thing is still open: the worst single-event latency in these runs is not yet explained. It is listed as unresolved on the project page until it is.
The measurements, charts, and setup are on the Jetson Edge AI Security project page.