A policy that passes in simulation has not passed yet

We built the lab’s robot arm and its workcell in Isaac Sim from the manufacturer’s URDF, described in OpenUSD, so the digital twin matches the physical bench down to the board the arm plays on. Then we evaluated an ACT manipulation policy in that twin: 35 of 36 turns completed, with a 19.2 mm median placement error.

That is a good simulation result. It is not a deployment result.

What it means

Simulation is where every run can be repeated, so it is the right place to find out whether a policy is worth putting on hardware. It answers one question: is this policy plausible enough to spend bench time on? It does not answer the question that decides whether a system ships: does it still work on embedded compute, against a real arm with its own kinematics, under real latency and heat?

What most teams get wrong

The sim result and the demo clip get treated as the finish line. Desktop GPU inference numbers are assumed to carry over to the edge device. They do not. On Jetson AGX Thor the same policy ran at 101.6 ms per inference with TensorRT, which is 9.8 Hz, and the board peaked at 42.0 °C while doing it. A controller that has to live inside 9.8 Hz is a different engineering problem from one that runs on a workstation, and no amount of simulated success tells you that.

What we do about it

Three things, in order. Build the twin from the manufacturer’s own robot description so the bench and the simulation agree. Evaluate in simulation first, because it is cheap and repeatable. Then measure on the target hardware, including thermals, before claiming anything.

And label what is not measured. The ROS 2 host on the Orin side of this program is implemented and not yet measured, and the project page says exactly that. A status label that is honest about what has not been proven is worth more to a partner than a demo that hides it.

The setup, the twin, and the measurements are on the Physical AI on Jetson project page.