Simulation Testing: How Miles in Software Compare to Miles on Roads

Ask an AV company how many miles its system has driven, and you'll often get two very different numbers back.

One is real-world mileage: miles logged by actual vehicles on actual roads, with all the cost and slowness that implies. The other is simulated mileage, and it's typically larger by orders of magnitude, because running a scenario in software is vastly cheaper than sending a car out to encounter it. A company can test thousands of variations of a single tricky intersection overnight in simulation. Doing the same thing on public roads would take years, assuming the rarer and more dangerous versions of that intersection ever showed up at all. It also means the same scenario can be rerun with small variations, a different weather condition, a different pedestrian walking speed, in a way no real street corner would ever repeat on demand.

That's the actual value simulation adds. It lets a company probe edge cases and dangerous scenarios that would be impractical, or simply unsafe, to seek out with a real vehicle and real pedestrians nearby. Nobody drives toward a jaywalking child on purpose to see what the system does. Simulation exists precisely so nobody has to.

The two totals feed into each other in a way that's easy to miss when they're presented side by side as if they were competing measures of progress. A real-world disengagement or an unusual encounter on the road doesn't just get logged and forgotten; it typically gets replayed in simulation with variations, different approach angles, different speeds, different lighting, to test whether a proposed fix actually resolves the underlying problem rather than just the one instance a vehicle happened to encounter. Simulation, in that sense, isn't a substitute for real-world driving so much as an amplifier for it, turning a single real event into many related test cases a fix has to pass before it ships back into the fleet.

The two totals also aren't built the same way from company to company, which undercuts any attempt to compare them directly. Some simulated mileage comes from fully synthetic scenarios generated from scratch; some comes from replaying and perturbing real sensor logs a vehicle already recorded, changing one variable at a time to see how the system reacts to a slightly different version of something that actually happened. Both count as simulated miles in a company's public totals, but they're testing different things and produced by different methods, which means one company's simulated-mile total and another's aren't necessarily measuring anything comparable, even setting aside the separate question of how faithfully either method reproduces real-world conditions.

There's a related asymmetry in what each method is well suited to sample from. Real-world miles are drawn essentially at random from whatever a vehicle happens to encounter on its assigned routes, which is a reasonable way to estimate how often ordinary situations go wrong but a slow way to accumulate examples of rare ones. Simulation lets a company target the rare cases deliberately, constructing a scenario a real vehicle might encounter only once in enormous mileage and running it repeatedly with small variations. That's a genuine advantage for finding failure modes, but it also means a company's simulated total reflects choices about what to go looking for, not a representative sample of ordinary driving the way a real-world mile count more nearly is.

The catch is that a simulated mile and a real mile aren't measuring the same thing, and the simulated number is only as good as how faithfully the simulation reproduces real sensor behaviour, real weather, real driver unpredictability. A simulation that's too clean, too well-lit, too idealized, tells you less than its impressive-sounding total suggests. That's hard to check. A company's own published simulation figures aren't independently audited the way its disengagement reports at least attempt to be. The mileage numbers make for a good headline, and companies know it, which is exactly why they keep leading with them. They don't, by themselves, answer whether the system is actually safe.