How the California DMV Disengagement Report Methodology Has Changed

California has required autonomous-vehicle testing permit holders to file an annual disengagement report since 2015, which makes it the longest continuously running public AV testing dataset in the United States. Ten-plus years of filings sounds like it should make year-over-year comparison straightforward. It doesn't.

Two problems complicate that comparison. The first is that the numbers are self-reported: permit holders file their own mileage and disengagement counts to the DMV, and the agency doesn't independently verify each figure before publication. The second is that the DMV's own definition of what counts as a reportable disengagement, and what has to be included in the annual filing, has been revised more than once since the program started.

Raw disengagement counts also don't account for how much mileage a given permit holder actually drove, which is a separate distortion from the self-reporting and definitional issues. A company running a small test fleet on a handful of familiar routes will naturally rack up fewer disengagements in absolute terms than a company driving far more miles across a wider variety of streets, even if the second company's software is performing better on a per-mile basis. Comparing this year's filing across companies by disengagement count alone, without normalizing for mileage, produces exactly the kind of misleading league table the data invites but doesn't actually support.

The reporting requirement also draws a line between planned and unplanned disengagements that isn't always visible from the outside. A disengagement triggered deliberately, to test how the safety driver responds, is a different event than one triggered because the system encountered a situation it couldn't handle on its own, and the annual filings haven't always made that distinction easy for an outside reader to separate out from the total count.

The dataset's scope is narrower than it can appear from a distance, too. It only captures mileage a permit holder chose to drive in California specifically, so a company concentrating its testing program in Arizona, Texas, or Nevada instead can show up in California's filings with only a small fraction of its actual nationwide testing activity, or none at all if it never held a California permit to begin with. Reading the California reports as a complete record of any single company's overall testing footprint, rather than as a record of what happened on California roads specifically, is a common misreading of a genuinely useful but geographically bounded dataset.

A disengagement that would have counted under one version of the reporting requirement might not count under a later one, or vice versa. That's not a scandal; it's what happens when a regulator tries to keep a reporting framework current for a technology that's changed substantially since 2015. But it does mean stacking company A's older filing next to company B's more recent one and treating the ratio as a clean apples-to-apples comparison is exactly the kind of league table the underlying data doesn't support.

None of this makes the reports useless. They're still the most granular public window into how permit holders describe their own testing programs, filed under legal obligation rather than volunteered for a press release. It just means reading them requires the same caveats this site's disengagement reports page lays out, rather than pulling a number out of a filing and running with it.

Whether a future revision of the reporting requirement will make the historical series more comparable, or whether the definitional drift is simply a permanent feature of a decade-plus dataset, is an open question the DMV hasn't addressed directly.