I generally spend the first half hour or so of my day (and a few minutes an hour throughout the day) digging into the latest news about various emerging technological fields that interest me. Today actually seems to be a bit of a #Robot heavy news day, but when I opened up my news feed it was full of news about the California DMV’s release of its“disengagement reports” for the dozen or so companies that are operating autonomous vehicles on California roads under permit.
This is an arcane topic, but basically the DMV requires these testers to record disengagements, or DE, which are incidents when the human test driver must take over from the autonomous systems. Many commentators, like Robert Ferris and Alex Davies underscored the progress made by Waymo in this metric since the last report, a fourfold improvement in average number of autonomous miles driven between DEs. That’s seems to show a marked improvement in the performance of Waymo’s vehicles, requiring far fewer human interventions. It’s not a clear scientific finding, but it certainly seems compelling.
But I also noticed a lot of people comparing Waymo’s numbers to other players in the field, notably Tesla. There’s a certain sense to this in that these two companies are both at the forefront of autonomous driving, albeit in different ways. But it got me to thinking about how fair a comparison this may be.
First off, let’s look at the numbers (taken from the Wired article):
|
Miles Driven
|
Disengagements
|
Miles per DE
|
2015 Mi/DE
|
|
|---|---|---|---|---|
| Waymo | 635868 | 124 | 5128 | 1244.4 |
| Tesla | 550 | 182 | 3 | – |
Obviously comparing the ‘headline’ miles per disengagement number shows an astronomical discrepancy. It’s quite impressive that a Waymo vehicle can travel more than 5,000 miles all by itself, on average, whereas Tesla’s meager 3 miles is not that thrilling. But there is no type of controlled experiment behind this data so my first concern would be the statistical significance of such a small sample size. We don’t know if the trip lengths or driving conditions were comparable, for instance. And note that Tesla had no data for 2015, indicating they may be at a different stage of testing than Waymo.
But perhaps there’s an even larger disconnect. As I talked about here a few weeks back, Google (now Waymo) started out by championing the direct route to fully autonomous cars due to concerns about the ability of drivers to effectively respond to emergency situations. These correlate to the disengagements measured above. So Waymo, I would expect, very much has the goal in mind of reducing (and ultimately eliminating) DEs.
On the other hand, Tesla, while also working hard to achieve full autonomy as well, was one of the earliest and loudest proponents of the more gradual approach. So in significant ways Tesla may not even be trying to improve this metric in its testing.
Another interesting thought is that Tesla’s vast number of Autopilot vehicles on the road can yield a much larger set of driving data than even the impressive numbers stacked up by Waymo. It would be very interesting to look at similar metrics in that data, although we would need to understand in both sets how to tell the difference between an emergency disengagement and just switching back to manual or ending the trip.
image source: Tesla