Wow. The future is here.
One situation that worries me about autonomous vehicles is driving in residential neighborhoods, because human and animal behavior is so erratic. Another thing that worries me is that frequently while driving, I am anticipating human behavior due to an understanding of human psychology, and reacting accordingly. For example, I see somebody at a cross street who looks like they're about to go through the intersection, and I can see that they're not even looking -- so I slow down in anticipation of them pulling out in front of me. Or I see somebody's head in their rear vision mirror, and they look like they're about to throw open the door and get out, but I'm passing a turning vehicle on the right, which would cause me to hit the door if they do throw it open. Or I see kids playing ball on the side of the street, and I know the ball could bounce out on the street at any moment. Or I see a biker who is clearly distracted, and know he could veer out of the bike lane at any moment. There is an asymptotically long tail of things that would be exponentially hard to code up heuristics to catch, and which even humans take years to learn how to drive defensively against. It's not very satisfying to me to know that Google's technology is able to handle maybe 75% of these contingencies by simply driving at safe speeds and obeying traffic laws.
Of course, since the possibility of accidental death in a self-driving car is non-zero, whoever ends up being responsible for the accident, it's only a matter of time before the first major death and corresponding lawsuit.
What we need is every single car on the road driving autonomously (so that cars can coordinate with each other over a wireless network), and walled-off freeway lanes exclusively for autonomous vehicles.