That's definitely going to happen. And it sucks.
The segment of outspoken techies who hate driving and are excited for this compare us enthusiasts to horse and buggy enthusiasts around when the car was invented - we're clinging onto a silly, outmoded and dangerous hobby. - They say we'll likely have "driving parks" to augment tracks for those regular people who miss driving.
It's still not the same though. Ex: Driving without necessarily a destination in mind but just to get out and just spend time reconnecting with you and the car.
You can't do that at a driving park.
The problem is that from a technological point of view it makes perfect sense to only have computer controlled vehicles on the road. You can make them communicate with each other which means they can drive much closer to one another because every car is aware of what every other car around it is going to do at any given time. Adding manually controlled cars in the mix screws this up big time, it's adding a very big and potentially very dangerous unknown variable to the system. It requires much more from an "A.I." to deal with complex and sometimes unpredictable human behaviour compared to just dealing with other A.I. controlled vehicles. It's not necessarily that the people developing/selling autopilots hate driving or the people doing it, but it makes their own job of putting computer controlled cars on the road much more difficult.
Many years ago people thought that teaching computers to visually recognize objects would be a moderately difficult, but ultimately solvable problem. Now we have 2016 and computers still can't even recognize a few letters in front of a scrambled background:
So how could you possibly expect to go from that to recognizing "the world around you" in real-time while you're doing 100 kph with a computer? Hence the computer has to rely on other sensory data that's easier to process and not based on visuals. But literally everything involving traffic right now is based on visual clues: traffic lights, signs, markings on the road, indicators and so on. Most of this is easy for us humans but extremely difficult for a computer to process. And I'm not a tech-hater, quite the opposite actually, I even try my hand at ANNs (Artificial Neural Networks) just to see what they can do, but one of the major challenges is to convert your data to something that your ANN can understand, and images are about the worst when it comes to converting them. Especially when we talk about images with megapixel resolution (which are obviously needed to identify objects further away from you and see further ahead), since basically each pixel needs its own input node, leading to an ANN with millions of nodes. Then there's the question of training such a network reliably so that it can make accurate predictions and decisions, which again is a gargantuan issue that only becomes more difficult when your data becomes more complex and difficult to understand.
So yeah, it might be a long way to true A.I. that can blend in with humans when it comes to driving. And we're probably going to see more than one half-assed attempt along the way.