I think he certainly has the most potential, and is actually the "best" driver in two ways. First, he is certainly the best driver when it comes to the boring stuff like driving etiquette, safety and sticking to the rules. Beyond that, he is probably the smartest when it comes to the technical bits and knowing what happens when you do a certain thing and can analyze it.
That said, I'm not sure he will always be the fastest. I don't think its down to lacking the ability, but more because he often doesn't seem to care all that much about it. Obviously he gets competitive and clearly can drive reasonably quickly, but that isn't a given. Jezza and Hammond are much more "into" being fast and both fancy themselves racing drivers. Hammond was the only one for a while that was actively seeking racing licensure and regularly goes and does the competition vehicles on the show. Jeremy is just bonkers and loves to drive fast and also is the only one of the three that is actually a dedicated fan of racing.
I put the win at Monaco down to Jeremy caring more about it then the other two, and since he is an F1 fan that watches the races, he knows the track. Plus, if the show was honest, Jezza spent the night grilling the F1 guys about how to drive the track and the other two were off staring at unobtainable booby's and getting drunk (is it bad that I don't know if I would choose Jezza's or Hammond/May's night out?)
So yeah, given an equal level of motivation, I would put James as the best driver full stop. Lacking that motivation, the other guys would probably come ahead. Jeremy is certainly the most flamboyant driver, but screaming around a track with your ass hanging out and yelling power in cool ways doesn't necessarily make you a great driver. The other thing I've noticed about him is that he finds it hard to drive any other way, and when a car isn't good at doing power slides he tends to get a bit grumpy about it and says it can't handle or that it is too clinical and boring.