Imho James May may be the fastest driver among them all - he just doesn't like to go fast.
This depends entirely on how you define fast. James has gone the fastest in a production car (in the Bugatti), but that was due largely to the car, not his skills. And it was a straight line.
I think, in terms of getting around a track (or anything else non-linear) James probably is the slowest, based on the few times we've seen him do it. Jeremy can powerslide like nobody's business, and Richard can at least kick the tail out and control under- and oversteer, but James seems to need a lot of coaching before a car will do what he wants.
I'm not saying he's slow, mind you. He could drive rings around me, I'm sure. I'm just saying, of the three, I think he's the slowest around a track.
As for not liking to drive fast, I'm guessing you're just wrong. He has a role to play on Top Gear, but in
Oz & James he tears it up. Just because he's cast as Cpt. Slow in TG doesn't mean that's who he is. Every comedy act needs a straight man, after all.