I have had the same argument with my friends as well... And I always compare kart driving. I am the most unfit person I know and as such am completely wasted after 15 minutes at the wheel of a kart car. But even the most fit people I know (and they are really fit, like pro atheletes comepting on national level) are out of puff after 30mins.
Now imagine driving at full tilt for 2hours, I cannot even imagine how hard that must be at the speeds they are going.
So to give my opinion, I think they are extremely fit. Their training is just really specialised so they might not be able to compete with other athletes trained secifically for their own events.