I'd have to agree with Kunai, the Scout and Spy are the hardest, closely followed by the pyro and medic. Medic because while you can stand behind a big meat shield, you are one of the most hunted classes in the game. If you can get the medic, you can get the heavy/soldier/whatever class he happened to be healing at that moment. Also you have to have a sense to know when to get out of a bad situation, even if it is a tough choice between whether you can make it by healing your guy, or if you'd stay alive by running away and sacrificing the guy you were healing.
In short, there is more to the medic than just point and heal, all I'm saying.