But why on earth didn't they just choice the MP5 or some other proven 9 mm SMG to start with?
Actually, not really. It may have looked cheaper on paper (I doubt it) but there's some issues with Colt having far too much influence with the government. This isn't really the place for it, but the bottom line is that Colt has historically had quite a lot of Senators and Representatives in their pockets for a long time - which has resulted in our government and others getting stuck with Colt-produced weapons platforms that weren't actually ideal, best performing, or even really suitable for the job. Further, Colt's managed to get many of the competitions to 'replace' their products killed off or suspended, though it looks like the latest one may not go their way.
You should also have heard the screaming from Colt when someone in the government noticed that Colt didn't actually own the design of the M16 (Armalite did) and that Colt was probably overcharging for their product due to their supposed (by them) monopoly. When they put it out for bid, Colt was taken completely by surprise. Not surprisingly, Colt wasn't the top finisher in the M16 re-let contract competition; their offerings turned out to be higher cost and lower quality than almost all the competition. FN (yes, the FN who made the FAL, P90, High Power, etc) makes the M16 for the US now, in a US plant. FN also supplies more than a few of the non-Colt AR-15 makers with the parts they use to make their AR-15 type rifles, such as upper and lower receivers, barrels, etc.
Colt responded by immediately patenting a bunch of things about the M4 so that contract couldn't be taken away from them. Colt basically hasn't made anything truly new in about 20 years. Funny thing is, them patenting the M4 and people finding ways around it has actually opened up the government contracting patterns, and moreover it's making covering for Colt politically expensive. Today, Knights Armament, Bushmaster, and many others are getting contracts for AR-15 pattern weapons that in the past Colt would have gotten by default. The only thing Colt still has a lock on is the general issue M4 weapons system for the military; most other departments who have a need will buy from someone else.
More than a little of the "Not Invented Here" problem involved, too. And in later years, when there wasn't domestic competition (after Armalite was out of the picture, TRW had given up, Ruger hadn't been able to sell the AC556, etc.), not all of the foreign makers had plants in the US that could make the weapons they offered - something that most corrected a long time ago.