I think three things killed the diesel in the US. They were actually briefly wildly popular in the 80s, but the below items were what killed them off:
1. The noise and smoke. Even the most advanced diesels from Mercedes or the Japanese makers in the 80s rattled loudly and produced clouds of smoke when accelerating away from a light or when getting on a freeway. The public learned to associate "small diesels" with "aural and visual polluter".
2. The performance - or lack thereof. 80's diesels were slow. Very slow. A lot of people that bought them felt that they'd been ripped off (in terms of performance), and frankly many of them were so slow they were unsafe to drive in many situations in the US. The public learned to associate "small diesels" with "deathly slow".
3. Repair costs and frequency. This is where the diesel got screwed in the US. First, diesels are not as simple or easy to repair as a gas engine; or rather, they weren't in the 80s. Therefore, you had to take your diesel to the dealership (and get raped) or to a diesel specialist mechanic (and get raped) for service. Many of the diesels sent to the US in the 80s weren't very good and broke down often. At the time, GM still had the majority of the US market, and they responded to the diesel craze by hastily converting a gasoline-powered Oldsmobile 350 cubic inch (5.7L) V8 engine to run on diesel with the advice of their then-subsidiary, Detroit Diesel. The resulting diesel was a total disaster. It would last until the end of the 2 year warranty, and then do things like crack blocks, shatter heads, and drop valves, if you were lucky. If you were unlucky, you had it break down inside warranty repeatedly, whereupon GM would eventually deny you warranty coverage because "you abused it". And GM sold millions and millions of them before people found out that their 30mpg full-size sedans had a huge drawback. So the public learned to associate "small diesels" with "hideously unreliable at worst, hideously expensive to maintain at best."
The American people were willing to put up with some pollution and a lack of performance to save money on fuel. They were not, however, willing to deal with a car that either emptied their wallets on top of all of this, or was so unreliable that it would leave you stranded (which costs the driver/owner money in lost time, if nothing else). Thus ended the diesel as a mainstream propulsion system in the US for the next 20 years.
Ironically, most European diesels are *still* too polluting to be allowed into the US, which is why the selection is still so slim.