Interestingly, one of the primary drivers of idle issues with diesels is soot accumulation in the intake and EGR tract. This is because the excess air at low load idle means lots of NOx production. To combat this, the EGR flow will be high. This reduces free O2 for NOx production. However the low flow rates in the tracts/tubes gives soot much more time to deposit, and thus produce the oil/soot cakes that cause trouble. Lower combustion temps at idle also mean higher soot production in general, combined with low EGTs, the DPF is not likely to regen properly. Hence the need for regular driving. For fuel efficiency, lower soot, and DPF regen high combustion temps are desirable. However, NOx rapidly increases with EGT (square of the temp change). So its a complex balancing game. The addition of SCR (urea catalytic reduction) has helped quite a bit, by allowing higher combustion temps, and still keeping NOx under control.
I am a trained powerplant engineer, and have some design experience. The idea that the diesel engine cycle (as opposed to the Otto cycle) is somehow inherently better is kinda ridiculous. Yes its slightly more efficient volumetrically, but much of that savings is wiped out by emissions scrubbing needs. There is no reason that a gasoline/Otto powerplant cannot perform the same tasks with equivalent reliability. In fact its fairly easy to engineer a gas powerplant to make similar torque down low. This is rarely done because it serves little benefit with a proper transmission. Diesel engines have slightly different needs geometrically, so they will generally produce more torque at low RPM. Because diesel fuels flame front speed is so slow, they are generally limited to about 4800rpm with cylinders up to 700cc, and well under 3000rpm at sizes over 1.2 liters per cylinder. Thus in order to make good power they must be turbocharged, to get maximum fuel per second at low RPM. This means heavier everything, from pistons to heads. More weight, more cost, etc.
For the same compression ratio, the otto cycle is actually more efficient than the diesel cycle. However, with gasoline auto-igniting much more easily, it has compression limits that don't apply to diesel cycle engines. However on the horizon are homogeneous charge compression ignition engines using gasoline. GDI technology has allowed compression ratios to rise further, improving efficiency in gasoline powerplants.
Diesels still have market in heavy transport. This is due to higher overall efficiency. In applications burning 100+ gallons per day, the savings adds up. In some industries the higher energy density of diesel has significant value as well.
I also find it quite funny when heavily emotionally invested folks argue that diesel is good because (insert random old engine/vehicle here). Heck, if I wanted to argue for the superiority of gas powerplants, I could pick any number of legendary engines from the last 30 years. We are talking modern on the lot engines with real world operating conditions. Not deleted/tuned units, not "bulletproofed" etc.
Its a typical mindset, that all non-proponents, must be opponents or enemies. Most of the non proponents in this thread aren't bashing diesels. They are just pointing out obvious economic, emissions, and reliability issues. Then again, for most its human nature to defend the position you've chosen, regardless of evidence. Doing so makes us feel good, as we are "winning/fighting" against the opposition. Even big fleets are that way at times, though eventually the almighty dollar begins to win out over tradition.