I'm sorry I didn't get back to this post yet, I had meant to, but with all the things going on lately it got lost in the mix.
Anyway, what MTRH said is basically correct except for a few points.
--r3mix is tuned for "acceptable" quality more than anything else I think. It may attempt to achieve "archive" quality but I believe the recent AQ test shows that it doesn't quite accomplish this to the degree which some may prefer. That being said, Roel often refers to the switch as being "reasonable" in relation to size/quality ratio and that to "most users" it is transparent. Again I'm not convinced this is entirely true (I know it isn't for myself, but I'm speaking for others here) according to the AQ test results, and other people's individual tests as well as my own.
This switch also seems to have problems with quieter music and certain samples like
serioustrouble and
2nd_vent_clip, which cause dropouts.
Summary: reasonable quality, good speed, may provide a more comfortable size/quality ratio for people who want good quality, but not necessarily absolute transparency.
The --dm-preset switches (or primarily --dm-preset standard) are different in that they were constructed from the ground up to provide the highest quality possible with LAME with NO regard to bitrate. Then, once quality was achieved, different methods were utilized to bring the bitrate down while still retaining that same level of quality. I believe this is basically the opposite approach that was taken with --r3mix, since bitrate was a concern from a start in that case.
As a result of this, --dm-preset standard performs better (sometimes significantly so) on critical test samples. The bitrate really is not so much higher than that of --r3mix, it is on average probably around 5-30kbps larger depending on the sample. Over a wide variety of music, it averages out to somewhere around 200-210kbps.
--dm-preset xtreme is fine for use, it is not necessarily experimental. Basically the only difference though is that it uses a lower ath curve. This will be changing shortly though.
However, "insane" is pretty much experimental for now. There isn't much of a reason to use it over standard or xtreme at the moment. The behavior of this switch is going to be changing soon as well though so hopefully the progression through the switches will provide a more linear increase in quality.