I tried to compare different mp3 decoders, in order to evaluate the benefits of dithering (MAD and foobar2000) and noise shaping (foobar2000 only), against simplier decoders, as LAME or Fraunhofer (Winamp 5.0). I didn't hope anything first. But I've finally succeed with one sample, and decided then to resume the test with others samples.
I've wrote a complete review in french, but I can't translate it properly. In few words:


• For this test, I didn't reproduced real listening conditions. Difference are so subtle that I have to increase the level of my amp in order to maximise the audible difference.
• In order to make it possible without being completely deaf in few seconds, I've selected very quiet samples. With classical music, it's not really hard to find such samples (even full tracks are sometimes totally quiet). I didn't evaluate any decoders on pre-instrumental silence, but only when music was playing


Four decoders were tested:

• MAD, basically in order to evaluate the real performance of this mythic decoder, and technically in order to rate the benefits of dithering WITHOUT noise shaping.
• LAME, to mesure the difference between MAD and a simplier decoder.
• Fraunhofer: in order to evaluate possible difference with LAME (both encoders are really similar, so difference may not be audible or existent)
• foobar2000: to see if noise shaping could make a difference with MAD dithering.

Eight samples were introduced:

I've mixed different musical genre: lyrical, orchestral, piano, chamber, electronic. I've tried to mixed dithered and undithered mastered CD too.


CONCLUSIONS:

MAD isn't impressive. On four samples, result was worse than simplest decoders: increase in noise, without audible benefit on other points of the signal (aliasing, higher definition, etc...). But on some other samples, real improvements were noticed (less aliasing, less naturel and less synthetic sound)

LAME and Fraunhofer (WA5) are close, but not totally identical. Both are noisy, but less than MAD (good point). With some samples, sound was irregular, unatural... MAD partailly correct this feeling, and foobar2000 was perfect in comparison. Something like aliasing or moiré was introduced by the decoding stage. Slight difference between LAME and Fraunhofer (hard to ABX : up to 30...50 trials), in favour of Fraunhofer, less synthetic and maybe slightly more noisy

foobar2000 : each time winner. The less noisy of all challengers on all samples. The only additionnal noise audible was a very high-pitched band, very typical, and audible on extreme conditions (near silence sound, and with my amp pump to the maximum). All aliasing, irregularities... were gone: instruments are smooth, naturel. Nothing synthetic at all. With one sample (the #7), the sound was clearly better than the reference file (CD!). The lack of dithering on mastering seem to be prejudicial on very low volume part, and foobar2000 decoding correct the aliasing present on the CD. Simply amazing conclusion....




All results are here :
www.foobar2000.net/mp3decoder
http://www.foobar2000.net/mp3decoder/conclusion.htm
(I've translated in english the summary of my comments).


Comments are welcome.


P.S. It's not a MAD CHALLENGE. I repeat again that the listening conditions of the test are not reproductible on daily music listening. I've just put the decoder behind a magnifying glass, by using low volume samples.

I suppose you tested with ABC/HR blind rating although you don't say it in the message?

Your pages here would be fantastic but too bad you chose to use french..

The "better than CD" sentence is maybe a bit controversial though, but I understand what you are after...

ABC/HR, yes (the log files are available on the site, or here.Of course, the notation or rank can't be the same with this test than for a 64 kbps listening test. Difference between different decoders was magnified, as well as samples were magnified by increasing the listening volume.

For french language, ask my parents or english teachers: I didn't choose my country

Impressive. Thanks for your effort. Several people before have noted that MAD's dithering is "sub-optimal" and inferior to fb2k's noiseshaped dithering.

See also this thread:
http://www.hydrogenaudio.org/forums/index....showtopic=14960

One "problem" with Guru's interesting results are that those are presented on www.foobar2000.net. Not very unbiased sounding place despite of all the blind ratings and ABX -results..

I agree

I'm amazed (but very glad) that you've managed to subjectively confirm my old objective test results: MAD was/is measurably inferior to the best.

http://mp3decoders.mp3-tech.org/24bit.html
(compare the various MAD compiles with l3dec)

I felt quite harsh for pointing this out at the time. I assumed no one would ever hear the difference with real music. It's interesting that it can be audible (in extreme conditions). FWIW the differences in dither algorithms could be greater with a very clean 24-bit source. (you state you used CDs)

It also suggests that the assumption which runs through the foobar2k GUI (which seems to be "this program performs very accurate processing, but it's probably impossible to hear the difference") is too modest!

Anyway, I wish I had your ears guruboolez!

Cheers,
David.

Golden ears would probably perceive a difference between different decoders on real listening conditions. Common ears, like mine, need to push the volume to extreme position. Try, please... I was the first to be surprise to hear differences.

At the beginning, I mainly heard different noise level... Then, I was able to ditinguish different things (that may explain why, at the end of the test, notation between decoders was more distant).

I've uploaded the lossless samples (which are -funny things- clearly inferior to the 320 CBR encoding filesize) in optimfrog format, so everybody could make the same test.

Good thorough test

Thanks for taking the time to do this... and all the other things you've contributed.

You are so modest, but this is pretty amazing stuff.

Thank you.

hmm. maybe it should be added that foobar is not the decoder itself. it uses the mpg123 library for decoding which is available for Winamp, too afaik. So the separation Winamp vs foobar is not correct and looks even more strange, as this test is provided on the foobar website

better would be: MAD vs LAME vs Fraunhoffer vs Mpeg123

Yes, and no.
I didn't test mpg123 alone, because there was dithering/noise shaping... from foobar2000. The most interessing thing to note is that mp4, vorbis, mpc... playback will profit from the same technique.

I explained that in french : the test isn't an audio player comparison (mainly because audio players are not only decoders). The test is not really a mp3 decoder test, but more a decoding technique test : dithered vs non-dithered decodings and noise-shaped dithering against simple dithering. LAME and Fraunhofer, sharing the same principles, are really close, if not the same on three or four samples. Dithering introduced a real difference (not always good, because MAD files were systematically the easiest to ABX), and noise shaping other differences.

I wonder why other audio players didn't use these techniques. Is it really hard to program? Dithering/noise shaping is very common on studio mastering: most of my CDs have visible noise shaping. So if studios use them, it must be good.

But are you sure that these improvements in sound quallity are the benefit of the dithering/noise shaping or the benefit of the decoder library itself. Maybe the same increase in quality could be heard by using the in_mpg123 plugin in winamp, too.

I would rather test Winamp in_mpg123 vs Foobar mpg123, so you can be sure that the dithering and noise shaping was the determining factor in increasing the quality. Now it can be the decoder itself.

Don't other players just do the obvious, simple thing: decode to 16-bits and truncate?

You can't noise shape and dither unless you have more than 16-bits available to start with. It must be there inside the decoder (well, at least partly, or at least in some decoders?) - but foobar is the first player where all these issues have been properly thought through.

Don't forget just how bad Winamp has been at times during its history. The old Nitrane decoder was barely 14-bit accurate at times - no point dithering it!


It seems that some players (did or do) think it's enough just to get something resembling music out of an mp3 - the quality wasn't/isn't important.

Cheers,
David.

P.S. surely mpg123 and lame give basically the same decodes (with occasional LSB differences), and all FhG decoders are likewise similar (with occasional LSB differences). FhG vs lame (=mpg123?) gives continuous LSB differences; whats more, the lame decoder apparently includes -90dB harmonic distortion when decoding to 16-bit - or at least it did when I tested it years ago.

huh? I cannot really follow your post.

I don't think the Shibatch mpg123 library and lame are related in any kind. 16+ bit output should be possible with winamp mpg123 plugin as well.

Are you confusing something?

Oh damn! I've always thought dithering was no good.

http://lame.sourceforge.net/

claims that

"LAME 3.xx uses software from:
Decoding engine (for the frame analyzer): mpglib, from the mpg123 project."

I've never investigated the details of this statement, but the lame output and the Winamp mpg123 output are amazingly similar.

Cheers,
David.

they adopted and improved hipp's mpglib and integrate for decoding

...and also made a hip interface (libmp3hip) for a straight decoding library


later

oops, then the fault was on my side. Good to know, though.

If you are correct, these conclusons can be drawn from the test:
- Neither MAD. nor mpg123, nor FhG can outperform its competitors (one better here, the other better there)
- only a combination with dithering and noise shaping will offer signifivantly better quality.

Right?

oh no, dithering is very important, especially when you reduce the resolution from 24 bits to 16 bits. it's consecutive re-dithering which is a questionable practice, especially when the source is only 16 bits. however, if we use mp3s as an example, if the bitrate is high enough, it may be possible to retrieve data above 16 bits, thus preserving more ambiance and detail which would surely be lost if we were to truncate at 16 bits. there is no guarantee, however, that you will hear a lot of difference or any difference at all. the side effect of re-dithering is more noise in the audio signal but this can be somewhat offset with proper noise shaping, although this is very subjective.

If the original dithering was noise shaped, it was mainly removed during mp3 encoding because of the applied lowpass filter.

Wrong.

Read this quote from an earlier post from guruboolez on this thread again:


There are no significant differences of any kind.

I've done a similar test (~ 1 year ago I guess, but I can't find where I posted it ):

• Take some mp3s (3.90.2 --alt-preset standard IIRC)
• Use mp3directcut to apply fadeouts (0 -> -40 dB IIRC)
• Decode using foobar2000, winamp, xmplay
• Apply fadeins opposite to the mp3directcut fadeouts (e.g. 0 -> +40 dB)
• Listen + compare

IIRC the results were these:
- With Winamp (native decoder, not MAD) there was clearly audible truncation distortion
- With foobar2000 (I can't remember what dither type I used - I don't even know if there were options to choose) there was no distortion at all but a clearly audible noise floor (of course exagerated by amplification)
- XMplay's decoder was some kind of compromise: Less distortion than Winamp and lower audible noise level than fb2k

This way a similar test can be performed without need for very good (=low noise floor) equipment. It's still hard to tell how these results are related to listening under real life conditions though.

Edit: Seems like I never posted about the test unfortunately. Some time before (I think even before I knew foobar2000) I did the same test XMPlay (2.5) vs. MAD, the result is here
Later, I repeated it with Winamp 2.8x, fb2k 0.dunno and XMPlay 2.6 as described above - well, if anyone's really interested, it should be very easy to repeat ....

They slow down the player for something which doesn't have a benefit in casual listening. This is why it defaults to off in foobar too.

The foobar decoder may have been based on mpglib, it has been improved quite a bit since.

The dithering/noiseshaping code is from me, based on SSRC and suggestions from KikeG.

I understand. But all players have a preference/option box. Allowing dithering isn't the same thing than forcing it. If people want a theoretical increase in quality, why not? It's not a bad thing... on a changelog

less natural and less synthetic seem to contradict Maybe you could clear up what you meant here?

Also I could not find which version of MAD you tested, (cause the PCM dithering algorithm was changed in version 14.0).

Edit: Ah it was in the French piece: est la 0.15.0 (beta, datant de juin 2003)

Thanks for testing and writing to this forum in English

Sorry, I did a mistake. MAD is sometimes MORE natural and LESS synthetic.
But this is feeling only. I don't really know if this 'natural' sound is a consequence of noise added by MAD dithering. Small amount of noise isn't always bad. For exemple, video decoders have 'film' or 'grain' effects: image is less clean, but looks sometimes better with than without.

Humm, interesting... Unfortunately I don't have any 24-bit audio to test it. Thanks a lot.

I've quickly tested MAD 24 bits (dithered I suppose) decoded files against FB2K 24 bits dithered/noise shaped through foobar2000 ABX component, and the two files were less noisy than the best 16 bits decoded files. Both files sounded similar.

It seems that ABC/HR don't support 24 bits input files.

So what is the point of application in plain terms for the non-technical person? Does this mean we should use Foobar to batch encode mp3 from waves OR that playback is best when using Foobar OR some box should/shouldn't be checked when playing them back or encoding or ABXing? Help the Average Joe here understand the real-world significance of the discussion, please. Thanks in advance.

Did you even try to read the discussion?
Did you even bother looking up what encoding and decoding are?

Yup but I don't get it - clearly you are much smarter than I. Or, you at least have years of training and/or experience I do not have (and can't get in the next 3 days before this thread dies off) that enables your participation in the discussion. Take it as a challenge. You genius try to explain, in non-tech terms, the real world application of the discussion for the average reader. Or if you are not interested in such a discourse, perhaps someone else is. Thanks in advance for your help.

Since you challenged me, I'll try to do so:
Decoding is the process of uncompressing compressed (encoded) data like MP3 back to uncompressed PCM Audio. Guruboolez' tried to objectivly measure the quality of different MP3 decoders with and without different types of dithering.

You are most likely not to hear any differences at all under normal listening conditions on normal music (especially loud pop/rock music will mask most differences), but Guruboolez tried to "magnify" the differences by choosing quiet passages of music and listening to them at a high volume.

Guruboolez' liked foobar2000's decoder and noise-shaped dithering best.

Only if you want to apply DSPs or volume changes (e.g. replaygain) before encoding. Volume changes can be applied after encoding as well (replaygain/mp3gain) BTW.


Theoretically, yes. That's what the test results suggest. Under real listening conditions there shouldn't be an audible difference though.


For theoretically best playback quality check "Dither" box in output settings if you use 16bit output. If you use 24bit output (and have a soundcard capable of 24bit ) dither will only (slightly) increase CPU load but have no advantages.


To understand the technical background better you might want to have a look at the FAQ or the Wiki ("Knowledge Base" on HA portal).

I think the 'real world' conclusion is:

1) Not all CD's are properly mastered

2) Differences between MP3 decoders are only audible in extreme circumstances

3) Taking 2 into account, foobar does well, and MAD isn't what it is hyped up to be

Would conclusions 2 and 3 be based only on Guru's test?

As in, I'm quite sure many of us would have already encountered many people on other forums who claim the direct opposite to 3. How are they going to be persuaded otherwise?

Thank you. I understand. Great article on dithering - very approachable by a lay person. Follow up questions then (if the answers aren't beyond the average Joe's discussion and I'm using the correct terminology): Why not do this on every recording? Is there a way to ensure that it is when recording digitally or is it dependant on choice of equipment? Does "upsampling(?)" to 24 bits during decoding/playback take care of the problem or is this artificial in some way?

I assume this explains some of the the background noise in some cases on soft recording passages (Jazz and classical is my main listening) or distorted/strange tone qualities/characteristic sounds of certain instruments played at soft volumes? Perhaps this problem isn't as uncommon as it seems to classical listeners, as turning up the volume to hear soft passages is frequently required to hear what is even being played...

Thanks everyone!

Not only, I knew of the differences before he did the test but it's nice to see it confirmed

Well, there's an objective blind test showing that foobar is better. I don't know any showing the opposite, do you?

Don't worry, your question is indeed rhetorical.

With the asterisk that this test was done at 16bit, and that based on gurobolez's 'quick test' MAD *might* well be about as good as fb2k at 24bit.

(edited for clarity)

Nonsense. You have made enough merits to be considered member of the "golden-ear" club. Some people being able to hear differences at real listening conditions? Maybe, for some really gifted people, and when listening in really good conditions (good headphones, low ambient noise), but even in this case audible differences would probably be very small. Anyway, without actual results it's hard to say.


I'm not very surprised, if using low amplitude music, amplifying the result, and using discriminating ears. When I tested it, MAD dithering showed to be clearly sub-optimal, due to lack of dither amplitude, that caused visible distortion and noise pumping on synthetic test signals.

I'm not sure I understand all of this, but I have this question.

All of my MP3s or MPCs are 16 bit. Now there are instances where there is more than 16 bits of information to decode at one time? The option to dither (in fb2k) would allow playback without truncating these instances to 16 bits. My audigy2 supports 24 bit playback, so if I chose 24 bit playback (in fb2k) then these instances would still not be truncated without the need of dithering?

Sorry if I have this completely wrong but I'm drinking beer right now. :X

does foobar also use the noise shaping (when enabled) on wav files or only on encoded material ? as i want bit identical output on my spdif out i guess i better leave it off.

They aren't.

In general, you can't say that a lossy file has a certain bit resolution.

Theoretically, an mp3 file can be decoded to any desired bit depth. And I think that by modifying e.g. gain information, you can create mp3s that would decode to something other than digital silence only, say, at 24 bits.

Whether decoding to a bit depth higher that 16 makes any difference in the real-world listening conditions is a different question, though.

Interesting discussion about dithering. Just a small off-point question. In the vorbis plugin for winamp 5, there is a 'dither' checkbox. Since lossy files are not bit resolution specific, does that mean when the vorbis plugin truncates to the (default) 16 bits, it does some sort of dithering? Also, in the Directsound SSRC output plugin, there is a dithering option and noise shaping. If I am doing dithering in the output plugin, does that mean the vorbis plugin dithering is redundant or vice versa?

@Quantum: I don't know if I fully understand your question. But I think that if Vorbis decoder truncate to 16-bit, then it won't be a form of dither. So I think you should enable Dither on it's decoder (on Winamp). I've always used it disabled and I really love my Vorbis. hehe

I haven't enabled dithering on fb2k either. I've always had a sort of predujice against this word hehe. Dithering for pictures su**s.

LOL! Quite a sensible question then!

The FAQ on these pages (scroll down) may help, though you've kind of got the answer already...

http://mp3decoders.mp3-tech.org/24bit.html
http://mp3decoders.mp3-tech.org/24bit2.html

If I could find the test files, I'd even think about testing foobar. But I have 1000001 things to do first.

Cheers,
David.

Ah ok, I'll rephrase it to see if its a bit understandable.

Essentially, there are two places in Winamp where there is a dithering option. One is in the Vorbis input plugin, which is a check box. I got that checked and using 16-bit decoding. The second place to dither is in Peter's DirectSound SSRC output plugin (the one which allows you to do resampling to 48 kHz). In this second one, you can choose noise distribution as well as dithering using a spectral shape (triangular).

Are these two dithering options different and independent of each other and should I have one turned on instead of the other?