IPB

Welcome Guest ( Log In | Register )

4 Pages V   1 2 3 > »   
Reply to this topicStart new topic
MP3 decoders test : MAD isn't so good! (for me...), MAD vs LAME vs Winamp 5 vs foobar2000
guruboolez
post Jan 18 2004, 13:58
Post #1





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I tried to compare different mp3 decoders, in order to evaluate the benefits of dithering (MAD and foobar2000) and noise shaping (foobar2000 only), against simplier decoders, as LAME or Fraunhofer (Winamp 5.0). I didn't hope anything first. But I've finally succeed with one sample, and decided then to resume the test with others samples.
I've wrote a complete review in french, but I can't translate it properly. In few words:


• For this test, I didn't reproduced real listening conditions. Difference are so subtle that I have to increase the level of my amp in order to maximise the audible difference.
• In order to make it possible without being completely deaf in few seconds, I've selected very quiet samples. With classical music, it's not really hard to find such samples (even full tracks are sometimes totally quiet). I didn't evaluate any decoders on pre-instrumental silence, but only when music was playing


Four decoders were tested:

• MAD, basically in order to evaluate the real performance of this mythic decoder, and technically in order to rate the benefits of dithering WITHOUT noise shaping.
• LAME, to mesure the difference between MAD and a simplier decoder.
• Fraunhofer: in order to evaluate possible difference with LAME (both encoders are really similar, so difference may not be audible or existent)
• foobar2000: to see if noise shaping could make a difference with MAD dithering.

Eight samples were introduced:

I've mixed different musical genre: lyrical, orchestral, piano, chamber, electronic. I've tried to mixed dithered and undithered mastered CD too.


CONCLUSIONS:

MAD isn't impressive. On four samples, result was worse than simplest decoders: increase in noise, without audible benefit on other points of the signal (aliasing, higher definition, etc...). But on some other samples, real improvements were noticed (less aliasing, less naturel and less synthetic sound)

LAME and Fraunhofer (WA5) are close, but not totally identical. Both are noisy, but less than MAD (good point). With some samples, sound was irregular, unatural... MAD partailly correct this feeling, and foobar2000 was perfect in comparison. Something like aliasing or moiré was introduced by the decoding stage. Slight difference between LAME and Fraunhofer (hard to ABX : up to 30...50 trials), in favour of Fraunhofer, less synthetic and maybe slightly more noisy

foobar2000 : each time winner. The less noisy of all challengers on all samples. The only additionnal noise audible was a very high-pitched band, very typical, and audible on extreme conditions (near silence sound, and with my amp pump to the maximum). All aliasing, irregularities... were gone: instruments are smooth, naturel. Nothing synthetic at all. With one sample (the #7), the sound was clearly better than the reference file (CD!). The lack of dithering on mastering seem to be prejudicial on very low volume part, and foobar2000 decoding correct the aliasing present on the CD. Simply amazing conclusion....




All results are here :
www.foobar2000.net/mp3decoder
http://www.foobar2000.net/mp3decoder/conclusion.htm
(I've translated in english the summary of my comments).


Comments are welcome.


P.S. It's not a MAD CHALLENGE. I repeat again that the listening conditions of the test are not reproductible on daily music listening. I've just put the decoder behind a magnifying glass, by using low volume samples.

This post has been edited by guruboolez: Jan 18 2004, 14:02
Go to the top of the page
+Quote Post
JohnV
post Jan 18 2004, 14:09
Post #2





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



I suppose you tested with ABC/HR blind rating although you don't say it in the message?

Your pages here would be fantastic but too bad you chose to use french.. sad.gif crying.gif

The "better than CD" sentence is maybe a bit controversial though, but I understand what you are after... wink.gif


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Jan 18 2004, 14:14
Post #3





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



ABC/HR, yes (the log files are available on the site, or here.Of course, the notation or rank can't be the same with this test than for a 64 kbps listening test. Difference between different decoders was magnified, as well as samples were magnified by increasing the listening volume.

For french language, ask my parents or english teachers: I didn't choose my country wink.gif
Go to the top of the page
+Quote Post
dev0
post Jan 18 2004, 14:48
Post #4





Group: Developer
Posts: 1679
Joined: 23-December 01
From: Germany
Member No.: 731



Impressive. Thanks for your effort. Several people before have noted that MAD's dithering is "sub-optimal" and inferior to fb2k's noiseshaped dithering.
Go to the top of the page
+Quote Post
Gabriel
post Jan 18 2004, 15:01
Post #5


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



See also this thread:
http://www.hydrogenaudio.org/forums/index....showtopic=14960
Go to the top of the page
+Quote Post
JohnV
post Jan 19 2004, 00:26
Post #6





Group: Developer
Posts: 2797
Joined: 22-September 01
Member No.: 6



One "problem" with Guru's interesting results are that those are presented on www.foobar2000.net. Not very unbiased sounding place despite of all the blind ratings and ABX -results.. wink.gif


--------------------
Juha Laaksonheimo
Go to the top of the page
+Quote Post
guruboolez
post Jan 19 2004, 00:37
Post #7





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I agree laugh.gif


dry.gif
Go to the top of the page
+Quote Post
2Bdecided
post Jan 19 2004, 00:44
Post #8


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



I'm amazed (but very glad) that you've managed to subjectively confirm my old objective test results: MAD was/is measurably inferior to the best.

http://mp3decoders.mp3-tech.org/24bit.html
(compare the various MAD compiles with l3dec)

I felt quite harsh for pointing this out at the time. I assumed no one would ever hear the difference with real music. It's interesting that it can be audible (in extreme conditions). FWIW the differences in dither algorithms could be greater with a very clean 24-bit source. (you state you used CDs)

It also suggests that the assumption which runs through the foobar2k GUI (which seems to be "this program performs very accurate processing, but it's probably impossible to hear the difference") is too modest!

Anyway, I wish I had your ears guruboolez!

Cheers,
David.
Go to the top of the page
+Quote Post
guruboolez
post Jan 19 2004, 00:58
Post #9





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (2Bdecided @ Jan 19 2004, 12:44 AM)
Anyway, I wish I had your ears guruboolez!

Golden ears would probably perceive a difference between different decoders on real listening conditions. Common ears, like mine, need to push the volume to extreme position. Try, please... I was the first to be surprise to hear differences.

At the beginning, I mainly heard different noise level... Then, I was able to ditinguish different things (that may explain why, at the end of the test, notation between decoders was more distant).

I've uploaded the lossless samples (which are -funny things- clearly inferior to the 320 CBR encoding filesize) in optimfrog format, so everybody could make the same test.
Go to the top of the page
+Quote Post
Althalus
post Jan 19 2004, 01:20
Post #10





Group: Members
Posts: 147
Joined: 14-December 01
Member No.: 647



Good thorough test

Thanks for taking the time to do this... and all the other things you've contributed.
Go to the top of the page
+Quote Post
AtaqueEG
post Jan 19 2004, 01:49
Post #11





Group: Members (Donating)
Posts: 1336
Joined: 18-November 01
From: Celaya, Guanajuato
Member No.: 478



QUOTE (guruboolez @ Jan 18 2004, 05:58 PM)
Golden ears would probably perceive a difference between different decoders on real listening conditions.

You are so modest, but this is pretty amazing stuff.

Thank you.


--------------------
I'm the one in the picture, sitting on a giant cabbage in Mexico, circa 1978.
Reseñas de Rock en Español: www.estadogeneral.com
Go to the top of the page
+Quote Post
amano
post Jan 19 2004, 03:06
Post #12





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



hmm. maybe it should be added that foobar is not the decoder itself. it uses the mpg123 library for decoding which is available for Winamp, too afaik. So the separation Winamp vs foobar is not correct and looks even more strange, as this test is provided on the foobar website dry.gif

better would be: MAD vs LAME vs Fraunhoffer vs Mpeg123
Go to the top of the page
+Quote Post
guruboolez
post Jan 19 2004, 03:20
Post #13





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Yes, and no.
I didn't test mpg123 alone, because there was dithering/noise shaping... from foobar2000. The most interessing thing to note is that mp4, vorbis, mpc... playback will profit from the same technique.

I explained that in french : the test isn't an audio player comparison (mainly because audio players are not only decoders). The test is not really a mp3 decoder test, but more a decoding technique test : dithered vs non-dithered decodings and noise-shaped dithering against simple dithering. LAME and Fraunhofer, sharing the same principles, are really close, if not the same on three or four samples. Dithering introduced a real difference (not always good, because MAD files were systematically the easiest to ABX), and noise shaping other differences.

I wonder why other audio players didn't use these techniques. Is it really hard to program? Dithering/noise shaping is very common on studio mastering: most of my CDs have visible noise shaping. So if studios use them, it must be good.

This post has been edited by guruboolez: Jan 19 2004, 03:21
Go to the top of the page
+Quote Post
amano
post Jan 19 2004, 03:41
Post #14





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



But are you sure that these improvements in sound quallity are the benefit of the dithering/noise shaping or the benefit of the decoder library itself. Maybe the same increase in quality could be heard by using the in_mpg123 plugin in winamp, too.

I would rather test Winamp in_mpg123 vs Foobar mpg123, so you can be sure that the dithering and noise shaping was the determining factor in increasing the quality. Now it can be the decoder itself.
Go to the top of the page
+Quote Post
2Bdecided
post Jan 19 2004, 04:33
Post #15


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



Don't other players just do the obvious, simple thing: decode to 16-bits and truncate?

You can't noise shape and dither unless you have more than 16-bits available to start with. It must be there inside the decoder (well, at least partly, or at least in some decoders?) - but foobar is the first player where all these issues have been properly thought through.

Don't forget just how bad Winamp has been at times during its history. The old Nitrane decoder was barely 14-bit accurate at times - no point dithering it!


It seems that some players (did or do) think it's enough just to get something resembling music out of an mp3 - the quality wasn't/isn't important.

Cheers,
David.

P.S. surely mpg123 and lame give basically the same decodes (with occasional LSB differences), and all FhG decoders are likewise similar (with occasional LSB differences). FhG vs lame (=mpg123?) gives continuous LSB differences; whats more, the lame decoder apparently includes -90dB harmonic distortion when decoding to 16-bit - or at least it did when I tested it years ago.
Go to the top of the page
+Quote Post
amano
post Jan 19 2004, 04:41
Post #16





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



huh? I cannot really follow your post.

I don't think the Shibatch mpg123 library and lame are related in any kind. 16+ bit output should be possible with winamp mpg123 plugin as well.

Are you confusing something?
Go to the top of the page
+Quote Post
Rash
post Jan 19 2004, 04:54
Post #17





Group: Members
Posts: 87
Joined: 9-December 03
From: Sao Paulo
Member No.: 10302



Oh damn! I've always thought dithering was no good.


--------------------
Liberate tutemae ex inferis
Go to the top of the page
+Quote Post
2Bdecided
post Jan 19 2004, 05:05
Post #18


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (amano @ Jan 19 2004, 03:41 AM)
huh? I cannot really follow your post.

I don't think the Shibatch mpg123 library and lame are related in any kind. 16+ bit output should be possible with winamp mpg123 plugin as well.

Are you confusing something?

http://lame.sourceforge.net/

claims that

"LAME 3.xx uses software from:
Decoding engine (for the frame analyzer): mpglib, from the mpg123 project."

I've never investigated the details of this statement, but the lame output and the Winamp mpg123 output are amazingly similar.

Cheers,
David.
Go to the top of the page
+Quote Post
xmixahlx
post Jan 19 2004, 05:32
Post #19





Group: Members
Posts: 1394
Joined: 20-December 01
From: seattle
Member No.: 693



they adopted and improved hipp's mpglib and integrate for decoding

...and also made a hip interface (libmp3hip) for a straight decoding library


later


--------------------
RareWares/Debian :: http://www.rarewares.org/debian.html
Go to the top of the page
+Quote Post
amano
post Jan 19 2004, 05:52
Post #20





Group: Members
Posts: 483
Joined: 1-December 02
Member No.: 3949



oops, then the fault was on my side. Good to know, though.

If you are correct, these conclusons can be drawn from the test:
- Neither MAD. nor mpg123, nor FhG can outperform its competitors (one better here, the other better there)
- only a combination with dithering and noise shaping will offer signifivantly better quality.

Right?
Go to the top of the page
+Quote Post
outscape
post Jan 19 2004, 06:45
Post #21





Group: Members
Posts: 374
Joined: 29-September 01
From: Toronto
Member No.: 30



QUOTE (Rash @ Jan 18 2004, 10:54 PM)
Oh damn! I've always thought dithering was no good.

oh no, dithering is very important, especially when you reduce the resolution from 24 bits to 16 bits. it's consecutive re-dithering which is a questionable practice, especially when the source is only 16 bits. however, if we use mp3s as an example, if the bitrate is high enough, it may be possible to retrieve data above 16 bits, thus preserving more ambiance and detail which would surely be lost if we were to truncate at 16 bits. there is no guarantee, however, that you will hear a lot of difference or any difference at all. the side effect of re-dithering is more noise in the audio signal but this can be somewhat offset with proper noise shaping, although this is very subjective.


--------------------
Be healthy, be kind, grow rich and prosper
Go to the top of the page
+Quote Post
Gabriel
post Jan 19 2004, 08:17
Post #22


LAME developer


Group: Developer
Posts: 2950
Joined: 1-October 01
From: Nanterre, France
Member No.: 138



QUOTE
the side effect of re-dithering is more noise in the audio signal but this can be somewhat offset with proper noise shaping, although this is very subjective.


If the original dithering was noise shaped, it was mainly removed during mp3 encoding because of the applied lowpass filter.
Go to the top of the page
+Quote Post
AtaqueEG
post Jan 19 2004, 08:18
Post #23





Group: Members (Donating)
Posts: 1336
Joined: 18-November 01
From: Celaya, Guanajuato
Member No.: 478



QUOTE (amano @ Jan 18 2004, 10:52 PM)
oops, then the fault was on my side. Good to know, though.

If you are correct, these conclusons can be drawn from the test:
- Neither MAD. nor mpg123, nor FhG can outperform its competitors (one better here, the other better there)
- only a combination with dithering and noise shaping will offer signifivantly better quality.

Right?

Wrong.

Read this quote from an earlier post from guruboolez on this thread again:
QUOTE
Golden ears would probably perceive a difference between different decoders on real listening conditions. Common ears, like mine, need to push the volume to extreme position. Try, please... I was the first to be surprise to hear differences.


There are no significant differences of any kind.


--------------------
I'm the one in the picture, sitting on a giant cabbage in Mexico, circa 1978.
Reseñas de Rock en Español: www.estadogeneral.com
Go to the top of the page
+Quote Post
tigre
post Jan 19 2004, 10:22
Post #24


Moderator


Group: Members
Posts: 1434
Joined: 26-November 02
Member No.: 3890



I've done a similar test (~ 1 year ago I guess, but I can't find where I posted it sad.gif ):
  • Take some mp3s (3.90.2 --alt-preset standard IIRC)
  • Use mp3directcut to apply fadeouts (0 -> -40 dB IIRC)
  • Decode using foobar2000, winamp, xmplay
  • Apply fadeins opposite to the mp3directcut fadeouts (e.g. 0 -> +40 dB)
  • Listen + compare
IIRC the results were these:
- With Winamp (native decoder, not MAD) there was clearly audible truncation distortion
- With foobar2000 (I can't remember what dither type I used - I don't even know if there were options to choose) there was no distortion at all but a clearly audible noise floor (of course exagerated by amplification)
- XMplay's decoder was some kind of compromise: Less distortion than Winamp and lower audible noise level than fb2k

This way a similar test can be performed without need for very good (=low noise floor) equipment. It's still hard to tell how these results are related to listening under real life conditions though.

Edit: Seems like I never posted about the test unfortunately. Some time before (I think even before I knew foobar2000) I did the same test XMPlay (2.5) vs. MAD, the result is here
Later, I repeated it with Winamp 2.8x, fb2k 0.dunno and XMPlay 2.6 as described above - well, if anyone's really interested, it should be very easy to repeat ....

This post has been edited by tigre: Jan 19 2004, 10:37


--------------------
Let's suppose that rain washes out a picnic. Who is feeling negative? The rain? Or YOU? What's causing the negative feeling? The rain or your reaction? - Anthony De Mello
Go to the top of the page
+Quote Post
Garf
post Jan 19 2004, 12:10
Post #25


Server Admin


Group: Admin
Posts: 4853
Joined: 24-September 01
Member No.: 13



QUOTE (guruboolez @ Jan 19 2004, 04:20 AM)
I wonder why other audio players didn't use these techniques. Is it really hard to program? Dithering/noise shaping is very common on studio mastering: most of my CDs have visible noise shaping. So if studios use them, it must be good.

They slow down the player for something which doesn't have a benefit in casual listening. This is why it defaults to off in foobar too.
Go to the top of the page
+Quote Post

4 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th April 2014 - 19:24