What are some of the ways used to determine the quality of a particular decoder?
P. S. I'm posting this topic here according to greynol's advice.
Full Version: Quality of Decoders in Audio Players
Have the decoders in the most popular audio players (Winamp, foobar2000, AIMP, iTunes...) ever been tested and have these tests been posted online?
What are some of the ways used to determine the quality of a particular decoder?
P. S. I'm posting this topic here according to greynol's advice.
What are some of the ways used to determine the quality of a particular decoder?
P. S. I'm posting this topic here according to greynol's advice.
QUOTE (Sylph @ Nov 20 2008, 15:55) 
Have the decoders in the most popular audio players (Winamp, foobar2000, AIMP, iTunes...) ever been tested and have these tests been posted online?
What are some of the ways used to determine the quality of a particular decoder?
P. S. I'm posting this topic here according to greynol's advice.
What are some of the ways used to determine the quality of a particular decoder?
P. S. I'm posting this topic here according to greynol's advice.
There are several ways to test decoders. Use search and you will find plenty of topics about decoders.
I was actually wondering whether someone has tested the latest versions of these applications.
The reason that you won't find much in the way of testing of decoders is that a decoder pretty much either works or it doesn't. Writing decoder software is actually fairly trivial, at least compared to writing an encoder. The ultimate test of a decoder is that it decodes to pcm audio with an error of no more than one lsb.
Are any of the decoders that you mentioned broken? Not that I am aware of. Will any of them ever produce data that is audibly different from the others? Not very likely.
Are any of the decoders that you mentioned broken? Not that I am aware of. Will any of them ever produce data that is audibly different from the others? Not very likely.
Actually it would be interesting if someone would repeat a test like this: http://mp3decoders.mp3-tech.org/decoders.html for modern versions of decoders.
QUOTE (rednyrg721 @ Nov 21 2008, 02:49)
Actually it would be interesting if someone would repeat a test like this: http://mp3decoders.mp3-tech.org/decoders.html for modern versions of decoders.
We should ask 2Bdecided (author of those test results) if he has any later data.
QUOTE (pdq @ Nov 21 2008, 14:38)
QUOTE (rednyrg721 @ Nov 21 2008, 02:49)
Actually it would be interesting if someone would repeat a test like this: http://mp3decoders.mp3-tech.org/decoders.html for modern versions of decoders.
We should ask 2Bdecided (author of those test results) if he has any later data.
Is he a member of this message board?
QUOTE
Is he a member of this message board?
Yes and he is into audio research. He put forth the original ReplayGain proposal. I have read some of his white papers on a number of different topics before in the past. I highly recommend them for the technical minded.
I'm afraid this just not interesting. These tests --
1) http://mp3decoders.mp3-tech.org/decoders.html
2) http://www.underbit.com/resources/mpeg/audio/compliance/
were conducted at ~2000, when mp3 was very popular (Napster, Metallica, and so on).
1) http://mp3decoders.mp3-tech.org/decoders.html
2) http://www.underbit.com/resources/mpeg/audio/compliance/
were conducted at ~2000, when mp3 was very popular (Napster, Metallica, and so on).
QUOTE (pdq @ Nov 21 2008, 14:38)
We should ask 2Bdecided (author of those test results) if he has any later data.
I don't, sorry. I don't even have the files from those tests.They are simple enough to do, but quite time consuming unless you automate them (I didn't).
e.g. the "sound quality" part...
http://mp3decoders.mp3-tech.org/objective.html
explains how it was done...
QUOTE
A wave file was created containing music, noise, test tones, silence, dither etc. It was encoded by each encoder, giving 8 mp3 files. Each of these files was decoded by each of the decoders, yielding a total of 216 wave files, 27 from each mp3 file. Each decode from a particular mp3 file was compared with every other decode from that file, by taking the difference between the two files. This gives 729 comparisons per mp3 file, or 5832 in all! Thankfully, trends soon became apparent, and most of these could be skipped.
Examining how the various decodes compared, certain things became obvious. Some decoders gave results that didn't match any of the others. mp3 to wave v1.04 wouldn't synchronise with any of the other decoders, and it was found to be skipping samples. Only two decoders were found to be identical (CEP FhG and Winamp 2.22). Sonique 1.51 always skipped near the start of the file. The difference between l3dec and Winamp 2.22 was only 1-bit, a few times per second - both were clearly based on the same decoding algorithm, but rounding at different points. The difference between Ultra Player, lame, and the Winamp mpg123 plug-in was similar, indicating that these three also had a common origin to each other (mpg123). However, the difference between the l3dec group and the mpg123 group was consistently a 1 sample signal which sounded like the original signal (but obviously very much quieter!). Which one of these two groups is more "correct" it is impossible to say, but for simplicities sake, l3dec was chosen as a reference, and all the other decoders were judged against it. This comparison yields the results shown in the above table. Had lame been chosen as a reference, the straight 6 and 7 results would be reversed, but all others would remain the same.
FWIW I subsequently decided that the l3dec "family" was objectively more accurate than the mpg123 family, at least with the samples I could generate. mp3 encoding dramatically alters the signal, but by using simple signals and high bitrates, I found that the l3dec decode was closer to the original .wav than the mpg123 decode. (IIRC - it was a long time ago!). The difference between scoring "6" and "7" in the objective tests is all in the LSB, so this really is splitting hairs.Examining how the various decodes compared, certain things became obvious. Some decoders gave results that didn't match any of the others. mp3 to wave v1.04 wouldn't synchronise with any of the other decoders, and it was found to be skipping samples. Only two decoders were found to be identical (CEP FhG and Winamp 2.22). Sonique 1.51 always skipped near the start of the file. The difference between l3dec and Winamp 2.22 was only 1-bit, a few times per second - both were clearly based on the same decoding algorithm, but rounding at different points. The difference between Ultra Player, lame, and the Winamp mpg123 plug-in was similar, indicating that these three also had a common origin to each other (mpg123). However, the difference between the l3dec group and the mpg123 group was consistently a 1 sample signal which sounded like the original signal (but obviously very much quieter!). Which one of these two groups is more "correct" it is impossible to say, but for simplicities sake, l3dec was chosen as a reference, and all the other decoders were judged against it. This comparison yields the results shown in the above table. Had lame been chosen as a reference, the straight 6 and 7 results would be reversed, but all others would remain the same.
Cheers,
David.
Wow!!! That's what I call dedication. 
I did recent tests of foobar, MAD (via the in_mad plugin for Winamp), and Winamp.
http://www.hydrogenaudio.org/forums/index....c=64098&hl=
http://www.hydrogenaudio.org/forums/index....c=64098&hl=
Would it be a problem to test the others as well, possibly when the new version of foobar2000 and possibly Winamp comes out?
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.