Public MP3 Listening Test @ 128 kbps - FINISHED Options

[Sebastian Mares]

The much awaited results of the Public, MP3 Listening Test @ 128 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available in the next few days. You can also download the encryption key along with the submitted results on the results page that is located here: http://www.listening-tests.info/mp3-128-1/results.htm

The results show that all encoders are tied on first place, except l3enc which of course comes out last being the low anchor.

What is interesting to see is how the MP3 codec actually evolved since its first days (l3enc was the first MP3 software encoder back in 1994 when it was released) and how it is still competitive with newer formats like AAC or Ogg Vorbis.

Another very interesting thing, which was also one of the goals for this test, is that Fraunhofer and especially Helix, which both outperform LAME in terms of encoding speed, are still very competitive. While statistically being tied to LAME on first place, Helix actually even received a higher rating than LAME 3.98.2 - and this at 90x encoding speed! Even FhG received a slightly higher score at least against LAME 3.97 which was the recommended encoder by the Hydrogenaudio community for a long time. But again, statistically, they are all tied so there is no quality winner.



The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test. This was the last test conducted by me at this bitrate. It's time to move to bitrates like 96 kbps or 80 kbps.

Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor l3enc.



Finally, I would like to thank everyone who participated!

EDIT: Whoops, the link to the results was pointing to the 64 kbps multiformat test by mistake. Corrected now.

This post has been edited by Sebastian Mares: Nov 24 2008, 22:46

[Neasden]

They are all techincally tied, but Helix outperformed all of them. Also, the encoding speed compared to LAME is absurd faster. Could these two arguments qualify Helix for the new recommended MP3 encoder? (LAME being the second recommended)

[halb27]

... Could these two arguments qualify Helix for the new recommended MP3 encoder? ...

I've never been too happy with recommendations especially when it's about just one encoder.
I was especially unhappy with recommending Lame 3.97. There was also a listening test where Lame 3.97 came out great, with a bigger quality difference against the contenders compared to the more or less equal scores in this test as far as average score is concerned. It was after the test that 3.97's 'sandpaper problem' became known. The question is how to weigh it, the question is: how annoying is it for the person who reads the recommendation? It may be negligible, it may be a big issue.

The problem is that we can't test encoders on the universe of music. We can get significant experience with encoders, that's why Sebastian's test is important. But we should always take the results with a grain of salt.
There's also the question what kind of a result you have in focus. Usually people concentrate on the average result of an encoder averaged over all samples. But is this really the real thing which is most important? That's a very personal question. You can look at worst case behavior which is what I do in the first place. To me it's more important that my favorite encoder has a low number of scores below 4.0, and - at best - there is no sample with a score below 3.0. But this too has to be taken with a grain of salt. A bad score on an (to me) exotic sample doesn't count much to me, but it has a very high impact if it happens with music of my favorite genre. So evaluating an encoder is more than just looking at the average scores of a listening test.

Instead of giving a rather strong recommendation as was done so far I'd prefer if we had a weaker suggestion, kind of:
When targeting at a quality which can be achieved with ~128 kbps on average the most recent mp3 listening test has shown that the current versions of Lame, Helix, Fraunhofer, iTunes all do an excellent job. Quality differences between them were negligible within this test as far as the average outcome was concerned, with XXX and YYY having the best consistency in high quality (in case it turns out that such a statement can be made).

[Pio2001]

Excuse me, but what is the correspondance between the files and the encoders ?

[lvqcl]

Excuse me, but what is the correspondance between the files and the encoders ?

As shown on the pictures above -- samplexx_1.mp3 is encoded by iTunes, samplexx_2.mp3 - lame 3.98.2 etc. Of course, ABC/HR tool randomizes order of samples every time you load abc/hr config file (Samplexx.ecf)

Sorry, it's not clear to me how many subjects participated. Can you point me to that in the graph?

Downloaded results.rar:
39 - 26 - 26 - 27 - 30 - 26 - 26 - 26 - 26 - 26 - 27 - 26 - 29 - 30.