IPB

Welcome Guest ( Log In | Register )

9 Pages V   1 2 3 > »   
Reply to this topicStart new topic
Public MP3 Listening Test @ 128 kbps - FINISHED
Sebastian Mares
post Nov 24 2008, 22:30
Post #1





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



The much awaited results of the Public, MP3 Listening Test @ 128 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available in the next few days. You can also download the encryption key along with the submitted results on the results page that is located here: http://www.listening-tests.info/mp3-128-1/results.htm

The results show that all encoders are tied on first place, except l3enc which of course comes out last being the low anchor.

What is interesting to see is how the MP3 codec actually evolved since its first days (l3enc was the first MP3 software encoder back in 1994 when it was released) and how it is still competitive with newer formats like AAC or Ogg Vorbis.

Another very interesting thing, which was also one of the goals for this test, is that Fraunhofer and especially Helix, which both outperform LAME in terms of encoding speed, are still very competitive. While statistically being tied to LAME on first place, Helix actually even received a higher rating than LAME 3.98.2 - and this at 90x encoding speed! Even FhG received a slightly higher score at least against LAME 3.97 which was the recommended encoder by the Hydrogenaudio community for a long time. But again, statistically, they are all tied so there is no quality winner.



The quality at 128 kbps is very good and MP3 encoders improved a lot since the last test. This was the last test conducted by me at this bitrate. It's time to move to bitrates like 96 kbps or 80 kbps.

Here is a zoomed version of the plot showing the competitors only and leaving out the low anchor l3enc.



Finally, I would like to thank everyone who participated!

EDIT: Whoops, the link to the results was pointing to the 64 kbps multiformat test by mistake. Corrected now.

This post has been edited by Sebastian Mares: Nov 24 2008, 22:46


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
/mnt
post Nov 24 2008, 22:52
Post #2





Group: Members
Posts: 696
Joined: 22-April 06
Member No.: 29877



Wow am really shocked, Helix (Xing) has performed well. Not only did GNR's new album came out; Xing (Helix) has outperformed LAME; Hell most be pretty cold now biggrin.gif

This post has been edited by /mnt: Nov 24 2008, 22:55


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
Sebastian Mares
post Nov 24 2008, 22:56
Post #3





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



I kept telling you guys that the results will be quite surprising. biggrin.gif If you analyze the decrypted results, you will see that for at least one sample, Helix is even statistically better than all other encoders.

This post has been edited by Sebastian Mares: Nov 24 2008, 22:57


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
Neasden
post Nov 24 2008, 23:00
Post #4





Group: Banned
Posts: 185
Joined: 1-July 08
Member No.: 55148



Does that make Helix the new recommended MP3 encoder, or has it to be LAME because it's open source?
Edit: Both are open source.

This post has been edited by Neasden: Nov 24 2008, 23:15
Go to the top of the page
+Quote Post
/mnt
post Nov 24 2008, 23:05
Post #5





Group: Members
Posts: 696
Joined: 22-April 06
Member No.: 29877



QUOTE (Neasden @ Nov 24 2008, 23:00) *
Does that make Helix the new recommended MP3 encoder, or has it to be LAME because it's open source?

Helix is open source aswell.


--------------------
"I never thought I'd see this much candy in one mission!"
Go to the top of the page
+Quote Post
Neasden
post Nov 24 2008, 23:14
Post #6





Group: Banned
Posts: 185
Joined: 1-July 08
Member No.: 55148



Yes it is, I just noticed it!
I can't believe how fast it is... here encodes at 33x realtime. LAME fastest speed here is 12x.
Foobar2000 parameters: -X2 -U2 -V150 -HF - %d
Would that be equivalent to LAME -V0 ?
Go to the top of the page
+Quote Post
greynol
post Nov 24 2008, 23:14
Post #7





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



I don't think open source has anything to do with it.


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
Sebastian Mares
post Nov 24 2008, 23:14
Post #8





Group: Members
Posts: 3629
Joined: 14-May 03
From: Bad Herrenalb
Member No.: 6613



If you submitted results, I recommend you look at them and choose your encoder of choice based on that. smile.gif


--------------------
http://listening-tests.hydrogenaudio.org/sebastian/
Go to the top of the page
+Quote Post
halb27
post Nov 24 2008, 23:22
Post #9





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



I am curious about the detailed results as my interest is in worst case behavior in the first place.
I guess Lame 3.98.2 and Helix will be the winners in this respect too, maybe quality difference towards the contenders will be more remarkable in this field.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Neasden
post Nov 24 2008, 23:25
Post #10





Group: Banned
Posts: 185
Joined: 1-July 08
Member No.: 55148



They are all techincally tied, but Helix outperformed all of them. Also, the encoding speed compared to LAME is absurd faster. Could these two arguments qualify Helix for the new recommended MP3 encoder? (LAME being the second recommended)
Go to the top of the page
+Quote Post
Canar
post Nov 24 2008, 23:51
Post #11





Group: Super Moderator
Posts: 3327
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



Thank you very much Sebastian. We have some things to carefully consider, I see.


--------------------
∑:<
Go to the top of the page
+Quote Post
DigitalDictator
post Nov 24 2008, 23:56
Post #12





Group: Members
Posts: 313
Joined: 9-August 02
From: SoFo
Member No.: 3002



This is indeed surprising. I'm sure I've seen smaller, recent, ABX-tests where Lame has outperformed Helix quite clearly. I think Guruboolez and maybe also Halb27 have done a couple, but I might be mistaken.

IIRC Helix has a very simple code. If it is open source, can it be tuned further by third party? The latest version is from 2005, or no?
Go to the top of the page
+Quote Post
ZinCh
post Nov 24 2008, 23:59
Post #13





Group: Members
Posts: 171
Joined: 28-September 06
Member No.: 35705



This is only 128k tests, so Helix in the winner in this niche. More sets needed smile.gif

Helix can be recomended for 128k encoding.
Go to the top of the page
+Quote Post
Neasden
post Nov 25 2008, 00:07
Post #14





Group: Banned
Posts: 185
Joined: 1-July 08
Member No.: 55148



I encoded a few tracks using -V150 (VBR range 0-150) with -HF (high frequencies encoding) enabled. The average bitrate goes up to 270 kbps. That would be equivalent to LAME -V0. People find -V0 and -V2 already excessive, and there is this ghost-case about the sbf21 bloat... many are -V3 advocates. Helix is a simpler encoder indeed with fewer switch options, but isn't its quality and speed an outstanding alert?
Go to the top of the page
+Quote Post
greynol
post Nov 25 2008, 00:10
Post #15





Group: Super Moderator
Posts: 10000
Joined: 1-April 04
From: San Francisco
Member No.: 13167



>Helix can be recomended for 128k encoding.

How can you say this when all the contenders were tied within the 95% margin of confidence?

Some other things to consider before leaping to such a conclusion:
- How many participants were there?
- Did Helix consistently score higher amongst all participants?

This post has been edited by greynol: Nov 25 2008, 00:13


--------------------
Your eyes cannot hear.
Go to the top of the page
+Quote Post
halb27
post Nov 25 2008, 00:11
Post #16





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (DigitalDictator @ Nov 25 2008, 00:56) *
I'm sure I've seen smaller, recent, ABX-tests where Lame has outperformed Helix quite clearly. I think Guruboolez and maybe also Halb27 have done a couple, but I might be mistaken. ...

Sorry: As far as I am concerned I didn't do a recent ABX test Helix vs. Lame 3.98.
A few years back after level and others stating Helix' remarkable quality in the 200 kbps area I did ABX Helix and some other encoders and I valued Helix' robustness against artifacts in this bitrate area very high. There were some more or less (to me) negligible HF issues however which Wombat found and I also found some cases with a subtle but ABXable lack of 'vividness' (don't know how to describe it). All these things relevant only in the very high bitrate range if at all. But as I don't care about bitrate much and as I was happy with Lame ABR 270 (3.90.3 then) I sticked with Lame with my mp3 encodings.


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Pio2001
post Nov 25 2008, 00:34
Post #17


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



AAAAAAAAAAAARRRRRRRRGGGGHGHHHHHH !!

They're all tied crying.gif

How can it be ? I ABXed nearly all the samples that I submitted, and there were important differences in quality between them. Could it be that for every sample, the best to worst order was different ? Or were there too many tied submissions ?

I must check my personal results right now !

Oh, and Greynol is right. Helix is not winner. It is tied. The differences are within the confidence intervals, which means that they are just random. If you redid the test with the same samples and same listeners, the simple fact the ABC/HR presents them in a different order every time would probably lead Lame, or Fraunhofer, or iTunes to get a slightly, but not significantly, superior score.

We must consider this to be chance, unless we have more information to backup further claims.
Go to the top of the page
+Quote Post
kwanbis
post Nov 25 2008, 00:42
Post #18





Group: Developer (Donating)
Posts: 2353
Joined: 28-June 02
From: Argentina
Member No.: 2425



Wow (even if the difference between LAME 3.98.2 and Helix is 0.08 and knowing that both (all) are statistically tied.)

EDIT: Congratulations to Sebastian for conducting the test.

This post has been edited by kwanbis: Nov 25 2008, 01:03


--------------------
MAREO: http://www.webearce.com.ar
Go to the top of the page
+Quote Post
halb27
post Nov 25 2008, 00:43
Post #19





Group: Members
Posts: 2414
Joined: 9-October 05
From: Dormagen, Germany
Member No.: 25015



QUOTE (Neasden @ Nov 25 2008, 00:25) *
... Could these two arguments qualify Helix for the new recommended MP3 encoder? ...

I've never been too happy with recommendations especially when it's about just one encoder.
I was especially unhappy with recommending Lame 3.97. There was also a listening test where Lame 3.97 came out great, with a bigger quality difference against the contenders compared to the more or less equal scores in this test as far as average score is concerned. It was after the test that 3.97's 'sandpaper problem' became known. The question is how to weigh it, the question is: how annoying is it for the person who reads the recommendation? It may be negligible, it may be a big issue.

The problem is that we can't test encoders on the universe of music. We can get significant experience with encoders, that's why Sebastian's test is important. But we should always take the results with a grain of salt.
There's also the question what kind of a result you have in focus. Usually people concentrate on the average result of an encoder averaged over all samples. But is this really the real thing which is most important? That's a very personal question. You can look at worst case behavior which is what I do in the first place. To me it's more important that my favorite encoder has a low number of scores below 4.0, and - at best - there is no sample with a score below 3.0. But this too has to be taken with a grain of salt. A bad score on an (to me) exotic sample doesn't count much to me, but it has a very high impact if it happens with music of my favorite genre. So evaluating an encoder is more than just looking at the average scores of a listening test.

Instead of giving a rather strong recommendation as was done so far I'd prefer if we had a weaker suggestion, kind of:
When targeting at a quality which can be achieved with ~128 kbps on average the most recent mp3 listening test has shown that the current versions of Lame, Helix, Fraunhofer, iTunes all do an excellent job. Quality differences between them were negligible within this test as far as the average outcome was concerned, with XXX and YYY having the best consistency in high quality (in case it turns out that such a statement can be made).


--------------------
lame3100m --bCVBR 300
Go to the top of the page
+Quote Post
Pio2001
post Nov 25 2008, 00:48
Post #20


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Excuse me, but what is the correspondance between the files and the encoders ?
Go to the top of the page
+Quote Post
krabapple
post Nov 25 2008, 01:02
Post #21





Group: Members
Posts: 2159
Joined: 18-December 03
Member No.: 10538



Sorry, it's not clear to me how many subjects participated. Can you point me to that in the graph?
Go to the top of the page
+Quote Post
lvqcl
post Nov 25 2008, 01:16
Post #22





Group: Developer
Posts: 3214
Joined: 2-December 07
Member No.: 49183



QUOTE (Pio2001 @ Nov 25 2008, 02:48) *
Excuse me, but what is the correspondance between the files and the encoders ?

As shown on the pictures above -- samplexx_1.mp3 is encoded by iTunes, samplexx_2.mp3 - lame 3.98.2 etc. Of course, ABC/HR tool randomizes order of samples every time you load abc/hr config file (Samplexx.ecf)

QUOTE (krabapple @ Nov 25 2008, 03:02) *
Sorry, it's not clear to me how many subjects participated. Can you point me to that in the graph?

Downloaded results.rar:
39 - 26 - 26 - 27 - 30 - 26 - 26 - 26 - 26 - 26 - 27 - 26 - 29 - 30.
Go to the top of the page
+Quote Post
Pio2001
post Nov 25 2008, 01:37
Post #23


Moderator


Group: Super Moderator
Posts: 3936
Joined: 29-September 01
Member No.: 73



Thanks, I analyzed my own results. That's what I feared. The ranking of the encoders is different for each sample. A Tukey HSD analysis on my ratings give them all tied too, except the low anchor.

Thanks again to Sebastian !
Go to the top of the page
+Quote Post
Raiden
post Nov 25 2008, 02:12
Post #24





Group: Developer
Posts: 224
Joined: 14-September 04
Member No.: 17002



QUOTE (Pio2001 @ Nov 25 2008, 01:34) *
Oh, and Greynol is right. Helix is not winner. It is tied. The differences are within the confidence intervals, which means that they are just random. If you redid the test with the same samples and same listeners, the simple fact the ABC/HR presents them in a different order every time would probably lead Lame, or Fraunhofer, or iTunes to get a slightly, but not significantly, superior score.

We must consider this to be chance, unless we have more information to backup further claims.

agreed. Probably the zoomed plot window should be removed, since it's quite misleading and doesn't show any useful information.
Go to the top of the page
+Quote Post
Canar
post Nov 25 2008, 02:32
Post #25





Group: Super Moderator
Posts: 3327
Joined: 26-July 02
From: princegeorge.ca
Member No.: 2796



QUOTE (Raiden @ Nov 24 2008, 17:12) *
agreed. Probably the zoomed plot window should be removed, since it's quite misleading and doesn't show any useful information.
Not really. It's just a zoomed version of the previous data. It is not misleading, it just represents the same data as the other graph and text in a slightly different form.


--------------------
∑:<
Go to the top of the page
+Quote Post

9 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 21st April 2014 - 09:18