Multiformat Listening Test @ 128 kbps - FINISHED Options

[Sebastian Mares]

The much awaited results of the Public, Multiformat Listening Test @ 128 kbps are ready.

Here is the results page: http://www.maresweb.de/listening-tests/mf-128-1/results.htm



Edit 1: A description of the mentioned Nero problem can be found here: http://www.maresweb.de/nero-problem

Edit 2: For people who want to decrypt their results, here are the encoder IDs:

1 = iTunes
2 = LAME
3 = Nero
4 = Shine
5 = AoTuV
6 = WMA Professiona

This post has been edited by Sebastian Mares: Jan 14 2006, 23:57

[minisu]

Great job!

*goes decrypting*

This post has been edited by minisu: Jan 14 2006, 22:06

[Ivan Dimkovic]

Just a quick info regarding the Nero bug - unfortunately it was found too late, as the encoder was given away couple of days before the test (*** too short ***) and it was completely "rushed in" to be ready for the test, even few months before the complete release. In the whole "rush" process, the bug was overlooked in the internal Nero Digital Audio QA as well by the external people testing the codecs (Guru and few others)

This bug reflects quality in the unpredictable way (as it does not allocate bits according to the psychoacoustics but drains the bit reservoir) - but in general we believe that the quality difference would not be significant (in fact it is my belief that it would be better without the bug as the extra bits would be allocated according to psychoacoustic model) - however Sebastian decided to exclude the Nero codec from the test, which is IMO unfortunate, but I can understand his decision.

Fortunately, the bug has been fixed (thanks to Guruboolez and his very good hearing) - and this kind of behavior will be included in pre-test screening of the codecs to seek for such obvious bugs.

I am dissapointed that this bug was found out too late, but hopefully for the next tests we won't be having such problems.

This post has been edited by Ivan Dimkovic: Jan 14 2006, 22:14

[skelly831]

Wow! awesome results, it's interesting to see how close iTunes and AoTuv are in the overall rating but they're somewhat disparate in the bitrate table.

[ff123]

How about making a hyperlink to the page describing the nero bug? Good explanation of what went wrong.

ff123

[Sebastian Mares]

By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.

[guruboolez]

Great job to everyone who participated to the test - especially to the conducer!

The first immediate lesson of this test is that ~132...135 kbps main encoders are very, very good. Even LAME at -V5 is close to transparency. I hope that -V2/--preset standard will progressively cease to be the automatic recommendation when new members are asking for good quality MP3 encodings: the first step to excellency is below the historical presets!

Developers must be celebrated: Apple's and Nero's developers for leading AAC to the best places; Aoyumi for having resurrecting Vorbis; Microsoft's developers for -at last- offering a very good encoding and free encoding tool; and of course the whole LAME team who are making MP3 better and better even if people are sometimes not realizing it or believing that MP3 couldn't be improved!

Last, people could compare the collective results to my individual test. I've conclude on aoTuV and iTunes superiority, with both Nero and LAME slightly lower. The group results are concluding on exactly the same order, with only less significance and an higher notation. It looks that my hearing thus my listening tests may be very representative from the group's one

This post has been edited by guruboolez: Jan 15 2006, 01:35

[Sebastian Mares]

By the way, Francis noticed another problem. The sample "Yello" submitted by Alex B was not lossless. By mistake, Alex B uploaded a file transcoded from a high bitrate MP3.

Guru noticed that the reference file he was listening to sounded too much like an encoded file. After looking at the spectral view of the track with CE, he noticed what he calls "an adaptive lowpass with some spectral 'holes'" which is typical to lossy encoders. I contacted Alex B about the problem and received the following mail today:

Hi again!

Here's what I found. It is a long story, but explains how accidents can happen when dealing with a large number of test files, even if you try to [be] careful.

I checked the reference and compared it with my ripped CD archive file, which is in Monkey's Audio disc image file & cue format and noticed that there really is a difference.

However, at first I couldn't understand why. It could not be the WMA 2-pass version I made for testing the bitrates. WMA 128 kbps has a lower lowpass. I have also a Musepack Q8 version on my home audio server, but that does not have such a lowpass.

Then I remembered that I cut the sample from the 25 files bitrate test archive I have stored. I have the separated track already there so it was faster than converting the big disc image ape file.

I selected these 25 files originally for testing LAME 3.97 VBR bitrates in September. When I made the different VBR sets I had the original cue files loaded in foobar and I converted each VBR set from the same playlist. Later when I cleaned the resulting 500 MP3 files I converted the 25 original cue tracks to separate lossless tracks for future use.

It appears that I have somehow accidentally loaded a high bitrate MP3 file instead of the cue to a foobar playlist and converted it to Monkey's Audio. The other 24 Monkey's Audio tracks seem to be fine, only this one is unfortunately different.

When I made this particular sample for testing WMA 2-pass in November I converted that ape file to wave and cut it. By looking the lowpass I believe the MP3 file was encoded at -V0. There is no way to tell if it was VBR new or old.

I am sorry about the mistake.

I think this sample will accidentally show that these new encoders are not bad transcoders, at least when the source is this kind of loud track. It is a different thing if you like to publish this information. It could be only meaningless clutter. It was never exactly mentioned how the source files were obtained. The test compared these reference files and the encoded versions.

This post has been edited by Sebastian Mares: Jan 14 2006, 22:38

[Shade[ST]]

The URL in the image is incorrect (www.maresweb.de/nero-problem.txt) -- might as well make the redirect at /nero-problem, too, anyways...

In any case, It's a shame I misunderstood the testing scenario : I did not know I needed to ABX the codecs before rating them :-/

All my results are invalid :-(

However, it's surprising to learn that all popular codecs around 128 kbps are of high enough quality to be practically imperceptible to everyone.

Great!

edit : de / net, whatever.

This post has been edited by Shade[ST]: Jan 14 2006, 22:41

[Lyx]

Pointing out an old issue with listening-test presentations: neither the plot, nor the detailed results page, mention clearly that "iTunes" means "iTunes AAC Encoder". Taking into account that results of ha.org listening tests are often posted elsewhere without all the necessary info, people may once more mistake iTunes (good) AAC performance with its (bad) MP3 performance.

[Sebastian Mares]

The URL in the image is incorrect (www.maresweb.net/nero-problem.txt) -- might as well make the redirect at /nero-problem, too, anyways...

maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

In any case, It's a shame I misunderstood the testing scenario : I did not know I needed to ABX the codecs before rating them :-/

All my results are invalid :-(

However, it's surprising to learn that all popular codecs around 128 kbps are of high enough quality to be practically imperceptible to everyone.

Great!

No, no, no... You don't have to ABX all files. ABX logs are only required when you ranked a reference.
IIRC, I used a large part of your results.

And BTW, this was a funny result:

CODE

ABC/HR for Java, Version 0.5b, 06 december 2005
Testname: DontLetMeBeMisunderstood

Tester:

1R = Sample05\DontLetMeBeMisunderstood_1.wav
2L = Sample05\DontLetMeBeMisunderstood_2.wav
3L = Sample05\DontLetMeBeMisunderstood_3.wav
4L = Sample05\DontLetMeBeMisunderstood_6.wav
5R = Sample05\DontLetMeBeMisunderstood_4.wav
6R = Sample05\DontLetMeBeMisunderstood_5.wav

---------------------------------------
General Comments: Focus on 4.16 - 6.46
---------------------------------------
1L File: Sample05\DontLetMeBeMisunderstood.wav
1L Rating: 4.8
1L Comment:
---------------------------------------
5L File: Sample05\DontLetMeBeMisunderstood.wav
5L Rating: 1.0
5L Comment:
---------------------------------------

ABX Results:

[Shade[ST]]

Also a question / comment. Is Itunes' encoder actually in iTunes, or is it in quicktime? What versions were used? Maybe you should list those on the results page (versions of each encoder..)

[Shade[ST]]

CODE

Testname: DontLetMeBeMisunderstood
Tester:  

1R = Sample05\DontLetMeBeMisunderstood_1.wav
2L = Sample05\DontLetMeBeMisunderstood_2.wav
3L = Sample05\DontLetMeBeMisunderstood_3.wav
4L = Sample05\DontLetMeBeMisunderstood_6.wav
5R = Sample05\DontLetMeBeMisunderstood_4.wav
6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
---------------------------------------
General Comments: Focus on 4.16 - 6.46
---------------------------------------
1L File: Sample05\DontLetMeBeMisunderstood.wav
1L Rating: 4.8
1L Comment:  
---------------------------------------
5L File: Sample05\DontLetMeBeMisunderstood.wav
5L Rating: 1.0
5L Comment:  
---------------------------------------

Wuups.. Thanks for hiding my tester name

This post has been edited by Shade[ST]: Jan 14 2006, 22:45

[minisu]

Ok, so this is one of my decrypted results... http://www.maresweb.de/listening-tests/mf-...12-result02.txt

How to know which encoders that matches which sample number?

This post has been edited by minisu: Jan 14 2006, 22:46

[Shade[ST]]

More comments : http://www.maresweb.de/listening-tests/mf-128-1/ Maybe you should update this page to point to the results.

[Sebastian Mares]

Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

Well, they're on the presentation page which I think is enough.

CODE

Testname: DontLetMeBeMisunderstood
Tester: 

1R = Sample05\DontLetMeBeMisunderstood_1.wav
2L = Sample05\DontLetMeBeMisunderstood_2.wav
3L = Sample05\DontLetMeBeMisunderstood_3.wav
4L = Sample05\DontLetMeBeMisunderstood_6.wav
5R = Sample05\DontLetMeBeMisunderstood_4.wav
6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
---------------------------------------
General Comments: Focus on 4.16 - 6.46
---------------------------------------
1L File: Sample05\DontLetMeBeMisunderstood.wav
1L Rating: 4.8
1L Comment: 
---------------------------------------
5L File: Sample05\DontLetMeBeMisunderstood.wav
5L Rating: 1.0
5L Comment: 
---------------------------------------

Wuups.. Thanks for hiding my tester name

I didn't say it was your result. Or to be more precise, it wasn't your result. It was from anonymous user.

Ok, so this is one of my decrypted results... http://www.maresweb.de/listening-tests/mf-...12-result02.txt

How to know which encoders that matches which sample number?

1 = iTunes
2 = LAME
3 = Nero
4 = Shine
5 = AoTuV
6 = WMA Professional

[Sebastian Mares]

More comments : http://www.maresweb.de/listening-tests/mf-128-1/ Maybe you should update this page to point to the results.

If you go one lever higher, you come to the listening tests page which lists all tests as pairs of presentation page and results page. AFAIK, Roberto does it the same way, too. But no problem, I can edit that page when the FTP server works again.

[fpi]

maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

You missed the .de on the graph.
Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.

This post has been edited by fpi: Jan 14 2006, 23:29

[IgorC]

First of all great test. Thanks, Sebastian.

However it's sad what happens with Nero encoder.
Maybe it will be a fair idea to cut 2/3 of encoded samples and see how much size of nero's samples is bigger. However it's not correctly 100% because of VBR distribution. I've already learned that the bigger/smaller size is not always the indicator of quality . However if distribution 150-130 (140) kbps is a good idea to check it. 10 kbit/s isn't issue 7% of total bitrate. It was admited that Itunes had 10 kbit/s extra of real bitrate that's ok because of VBR.

However I think it was right decesion to keep us informed about this issue.

This post has been edited by IgorC: Jan 14 2006, 23:31

[Sebastian Mares]

Ah, damn, you're right. Now I see what you guys mean. OK, going to PSP the graphs again... >_<

[Shade[ST]]

,Jan 14 2006, 10:42 PM]Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

Well, they're on the presentation page which I think is enough.

Actually, nowhere is that page linked on here... Also, If you wish, I can set up a nice 'design' image to show the results; Could you post the numbers (ranges), PM them to me or email them? I'll also include the necessary links and info in the image. And I can give you a PDF format of it, if you like.

[guruboolez]

I've already learned that the bigger/smaller size is not always the indicator  of quality 

That's right. But in this case, a careful listening reveals (revealed in my case) that this bug has an audible impact, leading to a brutal drop in perceived quality. The impact on quality varies from nothing to considerable.

However if distribution 150-130 (140) kbps is a good idea to check it. 10 kbit/s isn't issue 7% of total bitrate.  It was admited that Itunes had 10 kbit/s extra of real bitrate  that's ok because of VBR.

Even if bitrate stays within the tolerence of 10%, there's still the problem of unrepresentativity (the same one which decided to not use WMA Std 2-pass with short samples). The tested samples content and quality is different from what a user would get.

[JeanLuc]

Damn ... looking at the bitrate table and regarding the overall ranking, iTunes AAC seems so effective.

It is good to see, though that users can today chose between different formats at comparable bitrates without having to ask about a possible sacrifice in quality.

[IgorC]

there's still the problem of unrepresentativity (the same one which decided to not use WMA Std 2-pass with short samples). The tested samples content and quality is different from what a user would get.

yes , I had fear to it.

[Sebastian Mares]

maresweb.de is the correct domain. .net and .org are only redirecting to the .de domain. Also, I tested the URL from the picture and Apache redirects the users to the correct file. No problem I guess.

You missed the .de on the graph.
Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.

Well, if someone posts the image, he should also post to results page.
Adding the full encoder version / information is useless IMHO - it's stated already on the presentation page (which can be accessed if you are on the results page, that is supposed to be posted together with the plot).

,Jan 14 2006, 11:31 PM]

,Jan 14 2006, 10:42 PM]Also a question / comment.  Is Itunes' encoder actually in iTunes, or is it in quicktime?  What versions were used?  Maybe you should list those on the results page (versions of each encoder..)

Well, they're on the presentation page which I think is enough.

Actually, nowhere is that page linked on here... Also, If you wish, I can set up a nice 'design' image to show the results; Could you post the numbers (ranges), PM them to me or email them? I'll also include the necessary links and info in the image. And I can give you a PDF format of it, if you like.

Sent! Thanks.

BTW, I only changed iTunes and Nero to iTuness AAC and Nero AAC on the final plots. The sample plots still have iTunes only. I hope it's not such a big deal since those images are almost never posted alone.

That's it guys - a warm bed is waiting for me.

BTW... Gambit is screwed if his neighbors read his comments.

This post has been edited by Sebastian Mares: Jan 15 2006, 00:03