Multiformat Listening Test @ 128 kbps - FINISHED Options

[kwanbis]

could a ranking based on rating and bitrate be made? also as somebody already pointed, it would be good to include the version of the encoders used on the graphs, as many people would be linking to them.

edit:by the way, great job guys.

This post has been edited by kwanbis: Jan 15 2006, 00:13

[mdmuir]

When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

Choice for compatibilty:

1. lame

2. apple aac/nero aac

3. vorbis

4.wma pro-good codec, nothing but computer software plays it

[kwanbis]

When are we reaching the point that further tests in the future become superfluous? Just from looking at the results of this test, not one codec is significally better than another one. Can we safely say "stick to lame for universal compatibilty or take your pick for whatever your hardware device will suport"

i prety much agree ... ony thing is gapless option on some of them ...

[kwanbis]

[quote=Sebastian Mares,Jan 14 2006, 03:41 Can you also add more complete info of the encoder in the graph? e.g.: AoTuv -> Vorbis AoTuV 4.51, Nero -> AAC Nero 3.1.0.2, etc... I prefer first the format, then vendor and version. Many sites link only to the image and can give a confusing idea of which encoder was used. Also on that image should be a link to the full explanation of the results.

Well, if someone posts the image, he should also post to results page.
Adding the full encoder version / information is useless IMHO - it's stated already on the presentation page (which can be accessed if you are on the results page, that is supposed to be posted together with the plot).

i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.

[guruboolez]

When are we reaching the point that further tests in the future become superfluous?

At this bitrate, further tests are indeed questionable. Quality of the tested encoders is apparently too high for most listeners at ~130 kbps - at least for those interested by participating in such tests. The 192 kbps syndrom has now reached the 128 kbps area: it's beyond most listeners abilities, including HA.org members' one. At this stage, all people who can't differenciate MP3 from Vorbis or AAC and interested by these formats should try to lower the bitrate (I guess that it's already the case for many of them).
It's maybe the last 128 kbps multiformat collective test organized here. The next "mid/high" collective test should maybe lower the pretension and be limited to 100...112 kbps.

[Sebastian Mares]

i agree with fpi, i would add encoder, version, and probably command line. Even if i link to your page, people most of the time just look at the graph.

You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow.

This post has been edited by Sebastian Mares: Jan 15 2006, 00:27

[Halcyon]

Thanks for the results and work everyone!

I could only offer 6 results, not only because I fell ill, but because trying to spot the differences (assumed anchor excluded) it was really hard work!

It also showed me how some samples truly are more useful (at least for me).

The louder, more compressed and "busy" samples (like metal, rap and pop) were not nearly as useful for me as the acoustic and classical tracks. Not that they were easy either!

c. 128 kbps level has gone a long way in the past few years. I'm almost afraid to think when we'll reach the similar performance with 96kbps Not that I'm complaining, it's all for good.

This post has been edited by Halcyon: Jan 15 2006, 00:45

[kwanbis]

You know, space is limited. Writing "VBR/Stereo - Streaming, 100-120 kbps [LC AAC]" in the graph is pretty much overkill. Anyways, I will see what I can do tomorrow.

you can do better than this paint shop pro hack for sure, but its the idea:



This post has been edited by kwanbis: Jan 15 2006, 00:54

[vinnie97]

Vorbis' place at the top correlates with it being the highest bitrate.

Good results from everyone all around. Thanks for the test.

[sehested]

And BTW, this was a funny result:

CODE

ABC/HR for Java, Version 0.5b, 06 december 2005
Testname: DontLetMeBeMisunderstood
 
Tester: 
 
1R = Sample05\DontLetMeBeMisunderstood_1.wav
2L = Sample05\DontLetMeBeMisunderstood_2.wav
3L = Sample05\DontLetMeBeMisunderstood_3.wav
4L = Sample05\DontLetMeBeMisunderstood_6.wav
5R = Sample05\DontLetMeBeMisunderstood_4.wav
6R = Sample05\DontLetMeBeMisunderstood_5.wav
 
---------------------------------------
General Comments: Focus on 4.16 - 6.46
---------------------------------------
1L File: Sample05\DontLetMeBeMisunderstood.wav
1L Rating: 4.8
1L Comment: 
---------------------------------------
5L File: Sample05\DontLetMeBeMisunderstood.wav
5L Rating: 1.0
5L Comment: 
---------------------------------------
 
ABX Results:

I'm the culpit here

I started by listening to all encoders to locate the low anchor.
Apparently I pulled the wrong slider on sample 5, although I had it nailed.

Next time I better ABX the samples to lock the reference slider, in order to eliminate this kind of stupid "pressing the wrong button" mistakes.

I hope your are able to use my other test results, even though I'm positive that I on several occasions have rated the reference.

Anyway great work Sebastian.

[richard123]

Do these results help answer the question: Are these encoders transparent to most of those participating in the tests?

[westgroveg]

Does the results page state what settings where used for each encoder?

[Serge Smirnoff]

SoundExpert preliminary results on the same contenders are here. Alternative testing will end on 22 Jan. Details and discussion are in this thread.

[sehested]

By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.

Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?

I'm also interested in knowing how many result files have been discarded and for what reasons.

Previously discussions have taken place on criterias for discarding result files.

However I seem to have missed any discussions on the subject prior to this listening test.

How would the consolidated result look if you would have used results were the reference where rated slightly below 5.0?

In the AAC @ 128 kbps listening test only using "clean" files did not change the outcome of the test. Can the same be said for this test?

[Pio2001]

Thank you for the results. I was Anon08.
Here are my individual results. Every mark different from 5.0 has been validated with a successful ABX test.

CODE

Sample  AoTuV AAC-Itunes AAC-Nero Lame Shine WMApro
01      5.0   4.5        5.0      2.0  2.0   5.0  
03      5.0   5.0        5.0      2.0  2.0   5.0
04      5.0   5.0        5.0      5.0  2.0   5.0
05      5.0   5.0        3.0      5.0  1.0   5.0
06      5.0   5.0        5.0      5.0  1.0   5.0
07      5.0   5.0        5.0      5.0  3.0   5.0
08      5.0   4.0        4.0      3.0  1.5   4.5
10      5.0   5.0        5.0      5.0  2.0   5.0

I can tell that I dislike MP3

This post has been edited by Pio2001: Jan 15 2006, 05:48

[plonk420]

i only got thru 10 of the tracks.... but i could only ABX the diff on maybe 1 or 2 of the samples out of the 4 or 5 X 10 tracks i tried.....!! i blame my noisy DLP projector >_> (where i have my nice speakers setup) ;-\

[Raptus]

Pio, have you automated somehow the process of compiling this personal result table of yours?

[Raptus]

As I stated in the other topic, I was not very happy with the sample selection. Anyway I managed to descriminate more than average:

CODE

Sample iTunes LAME Nero Shine AoTuv WMA Pro
1      3,2    3,2  4    1     5     3,5
2      5      4,5  4    1     5     4,5
3      5      2,2  5    1,5   4     3,5
4      4      3,2  3,5  1     5     2,8
5      5      4    5    1,5   4     3,5
6      4,4    2,8  4    1,5   5     3
7      5      4    4,5  2,5   5     4,5
8      5      3,5  4    1     5     3,5
9      5      4    3,5  1     5     4
10     5      4    4    1,5   5     3,5
11     3,5    4    4    1,5   5     3,5
12     4,5    3,5  4    1,5   4,5   3,5
13     5      3,5  5    1     5     4
14     5      4    4,5  1     5     4
15     4,5    3,5  4    1     5     3,5
16     5      4    4    1     5     4
17     5      3    4    1     5     3,5
18     5      4    5    1     5     5

AVG    4,67   3,61 4,22 1,25  4,86  3,74

Congrats to the conducer and everyone who participated

EDIT: Corrected a number

This post has been edited by Raptus: Jan 15 2006, 09:49

[Sebastian Mares]

Does the results page state what settings where used for each encoder?

No, but on the presentation page.

By the way, results that were invalid (didn't meet ABX minimums) were not uploaded. Since I posted the encryption key, you can decrypt the results yourself if you are wondering why your result is not counted.

Sebastian,

Could you please elaborate on the criteria used for invalidating a result file?

If the results contain ranked references and no ABX logs, the results are invalid.

[sehested]

If the results contain ranked references and no ABX logs, the results are invalid.

Well that is new to me. I thought that slightly ranking a reference would result in a 5.0 rating, not invalidating the result file for that sample.

BTW: Were was this mentioned?

[Garf]

On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?

[sehested]

Is there a way to see your own results, including results for discarded result files?

[Sebastian Mares]

On the total results: how much % of the gradings gave a "transparent" mark, if we exclude shine?

The same question but adding the graded references as 5.0 for the codec in question?

Is there a way to see your own results, including results for discarded result files?

Sorry, I didn't understand both of you.

Anyways, I edited all graphs to show the encoder and its version. Additionally, I included a link to the full results page in the overall rankings and the zoomed plot. The encoder settings can be seen on the presentation page (which as I said can be reached from the results page).

Edit: sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.

This post has been edited by Sebastian Mares: Jan 15 2006, 11:04

[sehested]

sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.

Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?

[Sebastian Mares]

sehested, do you want to see the results you submitted or what? If you still have the encrypted files, you can decrypt them using ABC/HR and the encryption key I linked to on the results page.

Thanks, I will check it right away.

The other question I have is about the amount of result files that where discarded.

Can you give any numbers?

I am going to check ASAP.