Multiformat Listening Test @ 64 kbps - FINISHED Options

[Rio]

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

[naylor83]

Stupid question alert:

If I ranked the reference, will the result text file say so? Or will it just not show a result for that file?

[Sebastian Mares]

The decrypted result files will then contain the rating you gave for the reference.

Edit: It will look like this:

[...]
2L File: Sample08\Sample08.wav
2L Rating: 4.5
2L Comment: blah
[...]

This post has been edited by Sebastian Mares: Aug 16 2007, 16:00

[pdq]

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

I would say rather that A>C and the B is approximately equal to A and approximately equal to C, but is not necessarily A>B>C since there is a possibility that either B>A or B<C (but not both).

[benski]

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

I suggest it would be politically (and mathematically) correct that it is like if A>B and B>C then A>C.

No.

There is a chance that A>B but also a chance that A<B.
There is a chance that B>C but also a chance that B<C.
A>C


To rank them, A and B are tied for first. C is third.
Given the data set, the "true" rank has three possibilities. ABC, BAC, ACB. However, more samples would be necessary to determine this.

One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.

This post has been edited by benski: Aug 16 2007, 16:15

[naylor83]

Stupid question alert (again):

I'm trying to work out which samples are which contenders.

I realize number 3 is Vorbis, and that number 4 must be low anchor. But I'm confused about the others...

[guruboolez]

You can use MrQuestionMan, foobar2000 or several other tools to check these files :

1: WMAPro (losslessly compressed due to the lack of WMA CLI decoder)
2: high anchor (iTunes LC-AAC at ~100 kbps)
3: vorbis (ogg fileformat)
4: low anchor (iTunes LC-AAC at 48 kbps)
5: HE-AAC (Nero Digital AAC at ~64 kbps).

This post has been edited by guruboolez: Aug 16 2007, 16:19

[naylor83]

1: WMAPro (losslessly compressed)
2: high anchor (LC-AAC at 96 kbps)
3: vorbis (ogg fileformat)
4: low anchor (LC-AAC at 48 kbps)
5: HE-AAC

Thanx.

[ff123]

One thing I've always disliked about these tests is that, given the subjective nature of the ratings, the deviation in participants' rating style is likely larger than the standard deviation.

In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

[Whelkman]

In the analysis, each listener is treated as a separate "block", which takes into account the fact that different listeners have individual rating styles.

Thanks. I wondered about this. I doubt I applied consistent "objective" ratings across the board, but codecs were always ranked compared to each other.

[Sebastian Mares]

Does anyone know how to make Excel to refer to the current table when creating a plot? I have a document with 19 tables and I thought about plotting the results for the first sample and then copying and pasting this in the other 17 documents and then only changing the values. However, if I copy and paste a plot, the pasted plots still refer to the source table. Then if I change the data source, some of the plot formatting is gone, such as the margins, the vertical grid and the grid color.

[Sebastian Mares]

Uploaded the plots for each sample. The corresponding text is still missing, though, although there isn't much to say since all three were tied in almost every case.

Off-Topic: That listening test page needs rework badly. The design could be better and maybe offer some help for newbies.

[mezenga]

Does anyone know how to make Excel to refer to the current table when creating a plot?

Maybe joining all 19 tables in a big one and making a single one for the plot. This single table should change its content among one of the 19 blocks from the big table. That would be my approach for a dynamic plot.

This post has been edited by mezenga: Aug 17 2007, 00:19

[ff123]

Interesting. he-aac had some clear winners over wmapro10, whereas there were none the other way around. Poets of the fall and Bachpsichord are particularly striking. Choice of samples is pretty critical in these tests.

[echo]

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

not an expert, but at leas mathematically if A=B and B=C, A=C.

Mathematically yes, but this is not math, this is statistics.

To put it in simple terms, without any statistical talk, this means that A is probably equal to B, B is probably equal to C, while A is greater than C. Try to think "equal" like "roughly equal".

[rockcake]

I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend!

[TechVsLife]

I'd also like to give a big thankyou to Sebastian for organising another test (and publishing the results amazingly quickly!), especially under difficult circumstances e.g. HDD failure, widespread apathy, moving house etc. etc. You're a legend!

Or does such superhuman generosity border on insanity? Is his undying fame worth the terrible price he pays--with his very life etc. (Life itself is a 64 kbps lossy compression where you have to pick carefully what to carry to get to a half-decent harmony, but discerning ears will always be able to pick up the falseness, especially in critical passages.)


But seriously, thanks for the hard work, even if insane,
--and how about the next test! (128 kbps mp3?).

This post has been edited by TechVsLife: Aug 17 2007, 07:44

[IgorC]

Thanks for test. Nero has done a good work.

[Alexxander]

Congrat Nero!

I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

[halb27]

... I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

It's rather the other way around. The beleive in VBR's universal superiority has simply no good basis. Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR. But even without it there's really no reason to think that constant bitrate automatically means reduced quality.
There's no contradiction to the fact that Vorbis, NeroAAC, MPC, Lame 3.98 are good at VBR.
Everything depends on codec principles and - may be to a larger extent - implementation details.

This post has been edited by halb27: Aug 17 2007, 10:28

[muaddib]

Congrat Nero!
I can't believe WMA Pro 10 is true CBR because it has good results compared to the VBR samples. If it really is there would be room for improvement (by going VBR)

Thanks!

Considering CBR: CBR is more dependent on choice of samples. It is expected that Nero would perform on this sample set a bit better when CBR 64kbps is used (most probably not enough to be statistically better than WMA). From this test it can also be concluded that VBR mode in Nero doesn't have big flaws.

[Alexxander]

...Moreover there seems to be a common misconception that a constant frame bitrate (CBR) means constant audio data bitrate which is simply wrong. Maybe WMP10pro CBR offers a higher degree of audio data bitrate variation than for instance mp3 CBR...

So CBR actually means constant frame bitrate? I thought CBR referred to constant audio data bitrate, like plain old PCM: for example sampling 8000 times per second at fixed intervals with 8 bits per sample. Then, if frame bitrate is constant but audio bitrate varies within a frame it's actually VBR but only on a different timescale. It all depends on the exact definitions and the correct use of terms (as always).

Thanks for clearing up.

[Ivan Dimkovic]

CBR, in this context, means: "Fixed bit rate within a fixed (predictable) period, or fixed amount of data"

Most "CBR" codecs are actually variable bitrate, but they have relatively small "bit buffer" which is constant in size and known a-priori, and that provides variations in frame bit rate. Within those limits, codec has full freedom to allocate bits.

Even within a single frame, bits are allocated in the variable sense - depending on the psychoacoustic threshold, etc...

So, in a nutshell - "CBR" in the modern audio codec is way different than "CBR" in PCM sense - both frames and individual samples are coded with different, variable, accuracies.

[Alex B]

Sebastian,

Could you possibly post the average results per sample as a table like this (in the original sample order):

CODE

WMA    High    Vorbis    Low    Nero
2.60    4.00    1.70    1.00    3.30
2.00    3.50    2.00    1.00    3.00
2.80    4.00    2.30    1.00    2.70
3.40    4.00    3.10    1.00    3.70
2.40    3.60    2.20    1.00    2.30
2.10    3.50    1.70    1.00    2.50
1.70    2.50    2.00    1.00    1.70
2.20    3.40    3.00    1.00    2.60
1.60    3.20    2.30    1.00    2.60
3.10    3.50    2.80    1.00    2.60
2.60    3.50    2.40    1.00    2.80
1.80    3.40    2.00    1.00    1.80
2.90    3.80    2.30    1.00    2.60
3.00    3.90    2.00    1.00    2.70
2.00    3.70    2.30    1.00    1.70
3.00    4.00    2.10    1.20    2.10
2.30    3.50    2.80    1.00    1.80
3.40    4.00    3.40    1.00    3.10

I would like to draw a chart in the following format, but it would be quite laborious to grab the values from the result images.

Alex B's personal results:

[muaddib]

Could you possibly post the average results per sample as a table like this (in the original sample order):

It is possible to get that data using chunky on the complete test results which are available in .rar.