Multiformat Listening Test @ 64 kbps - FINISHED Options

[Sebastian Mares]

The much awaited results of the Public, Multiformat Listening Test @ 64 kbps are ready - partially. So far, I only uploaded an overall plot along with a zoomed version. The details will be available tomorrow. You can also download the encryption key on the results page that is located here:

http://www.listening-tests.info/mf-64-1/results.htm
http://www.listening-tests.info/mf-64-1/resultsz.png

Nero and WMA Professional 10 are tied and WMA Professional 10 is tied to Vorbis. Vorbis however performed worse than Nero. Of course, High Anchor is best and Low Anchor loses.

This one goes to the experts: How would you rank codecs in such a situation, where A=B and B=C, but C<A?

[lexor]

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

That is actually not a contradiction as such (though further expert opinion on the actual statistical metric used is needed).

You think that is a contradiction, because such situation doesn't happen in "normal" number systems, like integers, reals, etc. What you noticing is the property of total order breaking. However not all valuations have that property.

Take integers, if you take 2 integers at random there is the way to count from one to the other, precisely because there is a total order and you know what is less/greater than what, what equals what and what follows what.
On the other hand take Complex numbers, this is the first number system students usually exposed to in school that doesn't have total order on its elements (though school teachers don't usually mention that). Given 2 random Complex numbers there isn't "the" way to count from one to the other, in fact there are infinitely many ways, all correct in some sense.

So while I don't know if the underlying statistical measure produces set of values that has total order, your example (if not subject to some freaky error) shows that it doesn't, and should be read as raking:

1) HE = WMA
2) Ogg

My immediate intuition would be to use equivalence classes to solve this problem.
1) Make individual comparison between every possible pair
2) Look at the ones with strict inequalities
3) Pick the largest of them all (strictly greater, not >=)
4) Rank that first
5) Add all who are directly equal to it to its equivalence class (not ones that are equal by some chain of equalities)
6) Removing them from further consideration
7) From remaining, rank the next largest as number 2
8) Go to 5 and repeat for the rest of the ranking.

This post has been edited by lexor: Aug 19 2007, 16:00

[halb27]

This one goes to the experts:

How would you rank codecs in such a situation, where A=B and B=C, but C<A?

That is actually not a contradiction as such...

Same opinion for me.
For clearly defined objects A, B, C, a clearly defined identy and a clearly defined <-relation it would be a contradiction.
Here A and B correspond to the quality of Nero HE AAC and that of WMA pro, and C corresponds to that of Vorbis. Quality as measured with this test.
The problem is in the meaning of '=' and '<' as these are rough quality comparison operators which can easily make up for such a pseudo-contradiction.

The zoomed view is a major evil to me as it overestimates such a rough '<' comparison.
From absolute view ranging from 1.0 to 5.0 it's easy to say 'all these three encoders yield roughly the same quality with vorbis being a tiny bit behind.'
This is what is most important in practice, cause with these codecs you usually don't have the choice which one to use on a mobile device. No matter which one you use: you get state of the art 64 kbps technology regarding quality.

If it's up to elaborating differences between the encoders the very personal preferences are much more of concern than the overall small quality differences according to the test.

This post has been edited by halb27: Aug 19 2007, 22:39