Analyzing the Results of Listening Tests, ..."Composite Rating" meaningful? Options

[ScorLibran]

A side-topic began in this thread about how to interpret the results of Roberto's 128kbps listening test. It spurred some questions in my mind concerning where the "threshold of significance" exists for certain aspects of the test results. This is addressed in some ways in the "Test Finished" thread, but there are still some things I'm wondering about concerning how the results can be interpreted.

The top four rated codecs, AAC, WMA Pro, Vorbis and MPC were statistically tied, yet the overall average bitrates for each codec are still disparate. My contention is that the average bitrates are fixed values, and hence usable in further analysis of the test results.

Hence, is it feasible to measure the test results as "rating / average bitrate"? I did a quick calculation of this based on the published test results data, multiplying all my calculation results by 128 (the bitrate target in kbps) to get numbers >0, and came up with the following "composite ratings" addressing the test's quality ratings divided by average bitrates for each codec (results are rounded):

AAC = (4.42/129)*128 = 4.39
WMA Pro = (4.30/128)*128 = 4.30
MPC = (4.51/146.1)*128 = 3.95
Vorbis = (4.28/140.1)*128 = 3.91

...so does this interpretation have any significance? Or am I on the wrong track?

I hear many people saying that MPC was "better" than Vorbis in the test, but looking only at sound quality, since they were tied within ANOVA's margin of confidence this would not be the case. However, it may be possible to qualify that statement by defining a "composite rating", if the approach has any significance, that is.

A decision of either "Yes, Chris, this is usable", or "No, the resulting variance between these four codecs at 128kbps is still insignificant, even considering the average bitrate differences" may settle a lot of disputes on this subject.

Edit: Changed multiplier from "100" to "128", as noted above.

This post has been edited by ScorLibran: Nov 12 2003, 11:41

[Todesengel]

I have encoded about 100 albums in MPC. and I told you --radio gives bitrate around 140kbps!!!