Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[IgorC]

The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

[C.R.Helmrich]

Thanks for organizing the tests, guys! Sorry for being picky, but I'm not convinced about the analysis. To ease my mind, it would be great if you could comment on the following.

• Please provide the number of valid results (i.e. listeners) per sample (excluding "27", see below).
• How did you compute the overall average score of a codec and its confidence intervals? Taking the mean of all listeners' results? That would mean a sample with more listeners (i.e. probably sample01) has a greater influence than the last few samples (which still needed listeners shortly before the end of the test). This is probably not a good approach; weighting each sample equally in the overall score seems to be the way to go for me (but it probably doesn't make a difference here, but still...).
• Nothing personal, but if a listener like "27" consistently scores in opposite direction as the average (as shown by Igor), a thorough post-screening analysis (like Spearman rank correlation < some value) would - and has to - exclude such results.

Edit: Christoph, why are the samples you uploaded at 96 kHz? Did you do the test that way?

Chris

This post has been edited by C.R.Helmrich: Apr 12 2011, 20:17

[motion_blur]

Thanks for organizing the tests, guys! Sorry for being picky, but I'm not convinced about the analysis. To ease my mind, it would be great if you could comment on the following.

• Please provide the number of valid results (i.e. listeners) per sample (excluding "27", see below).
• How did you compute the overall average score of a codec and its confidence intervals? Taking the mean of all listeners' results? That would mean a sample with more listeners (i.e. probably sample01) has a greater influence than the last few samples (which still needed listeners shortly before the end of the test). This is probably not a good approach; weighting each sample equally in the overall score seems to be the way to go for me (but it probably doesn't make a difference here, but still...).
• Nothing personal, but if a listener like "27" consistently scores in opposite direction as the average (as shown by Igor), a thorough post-screening analysis (like Spearman rank correlation < some value) would - and has to - exclude such results.

Edit: Christoph, why are the samples you uploaded at 96 kHz? Did you do the test that way?

Chris

@Edit: Oh sorry, I just fast cut it with audacity, did not noticed it was still configured that way. But anyway I hope you can hear what mean.
Maybe the most people only concentrated on the beginning of the samples!? the part with the glitch is way in the sample.

Yes, I know that my results do not fit the standards and therefore are excluded.
And if I would be in, I am just one of the outliers and do not influence the median of the scoring much.
http://www.physics.csbsju.edu/stats/complex.box.defs.gif

But I want to know what I did different and what I can change next time.

This post has been edited by motion_blur: Apr 12 2011, 20:44