Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[IgorC]

The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

[Alex B]

Thanks guys! Interesting results.


One note though:

CODE

Read 5 treatments, 531 samples => 10 comparisons
    Means:
          Vorbis   Nero_HE-AAC  Apple_HE-AAC          Opus    AAC-LC@48k
           3.513         3.547         3.817         3.999         1.656

For processing the result .txt files with chunky I organized them to sample folders. I removed the results that were marked "invalid" and results that apparently had a fixed newer version (marked as such). I had a duplicate problem with romor's results (a couple of duplicates in a subfolder), but I decided to keep the newer result files. I got 566 remaining result files. Assuming I did not make lots of mistakes, I wonder what can cause the difference. Did you disqualify more results after creating the rar package or does "531 samples" mean something else than the total number of result files?

Here's how chunky parses the 566 result files I have:

CODE

% Result file produced by chunky-0.8.4-beta
% ..\chunky.exe --codec-file=..\codecs.txt -n --ratings=results --warn -p 0.05
%
% Sample Averages:

Vorbis    Nero    Apple    CELT    Anchor
2.56    4.28    4.19    2.67    1.87
2.95    4.20    4.03    2.36    1.68
3.42    3.51    3.98    4.73    2.51
4.12    3.84    4.49    4.64    2.18
4.18    3.59    3.87    4.52    1.95
3.35    3.68    3.34    4.00    1.56
3.86    2.98    2.96    3.50    1.85
4.03    3.78    4.09    4.49    2.02
3.60    3.71    3.89    3.94    1.51
4.28    2.78    2.19    4.12    1.44
4.12    3.93    4.17    4.39    1.70
3.25    3.18    3.20    4.14    1.77
3.83    3.63    3.86    4.56    1.41
3.49    3.81    4.01    4.27    1.37
4.15    3.84    4.08    4.76    2.04
3.97    2.74    3.09    4.38    1.74
3.35    3.24    4.15    4.44    1.56
2.68    2.96    3.63    4.10    1.51
3.58    4.37    4.88    3.73    1.76
3.40    4.10    4.68    4.26    1.61
3.80    3.49    3.55    4.43    1.38
3.81    3.30    4.27    4.26    1.13
3.59    3.14    3.51    4.09    1.18
3.29    3.61    3.88    4.16    1.36
3.66    3.84    4.37    3.86    1.55
2.78    3.99    4.18    2.82    1.57
3.62    3.88    3.92    3.93    1.34
3.39    4.03    4.39    3.96    1.46
3.61    4.12    4.36    4.09    1.54
4.42    3.48    4.29    4.68    1.82

% Codec averages:
% 3.60    3.63    3.92    4.08    1.65

[NullC]

For processing the result .txt files with chunky I organized them to sample folders. I removed the results that were marked "invalid" and results that apparently had a fixed newer version (marked as such). I
had a duplicate problem with romor's results (a couple of duplicates in a subfolder), but I decided to keep the newer result files. I got 566 remaining result files. Assuming I did not make lots of mistakes, I
wonder what can cause the difference. Did you disqualify more results after creating the rar package or does "531 samples" mean something else than the total number of result files?

Sounds like you didn't eliminate the listeners with more than 4 invalid results.

The filtering rules on the page are:

* If the listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
* If the listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
* If the listener ranked the reference below 5.0 on more than 4 samples, all of that listener's results were discarded.

You'll have to modify chunky to get the that behavior.


This post has been edited by NullC: Apr 12 2011, 15:12

[Garf]

Sounds like you didn't eliminate the listeners with more than 4 invalid results.

The filtering rules on the page are:

* If the listener ranked the reference worse than 4.5 on a sample, the listener's results for that sample were discarded.
* If the listener ranked the low anchor at 5.0 on a sample, the listener's results for that sample were discarded.
* If the listener ranked the reference below 5.0 on more than 4 samples, all of that listener's results were discarded.

You'll have to modify chunky to get the that behavior.

Ah, good point. There were two discarded listeners, I got those. I saw one result with a rated reference that didn't cause an invalidation, so got that correctly.

But there are a few results with 5.0's for the reference. After discarding those, I'm at 559 samples now.