Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[IgorC]

The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

[IgorC]

motion_blur,

You can download the results of all listeners and compare them with yours. http://listening-tests.hydrogenaudio.org/i...ous/results.zip
They are too different.

Also why do you post samples with samplerate 96 kHz?



Hi, Chris,
I had also hard time to understand the bootstrap analysis.
Please, wait for detailed answer on it

As of Christoph's results, all of them were excluded. http://www.hydrogenaudio.org/forums/index....st&p=751768

This post has been edited by IgorC: Apr 12 2011, 20:39

[_m²_]

Some presentation suggestions:
1. Codec versions and settings should be in the results or one clearly marked click away. I don't consider what is now to be clearly marked.
2. Links to results of older tests would be welcome.
3. I can't wait for the bitrate table.

[Alex B]

2. Links to results of older tests would be welcome.

http://listeningtests.t35.com.

I have mirrored Roberto's and Sebastian's old test sites. Sebastian's tests are also available here: http://listening-tests.hydrogenaudio.org/sebastian/

3. I can't wait for the bitrate table.

Actually, a more useful presentation would be a comparison like this: http://www.hydrogenaudio.org/forums/index....st&p=593735
I.e. bitrates that represent real life usage, not the bitrates of these short test samples.

I am planning to do it, but the lack of application support for Opus (CELT) will make the process quite a bit more complex than before.

This post has been edited by Alex B: Apr 12 2011, 21:24

[Alex B]

3. I can't wait for the bitrate table.

Actually, a more useful presentation would be a comparison like this: http://www.hydrogenaudio.org/forums/index....st&p=593735
I.e. bitrates that represent real life usage, not the bitrates of these short test samples.

I am planning to do it, but the lack of application support for Opus (CELT) will make the process quite a bit more complex than before.

I have done it now.

I have used these two test sets, Various and Classical, for a long time. To see how these test tracks are handled by other encoders see for example the above linked post and my "LAME 3.98.2 VBR bitrate test, all -V settings in 0.5 step increments" thread: http://www.hydrogenaudio.org/forums/index....showtopic=67523

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.











This post has been edited by Alex B: Apr 13 2011, 12:35

[NullC]

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

This post has been edited by NullC: Apr 13 2011, 12:54

[Garf]

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

Depends on the format. IIRC, AAC/MP4 bitrate includes the container, but Peter wasn't sure about Vorbis, and I'm not sure either.