Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[IgorC]

The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

[IgorC]

motion_blur,

You can download the results of all listeners and compare them with yours. http://listening-tests.hydrogenaudio.org/i...ous/results.zip
They are too different.

Also why do you post samples with samplerate 96 kHz?



Hi, Chris,
I had also hard time to understand the bootstrap analysis.
Please, wait for detailed answer on it

As of Christoph's results, all of them were excluded. http://www.hydrogenaudio.org/forums/index....st&p=751768

This post has been edited by IgorC: Apr 12 2011, 20:39

[_m²_]

Some presentation suggestions:
1. Codec versions and settings should be in the results or one clearly marked click away. I don't consider what is now to be clearly marked.
2. Links to results of older tests would be welcome.
3. I can't wait for the bitrate table.

[Alex B]

2. Links to results of older tests would be welcome.

http://listeningtests.t35.com.

I have mirrored Roberto's and Sebastian's old test sites. Sebastian's tests are also available here: http://listening-tests.hydrogenaudio.org/sebastian/

3. I can't wait for the bitrate table.

Actually, a more useful presentation would be a comparison like this: http://www.hydrogenaudio.org/forums/index....st&p=593735
I.e. bitrates that represent real life usage, not the bitrates of these short test samples.

I am planning to do it, but the lack of application support for Opus (CELT) will make the process quite a bit more complex than before.

This post has been edited by Alex B: Apr 12 2011, 21:24

[Alex B]

3. I can't wait for the bitrate table.

Actually, a more useful presentation would be a comparison like this: http://www.hydrogenaudio.org/forums/index....st&p=593735
I.e. bitrates that represent real life usage, not the bitrates of these short test samples.

I am planning to do it, but the lack of application support for Opus (CELT) will make the process quite a bit more complex than before.

I have done it now.

I have used these two test sets, Various and Classical, for a long time. To see how these test tracks are handled by other encoders see for example the above linked post and my "LAME 3.98.2 VBR bitrate test, all -V settings in 0.5 step increments" thread: http://www.hydrogenaudio.org/forums/index....showtopic=67523

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.











This post has been edited by Alex B: Apr 13 2011, 12:35

[NullC]

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

This post has been edited by NullC: Apr 13 2011, 12:54

[Alex B]

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

I don't know how foobar2000 gets the bitrate data, but here's a small comparison. The A values are from foobar2000. The B values are calculated from the file sizes and durations. (foobar2000 displays only integer values.)

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

The duration is 2 min 37 s. I have just rechecked the test files. The bitrate values appear to be correct. I'll send you a new PM soon.

This post has been edited by Alex B: Apr 13 2011, 15:11

[NullC]

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

The duration is 2 min 37 s. I have just rechecked the test files. The bitrate values appear to be correct. I'll send you a new PM soon.

Thank you very much for posting these figures. There was indeed a misbehavior in our encoder.

The current opus encoder has very simple VBR which is mostly designed for low latency constrained VBR usage. In full VBR mode we simply turn off the constraint, but otherwise it's the same. Because of this our VBR should be very constant compared to other VBR codecs— and you can see this on the samples used in this test. A few of your samples, however, showed bitrate spikes which should not have been possible with our VBR system.

The way the VBR currently works is that in the middle of encoding a frame the size of the current entropy coded variable-rate part of the frame is added to an offset value. This target is boosted for frames containing transients, then clamped to the permissible rates and then used as the target size for the whole frame. The error between the target rate and the requested rate is used to control the the offset with a simple low-pass linear controller. (Notice that there is no psy-model in any of this except a dumb transient boost, which is why we're confident that the Opus encoder will improve a lot for high latency uses like this in the future. This is also a good area if someone would like to help improve the Opus encoder.)

Opus can encode digital silence with two-bytes per frame and our encoder does so when it is fed digital silence. My explicit intention in the above model was to ignore these silence frames for the purpose of the closed loop rate control, so that files with lots of silence would end up undershooting the rate but the presence of silence would not cause huge rate jumps either. Somehow, I don't know if a patch was lost or I just had a blonde day, I actually failed do this. Instead, during long spans of digital silence the encoder would shoot the offset through the roof in a futile attempt to use more than two-bytes per frame. Once the audio began again it would _greatly_ overshoot the target for a little while until the closed loop control caught up with it. The pink panther file begins with 275 frames of digital silence, and the first non-silent frame was encoded at 423kbit/sec. The encoder it took hundreds of frames to get back to a sane rate.

Since this actually requires digital silence (not just quiet frames) and long spans of them to matter, it isn't actually triggered on many things. I'm pretty sure this change has no significant effect on any of the samples used in this test, for example... and on the collection of 20 or so CD's (commercial and live recordings) that I've been using for automated Q/A it never managed to trigger a jump large enough to get my attention.

The behavior is now fixed: http://git.xiph.org/?p=celt.git;a=commitdi...1dd48d13e734125

With the fix the file is encoded with an average rate of 62.890kbit/sec, instead of ~82kbit/sec.

If you're interested in measuring the bitrates on a fixed encoder, I'd be glad to build windows binaries for you. I can also trivially back-port this fix to the encoder version that was used in the test (though I think we haven't changed anything important since then) if you'd like to see that.

Thank you for taking the time to test it out and report results.

This post has been edited by NullC: Apr 13 2011, 22:50