Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[IgorC]

The test is finished, results are available here:

http://listening-tests.hydrogenaudio.org/igorc/results.html

Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

[googlebot]

I'm stunned by the CELT/Opus results! I would have assumed that your toolbox is smaller than usually when you are targeting low-delay. And now Celt even beats the others by lengths.

Thanks for the great work, guys!

[NullC]

I'm stunned by the CELT/Opus results! I would have assumed that your toolbox is smaller than usually when you are targeting low-delay. And now Celt even beats the others by lengths.
Thanks for the great work, guys!

We were surprised when we started getting competitive with high latency codecs too, but we've had a little while to get used to that. HE-AAC was a bit more surprising, especially since so many things were going against us (the immaturity of our encoder, it's inadequacy for this application (VBR), etc.)

Low-latency work implies some serious compromises, but it's not all bad. Small transform sizes automatically give you better time domain noise shaping, for example. There have been a lot of people that liked codecs like MPC at high rates for the reduced time-domain-liability they imply. Simply getting many little details right helps reduce the harm of low-latency. E.g. we use a trivial time domain pre-whitening before the transform so that the quantization noise from spectral leakage is less exposed.

We also expanded the low latency toolbox some. For example, we have short and long frames without window switching (and the look-ahead latency that requires). We're also using other things like range coding and high-dimensional vector quantization which might not have been great options a number of years ago. Our decoder is currently quite a bit slower than AAC decoders (though it's not optimized, so it's hard to say how much the ultimate difference will be), but since we're mostly targeting interactive use we were able to "pay" for decoder complexity increases with encoder decreases: We're an order of magnitude faster encode side than our high latency competition (no explicit psy-model!). With some fairly modest compromises you can make an Opus audio mode (the CELT part) encoder which is basically the same speed as the decoder. Though cpu cycle hungry, Opus uses a very small amount of memory, which eliminates one of the embedded hurdles Vorbis suffered.

I also like to think that the ultra-low latency support has fostered some beneficial discipline: Every single fractional-bit of signaling overhead and frame redundancy counts a lot with 2.5ms frames. While it's not so important with larger frames, waste is waste, and Opus has very little of it. A very high percentage of the bits in opus frames go directly to improving the audio band SNR, and very few go to shuffling around coding modes or switching between inconsequentially different alternatives. Signaling bits are sometimes helpful, sometimes _very_ helpful... but bits spent coding signal data are pretty much always helpful. We came up with a number of clever ways of eliminating signaling, and Opus is able to provide a true hard CBR (no bitres!) in audio modes which is super efficient (uses every bit in the frame) and actually sounds really good.

For music at lower rates, I expect that HE-AAC would win— we simply start to fall apart once the number of bytes in the frame gets too small. Speech is another matter and Opus should do quite well there down to very low rates, owing to the merger of Skype's SILK.

Opus should also scale well to higher rates— it is not using any highly parametric techniques that don't respond to additional bits— though the lack of a mature encoder will probably still give other codecs the edge in many cases. This is especially true for exposed multi-tonal samples like the sample 02 in this set, though multi-tonal killers are fairly uncommon and I expect that we can fix them with VBR in the same way large block codecs fix transients...

I also think that aggressive tuning of these HE-AAC encoders could put them back in the lead, or at least strongly tie opus. E.g. From my own listening I think a lot of the difference between nero and apple in the test was due to the lowpass difference for example. That said, the HE-AAC format is mature (and by some accounts now stagnant), and we have a lot of low hanging fruit.

I feel that Vorbis loses for a sad reason at this rate and lower: The Vorbis toolbox doesn't have any good tools to avoid having to low-pass at rates well below its initial design goals at higher rates it obviously does much better than it did here. The lack of efficient tools for low rate HF is real shortcoming at this rate, but not one which is all that interesting from an engineering/competitive basis.

Cheers,

This post has been edited by NullC: Apr 14 2011, 05:04

[saratoga]

We also expanded the low latency toolbox some. For example, we have short and long frames without window switching (and the look-ahead latency that requires). We're also using other things like range coding and high-dimensional vector quantization which might not have been great options a number of years ago. Our decoder is currently quite a bit slower than AAC decoders (though it's not optimized, so it's hard to say how much the ultimate difference will be), but since we're mostly targeting interactive use we were able to "pay" for decoder complexity increases with encoder decreases: We're an order of magnitude faster encode side than our high latency competition (no explicit psy-model!). With some fairly modest compromises you can make an Opus audio mode (the CELT part) encoder which is basically the same speed as the decoder. Though cpu cycle hungry, Opus uses a very small amount of memory, which eliminates one of the embedded hurdles Vorbis suffered.

Is the decoder spec finalized yet? Would be interesting to see about getting a fixed point version running in Rockbox.

[NullC]

Is the decoder spec finalized yet? Would be interesting to see about getting a fixed point version running in Rockbox.

We're in a soft-freeze right now. We're really not planning on changing it, but we're also not yet making any promises not to: if something awful comes up we will. The whole process is dependent on progress in the IETF now, which appears to have gone super-political and thus slow, for the moment. The Ogg encap for it is certainly not final at the moment, it's been due for a redo for over a year now, but that was pushed back because we were working on finishing the codec itself.

So for the CELT part the reference implementation (libcelt) is both a fixed point implementation and a floating point implementation, through the magic of unholy C macros. The SILK part has split fixed/float code, and the combination is float only at the moment, but I think this is mostly just a build system issue. I'd be glad to work with whomever to get it working on whatever. Feel free to hop into the #celt channel on irc.freenode.net.