Multiformat listening test @ ~64kbps: Results, Results and post-test discussion Options

[motion_blur]

Sorry, Christoph, can't reproduce it. What you describe must sound like a notch filter, i.e. frequency band missing. Haven't noticed anything of that sort during and after the test. What OS are you using? 64-bit?

Thanks, Garf and NullC, for the explanations.

Note that equal sample weighting, by only including complete results, does not change the results in the slightest.

That's good to hear. Still, if you find some time, would you mind creating a closeup average-codec-score plot using only the complete results, just like the plot on the results page?

Thanks,

Chris

Yes, Windows 7 64-Bit. But this is not an OS issue, I think. The files are as they are. If you like, you can try Audacity to take a look at the spectra, you can see the gap here to.
http://dl.dropbox.com/u/745331/spectrum2.png

[NullC]

Hi Igor,
on sample 10 I votes this way, because I found this part of the sample SUPER annoying:
http://dl.dropbox.com/u/745331/64kbs%20tes..._4_celt_cut.wav
http://dl.dropbox.com/u/745331/64kbs%20tes...e10_org_cut.wav
From this point the "glitch" gets less annoying but stays through the end of the sample.
maybe this is only for me that annoying or is it a decoding error?! can you please check this?
Thanks,
Christoph

So, even after listening to the difference signal in order to set my loop points, I had a somewhat difficult time A/Bing your sample vs the original.. I certainly don't hear anything super-annoying.

Can you try taking the two signals (Sample10r.wav and Sample10_4.wav) and lowering their volume level by 6dB in an audio editor, then try to ABX them on the segment where you thought the artifact was most obvious and see if you can find it?

The peak value of the signal is very high in this file, and I'm speculating your audio software might be adding a small amount of gain and causing clipping.

This would also be consistent with your overall preference ranking— since I think you managed to order the codecs in order from least peak-value amplification to most (then the low anchor).

[NullC]

The test is finished, results are available here:
http://listening-tests.hydrogenaudio.org/igorc/results.html
Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC.

Hey all, you may be interested in a presentation of the results that I did which summarizes some of the results I found while looking at the data.
http://people.xiph.org/~greg/opus/ha2011/

I'll probably be adding a bit to it over the next few days as I analyze some more things. I'm posting the link here first before linking it up elsewhere in the hopes that if I've done something obviously dumb that the folks here will cluestick me. Thanks!

[Garf]

Hey all, you may be interested in a presentation of the results that I did which summarizes some of the results I found while looking at the data.
http://people.xiph.org/~greg/opus/ha2011/

I'll probably be adding a bit to it over the next few days as I analyze some more things. I'm posting the link here first before linking it up elsewhere in the hopes that if I've done something obviously dumb that the folks here will cluestick me. Thanks!

One thing which catches my attention here is that the Nero VBR seems to flex much more than that of Apple VBR, but on this sample set it neglected to increase the bitrate, i.e. it somehow failed to recognize these samples were difficult.

What exactly does the "Per-sample distribution" graph show?

[motion_blur]

Hi Igor,
on sample 10 I votes this way, because I found this part of the sample SUPER annoying:
http://dl.dropbox.com/u/745331/64kbs%20tes..._4_celt_cut.wav
http://dl.dropbox.com/u/745331/64kbs%20tes...e10_org_cut.wav
From this point the "glitch" gets less annoying but stays through the end of the sample.
maybe this is only for me that annoying or is it a decoding error?! can you please check this?
Thanks,
Christoph

So, even after listening to the difference signal in order to set my loop points, I had a somewhat difficult time A/Bing your sample vs the original.. I certainly don't hear anything super-annoying.

Can you try taking the two signals (Sample10r.wav and Sample10_4.wav) and lowering their volume level by 6dB in an audio editor, then try to ABX them on the segment where you thought the artifact was most obvious and see if you can find it?

The peak value of the signal is very high in this file, and I'm speculating your audio software might be adding a small amount of gain and causing clipping.

This would also be consistent with your overall preference ranking— since I think you managed to order the codecs in order from least peak-value amplification to most (then the low anchor).

OK, cut me out :-)

I just found the problem! It's not my hears, it's not the decoder, It's my Hardware.

The first clue I got was when I played the Opus sample with my speakers (not headphones) and it did not sound bad at all. Than I experimented a bit and could locate the problem at the headphones jack of my sound system. This seems to be a common issue:
http://forums.logitech.com/t5/Speakers/Z-5...ack/td-p/440105

Sorry I caused you that much trouble. I want to thank all the people who prepared this test, especially Igor. This was really great work.
I hope next time I can participate with valid results. I should really put some tape on this darn headphone jack, so I do not use it again accidentally ;-)

Best,
Christoph

This post has been edited by motion_blur: Apr 13 2011, 10:08

[Alex B]

3. I can't wait for the bitrate table.

Actually, a more useful presentation would be a comparison like this: http://www.hydrogenaudio.org/forums/index....st&p=593735
I.e. bitrates that represent real life usage, not the bitrates of these short test samples.

I am planning to do it, but the lack of application support for Opus (CELT) will make the process quite a bit more complex than before.

I have done it now.

I have used these two test sets, Various and Classical, for a long time. To see how these test tracks are handled by other encoders see for example the above linked post and my "LAME 3.98.2 VBR bitrate test, all -V settings in 0.5 step increments" thread: http://www.hydrogenaudio.org/forums/index....showtopic=67523

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.











This post has been edited by Alex B: Apr 13 2011, 12:35

[Garf]

The result page is now updated with per-sample graphs.

[NullC]

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

This post has been edited by NullC: Apr 13 2011, 12:54

[Garf]

The bitrates are from foobar2000, except the Opus (CELT) bitrates, which are calculated from the exact file sizes and original durations. The encoding settings and encoder versions are the same as were used in the listening test. I resampled the source files to 48 kHz before encoding the Opus tracks (as was done in the listening test). The other test tracks have the original 44.1. kHz sample rate.

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

Depends on the format. IIRC, AAC/MP4 bitrate includes the container, but Peter wasn't sure about Vorbis, and I'm not sure either.

[Alex B]

Any idea if fb2000 is including the container overhead? It's not much, but e.g. on the opus files it's 1kbit/sec, which would explain the entire difference of the means here.

I don't know how foobar2000 gets the bitrate data, but here's a small comparison. The A values are from foobar2000. The B values are calculated from the file sizes and durations. (foobar2000 displays only integer values.)

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

The duration is 2 min 37 s. I have just rechecked the test files. The bitrate values appear to be correct. I'll send you a new PM soon.

This post has been edited by Alex B: Apr 13 2011, 15:11

[mixminus1]

Thanks much to all for their work in both setting up the test, and processing the results - it was very educational!

Is there a Win32 binary of the CELT/Opus encoder available? I'm only seeing source code packages on the CELT website.

[Garf]

Thanks much to all for their work in both setting up the test, and processing the results - it was very educational!

Is there a Win32 binary of the CELT/Opus encoder available? I'm only seeing source code packages on the CELT website.

It's linked from the test website, in the "Where can I download the encoders?" section.

[mixminus1]

:facepalm: Good God...

Thanks, Garf, I was scouring the results page like mad...clueless noob!

[romor]

@Garf: can you please reupload results you posted on page 1 yesterday
It was text showing all results merged (grouped by sample, columns by codec)
If it's possible and if you can include tester number as first column if would be great

Thanks

[IgorC]

AlexB, thank you for bitrate verification.
I really appreciate it.


Just a little observation. Your list is split in two parts (Various and Classic). Various contains many musical genres except classical. This way all genres have only 50% of weight and 50% belong to overweighted classic music.

Imagine situation if I will calculate average bitrate on (Various vs Metal) because it is my main musical genre.
Any person has his/her personal _Various vs most admirable genre_

All genres are enough popular and should be weighted equally.

I can't imagine that 50% of people listen classic (or metal) music.

P.S. It will be great to add a few (really brutal one like trash) metal albums to your bitrate table. There is only one metal album Evanscence which is not quite heavy enough. Some encoders tend to inflate the bitrate a lot on brutal genres.

This post has been edited by IgorC: Apr 13 2011, 17:39

[pdq]

Whether or not classical should be considered to be of equal importance to all other genres combined is a matter of opinion. For example, suppose you weight it by longevity?

[IgorC]

Whether or not classical should be considered to be of equal importance to all other genres combined is a matter of opinion.

No, it's matter of facts.
Only 5 of 30 tested sample were classic. Rock and pop are much more popular than classic nowdays.

I argue about genres not because I think some of them better or have more longivity but because it have influence on bitrates.

[pdq]

Whether or not classical should be considered to be of equal importance to all other genres combined is a matter of opinion.

No, it's matter of facts.
Only 5 of 30 tested sample were classic. Rock and pop are much more popular than classic nowdays.

I argue about genres not because I think some of them better or have more longivity but because it have influence on bitrates.

Sorry, that was meant to be a joke.

[Garf]

@Garf: can you please reupload results you posted on page 1 yesterday

I deleted that file because it contained mistakes in the post-screening, as discussed earlier in this thread.

The page NullC linked has the processed results in various formats with correct screening. You should be able to massage that into whatever format you want.

[romor]

file: http://people.xiph.org/~greg/opus/ha2011/2...it_test.tar.bz2

Sorry, found it in 'summary_complete' folder, separated sample per file
thanks

This post has been edited by romor: Apr 13 2011, 19:34

[NullC]

file: http://people.xiph.org/~greg/opus/ha2011/2...it_test.tar.bz2
Sorry, found it in 'summary_complete' folder, separated sample per file
thanks

Summary complete is the listeners which submitted results for 30 samples, summary all is all listeners. Both post-postscreening.

I'll be updating the file later today and will include a readme with descriptions.

[IgorC]

Bitrate verification on my set of albums:


http://www.hydrogenaudio.org/forums/index....st&p=752009

This post has been edited by IgorC: Apr 13 2011, 21:49

[C.R.Helmrich]

Thanks, Garf, for the plot! And thanks, Christoph, for clearing up your (very unfortunate) headphone jack issue.

Hey all, you may be interested in a presentation of the results that I did which summarizes some of the results I found while looking at the data.
http://people.xiph.org/~greg/opus/ha2011/

I'll probably be adding a bit to it over the next few days as I analyze some more things. I'm posting the link here first before linking it up elsewhere in the hopes that if I've done something obviously dumb that the folks here will cluestick me. Thanks!

*raising my hand*

Please don't write "HE-AAC" but something like "two popular HE-AAC encoders" in

Now, these results demonstrate that Opus's performance against HE-AAC, one of the strongest (but highest-latency) codecs at this bitrate, are very strong, besting its quality on the majority of individual audio samples and receiving a higher average score overall.

Sorry, but being an AAC developer I have to stress this. The nero and Apple encoders have never been proven to be the best encoders around.

Chris

[NullC]

Also, I'm quite perplexed by the file that runs at 82kbit/sec: Unless the file is quite short the Opus encoder shouldn't currently be able to do that.

The duration is 2 min 37 s. I have just rechecked the test files. The bitrate values appear to be correct. I'll send you a new PM soon.

Thank you very much for posting these figures. There was indeed a misbehavior in our encoder.

The current opus encoder has very simple VBR which is mostly designed for low latency constrained VBR usage. In full VBR mode we simply turn off the constraint, but otherwise it's the same. Because of this our VBR should be very constant compared to other VBR codecs— and you can see this on the samples used in this test. A few of your samples, however, showed bitrate spikes which should not have been possible with our VBR system.

The way the VBR currently works is that in the middle of encoding a frame the size of the current entropy coded variable-rate part of the frame is added to an offset value. This target is boosted for frames containing transients, then clamped to the permissible rates and then used as the target size for the whole frame. The error between the target rate and the requested rate is used to control the the offset with a simple low-pass linear controller. (Notice that there is no psy-model in any of this except a dumb transient boost, which is why we're confident that the Opus encoder will improve a lot for high latency uses like this in the future. This is also a good area if someone would like to help improve the Opus encoder.)

Opus can encode digital silence with two-bytes per frame and our encoder does so when it is fed digital silence. My explicit intention in the above model was to ignore these silence frames for the purpose of the closed loop rate control, so that files with lots of silence would end up undershooting the rate but the presence of silence would not cause huge rate jumps either. Somehow, I don't know if a patch was lost or I just had a blonde day, I actually failed do this. Instead, during long spans of digital silence the encoder would shoot the offset through the roof in a futile attempt to use more than two-bytes per frame. Once the audio began again it would _greatly_ overshoot the target for a little while until the closed loop control caught up with it. The pink panther file begins with 275 frames of digital silence, and the first non-silent frame was encoded at 423kbit/sec. The encoder it took hundreds of frames to get back to a sane rate.

Since this actually requires digital silence (not just quiet frames) and long spans of them to matter, it isn't actually triggered on many things. I'm pretty sure this change has no significant effect on any of the samples used in this test, for example... and on the collection of 20 or so CD's (commercial and live recordings) that I've been using for automated Q/A it never managed to trigger a jump large enough to get my attention.

The behavior is now fixed: http://git.xiph.org/?p=celt.git;a=commitdi...1dd48d13e734125

With the fix the file is encoded with an average rate of 62.890kbit/sec, instead of ~82kbit/sec.

If you're interested in measuring the bitrates on a fixed encoder, I'd be glad to build windows binaries for you. I can also trivially back-port this fix to the encoder version that was used in the test (though I think we haven't changed anything important since then) if you'd like to see that.

Thank you for taking the time to test it out and report results.

This post has been edited by NullC: Apr 13 2011, 22:50

[Garf]

Please don't write "HE-AAC" but something like "two popular HE-AAC encoders" in

Sorry, but being an AAC developer I have to stress this. The nero and Apple encoders have never been proven to be the best encoders around.

From the previous tests, the Nero and Apple encoders are proven to be the best AAC encoders, so with the data at hand I strongly believe you are wrong.

If you believe otherwise, feel free to give evidence to the contrary. Pointing out this hypothetical better encoder would be a good start, and is sure to generate a lot of interest on this forum.