Codecs and settings for 64kbit/s SE listening test, criticism required Options

[Serge Smirnoff]

As SoundExpert now has pretty stable flow of volunteer testers it is time to update codecs in all bitrate sections. After a short discussion at Russian bit-torrent tracker we decided to update 64-96-128-192-256-320 sections first. For the 64kbit/s section five codecs were chosen:

Fhg AAC (2012-06-24) - 59.2 kbit/s (fhgaacenc --vbr 2 se_ref.wav)
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)
Nero AAC (v1.5.4.0) - 60.1 kbit/s (neroAacEnc.exe -q 0.25 -if se_ref.wav -of out.mp4)
Vorbis (Xiph 1.3.3) - 60.3 kbit/s (oggenc2.exe -q-0.3 se_ref.wav)
Opus (libopus 1.0.1) - 59.9 kbit/s (opusenc --bitrate 59 se_ref48.wav out.opus)
Conversion chain for Opus: 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> foobar2000(48/24) -->> 44.1/16(Audition CS6)

Bitrates are calculated on the basis of nine SE test samples concatenated. Fortunately first two codecs have close values of resulting bitrates at their corresponding discrete quality settings. Other contenders were adjusted to have close values.

Did we miss something?

[yourlord]

Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

As for including mp3, it's still the most popular codec, and it would be useful to see the bitrate at which mp3 gains parity with the test cases.

This post has been edited by yourlord: Oct 15 2012, 01:10

[Serge Smirnoff]

Sounds like a useless test to me. There is nothing to be gained by exposing otherwise inaudible artifacts. These codecs make decisions based on the fact the artifacts this test seeks to expose would be inaudible.

Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.

[yourlord]

Below 128kbit/s artifact amplification is not applied. Outputs of codecs are used as is.

The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

If every codec other than mp3 becomes transparent at 128kbps then let your study confirm that so that it adds even more to the mountain of evidence that above those bitrates you can use any codec you like with no audible problems, and that mp3 sucks.

This post has been edited by yourlord: Oct 15 2012, 03:06

[Serge Smirnoff]

The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0

Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.

[Kohlrabi]

The fact remains I question doing it above 128kbps.. Amplifying inaudible artifacts to the point they become audible SERVES NO PURPOSE. You can't judge the quality of a lossy codec that way. The whole exercise will do nothing but provide misleading reference material which nimrods will use to base inaccurate claims that some codec is better or worse than another because of the INAUDIBLE artifacts this test needlessly exposes.

Please not here, there is more appropriate place for the discussion - http://www.hydrogenaudio.org/forums/index....=85182&st=0

While the concerns with the basic premise have not been addressed as far as I can tell, this is the thread you asked for input and comments in this thread, so let's stick with this one instead of reviving the old one.

Each codec should be included in all test samples. If for no other reason than to illustrate how badly mp3 works at those low bit rates compared to the more modern ones.

I'm not sure there is a need to prove shortcomings of mp3 at low bitrates over and over again.

I'm on the other hand not sure if there is any need to prove "shortcomings" of lossy encoders by trying to inflate certain, previously inaudible, artifacts in a listening test. How do you make sure thise method doesn't artificially bias towards certain encoders/artifacts? If you want to prove that lossy encodes differ form the original, you're done now, since they obviously do and have to. Another useful metric to me is the binary issue of transparency. Either the (unaltered!) encoder result is transparent or it isn't. In the real world you'll never have weird mixes where you superimpose difference signals onto the encoded signal. This method is completely artificial with no real world application or meaning. Another thing done regularly here are the ABC tests, but those are mainly useful to grade encoders on results with obvious audible flaws, to decide which encoder produces the less annoying results.

This post has been edited by Kohlrabi: Oct 15 2012, 12:34

[Serge Smirnoff]

I'm on the other hand not sure if there is any need to prove "shortcomings" of lossy encoders by trying to inflate certain, previously inaudible, artifacts in a listening test.

At 64kbit/s there is no need for artifacts amplification for sure. Above 128kbit/s meaningful results of ABX testing become more and more expensive (but still meaningful). SoundExpert proposes methodology that makes those tests less expensive. SE quality ratings of devices with small impairments could be considered as results of specially simplified listening tests. Results are experimental which is clearly stated on the site.

[yourlord]

At 64kbit/s there is no need for artifacts amplification for sure. Above 128kbit/s meaningful results of ABX testing become more and more expensive (but still meaningful). SoundExpert proposes methodology that makes those tests less expensive. SE quality ratings of devices with small impairments could be considered as results of specially simplified listening tests. Results are experimental which is clearly stated on the site.

If meaningful results of ABX tests above 128kbps become more and more expensive it's because the codecs are doing their jobs and producing audibly transparent output. At a point where normal ABX results become statistically insignificant then transparency has been reached and we're done. Artificially altering encoder output to highlight normally inaudible artifacts of the encoding process and then trying to assign a quality to a codec based on those artificially accentuated normally inaudible artifacts is a USELSS process. It has no application in the real world, it means nothing, and the results obtained from such "tests" are useless noise and best ignored.

You might as well subtract the lossy output from the original, post the spectrograms, and start running around screaming the sky is falling and vinyl is better than digital..

This post has been edited by yourlord: Oct 15 2012, 17:41

[Serge Smirnoff]

At 64kbit/s there is no need for artifacts amplification for sure. Above 128kbit/s meaningful results of ABX testing become more and more expensive (but still meaningful). SoundExpert proposes methodology that makes those tests less expensive. SE quality ratings of devices with small impairments could be considered as results of specially simplified listening tests. Results are experimental which is clearly stated on the site.

If meaningful results of ABX tests above 128kbps become more and more expensive it's because the codecs are doing their jobs and producing audibly transparent output. At a point where normal ABX results become statistically insignificant then transparency has been reached and we're done.

The problem is that there is no such "point" in practice. Another more seriously organized listening test moves the point of transparency to higher bitrates. Codecs at 256 are not the same even if your "normal ABX results" show that; another super-normal ABX results will reveal the differences for sure. In other words, differences between "equally transparent" codecs can be revealed by more thoroughly prepared listening tests. Besides codecs there is a lot of audio equipment with small impairments that requires evaluation and expensive listening tests. So, the purpose of SE testing methodology is to make such listening tests cheaper but still relevant.