Public Listening Test [2010], Discussion Options

[IgorC]

Many people will be agree that it’s time for public listening test.

Short Agenda:
1. Organization
2. Samples
3. Codecs
4. Bitrates

Detailed Agenda:
1. Organization.
People with experience of personal or public test are welcome here.
Personally I can afford help for conduction of test.
Unfortunately if I understand correctly that Sebastian informed that he don’t want to conduct public tests.

2. Samples.
Different styles of music, different levels of difficulty, pointing issues etc....? To be discussed here or in separate topic.

3. Codecs
3.a) Multiformat test.
3.b) AAC test.
I think that it’s more appropriate to conduct AAC test because there are at least 3 AAC encoders to test. Nero, Apple and Coding Technologies/Dolby. All these codecs were updated during this year. More information in List of AAC encoders

4.Bitrates.
96-100 kbits/s?
It's possible to perform test at 128 kbits/s with very hard samples. Aemese-like samples will be very easy to spot from original.


Sorry for my English.

All kind of thoughts and suggestions are welcomed here.

[rpp3po]

I don't think that a <128kbit/s listening test makes any sense nowadays. Nobody is using those bitrates and results cannot be extrapolated into regions of general transparency.

One encoder might be a worse low bitrate performer, but still have much better resilience against killer samples than another, once the bitrate passes a certain barrier. And since we have reached a point were most popular encoders are totally transparent for most music at 128-192kbit/s, we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome.

A lengthier version of this...

[halb27]

... we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome. ...

Yes, the value of the outcome of such listening tests is often rated too high IMO. Especially at such a pretty low bitrate. Especially as many people just take the average score of each encoder to get a quality order for the encoders.
But: Despite this listening tests do have a value IMO, though a restricted one.

As for your approach: this would give valuable information, though restricted information as well because the universe of killer samples is not known.

[C.R.Helmrich]

... we should now focus onto what encoder knows less killer samples. A listening test in its traditional form is not suited to evaluate that. The limited, initial choice of test samples will determine the outcome. ...

Yes, the value of the outcome of such listening tests is often rated too high IMO. Especially at such a pretty low bitrate. Especially as many people just take the average score of each encoder to get a quality order for the encoders.
But: Despite this listening tests do have a value IMO, though a restricted one.

As for your approach: this would give valuable information, though restricted information as well because the universe of killer samples is not known.

Well, an educated audio codec developer knows a lot about the universe of killer samples for his/her codec. I work as an AAC developer, and I've seen about two dozen high-bitrate killer samples until now. They all fall into very few specific categories of sounds, hence you could look for similar sounds (with high chances that they'll be killer samples as well), or even design your own. Today, one decade after version 1.0 of the encoder I work on, many initial killer samples are not killer samples any more. Of course, some still are, simply because AAC is not perfect, or because appropriate input analysis would make the encoder unacceptably slow. For example, you will never find an AAC encoder which at 128-160 kbps stereo will give you a transparent encoding of the emese sample.

So, rpp3po, I don't understand what you mean by "A listening test in its traditional form is not suited to evaluate that". I think it is. You just need to know the abovementioned categories.

Chris

This post has been edited by C.R.Helmrich: Jan 8 2010, 21:21

[rpp3po]

So, rpp3po, I don't understand what you mean by "A listening test in its traditional form is not suited to evaluate that". I think it is. You just need to know the abovementioned categories.

With "listening test in its traditional form" in this context I mean public listening tests as the one in discussion. A public listening test like this has usually a limited test of samples. And the more 'emesesk' samples the set includes the worse AAC will look in comparison to MP3 and vice versa. So the outcome is severely influenced by the choice of samples and thus of limited use as a tool to evaluate which is best overall. The encoder with the least problematic samples in the set wins. You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

For the individual developer, who knows its encoder inside out, a "traditional" listening test with select samples can be of great value, no question.

This post has been edited by rpp3po: Jan 8 2010, 22:48

[IgorC]

The encoder with the least problematic samples in the set wins. You can intentionally exclude problematic samples from the test, but what are you going to get then? Good results for all encoders for most music. But we already know that. My point is, isn't it time to move on with testing to compare which encoders have the lowest probability of failing over broad collections of music, even if that would be harder to accomplish than with traditional 8-16 sample shootouts?

For the individual developer, who knows its encoder inside out, a "traditional" listening test with select samples can be of great value, no question.

It's just different approach. Good one actually.

Somebody would say that testing killer samples wouldn't be representative but the same way light samples aren't representative neither.

I'm in favor to rise bitrate to 128 kbps and include more difficult samples randomly (and don't include samples which are only difficult for one specific competitor). Emese-like samples are horrible at this bitrate.

This post has been edited by IgorC: Jan 12 2010, 03:31