Public Listening Test [2010], Discussion Options

[IgorC]

Many people will be agree that it’s time for public listening test.

Short Agenda:
1. Organization
2. Samples
3. Codecs
4. Bitrates

Detailed Agenda:
1. Organization.
People with experience of personal or public test are welcome here.
Personally I can afford help for conduction of test.
Unfortunately if I understand correctly that Sebastian informed that he don’t want to conduct public tests.

2. Samples.
Different styles of music, different levels of difficulty, pointing issues etc....? To be discussed here or in separate topic.

3. Codecs
3.a) Multiformat test.
3.b) AAC test.
I think that it’s more appropriate to conduct AAC test because there are at least 3 AAC encoders to test. Nero, Apple and Coding Technologies/Dolby. All these codecs were updated during this year. More information in List of AAC encoders

4.Bitrates.
96-100 kbits/s?
It's possible to perform test at 128 kbits/s with very hard samples. Aemese-like samples will be very easy to spot from original.


Sorry for my English.

All kind of thoughts and suggestions are welcomed here.

[Sebastian Mares]

Are you sure 48 kbps LC is not too exaggerated? Do you expect any contender to be worse or as bad as 64 kbps?

[C.R.Helmrich]

IgorC: Yes, Fraunhofer's AAC encoder supports VBR coding. We don't distinguish between TVBR and CVBR, though, so that VBR mode is most likely a CVBR mode.

Sebastian: Maybe we should think about using a generic low anchor, as done in MUSHRA tests. How about a 7-kHz lowpass filtered version of the reference? Sonically, that should be quite similar to 48-kbps AAC LC. And we wouldn't have to worry about which encoder to choose for generating the anchor.

As mentioned in the AES Journal paper referenced here, the goal is to span the entire range of the grading scale with the codecs and anchors, in order to minimize bias, i.e. reduce the width of the confidence intervals. A 7-kHz anchor (and 48-kbps LC) should be about mid-way between "very bad quality" and the quality of the 96-kbps LC encoders. To define the lower end, we could throw in 8-kHz 8-bit a-Law PCM of the reference as "64-kbps phone quality low anchor". That will take very little listening effort to identify, so it doesn't make the test more difficult, but gives us the advantage of stabilizing the test results.

Chris

[IgorC]

A 7-kHz anchor (and 48-kbps LC) should be about mid-way between "very bad quality" and the quality of the 96-kbps LC encoders.

Are you sure 48 kbps LC is not too exaggerated? Do you expect any contender to be worse or as bad as 64 kbps?

LC-AAC 56 kbps or 8-khz anchor could be a good compromise. Both looks good for me.


As poll indicates
there are two widely used AAC encoder on HA.
Coding Technologies to be dropped as it's only CBR and not so popular here.

It's clear now at least for me speaking about propsal list:
1. Nero
2. Apple
3. FH
4. Divx

The idea was not to test LossyWAV transparency, but to download smaller files. BTW that could also apply to the proposed 7 kHz-filtered low anchor.

I think it's not good idea from point of view of credibility of final test. There will be rumors like "not 100% lossless references"
I'm completely against of lossyWAV here.

This post has been edited by IgorC: Jan 6 2010, 00:52