General discussion of future public test Options

[IgorC]

Hello,
Now when it’s sure that Apple encoder represents AAC standard very well the next step will be a public multiformat test.
No particular date or plan. It’s a general discussion with the purpose to hear the opinions, points of view, suggestions.

Items to talk about (these numbers are first approximations):
1.The number of codecs: 4+low anchor. 4 is affordable number for one test.
2.The selection of codecs (basing on curret previous votes):
http://listening-test.blogspot.com/2011/06...reparation.html

Apple LC-AAC - the most optimal AAC encoder. (Well, though FhG was quite good too).
MP3 LAME – 10 votes
Vorbis AoTuV – 8.5 votes.
Opus/CELT – 6 votes. (still no info when there will be Opus 1.0)

It’s possible to change your votes.

(?) USAC (working name of future AAC’s successor ). No information about future availability of it.
(?) low anchor
(?) Settings, versions, etc.

3. Bitrate:
VBR 96 kbps. The last AAC public test was at 96 kbps (well, it was actually 100 kbps).
Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency

4. Samples.
Last time we have applied the technic of random selection of the samples. 20 samples.

The developers are very welcome to participate in this topic.

This test should be more interesting as it will include interesting (for use) codecs.

[IgorC]

Some feedback was received on #Hydrogenaudio and I would like to bring some information and answers basing my opinion on past public tests.


1. Can we go for higher bitrate (>96 kbps)?
There were 531 results for 64 kbps test and only 280 results for 96 kbps AAC test while it was open during more time. 280 results were just fairly enough to close the test.
64 kbps test has continued during 20 days, 96 kbps – 35 days (hence more difficult).
It’s not recomended to keep the test open for more than one month or so.

So it’s hard to imagine a public test at higher bitrate.


2.a) Will be it more reasonable don’t discard all results of a listener who submit more invalid results that it’s allowed by rules? Keep “good” results and discard only erroneous results?
First of all it worth to mention that this question isn’t new.
The problem is that it’s impossible to know if it was a really “good” result or it was a “lucky guess”. So it’s not accepting the “good” results and discarding the “bad” results but rather accepting “lucky guessing” results (which is still invalid one) and discarding “not lucky guessing”.

Answer: it’s more correct to discard all results of the listener who has past the limit of the invalid results (submit too much of them). Also see (1º)

2b) One would say then we can check if the results of this one particular listener correlates enough well with the results of other listener then it’s ok to accept ¨only good results ¨.
The problem that later developers and memebers will complain why the rules weren’t applied homogeneously (to all listeners in the same manner).
Then the simple and effective decesion was made:
The rules were _strictly_ applied (as of previous public tests 64 amd 96 kbps). No context. You follow the rules – your results are in. If not – then no but you can start from zero. Simply as that.



Also my observation about placebo effect and the interpretation of the results during the public listening test.

The score 4.0 means “perceptible but not annoying”. And how should the results of the last AAC test be interpreted in case of 4.1 4.2 4.3 etc...?
2-3 steps (0.2-0.3) is considering like a “very few ones” (in general interpretation. It’s psychology so I won't opine here ). But more than 2-3 steps aren´t “very few” . So one could say that 4.2-4.3 is a limit score where the high quality begins.
I don´t know if it´s coincidence but this score (4.2-4.3) is also where placebo effect starts to reveal itself more. (º1) Another interesting observation is that placebo leads to lower score than normal. There were quite enough results where the listener has put lower score if it was actually placebo. It leads to “flipped results” . It means if average results were Codec A > B > C > D then flipped ones (with placebo effect) will be D>C>B>A.

(oh lord, my english )


Speaking of this test we can go for both MP3 96 and 128 kbps. It's a period to throw the ideas.

This post has been edited by IgorC: Dec 25 2011, 18:28