General discussion of future public test Options

[IgorC]

Hello,
Now when it’s sure that Apple encoder represents AAC standard very well the next step will be a public multiformat test.
No particular date or plan. It’s a general discussion with the purpose to hear the opinions, points of view, suggestions.

Items to talk about (these numbers are first approximations):
1.The number of codecs: 4+low anchor. 4 is affordable number for one test.
2.The selection of codecs (basing on curret previous votes):
http://listening-test.blogspot.com/2011/06...reparation.html

Apple LC-AAC - the most optimal AAC encoder. (Well, though FhG was quite good too).
MP3 LAME – 10 votes
Vorbis AoTuV – 8.5 votes.
Opus/CELT – 6 votes. (still no info when there will be Opus 1.0)

It’s possible to change your votes.

(?) USAC (working name of future AAC’s successor ). No information about future availability of it.
(?) low anchor
(?) Settings, versions, etc.

3. Bitrate:
VBR 96 kbps. The last AAC public test was at 96 kbps (well, it was actually 100 kbps).
Also I think it will be more interesting to compare MP3 128-135 kbps and AAC/AoTuV 96-100 kbps. Probably a lot of people are interesting to trade off between compability/compression efficiency

4. Samples.
Last time we have applied the technic of random selection of the samples. 20 samples.

The developers are very welcome to participate in this topic.

This test should be more interesting as it will include interesting (for use) codecs.

[IgorC]

There is a first issue if MP3 will be tested at 96 kbps. MP3 has a sample rate 32 kHz at 96 kbps while AAC, AotuV, Opus - 44.1 kHz.
Sampling issue.

Possible solutions are resampler or higher bitrate for MP3.

[NullC]

There is a first issue if MP3 will be tested at 96 kbps. MP3 has a sample rate 32 kHz at 96 kbps while AAC, AotuV, Opus - 44.1 kHz.
Sampling issue.

Possible solutions are resampler or higher bitrate for MP3.

I'm really not a fan of testing with multiple things unequal.

The reason for this is that the results lose their meaning and mean whatever the reader wants them to mean... or at least only one side is meaningful. If AAC with 96 beats MP3 at 128, thats potentially interesting— but if they tie or MP3 wins then "of course, it's running at a higher rate, what did you expect?".

Of course, this is also true if the rate changes the bandpass "It wasn't really better, people just preferred the other lowpass setting".

Bitrate cures a lot of sins too. A little extra rate can be the difference between transparency and not transparency.

What I would recommend is that the rates for the samples under test be run as close as possible to equal so long as there is still a good chance of any of them being a winner or loser. E.g. if we think mp3 will very likely lose if run at 96k, then we should use a higher rate— but only enough to correct that. If we run it at 192 and find that it does best, we'll have learned nothing from that effort.