Alternative Multiformat Listening Test @ 128 kbps, at SoundExpert with the same contenders Options

[Serge Smirnoff]

As Sebastian Mares have practically finished accepting test results from participants I would like to offer for all who care to take part in alternative listening test with the same codec contenders but different sound samples. Testing methodology is also a bit different and makes testing much easier because sound artifacts are clearly audible in most cases. The main task of a listener is to grade annoyance of those artifacts.

Test files could be downloaded from here – ftp://www.soundexpert.info
Each time you click the link you’ll get random test file for one of these codecs:
• Nero AAC 3.1.0.2
• iTunes AAC 6.0.1.3
• LAME 3.97 Beta 2
• Ogg Vorbis AoTuV 4.51 Beta
• WMA Professional 9.1
• Shine 0.1.4 (Low Anchor)
Inside zip file you’ll find brief instruction and form for submitting results. The whole testing procedure is not hard and takes approx. 2-3 min., so you could easily test 5-10 files in one session. Each test file is 2-3 Mb.

Results of this test will appear on this page the same time with Sebastian Mares ones. After that they could be monitored in real time as new participants will add their grades.

Testing methodology used is described here - http://www.soundexpert.info/prozone.htm

Hope, comparison of both tests results will be interesting. If someone has any questions I am glad to answer.

[Garf]

If you spend more bits on the right coefficients, the time smearing gets less. This is true even if the time resolution is low. Since MP3 is more efficient, it can afford to spend more bits to do this (remember GTune Vorbis?). That's why it's not so clear-cut.

On the page you link though, there's this:

To my mind only MPEG-4 AAC is capable to eliminates all disadvantages of the additional frequency resolution. The result is transparent coding at data rates around 120...130 kbps (instead of 170...180 kbps as MPEGplus). But a

    * high quality MPEG-4 AAC Encoder is much much more difficult to program and to tune than a MPEGplus encoder

I agree strongly with this (it's also true for MP3 vs MP2) and it points that the "theorethical" advantages any format may have can be problematic since they must be used fully and correctly. We have not reached this point with MP3 and it will take much longer with AAC.

That's why I also disagree with the statement that format X must be better than format Y. The implementation is the limiting factor.

This post has been edited by Garf: Jan 14 2006, 16:26

[guruboolez]

That's why I also disagree with the statement that format X must be better than format Y. The implementation is the limiting factor.

I also agree with it, of course
It's just that Frank Klemm comments are making sense. It's like He-AAC or Parametric Stereo: it outperforms LC-AAC or joint stereo for efficiency at low bitrate. But at higher ones, these tools are leading to artifacts or distortions you won't get with less efficient tools. This sudden change has nothing to do with magic. A sprinter is always a poor 10.000 meters runner.
There are several samples which may be transparent with layer 2 at high bitrate, even if the mp2 encoder is not intensively tuned - or may at least be less distorted than a very high quality implementation of MP3 such LAME. I can ABX castanets at 640 kbps with LAME freeformat, but I'll probably fail with most other formats at half this bitrate.
Implementation has a big part in quality, but some inherent flaw in design could definitely handicap a format. Klemm often criticized layer 3 (and not transform encoders by themselves) design in the past.

Actually, I woudn't exchange one LAME encoded file for 10 mp2 (toolame...) files at 224 kbps. There's maybe less pre-echo (it's not even sure), but there are several other form of distortions.

This post has been edited by guruboolez: Jan 14 2006, 16:57

[halb27]

... A sprinter is always a poor 10.000 meters runner. ....

I like this comparison for high / low bitrate encodings.

... I can ABX castanets at 640 kbps with LAME freeformat, but I'll probably fail with most other formats at half this bitrate. ...

As I'm pretty insensitive towards pre-echo would you be so kind to try castanets
with
- Helix -V140 -X2 -HF2 -SBT450 -TX0 -F18600 or similar
- Lame 3.90.3 or 3.91 --abr 270 -b224 -h --lowpass 18600 or similar ?

As I'm considering using bitrates lower than I did before Vorbis comes to my mind again (on my iRiver H140 battery drain with Vorbis unfortunately is rather high but I can compensate for it a bit by using lower bitrates). Can you say something towards aoTuv 4.51 pre-echo behavior for -q7 or -q6?

Thanks in advance.