Alternative Multiformat Listening Test @ 128 kbps, at SoundExpert with the same contenders Options

[Serge Smirnoff]

As Sebastian Mares have practically finished accepting test results from participants I would like to offer for all who care to take part in alternative listening test with the same codec contenders but different sound samples. Testing methodology is also a bit different and makes testing much easier because sound artifacts are clearly audible in most cases. The main task of a listener is to grade annoyance of those artifacts.

Test files could be downloaded from here – ftp://www.soundexpert.info
Each time you click the link you’ll get random test file for one of these codecs:
• Nero AAC 3.1.0.2
• iTunes AAC 6.0.1.3
• LAME 3.97 Beta 2
• Ogg Vorbis AoTuV 4.51 Beta
• WMA Professional 9.1
• Shine 0.1.4 (Low Anchor)
Inside zip file you’ll find brief instruction and form for submitting results. The whole testing procedure is not hard and takes approx. 2-3 min., so you could easily test 5-10 files in one session. Each test file is 2-3 Mb.

Results of this test will appear on this page the same time with Sebastian Mares ones. After that they could be monitored in real time as new participants will add their grades.

Testing methodology used is described here - http://www.soundexpert.info/prozone.htm

Hope, comparison of both tests results will be interesting. If someone has any questions I am glad to answer.

[guruboolez]

The testing procedure of this second listening test puzzled me.
I downloaded a file, and it appears that I can only rate one encoder for a given sample. It's a single A-B procedure. There's nothing wrong with that. What perplexed me is the role of the anchor. Isn't it intended to prevent the listener from temperamentic rating? It implies that the listener could access to the anchor while testing the other contenders. But in your testing procedure, you can only access to one encoded file and the hidden reference; the anchor, like all other contenders, are not accessible. The anchors can't therefore plays any role - at least not the anchor's one. It's just an additional contender.

It also mean that the listener can't rate all competitor in a same row. I can download:
- LAME => hearing a distortion => give it the note of 2/5
- then the ANCHOR => finding it awful => give the note of 1/5
- then download again LAME => hearing the same distortion as before => give the note of 4 because I would consider is as much better than the anchor quality I still have in mind.

It's clearly recommended to evaluate all encodings in a same raw, and to compare them each others before rating them all. It's like ABCHR softwares are working. Or at least to have the possibility to rank all encodings in a short amount of time. With SoundExpert's procedure, it looks impossible and the ranking could vary according to the testing mood, leading to uncoherent results.

This post has been edited by guruboolez: Jan 15 2006, 04:30