Codecs and settings for 64kbit/s SE listening test, criticism required Options

[Serge Smirnoff]

As SoundExpert now has pretty stable flow of volunteer testers it is time to update codecs in all bitrate sections. After a short discussion at Russian bit-torrent tracker we decided to update 64-96-128-192-256-320 sections first. For the 64kbit/s section five codecs were chosen:

Fhg AAC (2012-06-24) - 59.2 kbit/s (fhgaacenc --vbr 2 se_ref.wav)
QAAC TVBR (v1.42) - 59.6 kbit/s (qaac --he -v56 se_ref.wav)
Nero AAC (v1.5.4.0) - 60.1 kbit/s (neroAacEnc.exe -q 0.25 -if se_ref.wav -of out.mp4)
Vorbis (Xiph 1.3.3) - 60.3 kbit/s (oggenc2.exe -q-0.3 se_ref.wav)
Opus (libopus 1.0.1) - 59.9 kbit/s (opusenc --bitrate 59 se_ref48.wav out.opus)
Conversion chain for Opus: 44.1/16 -->> 48/24(Audition CS6) -->> opusenc -->> foobar2000(48/24) -->> 44.1/16(Audition CS6)

Bitrates are calculated on the basis of nine SE test samples concatenated. Fortunately first two codecs have close values of resulting bitrates at their corresponding discrete quality settings. Other contenders were adjusted to have close values.

Did we miss something?

[C.R.Helmrich]

Thanks! How does the reliability rating work? How many listeners are needed for, say, 5 percent? And why is Opus graded worse than mp3?

Chris

This post has been edited by C.R.Helmrich: Oct 28 2012, 17:44

[Serge Smirnoff]

How does the reliability rating work?

Each time a device under test receives a grade the rating is recalculated. A sequence of such ratings tends to some final value. ((Max-Min)/Last value)*100% over last N values is reliability. Now N = number of test files for a device. For low bitrates when artifacts amplification is not applied the number of test files equals to 18 (9samples*2).

How many listeners are needed for, say, 5 percent?

Usually 5-7 grades are necessary for each test file in order to achieve 5% reliability of rating. Due to above mentioned nature of the parameter there is no strict relationship between accuracy of ratings and number of returned grades.

And why is Opus graded worse than mp3?

At the moment Opus received 9 grades only and its rating is completely unreliable. I will check now why it shows 4% …

[skamp]

Usually 5-7 grades are necessary for each test file in order to achieve 5% reliability of rating. […] At the moment Opus received 9 grades only and its rating is completely unreliable. I will check now why it shows 4% …

I don't understand

[Serge Smirnoff]

Usually 5-7 grades are necessary for each test file in order to achieve 5% reliability of rating. […] At the moment Opus received 9 grades only and its rating is completely unreliable. I will check now why it shows 4% …

I don't understand

5-7 grades for each test file, number of test files is 18. Opus received 9 grades in total, i.e. not all test files received even 1 grade.