Pre-Test thread, for the 64kbps test Options

[rjamorim]

Hello.

As most of you already know, I am planning to start a 64kbps public listening test in September.

So here are the planned details. Nothing is definitive so far:

The test starts at September 3rd and ends at September 14th

The codecs that will be tested are:
- Ahead HE-AAC "Streaming :: Medium" VBR profile, high quality.
- Ogg Vorbis post-1.0 CVS -q 0
- MP3pro codec in Adobe Audition, VBR quality 35, high quality, m/s and is stereo, no CRC, no narrowing.
- WMAv9 Standard 64kbps (there's no PRO version at such bitrate, AFAIK)
- Real Audio Cook 64kbps (I didn't investigate other settings yet. Comments welcome)

The samples that will be tested have been announced on this thread. If you have concerns/comments about the sample suite choice, please post there.

The test results will be calculated the same way my former tests were. I don't plan to include bitrates in the formula. Comments are welcome now (they are of no use to me after the test has been started).

I haven't decided about anchors yet (my guru is traveling ), but someone suggested me that I use Lame ABR 128 as higher anchor, so that we can verify which one of these codecs really deliver the marketing of "sounds like MP3 at half the bitrate"

Then, the lower anchor would be a standard 3.5kHz lowpass, like it is done on most formal tests at low bitrates.

So, I'd like to know your thoughts on what I planned.

Thanks for your attention;

Roberto.

This post has been edited by rjamorim: Aug 22 2003, 07:53

[Dibrom]

As Roberto pointed out earlier, and I agree, I think that adjusting the settings away from the common incarnations, just to "set" the bitrate on this test, calls into question it's credibility.

Not at all. This should give the same result as 2-pass ABR, just done manually, instead of automatically.

I understand that some people do not agree with me. But that is where I stand, and there seem to be people that agree with me. *shrugs*

You seem to have missed the main point of my post:

This test should reflect a real world usage scenario as much as possible.

Artificially tuning the settings does not do this, not to mention that 2-pass ABR doesn't even exist for these codecs (with the exception of WMA maybe? even so, this is not case being called into question).

I do understand where Roberto and co. are coming from and saying that the mode used should be the VBR mode that produces an average of "x" kbps for a broad spectrum of music. The problem is that we're not testing that broad spectrum of music.  We're testing  several individual samples.

This is arguable. We are testing several individual samples comprising a broad spectrum of music. The two conditions are not mutually exclusive. Granted we are not testing every type of music, but that is impossible in any case, and if we're going to nitpick on that point, then we might as well forgo the test entirely.

No matter what the case here, people are going to have to accept that these codecs are not being tested under every condition (genre of music, or even sample for that matter) possible. That means that the results, same as with all tests, have to be taken with a grain of salt, and within context.

In terms of equality, ABR works much better for a small, focused listening test like this one, merely because it provides both a single, standard testing methodology, and uniform bitrates across samples.

I disagree. For one, most ABR modes are based upon VBR (so they share many possible flaws). They are further encumbered though by catering to bitrate first, and quality second (or quality restrained or superceded by bitrate if you prefer).

As I understand it, simply by virtue of emphasing VBR as the method of choice in this test, the point is to measure quality first and bitrate second here. I see no implicit virtue in testing ABR over VBR either -- it would seem to me to be a rather arbitrary choice and one not directly tied to the test's focus which, once again, should be related to provided quality in a real world usage scenario.

Most ABR modes in audio codecs are not entirely accurate in terms of bitrate either, so there is no guarentee that using such a mode will provide any more stable of a foundation than VBR in the first place.

The tester should make every effort to provide as little difference between codecs as possible.

No, the only real necessity is that the tester attempt to equalize the perceived classification of the codecs being tested, only making adjustments where gross mismatches occur.

This means that the absolute bitrate is not nearly as relevant as the wide-scale average bitrate (going beyond samples in this test) or usage scenarios.

What I believe we should be focusing on first is, again, the perceived classification (-q0 vorbis is said to compete with other codec at 64kbps avg mode, for example) first, and the average bitrate second, not worrying about the absolute sample by sample bitrate.

Roberto did not do that with the 128 test. Instead, he threw together a mismatch of VBR and ABR, rationalizing his use of VBR with the broad-spectrum tests.

I cannot speak directly for Roberto, but I believe this is because he was approaching the test from the perspective that I just laid out above.

It remains an inequality between codecs. This is the problem I have with the theory behind these tests.

The problem is that these codecs aren't equal, and so any test including them cannot expect the conditions of comparison to be pefect down to the attribute level of every single codec, only varying by common value. These codecs are different in abilities and behavior, and so if we are to test them and compare them meaningfully and representatively, we need to focus on measuring a generalized and abstracted point (quality levels) not a specific and particular point (performance at exact bitrates).