Pre-Test thread, for the 64kbps test Options

[rjamorim]

Hello.

As most of you already know, I am planning to start a 64kbps public listening test in September.

So here are the planned details. Nothing is definitive so far:

The test starts at September 3rd and ends at September 14th

The codecs that will be tested are:
- Ahead HE-AAC "Streaming :: Medium" VBR profile, high quality.
- Ogg Vorbis post-1.0 CVS -q 0
- MP3pro codec in Adobe Audition, VBR quality 35, high quality, m/s and is stereo, no CRC, no narrowing.
- WMAv9 Standard 64kbps (there's no PRO version at such bitrate, AFAIK)
- Real Audio Cook 64kbps (I didn't investigate other settings yet. Comments welcome)

The samples that will be tested have been announced on this thread. If you have concerns/comments about the sample suite choice, please post there.

The test results will be calculated the same way my former tests were. I don't plan to include bitrates in the formula. Comments are welcome now (they are of no use to me after the test has been started).

I haven't decided about anchors yet (my guru is traveling ), but someone suggested me that I use Lame ABR 128 as higher anchor, so that we can verify which one of these codecs really deliver the marketing of "sounds like MP3 at half the bitrate"

Then, the lower anchor would be a standard 3.5kHz lowpass, like it is done on most formal tests at low bitrates.

So, I'd like to know your thoughts on what I planned.

Thanks for your attention;

Roberto.

This post has been edited by rjamorim: Aug 22 2003, 07:53

[Canar]

@wildboar:
The multi-pass ABR method I described would work perfectly for a VBR system like Vorbis's or Musepack's. It is in no way akin to your analogy. I understand the need to mix ABR and CBR here. I do.

The analogy falls flat, though, because it is nothing more than superficial. It doesn't relate at all to the test other than in a very, very general sense. And it's not adequate for discussion.

Perhaps something more adequate in describing the diffence between ABR/VBR/CBR is as follows:

We're studying the genetics of the tail lengths of dark cats trying to find a breed that has the longest tail, whilst remaining suitably dark. Due to some bizarre reason, we can only pick breeding pairs that are dark, and we study the tail lengths of the kittens.

So, do we pick parent cats that have:

A great majority black , but several pure white kittens? (VBR)
All dark kittens, with some that are black, some that are lighter? (ABR)
Or kittens that are all a uniform shade of dark grey? (CBR)

If all the kittens coat colours were properly weighted (however that is done) and averaged, and they all worked out to the uniform shade of dark grey of the CBR kittens, would all breeding pairs be acceptable as parents to be tested?

It adequately analogizes in my mind. I may not have explained it thoroughly enough, so I hope you all can catch my drift.

@dibrom:
Thank you for the in-depth response, foremost. I did decide to pick one specific area and debate that. I got your main thrust, and I understand the need for similarity to the real-world. What I meant to do was describe the way an equivalent of a 2-pass ABR mode could be achieved using a numerical quality selector.

I'm going to forgo debating every single point of your response. You raised several more points that are opinion-based, and we could argue them all day and achieve nothing.

You did make some good non-opinionated points, though. I'll pick a few that stand out to me.

What I believe we should be focusing on first is, again, the perceived classification (-q0 vorbis is said to compete with other codec at 64kbps avg mode, for example)

It has this perceived classification? My understanding was that the coders intended -q0 Vorbis to work out to 64kbps on average, not compete with a 64kbps average codec. Furthermore, it was my understanding that everyone acknowledges that -q0 may be a little off in one direction or another.

The name of the test states that it's testing 64kbps codecs. I'd think that this implies that perceived classification does not enter into the picture; rather, that 64kbps should be the bitrate, or something thereabouts.

No, the only real necessity is that the tester attempt to equalize the perceived classification of the codecs being tested, only making adjustments where gross mismatches occur.

What's the point of the test then, if the codecs aren't on even ground? Attempt to equalize, so, in other words, set the codecs up so there's as little difference between them as possible? That's what I meant, if not what I said. We agree here.

we need to focus on measuring a generalized and abstracted point (quality levels) not a specific and particular point (performance at exact bitrates).

But ABR does not do that. ABR shifts the focus to the latter, not the former. CBR doubly so. This is exactly the point that bothers me.

I understand that Roberto's aiming for real-world results, and thus I can see why he does not wish to use anything other than the encoding methods directly available through the encoders, but there are some problems, that I think can have some detrimental effect on the overall test.

Dibrom, I apologize if I glossed over your message. It was long, and there was plenty of issues for me to address, so if I missed something, please tell me.

@rjamorim:

Why, thank-you for your kind words.

Let me quote my guru here, Darryl Miyaguchi:

"For those who would have done it differently: the opportunity is still there for you! If you can dish out the criticism, can you stand to take it too?"

Forgive me for wording that as strongly as I did. I didn't mean to be unkind, I meant to emphasize that I would have done things in a different manner, had I been the organizer. I'm not. You're putting in a great amount of time and effort to set the test up and to defend it. You have my respect for that, and I do not mean to seem otherwise. I can take criticism; I thrive on it. I do not presently get as much of it as I would like. I also greatly appreciate what you're doing. I forgot to emphasize that. If you weren't taking the time to do it presently, it wouldn't be getting done. That said, I had problems with the way the test was performed, although I think the LAME 128ABR anchor and the reasoning behind it was a stroke of genius, and added a human touch to all the dry science and ABX tests.

I read through the 128 test's explanations. I disagree with some of their assertions. Ultimately, what we're dealing with here is a difference of opinions. I suppose I've made enough noise about this to last for some time. I won't bring this topic up again, I just hoped I could make a difference and explain to other people the way I saw things. We'll see how you all take my cat analogy.

EDIT: And yes, I did see that response. I just disagree.

This post has been edited by Canar: Aug 23 2003, 06:11