I keep wondering if I shouldn't get rid of Waiting once and for all.

But it's too good to let go

I've PM'd aoyumi, asking him if its ok for aoTuV beta 1 to be included in the Vorbis test or whether he has a beta 2 coming out soon.

So the encoders list consists of: 1.0.1, mtb3, QKTune b3.2, aoTuV beta 1

Anyway, since we've got the encoders sorted out, the next issue is the samples to be used. Someone suggested we use the samples in ff123's 64 kbps listening test, so that they are different to what Roberto's been using in his recent ones. The files can be found HERE (at the bottom of the page)

Does everyone, who is interested in testing, have a copy of and/or are happy with using these samples?

It answered QuantumKnot in PM. I submit beta1 after all. The following version cannot immediately be released yet.

Since it is an interesting test, I am also going to participate.

Thanks Aoyumi.

Well, Roberto mentioned a date for the 128 kbps multiformat test in http://www.hydrogenaudio.org/forums/index....ndpost&p=191805

April 7

So have I got unanimous agreement in using ff123's 64 kbps listening test samples?

Please keep in mind it's not definitive. I plan for April 7th, but it could be postponed, or even not happen at all if people decide that the few development that happened to several participants doesn't justify a test.

Test discussion should start in one or two weeks

I wonder if someone could make following version:

latest cvs + all known tunings from different versions, ultimate version and then send it to test

I'm sure it's not too hard to do that, just as throwing GT3b1 and 1.0.1 together was quite easy, since there were no conflicts. But whether each of the tunings can cooperate together constructively is another matter. My guess is that it will take a lot of time to fine tune it, and perhaps by the time we do complete it, Vorbis 1.1 will be out and.....

What's the lookout on Vorbis 1.1 anyway ?
Are they going to merge the Flac tools in to the standard vorbis toolset ?

Multichannel Ogg, which I thought was already a reality for OGM containers.

If there's a listening test for the next version, I'd like to participate, but don't know how. Any pointers would be apreciated.

I didn't have too much time for testing these last days. But I've tried to elaborate an approximate grid of bitrate for each vorbis encoder, by choosing 160 different tracks from ~150 classical music CD.

The complete table is accessible here:
for IE6: http://www.foobar2000.net/divers/vorbis/playlist160_IE.htm
for Mozilla: http://www.foobar2000.net/divers/vorbis/playlist160_MZ.htm

Excel and OpenOffice table are also available:
http://www.foobar2000.net/divers/vorbis/


At the end, I've finally encoded 4 discs of non-classical music for comparison. Results are different, but I don't have enough material for this kind of comparison.

Is the idea of a listening test still alive? I suggest:
- vorbis CVS -q4.25
- vorbis GTQK [3.2?] -q4.25
- vorbis Modest Tuning beta3 q-4
- vorbis aoTuV -q4
- maybe an uncoupled version, but it's maybe not a promising solution to put a full stereo encoder in the ring

These setting suggestions are based on my statistical results (classical only)

For the sample, the ff123 12 samples from 64 kbps listening test are still OK?
Or are they new versions of QK, MT or aoTuV to introduce in the test in the next hours/days?

I don't know if it is, but I would like to inform that I decided to start the test on April 14th. No longer april 7th because that will be just before Easter, and I will travel.

Test discussion will start really soon.

As a point of curiosity, not just related to this proposed test, why are most listening tests performed on bitrates of 128K and lower? It seems to me that many (most?) HA members encode at higher bitrates. Would it not be more practical to test higher rates? It has been noted that just because an encoder is good at 128K, it's not always true it's better at higher bitrates. Or is it because at higher rates, many listeners can't distinguish differences?

While I understand why testing 128K-ish is useful, I also notice a great deal of posts, and "unscientific" polls, asking what the best high bitrates are, and generally speaking, the best information I can find is purely subjective. True, I can test it myself, but the same can be said of 128K.

There you go.

Yes, the Vorbis listening test is still alive, though it lay dormant for a while until we got a date on Roberto's multiformat test. It seems April 14 is the date of when it starts so we should have a Vorbis encoder by that time. Essentially, it gives us two weeks, which should be enough I guess.

The competitors list hasn't changed from the previous one. I submitted QK3.2 for testing. Nyaochi also submitted MTb3. Aoyumi submitted aoTuV beta 1.

I suggested ff123's 64 kbps listening test samples be used and so far I haven't had any objections. So I think we'll stick with those samples unless someone has objections to that.

We'll finalise and freeze everything soon so suggestions on anything (competitors, samples, bitrates, etc.) should be made ASAP

Please consider I need to know the chosen Vorbis codec by the 11th-12th of April, so that I have time to do bitrate deviation tests, prepare the packages and upload them.

Roger that.

Well, I guess we should start very soon then.

Even though I'm trying to organise a listening test here, I've little to no experience with how to run one, gather the various ABC/HR results and give a final score etc. Anyone wish to volunteer to do this?

I can help with instructions and hints whoever volunteers to do so. But I won't conduce this test.

My guess is that we probably won't have enough listeners to get enough results to do any confidence interval stuff. I'm not too familiar yet.

But since this is going to be a sort of 'informal' listening test, where essentially each listener will perform his/her own ABC/HR test and post their results, I guess we can just pick the Vorbis encoder that gives the highest average score? It's quick, simple, no fuss. How does that sound?

If possible, I will want to send the test version of aoTuV. Although it is not still sufficient tuning, it is different in beta1 and is not troubled with HF boost. Moreover, probably, some noises are not conspicuous from the beta version.
Now, here is the binary(aoTuV experiment [20040402]) which can be submitted. aoTuV test page





Although this is a problem difficult to be sure, it thinks that it should judge by score fundamentally. There is little time anyway.

If there's no objection to it, we'll include aoTuV experiment.

As for how to choose the encoder, there should be time to agree on how we will do this after the raw results from the listening tests have been collected. What we need to get are the raw scores. Deciding how to interpret them can come later.

One very final issue that I need some feedback on, before launching the test. Can someone, who has the samples from ff123's 64 kbps listening test, do a quick test with the encoders using guruboolez' quality values to see if it gives on average, approximately 128 kbps?

The encoders that will be tested are:

- Vorbis 1.0.1 CVS
- aoTuV 20040402: http://www.geocities.jp/aoyoume/aotuv/aotuv_experiment.zip
- Modest Tuning beta 3: http://homepage3.nifty.com/nyaochi/soft/dist/oggencmtb3.zip
- QKTune beta 3.2: http://www.rarewares.org/quantumknot/oggencqk32.exe

Once we can determine the quality values to use for each encoder, we will be able to start right away.

EDIT: I did an encoding of two encoders on the test set and calculated the average bitrate

1.0.1 CVS (q 4.25) => 135.7 (max=171, min=114)
1.0.1 CVS (q 4) => 129.6 (max=163, min=109)
QKTune beta 3.2 (q 4) => 135.33 (max=169, min=106)
QKTune beta 3.2 (q 3.75) => 131.6 (max=164, min=102)
QKTune beta 3.2 (q 3.5) => 127.3 (max=158, min=99)

great! i am very interested in that vorbis does the best possible

I always thought a setting should be used that gives ~128 kbps on "normal" music (whole albums of rock, metal, pop, classic) and not a setting that gives ~128 kbps on the test set... am I wrong?

*smacks forehead*

You're right. I can't believe I didn't realise that. I guess I got too much going on these days to think straight.

In that case, I think guruboolez' determined quality levels should be okay then.