*clears throat*

Well, Roberto in this thread hinted that the next multiformat test will start next month and he also said he has no problem including the non-official Vorbis encoders.

Obviously, us Vorbis enthusiasts want our favourite encoder to do its utmost best in the tests (personally I'd be happy just to see progress rather than victory) so I'd like some feedback on how we shall select the encoder. For the developers, we have a month to fine tune our encoders. So is there anyone who would be available to prepare/participate in a small listening test, similar in style to the official 128 kbps multiformat tests on all the Vorbis encoders (1.0.1, GT3b2, aoTuV, MTb2, QKTb3.2) just before Roberto starts to organise the multiformat test next month? The target bitrate is 128 kbps so q 4 is the likely range. The samples used in the listening tests have a wide range of genres so we want the best ranked encoder.

Any thoughts?

That's true. Since I don't know much about Vorbis, I think it's best if the format enthusiasts decide what to test. (I already feel the wrath of purists complaining I should have used the official encoder, but oh well... )



Well, I'm not sure if it'll be held next month or this month. My initial intention was to start it on march 24th, but I guess I should give you guys more time to rest from the previous listening test. And I also think it's fair to give the unofficial developers some time (although I do get shivers thinking of encoders specifically tuned for the tested samples. I mean, that would be quite unfair).

I'll decide about the best schedule for me and for you (developers, listeners, etc.), and will announce my decision until the weekend.

Regards;

Roberto.

That is based on the assumption we are using exactly the same test samples to tune though, which is perhaps partly true. I used castanets a lot to do pre-echo tuning but I think these are rather generic samples. From the various listening tests in the Vorbis-tech forum, I think it is safe to say that we have our own samples so no need to fear.

I'd definately participate.

Although I'm not a Vorbis' enthusiast (99% of my archiving are *.mpc), I have some comments.

At the moment, I personally would like to see qk3.x(I don't know which one is better, I'll test them though.), otherwise, I'd like to see mtb2 as well, if the average bitrate doesn't matter.

Anyway, I think Xiph's official will never be a good competitor for some reasons.
1. At least to me, it's inferior to recent encoders(aoTuV, Modest Tuning, QK) even worse than some its older versions(RC3) to some people.
(People can see some results of mine. tests.html)

2. We already know its quality in the previous test. I don't think it's good that wasting the space for it.


Completely agree.

Oh, everyone is free to discuss here. It's not just for Vorbis enthusiasts.

With regards to the QKTune series, beta 3.2 is the only coder which has been tested more and done reasonably well. The latter ones (3.3 and 3.4) are experimental and are what I would call 'disposable' releases...."test them, if they don't do well, discard" Unfortunately it doesnt fix every problem in Vorbis (best example is the Brahms3 sample) which is why mtb or aotuv are very strong contenders.

Just wanted to make something clear: Having two different Vorbis tunings is definitely not an option. Only one tuning will be tested, the same way as only one AAC implementation will be tested.

Yes, obviously. Eventually we will select one Vorbis encoder to participate in the multiformat test. The question is...which one?

The first thing to do is to find the quality setting for each fork which gives ~128 kbit over a wide range of samples. Only then, a useful comparison is possible.

I know where to get the GT3b1 and GT3b2 encoders from. Where do I get the others?

aoTuV
1.0.1 Modest Tuning beta2
QKTune
GT3b2 with HF Reduction

Or you could try search function.

Thank you - as for the search function I was not to sure what i was looking for. Consider wrist slapped.

As I already mentioned in the AAC listening test thread, my vote goes to oggencqk32 AKA QuantumKnot's tuned encoder version 3.2. At -q4, the bitrates are pretty close to 128kbps.

From the ABC/HR results I've seen so far (comparing different tunings), this encoder seems to consistently achieve the highest quality (together with Modest Tuning, but this one bloats the bitrates way to much to be useful in the listening test).

Except on the classical samples

I have some fears.
1/ A public listening test seems to be a lot of work. Choosing samples, finding the good presets, building the config txt files, uploading them, collecting informations, motivating people before the end of the test, fighting against criticism after the test... I fear that nobody will accept to conduct this kind of test (I prefer to see you, Nyaochi or Aoyumi spending their limited time on improving vorbis codec )

2/ A listening test with vorbis 1.01 as expected "loser" is certainly difficult for most people. But we need significant results. At the end, we need to see clear differences (even if differences are small). Otherwise, the test is useless. I fear that few people could be helpful here... Willingness people are maybe tired, with MP3@128 than AAC@128 and soon multi@128. Add a vorbis@128 in the same month, and I fear that people are going to be mad.


I suggest the following thing. If few people could send useful results, I don't think it's really necessary for someone to prepare archive samples, abc/hr material, etc... all the painful work. The challengers are not related to different editors (with "good" one and "evil" one), or linked to different ideology ("drmed crap format" against "paradisiacal free format"). I don't think that people are going to spend their time cheat in order to favour a specific encoder: all codec seems to be neutral.
Therefore, I suggest that people should prepare themself their own test material. All we need are to specify the conditions :
- samples
- challengers and settings
- if necessary, some rules if listeners are confront to some problems (ex. if you failed on ABX tests, listener must put the slider on 5.0, or eventually on 4.9, but not keeping the initial note of 3.7... seems logical to me, and this will avoid some epistemological problems encountered by Roberto). Other rule: testing all samples, and not only three or four.

I suppose and expect that 6 or 7 trained people will participate to this test. Nyaochi, Aoyumi, QuantumKnot, me. Maybe [proxima] or some other good listeners interested by vorbis, or simply interested about quality. Of course, it's not fully scientific. Just something pragmatic. We need good leads about quality of current vorbis encoders, before chosing one for the next multiformat test. Results should be available for public of course, and "winner" should be considered as the next recommanded version for daily encoding.

What do you think?

I agree, another "large scale" listening is stressing for the listeners who partecipated in the past tests and there are also a lot of time to invest (the results have to be ready before the multiformat test v2 launch) for the organization. Better to invest this time in a useful manner improving Vorbis.

I think i will partecipate if we decide to go with this sort of "self-prepared" test.
As Continuum said, the first thing to be attentioned is surely the bitrate, we need to know the exact quality setting for 128 kbps average to prevent unfair comparison. Next, we will continue with samples and rules.

Very good points, guruboolez. It didn't come to my mind that organising a large scale public listening test involves a lot of work. I agree with your idea that each of us should prepare our own test/samples and report our results. For me, I haven't participated nor done an ABC/HR test before so I, unfortunately, won't be one of those testers. My ears are only just good enough to ABX some samples but not good enough to rate samples (which is why I have to rely on other members with good ears to help me).

One question that I have is with regards to finding the quality where the average bitrate is approximately the same. It may not be that much fairer to scale back quality in order to match bitrate since VBR encoders, that are esp. tuned for short block switching, will jump high in order to maintain the same quality. We run into the dilemma of whether to use the same quality (q 4) or same bitrate (different qualities).

We need the same bitrate. If we don't do that, don't expect from Roberto or anybody to put an encoder with 160 kbps as average bitrate for -q4 on a (explosive) multiformat at ~128 kbps. Furthemore, if bitrate doesn't matter, I also suggest an uncoupled encoder: one big problem is removed with them.

We need an approximate average bitrate for each vorbis encoder.
I remember that in the last test, -q 4,25 was used for official 1.00 encoder. I suppose that if QK32 increase the bitrate on sample with a lot of transients, -q4 will be OK.
Most problematic encoder is maybe ModestTuning: a part of the bitrate/quality scale is totally different from official encoder (at least with MT v.1).

Yes. I don't think MTb2 is suitable for 128kbps listening test because of bitrate inflation and uncontrolable bitrate (bitrate is dependent on samples). Target bitrate of MTb2 is much higher (around 160-200kbps) than 128kbps. So I have a feeling that we should remove MTb2 from this test.

Of course I will participate this Vorbis only listening test which leads to Vorbis improvement.

But couldn't we use MT with lower quality setting? Something like -q2 or -q3? Your encoder is maybe very competitive at lower setting.
For exemple, I suspect your "uncoupled channel" release at -q2 to perform better on classical samples than any other official or experimental vorbis encoder. I don't know how it would perform on metal compared to QK32, but for classical (and related problems), -q2 is still very good.

If there's an encoder to remove, it's probably GT3b2. Garf modifications don't directly affect -q 4, and quality is certainly very close to 1.01 one.

Yes, GT3b2 will perform similar to 1.0.1 but I was hoping we could use this chance to see if the merging of GT3b1 and 1.0.1 was any good. But in order to keep the testing simple, we should try to keep the number of encoders to a bare minimum.

Just to summarise the good points made so far:

- The contenders will be:
1.0.1 (probably as an anchor)
aoTuV
MT
QKT

- The quality for each encoder should be adjusted so they give an average of about 128 kbps
- We'll allocate a certain timeframe (which is uncertain until we get a date from Roberto) to give developers a chance to continue tuning their encoders.
- For those interested in testing the quality of these Vorbis encoders (and I take the precedent in thanking them greatly for their interest and effort), they are free to prepare it themselves (aka the 'self-prepared' tests), subject to the conditions, which guruboolez listed above, namely:

- samples
- challengers and settings
- if necessary, some rules if listeners are confront to some problems (ex. if you failed on ABX tests, listener must put the slider on 5.0, or eventually on 4.9, but not keeping the initial note of 3.7... seems logical to me, and this will avoid some epistemological problems encountered by Roberto). Other rule: testing all samples, and not only three or four.

How does that sound?

I think that, fewer are the number of the encoders tested, better we can judge each encoder. The differences could not be so striking for some of them because of the common 1.01 base and with more than 4 encoders the rating become very difficult or imprecise. I'm for dropping gt3b2, we have to pay attention on quality more than usability so we can test successul merging later.
Nevertheless, i agree with the list of contenders. In the past test, i found nyaochi's uncoupled very good but i did'nt noticed the increased bitrate. I'm sure guruboolez, that tested -q2, is much more aware of the quality of uncoupled compile at ~128 kbps. If he think that uncoupled should be tested at the cost of one more contender we can go with it.

Obviously, personal testing is always appreciated.
This is not related to this "self-prepared" test where we have to decide the samples, the same for all listeners, isn'it ?.

Oh sorry, I got mixed up there. I forgot that the samples will have to be chosen for all to use.

(As you can tell, I don't have much experience with these things)

The previous sample suit of Roberto MP3 or AAC 128 kbps test is maybe a good basis. For me (I must admit that), it's very confortable: I don't have to download ~30 MB with my poor dial-up
What do you think?

I think that is the most attractive solution right now, since you, harashin, [proxima], already have these files, while for those who didn't participate in those tests, they will still need to download lots of files anyway, so might as well reuse these samples.

Are these sample suits still available for download now? I might get a copy soon.

From the last test:
http://rarewares.hydrogenaudio.org/samples/Sample01.zip BigYellow
http://rarewares.hydrogenaudio.org/samples/Sample02.zip bodyheat
http://rarewares.hydrogenaudio.org/samples/Sample03.zip DaFunk
http://rarewares.hydrogenaudio.org/samples/Sample04.zip gone
http://rarewares.hydrogenaudio.org/samples/Sample05.zip Hongroise
http://rarewares.hydrogenaudio.org/samples/Sample06.zip Mahler
http://rarewares.hydrogenaudio.org/samples/Sample07.zip mybloodrusts
http://rarewares.hydrogenaudio.org/samples/Sample08.zip NewYorkCity
http://rarewares.hydrogenaudio.org/samples/Sample09.zip OrdinaryWorld
http://rarewares.hydrogenaudio.org/samples/Sample10.zip Quizás
http://rarewares.hydrogenaudio.org/samples/Sample11.zip velvet
http://rarewares.hydrogenaudio.org/samples/Sample12.zip Waiting

I've tried one link, it seems to work.

Is there a prevision to change the sample set for the incoming multiformat test ?
Using the previous sample suite is a quick and good solution but there is the risk that probably some people, reading our comments, will identify Vorbis and cheat the final results of the multiformat v2 test. Am i maybe too paranoic ?

I have the samples too. But doesn't this favor Vorbis codec in multiformat test if Roberto has a plan to reuse the samples for multiformat test? I think we should separate test collection from learning (tuning) collection in order to keep fairness.

Well, lame mp3 and iTunes AAC were also tested by a lot of people with the same samples. Same risk apply to them

I'm not that familiar with how these tests are run though I have noticed the Waiting sample, to have been used a few times (understandably). But do they reuse a lot of past samples?

Yes, i suppose. We can't know this for sure but, generally, only few samples were changed between different tests. The whole sample set was changed only once.
Maybe we can go with the past sample set that was used with ff123's 64 kbps listening test. The samples are still available at the end of this page: http://www.ff123.net/samples.html
I suppose guruboolez already own these samples. What do you think ?

I have them.
But don't bother too much about me. If I have to download 30 MB, I'll do it.
Nevertheless, I like your idea. There are also on the same page the four files related of the 128 kbps (for the brave).

http://www.hydrogenaudio.org/forums/index....ndpost&p=189475

I guess I'll have to do a new call for samples >_<

I am a newbie to the Vorbis world...so please bear with me.

It the purpose of this test to find the optimal code base that can be consolidated into a future Vobis release? Is the plan to combine all these "forks?"

Also, if GT3b2 is the current "Recommended Ogg Vorbis Encoder," why would it be excluded from this test?

The codebase for all the tunings is basically the same - they all are based upon xiph.org´s libvorbis. For example qkt32 only differs in three files.




Garf won´t work on Vorbis anymore for various reasons.

Maik

Maybe this test will help to determine if it should still be the recommended version.

...i'm assuming that this "preparation" will eventually borrow from the various forks to create an optimal libvorbis (and, obviously, create new code...) --> which will be used for the official multiformat test

is this what you guys are expecting (qk, harashin, guru, nyaochi, etc, etc) ?


later

My suspicion is that we won't have enough time to try merging the various forks. It will require some fine tuning so this is something for the not-too-distant future.

The main goal of this whole exercise is to select the Vorbis encoder which has the best quality and submit that to Roberto's upcoming multiformat test.

The results from this will obviously affect very much what is to be the next 'recommended Vorbis encoder' but that's in the future.

ok, first, before we get ahead of ourselves, we need to decide on the forks (i think thats been decided). then the samples (roberto's test i guess) and the settings for each (so we know what q produces ~128).

after, i suggest we determine what we like/dislike about each encoder's results and choose the best "base" encoder. then its a mad dash to incorporate anything we feel can improve upon that encoder.

i feel kinda sleezy using previous test samples to tweak so perhaps after this initial test we can choose some different samples so we aren't accused of tweaking the test. it wouldn't be fair.

im going to DL the samples and run some tests to determine some good ~128 settings. I'll post my findings after.

Don't try to determine an average bitrate value with short samples. Most public samples are problematic one. Often, problematic mean high bitrate with vbr encodings.
If you want to help, better take some CD of your, and encode them with different encoders/settings.
I'll do that (later, I can't for the moment because I don't have my lossless library on my current computer), but I know that encoders generally react differently with classical music. I fear that my results won't be representative.

/me is very, very afraid.

People, developing encoders is not done in mad dashes. When you implement something you feel will improve a sample, you sometimes break everything else. If you just "throw everything together hoping each technology will improve the outcome a little", you end up with On2. Specialy if your deadline is less than one month from now.

Now, if you come up with an encoder "tweaked" without any method and without wide testing, you will harm vorbis - because it might well get a score it doesn't deserve - and you will harm my test, because the vorbis scores will be meaningless for people in the know.

When I announced I was going to accept unofficial tunings, I was hoping you guys would choose among what is already out. I never asked or hoped for a tune specially done for my test. Actually, me accepting an encoder tuned specifically for my test would be arguable at best (and immoral at worst)

Sorry to say this, but I'm afraid I won't be able to accept this encoder you are planning. There are several reputations at stake (Xiph, Vorbis, mine, yours...).

Regards;

Roberto.

Roberto's right. Having worked in IT for 13 years, I can testify to the need for extensive testing to insure the best-possible quality (not just sound quality, but overall quality of the programming). "Unofficial" tuned encoders should be used in this test to try out one of the latest-and-greatest (but also proven) encoders for each codec. But the key word is *proven*. Quick "throw-togethers" can't be tested well enough in the time before the listening test.

Using encoders of established quality is the only fair way to represent each format.

In that case, I'll put a freeze on beta 3.2 and submit that for testing. It has already done well in some of the personal listening tests so there is little more that can be improved and frankly, I don't really have the time to work on it.

I'll upload a fresh compile of it, based on the source code I publicly posted, which had some last minute pre-echo adjustments (incl. in John's OggDropXd compile) compared with the first binary I uploaded. I also encourage other Vorbis developers to confirm that their latest code is finalised and ready for testing.

That should put an end to the uncertainty here.

Thank-you for understanding

Yes... and no.
Most encoders used in tests don't have established quality. They are just the lastest version of each format. Think about mpc for exemple: each new version automatically became the recommanded version, without public testing or collective founded proof. Same thing for Nero AAC, QT AAC, etc... Only exception is maybe lame.

Our latest tests tend to the following suspicion: official 1.01 is maybe the worse challenger of the vorbis team.

Roberto: "Understanding" is my middle name. jk

OK, here we go:

http://www.hydrogenaudio.org/forums/index....ndpost&p=187669

Or if that page is too cluttered and confusing, the binary can also be downloaded from:

http://steve8988.homestead.com/files/vorbis/oggencqk32.zip

Final and frozen beta 3.2 reloaded. I officially submit this encoder and binary for the great Vorbis listening test. May it perform graciously and admirably. God Bless. *swings a champagne bottle*

That's true. But then again, we know more about the "background" of an encoder's architecture if it's developed by the same group, and using the same code base development path. If someone not involved with a codec's original development makes changes, those changes need more testing, in my opinion, to be sure that quality hasn't been negatively affected while they tried to improve it in specific ways.

It's good that people pick up an open source format and start tweaking...that's one way formats get improved in the long run. But only after fairly extensive testing can this be confirmed. I believe that an encoder that's just a new version in the main development path can usually be better "trusted" to not have bugs. But all code changes need a certain amount of testing, of course.

I released MTb3. I tuned around 128kbps a little and remapped (corrected) quality-bitrate mapping in order to coordinate with the official encoders. We had to set quality 3 to get 128kbps with MTb2 due to the bitrate inflation, but now, we can set quality 4 to get 128kbps with MTb3, which is the normal behavior of the official or other tuned encoders. MTb3 -q4 is equivalent to MTb2 -q3 and MTb3 -q5 be to MTb2 -q4. I believe MTb3 is free from quality regression judging from my personal listening result with MTb3 -q4, official -q4, aoTuV -q4, QK3.2 -q4, UC -q4, and MTb2 -q3 (Japanese only but ABC/HR listening results are there).

I submit this encoder for the test. Please contact me if you experience big quality regression from MTb2.

I'm glad to see it. Your encoder sounds interesting, and I couldn't seriously consider a listening test without ModestTuning

Nice work, nyaochi. I see you tested that evil sample, "Waiting".