Check the results page again: http://audio.ciara.us/test/64test/results.html. Roberto recently added a table of bitrates to the results and overall vorbis used the lowest bitrate of all the codecs.

In fact MP3Pro on average used nearly 12% more bits than Vorbis! I bet that if the bitrates had been the same Vorbis would have beaten MP3Pro. Also, real-audio, QT AAC and windows media were rated about the same but windows media guzzled a lot more bits!

The problem is that the test used VBR modes so the bitrates can't be controlled precisely. What you could try is tweaking vorbis's quality setting until the bitrate equals the target. Unfortunately I don't think all the others allow such precise control of the quality setting so you could never get them all equal.

Maybe ABR modes would be better in future tests. I know it's always whats used in the real world but it would be fairer.

Holy Jesus, help me. Not again. >_<

There were discussions of this exact problem before the test. You should have brought up your objections then.
pre-test discussion
discussion of samples

With a different set of samples, it's quite possible that vorbis could have used MORE bits than the others. If the failure of vorbis to use enough bits caused it to perform poorly on a particular sample (and I believe it did, for the sample that accounts for most of that 12%), I would say it's the fault of the encoder for missallocating bits.

Also, there were discussions of the subject before and after the 128kbps test. It was generally agreed that settings should be used which produce the correct bitrate when averaged over a wide variety of music. -q0 is supposed to be that setting, so it was used.

Worry not, Roberto! Excalabur and I shall fend them off. Avast!

Don't complain, you ask for it buy arranging a test that relates to the real world.
As long as you don't listen to these advices everything is just fine...

Jesus is one of my IRC nicks! I guess that means I should help you.



So what if WMA had an ABR of 88kbps whereas Vorbis had 62kbps? Good for WMA; that means it's capable of raising the ABR for a song as high as 88kbps while still maintaing an overall of 64. The point is, it's 64kbps average. Sure, the average for this file isn't 64, but that's because it wasn't designed to encode only one file. If you take 1000 CDs and encode them all, I'm sure you'll find files where WMA went as high as 88 and as low as 32 (probably even higher and lower). But overall, for all those files, the average will be somewhere around 64kbps.

The only way when everyone would be satisfied is 2-pass vbr, so the codec wouldn't be handicapped with ABR, but would reach the correct bitrate. Hopefully the Vorbis developers consider it to include in the next release, i don't think this is a difficult task to write.

whenever that will be..... *evilgrin*

- Lyx

Boy, you have a lot of faith. If Msoft can claim their 64k encoding rate sounds as good as
128k mp3 by *actually* coding at over 64k, would they do it?

Processor manufacturers have a history of doing whatever they can under the rules to
win at speed benchmarks, including compilers specifically optimised for the benchmark.

So for encoders, if the rule for your test is just how it sounds under the "64k" setting no matter the actual rate, you get what you get.

Not that I think Msoft would change anything for a single forum's test, but still...

I accept that with a different set of samples the bitrates would have been different. But how well the VBR modes are tuned to the 64kbps target probably varies from algorithm to algorithm. MP3Pro used more bits on all 12 samples. if there was an equal chance of MP3Pro or vorbis using more bits for any sample in the world then the probability of vorbis using fewer bits on all 12 samples is 0.5^12 = 0.00024414

I didn't want to cause trouble, I thought that it was important to note that vorbis had achieved its good result with fewer bits than the rest of them. And bitrate does matter. If it didn't we wouldn't be encoding at 64kbps.

Maybe vorbis didn't use enough bits on sample 9 (chopin)? Or maybe it used the right amount and the other codecs used too many bits, making vorbis bad by comparison?

I should add that I think the tests were excellent and the result are still very useful and very interesting.

I find the Post 1.0 CVS to be a bit conservative sometimes when it comes to upping the bitrate when it is perhaps needed. Perhaps the Vorbis developers should shake off this fear of deviating a lot from the quality versus bitrate scale. Most people are accepting the quality numbers better now and dont necessarily equate them to average bitrates so its safe to be more 'free' now rather than being this constrained.

Roberto, no good deed goes unpunished. Eh, mi amigo?

I second Phong's statement about reading the test preparation threads before treating something like "MP3Pro on average used nearly 12% more bits than Vorbis" as news. It's not. There was a lot of thought and pre-testing and adjustments that were performed prior to the official execution of this test to try to make the bitrate targets as accurate as possible for each codec. It's not so easily done with quality-based VBR encoding modes.

Trying to predict the precise "center" bitrate of VBR across many samples is kinda like trying to lasso a herd of hyperactive kittens. They're all going in different directions at once, some codecs dropping perhaps to 54kbps on one sample, while another codec spikes to 80kbps+ on the very same sample.

Kudos to Roberto and everyone else who worked on such preparations ahead of time to make these test results meaningful...because meaningful they are! And all the same was true for the 128kbps test (which sadly I missed out on).

Even if a person fell off the turnip truck this morning, that still gives them time to read the pre-test thread and the samples thread before reading the results thread and making commentary on it.

As for me, I just fell off the turnip truck a few days ago, just in time to read the pre-test threads and then perform in the test as well.

yes, that means that it is worse than some of the others.

I've already suggested manually adjusting q levels for VBR codecs to achieve the same filesize as the CBR codecs, and this is so obviously the best and most fair method. But I've always been ignored. Ah well.

Previously addressed.

Yey, two-pass audio! Though it isn't needed so much as video because you don't generally have a requirement to fit the file to a certain size. Unless using portables of course.

OT: Hey Tommy, nice to see you here.

It was kinda previously addressed. The main weakness of oggenc (maybe not the very latest) is that on very quiet passages the bitrate drops through the floor (10 kb/s on one of my samples) for a q0 setting. I find it worthwhile to set a minimum bitrate to prevent this. It does not have much of any effect on non-problem samples, so I can use it all the time.

So, though I use it because of high dynamic range music, I to not have to diddle the settings for each type of music. Though it is not stock block q0, it is valid in being a single optimum setting IMHO.

edit: and I did raise this issue in the pre-test discussions.

Exactly, thanks for pointing that out.

Can you use -m with -q? I thought those two setting didn't work together.

I tried it with Post 1.0 CVS and GT3b1, and -m was ignored on a line which also specifyed a quality level. I had to either use -m and -M (without -q) to control a bitrate "window", or -q without -m or -M since they'd be ignored anyway.

Am I wrong on this (or missing something)?

Well, it is an optimum setting, but it's not a default -> no real world usage.

Definitely not the very latest. This has been corrected over two weeks ago.

This probably has been previously addressed, but it didn't seem to make an impression on you, so here we go: Is that what you do to get each and every single Vorbis file you have to exactly 64kbps? No? Thought not.

The point of a listening test is to see how codecs perform in a real-world scenario, not some kind of irrelevant lab setting. The real world scenario is that people use -q0 to get about 64kbps.

By trying to argue that Vorbis could have come out on top if the test had been fairer, you're not doing anybody any favors. The test has shown that Vorbis has shortcomings that need to be addressed. Trying to argue the shortcomings away isn't helping anybody.

Your suggestion is not the best method and I don't see it very fair either, considering the nature of VBR. VBR is by nature different than CBR. When we are measuring VBR codecs, we are measuring certain quality settings. Different vbr codecs are of course acting slightly differently when encoding 20-30s clips, but fundamentally we are testing one quality setting which gives certain average bitrate with lots of material.

Tweaking manually 20-30 sec clips not only goes against vbr's nature, it would simply lead to a mess where we are having arbitrary number of results out of many different vbr quality settings. If a certain vbr codec has a deficiency that it gives too low bitrate with some 20-30 sec clip, how so it would be fair to go manually fix this deficiency? How does it for example help codec developers to see deficiencies and fix those, if we manually and artificially compensate the codec's deficiencies?

Also none of the codecs give insane high bitrates with measured samples, and if so that would of course reflect to the average bitrate testing phase which would be used to compensate... All in all we are measuring the nature of vbr: certain quality setting, which on average approaches certain bitrate. We are not measuring millions of different quality settings per codec, manually adjusted to compensate any bitrate deficiency or advantage the vbr codec may have with certain 20-30sec long samples.

Anyway, this same issue has been covered in dozens and dozens of messages before.. I suggest you go read those also. No need to start this again here.

OK, I see the point of comparing presets rather than bitrates, but this only allows you to say one preset in one codec is better than another preset in another codec. It does not allow you to say that one algorithm is more efficient at compressing music than another, which is a conclusion that someone might arrive at if they didn't see the bitrates.

However, since windows media and real scored virtually the same you could say real is better because it achived the same quality with fewer bits. If only it was easy to arrange for all codecs to give the same quality then we could compare bitrates...

Ideally, If testing wasn't so time consuming we could have both types of test - a bit like on food packaging where you get energy values 'per serving' and 'per 100g'.

Tweaking the settings around to hit a bitrate target for each *song* is not realistic usage.

I don't see what's so unrealistic about tweaking the settings around so that you hit an average bitrate target for a whole compilation (e.g. 12 test songs).

E.g. The user might have a collection that he figures would fit in one CD at an average of 64kbps. Using the default '64kbps' setting for Vorbis he ends up only filling 2/3 of the CD. He wants to increase quality, and he doesn't want to favour any particular songs in particular, so he re-encodes the compilation until he hits that 64kbps target. Plausible, if the compilation means a lot to him (and I think entering for a competition would qualify as 'meaning a lot'!)

But what's all this crap about 'real world conditions' anyway--if you want to see which product gives you the most bang for the buck will you let 'real world conditions' constrain you such that you don't calculate the value for each product using the same amount of money?? (e.g. buying detergent--both brand X and Y come in 1 litre packages, but X costs $10 while Y costs $7--in the real world you can't buy 0.7 of a package, so you buy a litre of both X and Y and say they are of equal value in real life, coz you used the closest to equal money in the real world and bought the same amount of product for both brands! Sounds ridiculous, but if you substitue brand for codec, amount of detergent for sound quality and money for bitrate, you get your competition situation )

A respectable idea, but my thought is..

#1 How many people would care enough to try to "hit" a specific nominal bitrate with an entire CD-R's worth of their collection? A small percentage of the population, I'd think.

#2 How many people from #1 would also have the time required to actually re-encode from WAV files over-and-over until the target was reached as averaged across all songs that would fit into 700MB at 64kbps nominal? A small percentage of the already small first group, I'd venture.

So, you'd probably have a few dozen people in the entire world for whom this might be "real-world" use of the codec. But not enough to call it "real-world" in general.

Well, you see, in this test I used the "endorsed" mode - the q settings - that happens to be the mode suggested by Xiph for home encoding (not for streaming)

In this aspect, you could say that Vorbis performs worse than the other codecs. Because it's failing in the suggested mode, not in a tweaked mode.

So, indeed, we are not only testing the compression algorithms here, we're testing the VBR algorithm. And that's exacly what needs to be tweaked it seems.



You can also say Real is better because it's a CBR-only codec (maybe it's even 100% CBR - no bit reservoir, although I have no details on that), while WMA could scale bitrate accordingly to the complexity of each second of audio.



Not only time consuming, but there are much more interesting tests that have higher priority, like the ones I'm already planning (vocoders, MP3 encoders at 128kbps, and again several encoders at 128kbps)

Regards;

Roberto.

What's with this analogy trend?


Anyway, I got lots of criticism in my former test about chosing 4.25 instead of 4 for Vorbis.

Before the test, ff123 suggested that I ask the help of several users to encode some albums with q 3.5, 3.75, 4.0, 4.25 and 4.50. From the outcome of this test we concluded that the q setting that most closely matches 128kbps over a wide range of musical genres is -q 4.25

Then, I started getting e-mails from people that were annoyed for three reasons:

1 - Most people don't care for exact bitrate matching. the read that -q 4 usually averages 128kbps, so they use it, and don't tweak the setting to, say, -q 4.16, that will exactly match 128kbps over his entire music collection.

2 - If vorbis doesn't know how to properly choose how to spend bits and ends up using a lower bitrate, it's vorbis' problem. They are announcing it as 128kbps average, and that's the information the user gets, so it's not the test conducer's duty to correct flaws in vorbis' implementation.

3 - It didn't help much because Vorbis bitrates ended up at 140kbps average, and that raised countless fairness issues.

So, there you have it why it's not really a good idea to try to adapt -q settings to match a desired bitrate.

Regards;

Roberto.

Of course, it all depends on the purpose of the test...

And I read Dibrom's argument about showing and hiding a codec's flaws...

BUT...

IF, on a wider sample, mp3pro and Vorbis get the same average bitrate of 64kbps, while in the test range there is this difference...

You could say this test is showing Vorbis's flaw in not allocating enough bits for this test range, but you could just as well say that it is showing mp3pro's flaw in allocating too many bits for this test range! (if it *had* to bloat in order to maintain quality, that's another kind of flaw again, so that argument doesn't really work) And it's unfair to penalize Vorbis for the former but reward mp3pro for the latter.

And the sample set would be biasing in the favour of mp3pro for choosing samples it bloats on and against Vorbis for choosing samples it shrinks on. Moreover, going on the assumption in this section that their average bitrates over a wide selection are the same, the test will not be 'realistic' because people won't be getting a lower bitrate in general with Vorbis and getting these shrinking problems!

If you really want to test the weaknesses of mp3pro and Vorbis in bloating and shrinking, you should include in your sample set equal amounts of bloating and shrinking files for both mp3pro and Vorbis, and see whether mp3pro suffers more from its shrinking than Vorbis, and mp3pro benefits more from its bloating than Vorbis, or is it the other way round? I.e. who is more justified in their bloating and shrinking?

But obviously this is not an alternative. The time it would take to find such a sample set would be unimaginable and the sample size would be too large to make a practical test.

If it turns out that even in a wide selection mp3pro generally gets more bitrate than Vorbis, the answer is even more clear: relative to Vorbis mp3pro is cheating across the board! (EDIT: or maybe not, after reading rjamorim's post. But I believe the rest still applies if you believe the codecs are generally matched at 64kbps)

Even if you think that bloating is always justified while shrinking is always a flaw (to justify Dibrom's argument?), the test is still biased against Vorbis for choosing samples IT shrinks on instead of samples that some other codec shrinks on!

If the goal were to see which codec gives the most bang for the same buck, the approach to take is even more clear (see my last post).

So, IMHO it is a better idea to standardize the bitrate in the test and then put a note in the results saying that you had to adjust Vorbis up, and/or mp3pro down, etc. and let people draw their own conclusions. After reading the test results people might even start to use q0.2 instead of q0, *making* the results more applicable to real life!

The test is all over and I don't even use any of these codecs, so I don't know why I'm so serious about it But I'm very convinced that I am right

If vorbis habitually under-bits on classes of music I prefer (high dynamic range) compared to another coder, and the artifacts are more apparent on standalone listening (not abx compare) because the program material of those sections is quieter/simpler, than I am
less sympathetic to that flaw... except that I know I can option my way out of it.

At any rate, until I get a limited memory vorbis player (maybe soon) I won't have much
64kb material other than monophonic..

I'm very convinced you are not!
Using the same reasoning, you could say that any test is biased against a losing contender, for using samples on which this particular encoder fails.

Is this a flaw, when on average it has been found to be still 64 kbps? According to this, VBR compression is flawed by definition.

I believe test encoder options are realistic, for getting an average of 64 kbps. Are test samples bitrates realistic? I mean, are they representative of actual average bitrates of actual average music? (not just short test samples).

Boring discussion.
A cool codec test would consider mutch more aspects then audio quality only (that is often subjective).
Yes, this could sound more like an "encoder test" but i think it could be useful for common peoples (for many peoples encodig time and file size matters).

Try to keep in mind A.C.T.

I propose to the testers to split their works in two parts:
1. a set of benchmarks designed to show the audio compression speed and ratio;
2. blind tests to measure the audio quality;

Mixing these results would be a more meaningful total appraisal.

Just a suggestion. B)

Roberto - you can't win!

But I've got a spare brick wall here that I can send to you - if you want something to bang your head against?

Cheers,
David.

First of all I think you meant my arguments, not Dibrom's. What comes to your second claim, I think KiKeG already answered to that. And the 3rd claim about choosing on purposes samples which are not favorable to Vorbis; There was a pre-test discussion and afaik the samples were chosen pretty much publicly and samples from different people were included.

Here's how Roberto chose the modes:

1. Use the VBR mode that's programmed to give 64kbps output over "all" music samples.

2. If that is unavailable, use ABR to give ~64kbps output over a specific sample.

3. If that is unavailable, use CBR to give constant 64kbps output.

No "hacks" or non-standard usage tricks are to be used to approximate one mode with another (ie. manually tweaking Vorbis -q settings to get a target bitrate), just standard, "real-world" settings.

Although I'm not 100% certain this is the best way (and that's a point of personal preference that will not be debated any more by me), it's certainly a very good way. Good enough to give meaningful real-world results, perhaps more so than my "best way", despite the fact that the codec modes are less consistent.