128kbps Extension Test - OPEN Options

[spoon]

After some thought: add up all the average bit rates for each of the samples and lets say for arguement that one codec comes out at 128Kbps and the next 256Kbps, the results of the 256Kbps codec should be scalled back by 50%. Present both sets of results with an explination. If all the Kbps comes within say 5% of 128Kbps then there wouldn't really be a need to adust the figures.

[Gabriel]

The currently used test samples are note necessarily representative of overall music. They are chosen test samples. What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.

regarding vbr codecs vs abr/cbr codecs:
As a Lame developper, I know that Lame in abr mode will be penalized in this test against, as an example, Vorbis which is a vbr codec.
But (unfortunately for Lame) this is perfectly fair as it is representative of codecs abilities.
Lame has currently no high quality vbr mode averaging 128kbps, but this is not a point that should be used to penalize other codes which have such a mode. We should not contrain codecs to use the lowest mode just because some of them are not able to use better modes.

[bawjaws]

I added this text to the presentation page:

"I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate"

Any remarks?

I think the problem some people have with this is they think the VBR codecs are cheating by going higher than 128kbps because they are focusing too tightly on the test alone. You may need to broaden their perspective.

I think it might make sense to people if you explain that the VBR codecs are being rewarded for the savings they make in the easier to encode sections where the 'less intelligent' CBR encoders are wasting bits.

If the bitrate for a given quality level averages 128kbps in ordinary use but is variable then obviously it must go both higher and lower than 128kbps. The reason you don't test on the sections that go lower is because those are the parts that the codec finds easy and so you would be less likely to spot the flaws in the encoder.

Just another angle on this obviously un-intuitive point.

This post has been edited by bawjaws: Jul 25 2003, 09:56

[askoff]

Does WMA9 include free VBR mode?

This post has been edited by askoff: Jul 25 2003, 10:45

[danchr]

A small pat on the back for Roberto:
A search for "listening test" on Google reveals "Roberto's public listening tests" as the best match.

That means there are plenty of links to it out there

[bond]

I noticed that, on some files, bitrate goes very high, nearing 200kbps sometimes. Is that right?

i wouldnt write it that way as it will make people think that there are many files which are around 200, which isnt the case
i wouldnt write 200kbps also, i would only write:
"I noticed that, on some files, bitrate can go up pretty high. Is that right?"

Yes, and that's the beauty of VBR encoding. It will simply ignore bitrate limitations (whenever possible), using as much bits as needed to encode a problematic sample.

i wouldnt write that, as talking about that is "beautifull" that codecs ignore bitrate goals (whenever possible) isnt really an argument against people who claim that the vbr codecs "are cheating" imho

Although that raises issues of fairness, it's the best way to compare modern codecs that shine the most in VBR mode, like Musepack and Vorbis. Trying to force a VBR setting to match as best as possible a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that an user just stick to a quality setting, not caring much about a specific bitrate

1) now this sounds like you think for yourself, that the test is unfair, but also that it has to be unfair as the vbr codecs "shine the most in VBR mode". this is also no argument against people claiming imho
2) i wouldnt tell the reader about his behaviour as everybody behaves different (for example i think that the usual practice in audio encoding for the masses is that people care to stay around a specific bitrate on the long run)

quality settings for each codec were chosen because they average to 128kbps over a number of encoded albums. It would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.

now this is the most important part! and the only part which should be mentioned on the homepage imho!
as it says that

the vbr codecs will reach, with this settings, normally the average of 128kbps!!!

that's the argument that counts (no need to talk about fairness or user behaviour or the beauty of vbr codecs)
like what gabriel wrote:

What is important is that the overall bitrate for overall music, using an encoder setting, would be 128kbps on average.

If the average bitrate of those test samples for a specific codec is 180kbps, it is still fair is the overall bitrate for overall music is 128kbps. What a user wants is 128kbps as an overall. A user does not care if 10 seconds of his track are encoded using 250kbps while the next 10 seconds are encoded using 90kbps, as long as overall it is still 128kbps.

sorry for my bad english

This post has been edited by bond: Jul 25 2003, 12:05

[bond]

so all in all i would write it that way:

Q: I noticed that, on some files, bitrate can go up pretty high. Is that right?

A: Yes, this can happen for samples with hard to encode content, and this is wanted that way as vbr codecs are able to put more bitrate where it is needed and less bitrate where it is not needed that much.

The settings for each codec were chosen because in the long run they average to 128kbps!

nothing less nothin more (a short but clear statement), no need to talk about 200kbps, about fairness, defending vbr codecs against anything, about how users behave (everybody has different opinions here)...

in the long run every codec setting averages to 128kbps - thats the only important message!

(and i wouldnt put it as question 1, as this is the presentation page where people can read how to participate, i would put it as question 5 (after linux) no need to push people on this info )

This post has been edited by bond: Jul 25 2003, 12:12

[ilikedirtthe2nd]

sorry for being off-topic here, but: why were nero/psytel forced to cbr 128 in the prior aac test? the arguments given for this test are also suitable for the aac test. nero -streaming also reaches an average bitrate of 128 on long distance (i suppose).

if you put this into relation, ogg vorbis should have been forced to cbr in this test.

regards; ilikedirt

This post has been edited by ilikedirtthe2nd: Jul 25 2003, 12:42

[AstralStorm]

Heh, this arguing is of no use

Rjamorim, have you taken my batch file to hide the results?
You'd need to modify and reupload all packages, but cheating in the test won't be as easy.

[ilikedirtthe2nd]

Heh, this arguing is of no use

i am not arguing against this test here, but maybe we are testing against the wrong aac encoder? maybe psytel -streaming would have won the prior competition if it were allowed to show the beauty of it's vbr algorithms

please don't hit me

[bond]

ilikedirtthe2nd
when you read the test again you will find the info that guruboolez (sorry if i misspelled the name) also tested the nero vbr options and came to the conclusion that quicktime is still better!
you'll also find more info about that on the aac test results page http://rarewares.hydrogenaudio.org/test/aa...st/results.html and especially here: http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt

This post has been edited by bond: Jul 25 2003, 13:07

[AstralStorm]

Oh, when I can continue the test not afraid I'll need to redo it?

[loophole]

At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.

Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)

It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>

[AstralStorm]

We're testing quality of settings averaging 128kbps on multiple different albums.
Not on short samples. That's all. LAME and Quicktime are using ABR,
because they either don't have VBR modes or VBR modes perform worse than ABR.

[JohnV]

At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be.

How do you know that it's 50% exta of what the original bitrate was targeted to be, you are testing just short hard-to-encode clips. The targeted quality setting tested produces 128kbps average, and we are testing the quality of specific quality setting of a vbr codec which gives this average bitrate. We are not testing qualities of 12 different quality settings of one vbr codec in one test.

Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

Like both Gabriel and me have said in this thread already, so this is prolly 4th time in this thread alone: There's no high quality low bitrate Lame vbr setting. ABR mode in Lame averaging 128kbps performs better.

[ff123]

At the end of the day, if the vorbis encode has an average bit-rate of 190kbps over QuickTime or WMA's encode of 128kbps, that's still 62kbps extra it's allocating itself which is almost 50% extra of what the original bit-rate was targeted to be. Vorbis has managed bitrates, it should be forced to use it - after all, LAME was.

Lame uses ABR because it's generally agreed that ABR sounds best at bitrates which average about 128 kbit/s. Vorbis uses -q (VBR) because that's the mode it generally sounds best using.

If you're against forcing Vorbis to use managed bitrates, maybe you should have encoded them at a given quality level (say they averaged 190kbps), then used the 192kbps setting for the ABR codecs. Seems fairer to me.

If one uses the Vorbis quality mode, and for a particular sample it averages 190, then the ABR codecs shouldn't be bumped up to 192 for that sample -- that's exactly the same thing as forcing the VBR codecs down to 128, except in reverse!

Also, Compressor which comes with Final Cut Pro 4 seems to have VBR options for AAC, not just ABR.

(for those who think QuickTime Pro is CBR, it isn't - it's ABR)

The purpose of the last test was to find the AAC codec to be used for this test. Final Cut Pro 4 wasn't tested.

It just does sound unfair to the ABR codecs to me. It seems to me like you're testing the accuracy of the bitrate engine, not which codec sounds better for any given bitrate.

Taken to the extreme, codec x in VBR mode y could allocate whatever it wants, and end up an average of 300kbps but that would be a fair comparison because it's engine was smart enough to allocate more bits. </SARCASM>

The VBR codecs in this test are quite stable in average bitrate on a per-album basis. If a VBR codec can manage to produce 128 kbit/s on average, but ends up with 300 kbit/s on a difficult sample, it should not be penalized for doing its job! That's why VBR exists. The caveat to this is that the VBR codecs should not sound worse on "easy" samples.

ff123

[upNorth]

This reminds me of discussions about "optimized" command lines for encoding, that people that don't know the first thing about compression come up with. Especially the ones that "improve" the -presets. How come people trust the developers (and the people helping them) to develop the codec, but not at all when it comes to using it. It makes no sense to me and I hope this test will be performed as intended and not in an artificial and "fair" way. Everything always seem very easy when you don't know nothing about it...

I did the test. To the best of my knowledge and capacity. If you dislike it, either run your own test or f*ck off.

B)
Thanks for arranging this test and keep up the good work

edit: typo

This post has been edited by upNorth: Jul 25 2003, 14:54

[ff123]

I've completed 9 samples, and I'm tracking my preferences:

http://ff123.net/friedman/stats.html

(Blocked ANOVA analysis)

The losers don't surprise me, but my apparent preference does (although it's not clearcut by any means).

ff123

[rjamorim]

Thanks for arranging this test and keep up the good work 

Thanks

[atici]

Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA

[Gecko]

Does WMA9 include free VBR mode?

After reading this thread, that's what I thought. If you use 2 pass you are forcing the encoder to a specific bitrate. If WMA9 Pro had a proper VBR implementation, it would maybe have increased the bitrate on critical samples as well.

This thread shows one thing: correct testing of VBR codecs, especially when mixed with cbr/managed bitrate ones, is all but trivial. Maybe this should be emphasized on the main page and then quickly explain why this method was chosen.

Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate. That would make the testfield more leveled and you are testing what a codec can do at a certain bitrate without tweaking.

But of course it will be tedious to find samples that will produce nearly identical bitrates over a range of codecs and you will probably end up with samples that are easy to encode and harder to identify by ear.

This post has been edited by Gecko: Jul 25 2003, 18:20

[rjamorim]

and you will probably end up with samples that are easy to encode and harder to identify by ear.

That's exactly the biggest issue.

Maybe ATRAC3 LP2 mode could've been added to this test too. And we could have referred to the results every once in a while MiniDisc zealot shows up in HA

Problem is, I already had my share of bad issues with SonicStage (nearly fuxored my Win2k installation), so I'll only test Atrac3 once I create a VirtualPC partition only for installing and running it. :-P

[atici]

Here's an idea for future tests. Instead of using regular difficult test samples, you would use samples that all come out at ca the same bitrate.

I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

This post has been edited by atici: Jul 25 2003, 18:32

[rjamorim]

I haven't read in detail how this test is performed. But in general isn't it a better idea for testing different codecs to fix a bitrate and adjust the quality level (for mpc and vorbis) so that the output is going to be exactly equal to that bitrate? This way every codec will be given the same amount of space to demonstrate their skills. Let's say 128kbps lame, q4.3 MPC, q4.5 vorbis for a specific sample, but 128kbps lame, q5 mpc, q5.5 vorbis for another sample...

JohnV will shoot you

[JohnV]

Does anybody here read what ff123, Gabriel, Guruboolez and I are writing in this very thread at least 5 times each so far? I can't BELIEVE THIS!!