In most listening tests, two very different beasts compete: VBR codecs and CBR codecs.
1. A CBR codec will allocate the same bitrate to any song and to any subset of that song: The codec will output 128kb for every second of a sample test.
2. VBR codecs are most often quality based, meaning that the bitrate will vary, depending on how complex the song is, and how complex a tiny subset of the music is.

One problem of these tests is to ensure a fair comparison. Hence, beforehand, VBR codecs are tested/tuned to find a quality parameter that will give an average of 128kbps. This parameter is then used to encode the test samples. Or... it should.

I didn't read them all of course, but the latest 128kbps is a good example, representative of everything I have read on the subject for many years now.

In this test, there is a table at the bottom summarizing bitrates for all codecs on all samples. Not surprisingly, VBR codecs have different bitrates over different songs, while CBR ones have the exact same bitrate no matter what.

But then there is a line that shows the average bitrate across all samples: 128 136 135 134 128 132. How is this fair? How can one compare a codec that averages - on the samples tested - 136kbps and still call this test fair against another one that averages 128kbps?

I've already heard some answers to that question:

A) 'we are testing only difficult parts of the music'
By design, a VBR codec will be better than a CBR codec on complex parts. That's the design of it. But, by design also, it should sound worse on 'non-complex' parts of the music.
Let's take an hypothetical 2min song, that would be 'non-complex' for the first minute and then would be 'complex' for the second minute. A lame encode of that song in CBR will give a contant bitrate on both parts: 128kbps. Now a lame VBR encode of that music will give (for example) 80kbps on the first minute and 176kbps on the second part.

Now, if you compare the second minute of the song only, you effectively compare lame@170 vs lame@128. Guess who will win? VBR.
Now, if you decide to compare the first minute of the song, you do compare lame@80 vs lame@128. Guess who will win? CBR.

While this example is extremely theoretical and probably pushed a little far, it just show that by choosing only complex parts to do your test, you naturally favor VBR codecs over CBR ones.

This result can even be proved within the test linked above. The sample called Debussy clearly show that all VBR codecs considered it 'not complex'. All three of them were under 128kbps... And two of them showed that they sounded worse than their CBR versions. Unfortunately, there were only two of these 'non-complex' samples in the samples selection.

B) 'on a larger sample, these parameters would give an average of 128kbps'
How relevant the average bitrate on another sample is to the test at hand? If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place?

How to select samples then ?
Note: I am not an expert in the domain, although I am not completely clueless either. I just try to let my common sense do the job for me. Therefore, you will be kind enough to consider this a proposal rather than a blind assertion .

1. VBR codecs will react differently on a song, based on a criteria that we will define as "complexity". Therefore it sounds natural to me that a 'general purpoose' listening test would include a representative sample of the typical variations in this criteria. If you compare only complex songs, you will favor one behavior of a VBR codec, while dismissing the other.

2. The average bitrate of ALL codecs should reach the same bitrate on the test samples

Any comments? Did I miss something huge? Am I a moron? Am I the new messiah of listening tests? Has this been covered earlier elsewhere?

Please give me your impressions/feeback

Sidenote: Why is Atrac encoded at 132kbps instead of 128? Didn't find any explanation on this....

comments on Answer A:

That's not true - not always at least. Low bitrate doesn't mean lower quality. By experience, I know that parts which are encoded with VBR at higher bitrate (than average) are often worse than parts encoded at lower bitrate (than average). Why? Probably because complex parts are often too complex, and therefore additional bitrate allocated by the encoder is not enough to avoid distortions and artefacts. But again, this is not a law. Some VBR encoders have problem with VBR and they tend to lower bitrate with bad impact on audible quality with musical parts which, according to low bitrate, shouldn't be "difficult".

I suppose so. Fortunately yes, VBR should be better otherwise VBR would be pointless.
You can't say that without testing it before. Some parts are so cool for encoders that they could be transparent, even at 80 kbps, and even for MP3. A good VBR implementation will allocate 80 kbps frame only if it doesn't harm.

.No, this test could only prove that MP3 and MPC don't have a perfect VBR model, but not that VBR mean low quality with low bitrate. In the same test, you have Vorbis which allocate 110 kbps only on the sample named ItCouldBeSweet. And guess what: Vorbis obtained the best note (4,90) despite of lower bitrate.


comments on Answer B:

It's not correct. The VBR settings used in this test was not tuned to reach 128 kbps on average with the tested sample. The VBR commandline produces on average ~128 kbps with a large number of discs. That's why you can use this command line with every samples, even if average bitrate of the small sample gallery is close to 100 kbps or superior to 180 kbps.


Here is the real problem. But if "complex" samples were used, it was to lower the difficulty of the test. Try to organise a collective test with non-complex samples, and you won't obtain enough results. Not at 128 kbps at least... However, the problem is still here: the previous listening test doesn't tell us how will react all VBR encoders with "non-complex" parts.

Ideally, probably. But you're forced to allow a margin of tolerence. Roberto had fixed this margin to 10% IIRC, which is an acceptable one, and in practice the deviation is even inferior to this (128 -> 136 kbps i.e. 6,25%). Few kbps won't significantly change the results: difference is very subtle, at least at mid/high bitrate.


N.B. atrac3 doesn't offer 128 kbps encoding. 132 kbps is the closest one.

That is an interesting thinking, however...

When listening to audio, what is usually noticed is the parts where the encoder intruduced obvious artifacts, not the parts where everything was fine. In the "easy" parts, we just listen to the music, without thinking about the codec. I think that for casual listening, how the codec will handle difficult parts is way more important regarding overall subjective quality than how it will handle the music overall.
Thus, testing mainly difficult parts is, in my opinion, quite valid.
You are right in the fact that it does not fully represent the overall reality, as it "scales" it. Testing difficult parts enhance the differences between contenders (encoders), but it does not change the ranking between encoders.

Thanks guru. I agree with most of your comments. For the 'lower bitrate means lower quality' rebutals, I do agree, and I was trying to be theoretical. Just to show a test is needed to prove it ;-) And yes, the Debussy sample shows a flaw in both VBR encoders. This is why we need more tests on 'low complexity' samples as well as 'high complexity ones'.



I tend to both agree and disagree to that. If you have a codec that gives overall 128kbps but the samples you choose are all above, then you show that your selection was based on some criteria that is not statistically insignificant. Therefore, your listening test is not 'general purpose', it is already tainted. The test remains valid, but only if you qualify it of 'high complexity samples listening test', not if you call it 'listening test'.


The Debussy samples tends to prove otherwise... This was clearly a 'low complexity' sample as far as all three VBR codecs were concerned (bitrate lower than their average with these settings) and yet, only one codec provides near-transparency: Ogg. Your argument would stand if these codecs were all perfect/good enough. But we clearly see it otherwise.


For the bitrate comment, I find 10% a bit high, but that's a matter of choice. This test wasn't too bad on this perspective. I have seen some tests where some VBR codecs would go up to 146kbps...

Thanks for the info on Atrac. Being worse than all at a greater bitrate, it didn't biaised anything anyways... In theory

Sure, but the Debussy sample in the linked test clearly shows everything was not fine for LAME, even though LAME did consider it to be an "easy" piece.

This sample shows that LAME and MPC did wrongly consider the sample to be an "easy" one. I would completely agree with you if these kind of mistakes didn't happen
Until then, we need to test these more, as it seems that some bit-allocation mechanisms are a bit thrown off balance by both Debussy and ItCouldBeSweet. Both were considered 'simple' by all three VBR encoders, and yet LAME and MPC did produce poorer quality than all other codecs.

So this was a test with 18 pieces of music, 16 of which were considered 'complex' or 'average' by VBR encoders. Only two were considered 'easy' and yet, on these, both show clear problems with LAME and MPC. I find it hard to dismiss it! There is a 100% correlation! (albeit on only two samples)

Wasn't Debussy partially chosen because of its low volume, potentially beeing source of difficulties for vbr encoders?
Need to dig the test preparation threads...

This sample, yes. But keep in mind that bitrate deviation is very high with MP3 and MPC and drops to an unusual value. Situation may be very different with other non-complex samples. By the way, Debussy.wav is a very quiet sample. MPC has problem to handle this, and not only with --radio and inferior profile.


This is exact. But I've only commented you point, which was "If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place?". VBR tuning was not based on few and short samples, but on much larger library. Whay I meant is that the VBR setting would be exactly the same, even if all samples are not complex at all or on contrary ultra-complex. But your right, there's a serious possibilty of problem if the tested samples are all falling into the same category.

Now we know that some VBR encoders could lower the bitrate and the quality in the same time, we should also include "non-complex" (i.e. low bitrate VBR) samples. But I repeat: the risk is that many people won't hear any difference with most if not all encodings, and quickly give up the test.

I've sent this sample to Roberto. I asked him to join this sample, which contrast with other samples/problems usually tested (transients mostly). Previous tests missed some problems, very common with instrumental or classical music: issue specific to tonal samples (organ, solo string instrument - not tested yet) or to low volume parts (tested with Debussy.wav).
This sample was also very accomodating, because it seriously lower the average bitrate of the complete set of sample for both MPC and MP3

As always, there is not a clear-cut answer. But I find on this 128kbps listening test that 2 out of 2 "easy" samples are clearly differenciated by most. That deserves at least an investigation

If you only include samples that are close to 128kbps, it's no longer representative of a real world comparison. A metal or classical test is valid because people like to listen to particular genres, but nobody likes listening only to music that gives an output bit rate of x kbps with a given setting. It would be useful for a technical comparison but wouldn't be relevant to any real word VBR vs. CBR scenario.

That's why I originally said:


So in an ideal test, you would have 160kbps, 130kbps and 90kbps samples. THAT would be representative, not merely 128kbps.

One important fact that people seem to forget here is that it is not the fault of VBR encoders that CBR encoders use an inferior bitrate model.

Of course VBR encoders use a higher bitrate on complex samples. Which simply shows that they are working the way they are supposed to.

What should we do about this? Disqualify CBR encoders from any listening tests right from the start? I don't really think that would be a good idea and would probably only lead to people whining: Why didn't you include this codec or that codec in the listening test.

Whilst vbr codecs might choose a higher bitrate this does not show any flaws in the testing procedure but only points out flaws in the chosen CBR encoders.

VBR encoding definitely has been around for a long time by now. If certain codecs are still using an outdated bitrate model they have only got themselves to blame for it.

picmixer, did you read the thread or just the topic?

No of course I didn't read the thread. I always browse Hydrogenaudio blindfolded, try to find the reply button by chance and tap in a random reply on the keyboard.

May I ask if you even read my post before replying?

Why reading posts when topics would suffice?

To get back on topic, I just don't understand anything about your post.


it is not the fault of VBR encoders that CBR encoders use an inferior bitrate model.
No one implied that.

Of course VBR encoders use a higher bitrate on complex samples.
No one denies vbr is superior to cbr. At least when properly implemented.

Which simply shows that they are working the way they are supposed to.
The bitrate alone doesn't show anything. Listening only can tell if they are not overdoing it or underdoing it.

What should we do about this? Disqualify CBR encoders from any listening tests right from the start? I don't really think that would be a good idea and would probably only lead to people whining: Why didn't you include this codec or that codec in the listening test.
We're just trying to find a fair way to compare both. Since they use different models - as you pointed out - it is not necessarily easy.

Whilst vbr codecs might choose a higher bitrate this does not show any flaws in the testing procedure but only points out flaws in the chosen CBR encoders.
So I am going to try and explain it again: VBR is supposed to be better than CBR because it allocates higher bitrate to complex parts and lower bitrate to simple ones. But is the algorithm is not perfect - and which one is - it could overdo it. Meaning, it could allocate way more bits into a complex part where that might not be justified and allocate too few bits in a part detected as 'easy', when it might there induce audible artifacts. As a result, complex parts would sound very good and simple ones would sound bad.
In this sense, such a codec woudn't be any better than CBR. But if you test only the complex parts, it would just shine like the sun. Hence, doing a test on complex parts alone would unfairly put this one on top. And it would not only be unfair to CBR codecs, but to all properly coded VBR codec as well.

VBR encoding definitely has been around for a long time by now. If certain codecs are still using an outdated bitrate model they have only got themselves to blame for it.
Where did you get the idea we were saying anything else


There, this is the main reason I asked if you did read the thread. Most of your comments are just pulled out of a hat as nobody even remotely suggested them in the thread... Where did you read we blamed VBR codecs for anything?

Well, no one else might have said these things, but I did.

You are trying to say that the current way of comparing vbr vs cbr codecs doesn't seem fair to you. All I was doing is trying to state why I think the current system is perfectly fair.

You see how you can find the command line with vbr codecs that produces a bitrate as close to 128 kbps as possible on a very wide selection of music.

Then you let them run the test samples with exactly the same setting. I personally can't see any inconsistensies with that.

Of course it coud be that vbr codecs sometimes overdo the bitrate. For higher quality settings that certainly is true at times (mpc braindead). But finding that out is not the point of these kind of listening tests.

The point is to find out how well do these codecs perform with the same average bitrate that is based on a wide selection of music. If those vbr encoders choose a somewhat higher bitrate on those problem samples that only shows they do what they were designed to do.

EDIT: Fixed wrong phrasing.

Not if they choose a higher bitrate at the cost of a more "simple" part. A VBR codec will react differently on two different songs. Song A might be heavy metal and Song B will be just plain guitar. The LAME VBR encoder will decide (with the fine tuned parameters) to allocate 160kbps to one song A, and then only 96kbps on Song B, while LAME CBR will encode 128kbps to both.

Obviously the VBR version of Song A will sound better: More bits. Obviously the CBR version of Song B will sound better: More bits, same encoder!

Now, the idea of VBR is to say that the difference between both versions of song B will be far less noticeable than the diff btw both versions of song A. This is why the VBR encoder chose to 'sacrifice' bits on song B in favor of song A.

There, I say: Where is the ABX test that proves this?

Well, I have one ABX test at hand, the latest 128kbps one. There are 2 songs in the sample for which all three VBR encoders allocate less bits than their CBR opponents. For both of them, in the three worst codecs, there are two VBR ones, and in the top three there are 2 CBR ones. This just proves that the above assertion is false, at least with LAME VBR and MPC. (It doesn't proves anything, it is only two samples, but it tends to prove!)

The point being: Allocating more bits on complex parts (or 'problem samples' as you call them) is only one part of the job of a VBR encoder. Allocating enough bits on simple ones is another one. That's the one I claim all these tests just overlook.

Yep, I somewhat agree on that part. That only shows though that the current model for choosing bitrates seems perfactly fair to me. CBR encoders do also get to show where their strong points are in comparison to VBR ones.

A listening test based on "easy" examples might provide some interesting results then. Feel free to organize one in case you have the time for it.

I only hope no one will say then "This test isn't fair, CBR encoders all use higher bitrates on those samples"

Such a test would overlook the exact opposite thing as the current one did. Hence, it would be 'as bad'.

However, a test with a fair balance of "easy", "complex" and "average" samples might provide good 'all purpose' results.

I am astonished at what I am saying. My original post did say the exact same thing.

The issue that is raised here is an interesting one, and to restate, it's basically saying that when a CBR setting is tested against a VBR setting which comes out, on average, about at the bitrate of the CBR setting, then the VBR encode has an advantage of using more bits on complex stuff, while the CBR encode should actually sound better (assuming that the golden ears can make out slight variations) on part that are less difficult to encode.

Which is to say, if I encode some very-low-complexity music with Lame -V3, and the average bitrate is 90 kbps, then a Lame CBR 128 kbps encode is probably actually higher-quality for that sample.

Which is just to restate (or iterate out the ramifications) that CBR encodes at a constant bitrate, whereas VBR encodes at a constant quality level. Hence, if your hearing is good enough that you start noticing a difference at a certain level of quality, VBR makes more sense than CBR because it sticks about at that level of quality.
Stated another way, VBR should be noticeably better than CBR 128 when VBR jumps to 170 kbps on a complex part of the music. CBR at 128 will be better than VBR at 90 kbps, but the difference is going to be a lot more subtle.

You got it just right, but for the fact that this is just theory. A huge number of listening tests have been proving the first part, while a fewer number have been testing the second one.

I have actually been noticing the same idea with video. I used to back up video content from DVD to SVCD using TMPGEnc, and at some very low bitrate, VBR-CQ would be just deceiving. The point is that you cannot tell a few macroblocks when in a battle scene, but you can easily when in a huge closeup where nothing is moving. Because the brain will be less sensible to raw image quality when bombarded with lots of stuff, and more when nothing else is disturbing the viewing experience.

So Constant Quality is not an absolute thing in video, because the brain won't see details on an action frame 10x times bigger than the one on the closeup.

Of course, video and audio perception has little to do with each other, other than the fact that it is highly subjective. That was just for the story

I think you are making a fundamental mistake here. As guruboolez posted before, you are confusing quality with bitrate. More bits doesn't necessarily mean that quality is better and fewer bits doesn't necessarily mean that quality is lower. So you can't really say that a 96kbps VBR encoding is worse than a 128kbps CBR encoding. You would have to do a listening test to see if this is true. Although, I understand your scepticism about having a VBR encoder which comes out too strong (allocates more bits than needed) on difficult parts and then comes out too weak (allocates less bits than needed) on easier parts. But if the easier parts are harder to ABX anyway then does it really matter? It all comes down to how well tuned is a VBR setting and in my experience most codecs do a pretty good job at the task.


No. The encoder does not sacrifice bits from one song to give them to another song. Neither from a certain part of a song to give them to another part of the same song. I think you are thinking in terms of 2-pass video encoding. A 1-pass audio encoder (as most) allocates bits making decisions for each frame without taking into account how many bits were spent on previous frames or how many it will spent on next frames. It just makes decisions to keep quality at a constant level. Other encoders are more aggresive (I've seen vorbis go up to 500kbps for a certain part with -q5) and others are less aggresive.

Now regarding the selection of VBR settings to use in a listening test you should try to think in a more practical way rather than a technical way. What is the question you are trying to answer with a listening test? Is it "Which encoder encodes samples A and B at bitrate X with better quality?" or is it "Which setting should I use to encode my music collection with optimal quality while taking into account storage space considerations?"

But are they? clearly, the listening test I was linking to shows otherwise. And while I agree that the two samples in there were carefully chosen, they clearly highlighted that not all VBR codecs do this fine all the time. Your argument is opposed to the results of the test!


Totally agreed, very very poor wording on my part. No, I am not confusing this with a two pass encoding... It just came out wrong.


While I do agree with that, you also need to compare apples with apples. If the test is titled '128kbps listening test', you need to make sure you are comparing stuff at 128kbps. If the test is entitled 'recommended transparency settings listening test' then you don't have to make sure bitrate is the same, and you will compare quality AND bitrate. It all boils down to make sure you don't confuse people and call your listening test by its right label.

I think we just have a terminology problem here, but I strongly disagree with that. In my terminology, more bits will mean better quality (assuming the same encoder). The ABX test will be here to prove that this difference in quality is audible or not.

It's the same old idea of asking: If I leave the room and close the door, it the light still on even though there is noone inside? My answer to that is yes (unless the bulb burned of course )

So if you throw more bits, you will have a better quality (in theory). Is it audible or not is a different problem.

Also, don't forget that even though the difference btw A and B might be non-audible to most under standard listening conditions, the simple fact of having a bass/trebble button, an equaliser or just doing a recoding (of an MP3 in to Ogg for example) will degrade the quality of the file further. Starting with more bits will help, even though these bits looked overkill in an ABX test. More bits into the original will mean fewer distortions/artefacts/frequencies cut out, hence a more versatile file.

I agree that these two samples show that VBR codecs don't do a good job all the time. But I believe that they do a good job most of the time. Of course it would be better if we had more samples like that for the listening tests. But finding more such samples is difficult, they are so rare (well maybe not so much with classical or other tonal or low volume parts).

But you see you are comparing apples to apples. If I encode my whole music collection with the settings used for the listening tests it would come out at exactly or really close to the target bitrate. Deviations (even huge ones) within samples or even within whole albums don't really matter. My whole collection would meet my bitrate/space criteria.


This is the part that we disagree. There are so many more things that an encoder has to decide apart from bitrate. All those psychoacoustic settings play a big role to quality as well. Throwing more bits while using a flawed psychoacoustic model won't help you much. Try a small example: Pick a sample and encode it with lame --preset 128. Then do another encoding at --preset 160 but set the lowpass cutoff to a high frequency (say 22khz) or even use the -k switch. My bet is that the 128kbps encoding will sound better (meaning closer to the original) than the 160kbps one. And that with just playing with the lowpass using the same encoder. I remember vorbis having severe pre-echo problems up to -q7,-q8 with older (version 1.0?) encoders. These were for the most part fixed by later tunings that had little to do with bitrate inflation. Now these same problems are not that serious at all even at lower -q settings.


Well, that has do only with somebody's listening habits. But being perfect with any eq setting is not what a lossy encoder tries to do. Most people here would advise you not to use any eq with your music. If you want to use extreme eq settings (small ones would not matter) or you are conserned about transcoding to another lossy format then my advice is to use a lossless encoder rather than a lossy one.

I don't think you understand what I'm saying. Let's take a real world situation. Person A wants to compress MP3s at 128kbps to get a lot of storage out of his small flash based DAP. He can use CBR at 128kbps or VBR at a target bit rate of 128kbps. You're saying comparing CBR at a given bit rate to VBR at a given target bit rate is unfair.

I'm saying while it may technically be unfair it's representative of what someone in the real word who was deciding between CBR and VBR would do. They aren't going to try to match bit rates to make the comparison fair, they are going to use preset x and be done with it.

I completely agree with you. My point focus on the selection of the samples with which you will conduct the test. Let's say you have a 5000 songs collection. With your VBR presets, you will get an average bitrate of 128kbps. Now I can safely bet that you will have at least 1000 songs under 128kbps (we will call them 'Set A') and 1000 songs over 128kbps (we will call them 'Set B').

What I am saying is that if you conduct a listening test and you include only samples from 'Set A' in your listening test, you're biaising the test, bacause this set is NOT representative of your entire collection as far as 'complexity' is concerned. You will test only the behavior of vbr encoders on 'low complexity' songs. Hence CBR encoders have an unfair advantage because they use more bits on your samples than your VBR encoders.

The samples used in a test must be representative of the entire collection on which you tuned the '128kbps presets' of your VBR encoder.

If you compare the most complex song of your collection (which will be encoded at 256kbps for example), of course VBR will win. I don't need a listening test to confirm that: It uses twice as many bits!

Now from that comparison, would you deduce that the VBR encoder is always better? I hope not!

Now if you compare your 'simplest' or 'least-complex' song (excluding silences which are meaningless), your VBR encoder will have thrown only 80kbps at it. Chances are it will sound worse than the CBR encoder which is throwing in 48kbps more!!! Of course, this is theory bacause no test allow us to conclude so...

So this clearly shows that if your sample selection tends to be only in 'Set A' or 'Set B', the test tend to be biaised.

Note to the picky people: When I make such statements as 'more bitrate is better', of course, I mean with the same encoder and with the same parameters!

Let's make a quick statement that I think will explain my thinking:
I do a test between two files, one encoded with LAME VBR 64kbps, one with LAME CBR 64kbps. This is the same encoder, same parameters (psy model, low pass, etc...). Chances are that a higher bitrate will lead to higher quality. Hope you agree on that one.

Now, how does VBR works: It will allocate more bits to the 'complex' files and less bits to the 'easy' files. Hence:
1. If you do this test on a 'complex' sample, VBR will sound better.
2. If you do this test on an 'easy' sample, CBR will sound better.

What have test 1 proven? It has proven that the VBR encoder did its job.
What have test 2 proven? It has proven that the VBR encoder did its job.

But none of these tests have proven anything as far as CBR vs VBR is concerned. If you test only complex files, VBR will win hands down every time.

Now of course, while this is true with LAME vs LAME, it might not be true with Ogg vs AAC or any others, but still what was an absolute in my example becomes a bias/trend. You cannot compare fairly VBR vs CBR only on complex samples.


You guys are incredible. In all my posts I try to be overly cautious and try not to say 'So if you throw more bits, you will have a better quality' but 'So if you throw more bits, you will have a better quality (same encoder, same parameters)'. Now in the last one I forget it and then you just attack me on that point... Did you read the rest of the thread? My other messages? I've repeated so many times the same thing that all my thought is in there. Yet you have to fight on a grammar mistake on my very last message...

Of course, if I compare a blue toyota with a red Ferrari, it could be erroneous to conclude that red cars are faster than blue ones. I thought we would be on the same paghe there. My point is that A GIVEN CODEC WILL SOUND BETTER WITH MORE BITS. How can that be wrong?

Are you sure about that? I think there are more settings changed than a CBR/VBR switch. I don't really know, I never have looked at a single encoder's code. Maybe a developer could tell us what happens.



Are we talking about files (songs) here or are we talking about samples? These are two different things. If you are talking about songs, then your logic is flawed (I know I sound like a Vulcan ) That is because in a song could not be characterized as 'complex' or 'easy'. There could be difficult parts and there could be easy parts. And if your resulting VBR encoding has a lower average bitrate than your CBR file, there could be parts that have a much higher bitrate, even if 90% of the song is what you call 'easy'. Now which parts are going to sound worse? The 90% of the VBR song with the lower bitrate as the encoder decided or the 10% of the CBR song that has less bits than the VBR one? I think you assume that lower average bitrate means that the bitrate never exceeds the CBR bitrate.



Hey lighten up. Noone is fighting nobody here. And yes, I did not bother to read anything and started posting just because I felt like it like picmixer did and I don't know who else... Try being less edgy OK?



Well, that's easy. You already answered it yourself in a previous post I didn't read.

A certain bitrate could be transparent for a given sample. Raising the bitrate won't help any more because it is already transparent! So if better 'quality' is not perceived as better is it really any better?

EDIT: Hey, why don't the quotes show up properly?

Because you forgot one quote start ([quote]). I have fixed your post.

Possibly, in terms of post processing.

Of course it does! If all you plan to do is listen to your file on a crappy headphones, then I agree that there is no point. If you plan to do anything else with the files, then there is a point.

Let's take my example: I ripped all my CDs to my HDD, encoding with LAME insane (CBR 320kbps). I sure could have ripped it at around half the bitrate and wouldn't have noticed the difference in a casual listening environment. But then now, all my CDs are in my parent's basement: This was one of the objectives. So when I want to encode 64kbps Ogg or AAC files for my Palm/iPod, I get them from the MP3 already ripped, cause I don't feel like driving 1/2 hour and dig in dusty boxes.

I know, I know, I shouldn't transcode, quality is worse. Who cares? Can anyone really pretend to hear a difference between CD->Ogg@64 and CD->MP3@320->Ogg@64? And what about CD->MP3@128->Ogg@64? I'm sure there are higher chances there.

And what If I decide to buy these $3000 speakers next year? I am to re-rip all my CD collection because all of a sudden 128kbps is not transparent anymore for me? And what if I invite this friend to my place that can detect a 320kbps from the original in 1/2 second (I have one) and just cannot listen to a 160kbps LAME encode? Am I to stop playing music when he is around?

Ripping a CD to MP3 is not something absolute. It is a convenience. Higher quality means higher quality, even though you might not be able to hear it in a test does not mean it is a waste. Whan I ripped my collection, it was for archival purposes and I don't feel like having wasted any disk space.

The problem is the same. In a file that will average 128kbps, the parts encoded > 128 will sound better in the VBR version end the parts < 128 will sound worse. Assuming - once more - the same encoder and the same settings.

As far as LAME CBR vs LAME VBR is concerned, AFAIK, in a VBR mode you activate a bit allocation mechanism that will decide for each frame how many bits it should consume. After that you get to the same algorithm that will encode a frame to a specific bitrate.

Of course, it is a bit oversimplifying, but higher bitrate on a portion of a song will usually mean higher quality. Of course a LAME dev could enlighten us more on that subject.

I always thought, that VBR schemes aimed at a constant quality. So in theory, the parts > 128 should sound as good/bad as the parts < 128, except for extreme killer samples of course.

A snare drum will be encoded with 160 kbs, for example, and a vocal part with equal length gets 96 kbs, resulting in an avarage of 128 kbs. The vocal part *should* sound the same as if it was encoded with 128 kbs, so we "saved" a number of bits on that part. I thought that was the idea behind VBR.

The equivalent CBR file will allocate 128 kbs to both parts. However, in the part of the snare drums that is not enough and in the part of the vocals it is too much. So, this file will sound worse.

As a general rule, CBR will always sound worse than its VBR counter part, providing that the algorithm does its job. Therefore, in listening tests almost all the time VBR is chosen if it is a available for a certain codec.

What is 'too much'? How can you say bitrate is too much? At 32kbps (for example), nothing is transparent, so the parts of the vocals will sound better with a CBR encoder than with a VBR one. (again, assuming same encoder, same parameters). So short of listening to it, how can you assert 'this file will sound worse'? All that you can assert is 'parts of this file is likely to sound worse, part of this file is likely to sound better'.


Emphasis mine. But isn't that the point of a listening test to confirm this assertion?

My point is that if you compare only 'high complexity' samples, you cannot say 'Ogg is better than Atrac3'. However, you can say 'Ogg is better than Atrac3 on high complexity samples'. And, by the way, high complexity samples are anything but representative of anyone's music collection as a whole.

For a pure sine wave for example, a limited (low) number of bits is necessary to encode it lossless, let alone transparant. So, throwing more bits at it is what I mean with too much.


I agree with you that the sample set should be chosen carefully, and it should depent on the bitrate. For example, if I do a listening test at ~140 kbs (which I've done recently), I choose samples with different levels of "problems": samples that are likely to result in severe problems at that bitrate, slight problems and samples that should do fine at that bitrate.

I think you have a point that on avarage, the samples should avarage the same bitrate as close as possible (by mixing problem, normal and easy samples).

However, it is also true that the problem samples are the "bottleneck", especially for the higher bitrates (> 160 kbs). The easy samples will sound (close to) transparent anyway.

Yes, but I thought we were talking about music here. This is a corner case, and transparency is rarely achieved even on 'easy' samples at bitrates < 80kbps, so the point is moot.

A quality based codec will generate (in theory) a file that sounds as good or as bad all along. A CBR codec can be transparent on half the file and worse than its VBR counterpart in the rest. Which one is 'the best sounding file' is a matter of choice...

Agreed.

[offtopic]Sine waves are not music, but maybe a new group of musicians stands up in the future, making avant garde sine music or something [/offtopic]


I think that's the point, yes. Probably the best way to select a set of samples is the following:

1) Determine at which setting the VBR algorithm results in an avarage bitrate equal to the CBR one, using a large number of "normal" samples (a "representative" set).

2) Next, choose a certain number of samples out of that set randomly. To make sure that all kinds of samples are included (I'm convinced that problem samples should at least be part of the test set), the number of final samples should be high enough.

One problem of this method is, that only experienced testers can participate, because most samples will be (almost) transparent to the general listener.

Therefore, current listening tests are a trade-off, I think.

Agreed at 128kbps, not so much at lower bitrates such as 64, 48, 32 etc...

Are lossy encodings only for listening to crappy headphones? And I thought that some settings were transparent for most music on any hi-tech gear... You see the only point of having lossy files is listening to them, at various settings on any kind of equipment. If you want anything else go with lossless. End of story.


Well that's the whole point. VBR is about having constant quality. I don't care if some parts sound worse than their CBR counterpart and some better. All I care about is that I decided on a setting, and all the parts in all my music have the same quality. The same quality I decided to use as a space/quality compromise. Because lossy is about making compromises. If you don't want any, then again go with lossless.


Of course someone can hear a difference! Very severe artifacts are being created from transcoding. There were several listening test in this board with this subject from guruboolez and some other members. Try doing a search, you'll see that audible differences exist even when transcoding from a lot better lossy sources than mp3@128 or mp3@320.


Well, once again, use lossless...


Heh, that's exactly what I said once and had rjamorim correct me that digital silence is totally transparent at 32kbps. Hell, you could even go a lot lower and still won't notice any difference... The point is that if the encoder decides that it should allocate 32kbps for a part, then that is because it decided that 32kbps is enough to keep the quality at a constant level.


You assume that the encoder doesn't do a good job at allocating bits. If you think you can help making things any better, the sources are there, take a look...


Well, feel free to organize a listening test with any kind of samples you decide are best. Good luck trying to get any meaningful results with the low complexity samples though (well you could make an ultra-low bitrate listening test and have meaningfull results...). People (including me) already have a hard time ABXing critical samples at medium (~128) bitrates.

At some point, I decided to reduce my postings about listening test conduction to the minimum. I was becoming too much of "HA's listening test authority", and I think that is not good. Other people need to come up with their own expertise on the subject.

Anyway...



I am amazed you guys can waste so many words on a non-issue. This is sickening.

This thread starts off from a flawed idea: That test conducers (that is obviously me and, to a lesser extent, Darryl), chose their samples based on complexity.

This is false.

In all my tests, what I worried the most was not if sample x or y were too much or too little complex. My worries were that the sample selection reflected a wide range of styles: rock, pop, classical orchestral, classical solo, a capella, jazz, hard rock, choir... (no J-Pop, ever!).

Why? Because my tests were always meant to represent reality. People don't build their music collections thinking "oh, will this track be a complex one on LAME, or will it go easy on it?". They have their collections built on personal taste, and that's it.

Of course, a side advantage of that decision was that complex samples and easy samples were more or less evenly distributed. A capella and classical solo are pretty much unavoidably easy. Likewise, hard rock and rock can be complex if correctly chosen (Waiting anyone?)

Then again, most of my tests were done on 128 kbps. If you focus on easy samples at that bitrate, you'll only end up with a handful of useless results, where all codecs are tied near the top. So, I'm only to be expected to favour complex samples there. To create a balance of sorts, I focused more on easy samples at the 64 and 32kbps tests I conduced. But I always favoured variety of styles over complexity issues.


Last but not least: I didn't read the whole thread. So, I won't comment on why your solutions won't work, as I suspect other knowledgeable people already did so. I'm on vacations now, on dial-up, and can't be arsed to read everything written here.


Peace;

Roberto.

My point is that MP3s that sit on my HDD are there for different purposes than just pure listening. It's a convenience.


We are obviously not talking about the same thing...


I am just saying that nothing is perfect, and encoders like everything else makes mistakes at times. I thought the point of a listening test was to point these mistakes out. If your preferred encoder decides to allocate more bits to the complex parts, at the same average bitrate it will allocate less bits to other parts. So there is a give, it is a tradeoff. By ABXing only complex samples, you just ignore it.


It's funny how this argument comes up every time someone says any kind of criticism about anything else. I was just trying to help by pointing out something that seems a little biaised. People seems to think I am attacking them and I am trying to discredit their test. I am just trying to help!

I am aware that 128kbps 'easy' samples are next to impossible to ABX. This is less true at 64kbps and certainly mostly false under.

rjamorin, thanks for your input. Here:

I see some test being conducted on only killer samples, hence testing only the codecs on those killer samples. When I see results from tests where all VBR codecs allocate more bits on 16 samples out of 18, I tend to think something is not quite right. Especially when on both other samples (where VBR codecs are under the target bitrate) they score lower than the CBR codecs.

Now don't get me wrong, I'm not saying the test is a fraud or something. I am just trying to understand...

pieroxy: killer samples are not sample encoded with only +10% than the average bitrate. The recent collective tests were never performed with killer samples. Some are "difficult"¹, some are less "difficult" but still difficult, and others are not difficult at all. There are not killer samples.
If you want a killer stressing some VBR encoders, try maybe this one.


¹ if by difficult we mean "encoded with high bitrate".

Thanks for shedding some light ont the subject. So you mean "killer" samples are not necessarily "difficult" samples... Damn terminology

To add to the confusion, in my opinion, the kraftwerk sample used in the multiformat 128k test is a killer sample for mp3, wma and atrac3.

Yes, it turned out to be a killer sample, but I didn't know it beforehand. I chose it because a) I only had DaFunk and wanted another electronic sample, b) Man Machine rocks and c) it was proposed to me by my master himself.

Interestingly, iTunes performed very, very well on this sample, considering it faithfully stuck to 128kbps.