Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Listening tests sample selection (Read 26056 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Listening tests sample selection

In most listening tests, two very different beasts compete: VBR codecs and CBR codecs.
1. A CBR codec will allocate the same bitrate to any song and to any subset of that song: The codec will output 128kb for every second of a sample test.
2. VBR codecs are most often quality based, meaning that the bitrate will vary, depending on how complex the song is, and how complex a tiny subset of the music is.

One problem of these tests is to ensure a fair comparison. Hence, beforehand, VBR codecs are tested/tuned to find a quality parameter that will give an average of 128kbps. This parameter is then used to encode the test samples. Or... it should.

I didn't read them all of course, but the latest 128kbps is a good example, representative of everything I have read on the subject for many years now.

In this test, there is a table at the bottom summarizing bitrates for all codecs on all samples. Not surprisingly, VBR codecs have different bitrates over different songs, while CBR ones have the exact same bitrate no matter what.

But then there is a line that shows the average bitrate across all samples: 128 136 135 134 128 132. How is this fair? How can one compare a codec that averages - on the samples tested - 136kbps and still call this test fair against another one that averages 128kbps?

I've already heard some answers to that question:

A) 'we are testing only difficult parts of the music'
By design, a VBR codec will be better than a CBR codec on complex parts. That's the design of it. But, by design also, it should sound worse on 'non-complex' parts of the music.
Let's take an hypothetical 2min song, that would be 'non-complex' for the first minute and then would be 'complex' for the second minute. A lame encode of that song in CBR will give a contant bitrate on both parts: 128kbps. Now a lame VBR encode of that music will give (for example) 80kbps on the first minute and 176kbps on the second part.

Now, if you compare the second minute of the song only, you effectively compare lame@170 vs lame@128. Guess who will win? VBR.
Now, if you decide to compare the first minute of the song, you do compare lame@80 vs lame@128. Guess who will win? CBR.

While this example is extremely theoretical and probably pushed a little far, it just show that by choosing only complex parts to do your test, you naturally favor VBR codecs over CBR ones.

This result can even be proved within the test linked above. The sample called Debussy clearly show that all VBR codecs considered it 'not complex'. All three of them were under 128kbps... And two of them showed that they sounded worse than their CBR versions. Unfortunately, there were only two of these 'non-complex' samples in the samples selection.

B) 'on a larger sample, these parameters would give an average of 128kbps'
How relevant the average bitrate on another sample is to the test at hand? If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place?

How to select samples then ?
Note: I am not an expert in the domain, although I am not completely clueless either. I just try to let my common sense do the job for me. Therefore, you will be kind enough to consider this a proposal rather than a blind assertion .

1. VBR codecs will react differently on a song, based on a criteria that we will define as "complexity". Therefore it sounds natural to me that a 'general purpoose' listening test would include a representative sample of the typical variations in this criteria. If you compare only complex songs, you will favor one behavior of a VBR codec, while dismissing the other.

2. The average bitrate of ALL codecs should reach the same bitrate on the test samples

Any comments? Did I miss something huge? Am I a moron? Am I the new messiah of listening tests? Has this been covered earlier elsewhere?

Please give me your impressions/feeback

Sidenote: Why is Atrac encoded at 132kbps instead of 128? Didn't find any explanation on this....

Listening tests sample selection

Reply #1 –
comments on Answer A:
Quote
But, by design also, it should sound worse on 'non-complex' parts of the music.

That's not true - not always at least. Low bitrate doesn't mean lower quality. By experience, I know that parts which are encoded with VBR at higher bitrate (than average) are often worse than parts encoded at lower bitrate (than average). Why? Probably because complex parts are often too complex, and therefore additional bitrate allocated by the encoder is not enough to avoid distortions and artefacts. But again, this is not a law. Some VBR encoders have problem with VBR and they tend to lower bitrate with bad impact on audible quality with musical parts which, according to low bitrate, shouldn't be "difficult".

Quote
Now, if you compare the second minute of the song only, you effectively compare lame@170 vs lame@128. Guess who will win? VBR.
I suppose so. Fortunately yes, VBR should be better otherwise VBR would be pointless.
Quote
Now, if you decide to compare the first minute of the song, you do compare lame@80 vs lame@128. Guess who will win? CBR.
You can't say that without testing it before. Some parts are so cool for encoders that they could be transparent, even at 80 kbps, and even for MP3. A good VBR implementation will allocate 80 kbps frame only if it doesn't harm.

Quote
This result can even be proved within the test linked above. The sample called Debussy clearly show that all VBR codecs considered it 'not complex'.
.No, this test could only prove that MP3 and MPC don't have a perfect VBR model, but not that VBR mean low quality with low bitrate. In the same test, you have Vorbis which allocate 110 kbps only on the sample named ItCouldBeSweet. And guess what: Vorbis obtained the best note (4,90) despite of lower bitrate.


comments on Answer B:
Quote
B) 'on a larger sample, these parameters would give an average of 128kbps'
How relevant the average bitrate on another sample is to the test at hand? If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place?

It's not correct. The VBR settings used in this test was not tuned to reach 128 kbps on average with the tested sample. The VBR commandline produces on average ~128 kbps with a large number of discs. That's why you can use this command line with every samples, even if average bitrate of the small sample gallery is close to 100 kbps or superior to 180 kbps.

Quote
1. VBR codecs will react differently on a song, based on a criteria that we will define as "complexity". Therefore it sounds natural to me that a 'general purpoose' listening test would include a representative sample of the typical variations in this criteria. If you compare only complex songs, you will favor one behavior of a VBR codec, while dismissing the other.

Here is the real problem. But if "complex" samples were used, it was to lower the difficulty of the test. Try to organise a collective test with non-complex samples, and you won't obtain enough results. Not at 128 kbps at least... However, the problem is still here: the previous listening test doesn't tell us how will react all VBR encoders with "non-complex" parts.

Quote
2. The average bitrate of ALL codecs should reach the same bitrate on the test samples
Ideally, probably. But you're forced to allow a margin of tolerence. Roberto had fixed this margin to 10% IIRC, which is an acceptable one, and in practice the deviation is even inferior to this (128 -> 136 kbps i.e. 6,25%). Few kbps won't significantly change the results: difference is very subtle, at least at mid/high bitrate.


N.B. atrac3 doesn't offer 128 kbps encoding. 132 kbps is the closest one.

Listening tests sample selection

Reply #2 –
Quote
By design, a VBR codec will be better than a CBR codec on complex parts. That's the design of it. But, by design also, it should sound worse on 'non-complex' parts of the music.
Let's take an hypothetical 2min song, that would be 'non-complex' for the first minute and then would be 'complex' for the second minute. A lame encode of that song in CBR will give a contant bitrate on both parts: 128kbps. Now a lame VBR encode of that music will give (for example) 80kbps on the first minute and 176kbps on the second part.

Now, if you compare the second minute of the song only, you effectively compare lame@170 vs lame@128. Guess who will win? VBR.
Now, if you decide to compare the first minute of the song, you do compare lame@80 vs lame@128. Guess who will win? CBR.

While this example is extremely theoretical and probably pushed a little far, it just show that by choosing only complex parts to do your test, you naturally favor VBR codecs over CBR ones.

That is an interesting thinking, however...

When listening to audio, what is usually noticed is the parts where the encoder intruduced obvious artifacts, not the parts where everything was fine. In the "easy" parts, we just listen to the music, without thinking about the codec. I think that for casual listening, how the codec will handle difficult parts is way more important regarding overall subjective quality than how it will handle the music overall.
Thus, testing mainly difficult parts is, in my opinion, quite valid.
You are right in the fact that it does not fully represent the overall reality, as it "scales" it. Testing difficult parts enhance the differences between contenders (encoders), but it does not change the ranking between encoders.

Listening tests sample selection

Reply #3 –
Thanks guru. I agree with most of your comments. For the 'lower bitrate means lower quality' rebutals, I do agree, and I was trying to be theoretical. Just to show a test is needed to prove it ;-) And yes, the Debussy sample shows a flaw in both VBR encoders. This is why we need more tests on 'low complexity' samples as well as 'high complexity ones'.


Quote
It's not correct. The VBR settings used in this test was not tuned to reach 128 kbps on average with the tested sample. The VBR commandline produces on average ~128 kbps with a large number of discs. That's why you can use this command line with every samples, even if average bitrate of the small sample gallery is close to 100 kbps or superior to 180 kbps.

I tend to both agree and disagree to that. If you have a codec that gives overall 128kbps but the samples you choose are all above, then you show that your selection was based on some criteria that is not statistically insignificant. Therefore, your listening test is not 'general purpose', it is already tainted. The test remains valid, but only if you qualify it of 'high complexity samples listening test', not if you call it 'listening test'.

Quote
But if "complex" samples were used, it was to lower the difficulty of the test. Try to organise a collective test with non-complex samples, and you won't obtain enough results. Not at 128 kbps at least...

The Debussy samples tends to prove otherwise... This was clearly a 'low complexity' sample as far as all three VBR codecs were concerned (bitrate lower than their average with these settings) and yet, only one codec provides near-transparency: Ogg. Your argument would stand if these codecs were all perfect/good enough. But we clearly see it otherwise.


For the bitrate comment, I find 10% a bit high, but that's a matter of choice. This test wasn't too bad on this perspective. I have seen some tests where some VBR codecs would go up to 146kbps...

Thanks for the info on Atrac. Being worse than all at a greater bitrate, it didn't biaised anything anyways... In theory 

Listening tests sample selection

Reply #4 –
Quote
That is an interesting thinking, however...

When listening to audio, what is usually noticed is the parts where the encoder intruduced obvious artifacts, not the parts where everything was fine.
Sure, but the Debussy sample in the linked test clearly shows everything was not fine for LAME, even though LAME did consider it to be an "easy" piece.

This sample shows that LAME and MPC did wrongly consider the sample to be an "easy" one. I would completely agree with you if these kind of mistakes didn't happen
Until then, we need to test these more, as it seems that some bit-allocation mechanisms are a bit thrown off balance by both Debussy and ItCouldBeSweet. Both were considered 'simple' by all three VBR encoders, and yet LAME and MPC did produce poorer quality than all other codecs.

So this was a test with 18 pieces of music, 16 of which were considered 'complex' or 'average' by VBR encoders. Only two were considered 'easy' and yet, on these, both show clear problems with LAME and MPC. I find it hard to dismiss it! There is a 100% correlation! (albeit on only two samples)

Listening tests sample selection

Reply #5 –
Wasn't Debussy partially chosen because of its low volume, potentially beeing source of difficulties for vbr encoders?
Need to dig the test preparation threads...

Listening tests sample selection

Reply #6 –
Quote
Quote
But if "complex" samples were used, it was to lower the difficulty of the test. Try to organise a collective test with non-complex samples, and you won't obtain enough results. Not at 128 kbps at least...

The Debussy samples tends to prove otherwise...

This sample, yes. But keep in mind that bitrate deviation is very high with MP3 and MPC and drops to an unusual value. Situation may be very different with other non-complex samples. By the way, Debussy.wav is a very quiet sample. MPC has problem to handle this, and not only with --radio and inferior profile.

Quote
If you have a codec that gives overall 128kbps but the samples you choose are all above, then you show that your selection was based on some criteria that is not statistically insignificant. Therefore, your listening test is not 'general purpose', it is already tainted.

This is exact. But I've only commented you point, which was "If you tune your VBR settings to reach an average of 128kbps on a sample and then do your test on another sample, what was the point of tuning it in the first place?". VBR tuning was not based on few and short samples, but on much larger library. Whay I meant is that the VBR setting would be exactly the same, even if all samples are not complex at all or on contrary ultra-complex. But your right, there's a serious possibilty of problem if the tested samples are all falling into the same category.

Now we know that some VBR encoders could lower the bitrate and the quality in the same time, we should also include "non-complex" (i.e. low bitrate VBR) samples. But I repeat: the risk is that many people won't hear any difference with most if not all encodings, and quickly give up the test.

Listening tests sample selection

Reply #7 –
Quote
Wasn't Debussy partially chosen because of its low volume, potentially beeing source of difficulties for vbr encoders?
Need to dig the test preparation threads...
[a href="index.php?act=findpost&pid=319041"][{POST_SNAPBACK}][/a]

I've sent this sample to Roberto. I asked him to join this sample, which contrast with other samples/problems usually tested (transients mostly). Previous tests missed some problems, very common with instrumental or classical music: issue specific to tonal samples (organ, solo string instrument - not tested yet) or to low volume parts (tested with Debussy.wav).
This sample was also very accomodating, because it seriously lower the average bitrate of the complete set of sample for both MPC and MP3 
 

Listening tests sample selection

Reply #8 –
Quote
But I repeat: the risk is that many people won't hear any difference with most if not all encodings, and quickly give up the test.
As always, there is not a clear-cut answer. But I find on this 128kbps listening test that 2 out of 2 "easy" samples are clearly differenciated by most. That deserves at least an investigation

Listening tests sample selection

Reply #9 –
If you only include samples that are close to 128kbps, it's no longer representative of a real world comparison. A metal or classical test is valid because people like to listen to particular genres, but nobody likes listening only to music that gives an output bit rate of x kbps with a given setting. It would be useful for a technical comparison but wouldn't be relevant to any real word VBR vs. CBR scenario.

Listening tests sample selection

Reply #10 –
Quote
If you only include samples that are close to 128kbps, it's no longer representative of a real world comparison. A metal or classical test is valid because people like to listen to particular genres, but nobody likes listening only to music that gives an output bit rate of x kbps with a given setting. It would be useful for a technical comparison but wouldn't be relevant to any real word VBR vs. CBR scenario.
[a href="index.php?act=findpost&pid=319051"][{POST_SNAPBACK}][/a]

That's why I originally said:
Quote
VBR codecs will react differently on a song, based on a criteria that we will define as "complexity". Therefore it sounds natural to me that a 'general purpoose' listening test would include a representative sample of the typical variations in this criteria


So in an ideal test, you would have 160kbps, 130kbps and 90kbps samples. THAT would be representative, not merely 128kbps.

Listening tests sample selection

Reply #11 –
One important fact that people seem to forget here is that it is not the fault of VBR encoders that CBR encoders use an inferior bitrate model.

Of course VBR encoders use a higher bitrate on complex samples. Which simply shows that they are working the way they are supposed to.

What should we do about this? Disqualify CBR encoders from any listening tests right from the start? I don't really think that would be a good idea and would probably only lead to people whining: Why didn't you include this codec or that codec in the listening test.

Whilst vbr codecs might choose a higher bitrate this does not show any flaws in the testing procedure but only points out flaws in the chosen CBR encoders.

VBR encoding definitely has been around for a long time by now. If certain codecs are still using an outdated bitrate model they have only got themselves to blame for it.

Listening tests sample selection

Reply #13 –
Quote
picmixer, did you read the thread or just the topic?
[a href="index.php?act=findpost&pid=319087"][{POST_SNAPBACK}][/a]


No of course I didn't read the thread. I always browse Hydrogenaudio blindfolded, try to find the reply button by chance and tap in a random reply on the keyboard. 

May I ask if you even read my post before replying?

Listening tests sample selection

Reply #14 –
Why reading posts when topics would suffice?

To get back on topic, I just don't understand anything about your post.


it is not the fault of VBR encoders that CBR encoders use an inferior bitrate model.
No one implied that.

Of course VBR encoders use a higher bitrate on complex samples.
No one denies vbr is superior to cbr. At least when properly implemented.

Which simply shows that they are working the way they are supposed to.
The bitrate alone doesn't show anything. Listening only can tell if they are not overdoing it or underdoing it.

What should we do about this? Disqualify CBR encoders from any listening tests right from the start? I don't really think that would be a good idea and would probably only lead to people whining: Why didn't you include this codec or that codec in the listening test.
We're just trying to find a fair way to compare both. Since they use different models - as you pointed out - it is not necessarily easy.

Whilst vbr codecs might choose a higher bitrate this does not show any flaws in the testing procedure but only points out flaws in the chosen CBR encoders.
So I am going to try and explain it again: VBR is supposed to be better than CBR because it allocates higher bitrate to complex parts and lower bitrate to simple ones. But is the algorithm is not perfect - and which one is - it could overdo it. Meaning, it could allocate way more bits into a complex part where that might not be justified and allocate too few bits in a part detected as 'easy', when it might there induce audible artifacts. As a result, complex parts would sound very good and simple ones would sound bad.
In this sense, such a codec woudn't be any better than CBR. But if you test only the complex parts, it would just shine like the sun. Hence, doing a test on complex parts alone would unfairly put this one on top. And it would not only be unfair to CBR codecs, but to all properly coded VBR codec as well.

VBR encoding definitely has been around for a long time by now. If certain codecs are still using an outdated bitrate model they have only got themselves to blame for it.
Where did you get the idea we were saying anything else


There, this is the main reason I asked if you did read the thread. Most of your comments are just pulled out of a hat as nobody even remotely suggested them in the thread... Where did you read we blamed VBR codecs for anything?

Listening tests sample selection

Reply #15 –
Well, no one else might have said these things, but I did.

You are trying to say that the current way of comparing vbr vs cbr codecs doesn't seem fair to you. All I was doing is trying to state why I think the current system is perfectly fair.

You see how you can find the command line with vbr codecs that produces a bitrate as close to 128 kbps as possible on a very wide selection of music.

Then you let them run the test samples with exactly the same setting. I personally can't see any inconsistensies with that.

Of course it coud be that vbr codecs sometimes overdo the bitrate. For higher quality settings that certainly is true at times (mpc braindead). But finding that out is not the point of these kind of listening tests.

The point is to find out how well do these codecs perform with the same average bitrate that is based on a wide selection of music.  If those vbr encoders choose a somewhat higher bitrate on those problem samples that only shows they do what they were designed to do.

EDIT: Fixed wrong phrasing.

Listening tests sample selection

Reply #16 –
Quote
If those vbr encoders choose a somewhat higher bitrate on those problem samples that only shows they do what they were designed to do.

Not if they choose a higher bitrate at the cost of a more "simple" part. A VBR codec will react differently on two different songs. Song A might be heavy metal and Song B will be just plain guitar. The LAME VBR encoder will decide (with the fine tuned parameters) to allocate 160kbps to one song A, and then only 96kbps on Song B, while LAME CBR will encode 128kbps to both.

Obviously the VBR version of Song A will sound better: More bits. Obviously the CBR version of Song B will sound better: More bits, same encoder!

Now, the idea of VBR is to say that the difference between both versions of song B will be far less noticeable than the diff btw both versions of song A. This is why the VBR encoder chose to 'sacrifice' bits on song B in favor of song A.

There, I say: Where is the ABX test that proves this?

Well, I have one ABX test at hand, the latest 128kbps one. There are 2 songs in the sample for which all three VBR encoders allocate less bits than their CBR opponents. For both of them, in the three worst codecs, there are two VBR ones, and in the top three there are 2 CBR ones. This just proves that the above assertion is false, at least with LAME VBR and MPC. (It doesn't proves anything, it is only two samples, but it tends to prove!)

The point being: Allocating more bits on complex parts (or 'problem samples' as you call them) is only one part of the job of a VBR encoder. Allocating enough bits on simple ones is another one. That's the one I claim all these tests just overlook.

Listening tests sample selection

Reply #17 –
Quote
Allocating enough bits onsimple ones is another one. That's the one I claim all these tests just overlook.
[a href="index.php?act=findpost&pid=319116"][{POST_SNAPBACK}][/a]


Yep, I somewhat agree on that part. That only shows though that the current model for choosing bitrates seems perfactly fair to me. CBR encoders do also get to show where their strong points are in comparison to VBR ones.

A listening test based on "easy" examples might provide some interesting results then. Feel free to organize one in case you have the time for it.

I only hope no one will say then "This test isn't fair, CBR encoders all use higher bitrates on those samples" 

Listening tests sample selection

Reply #18 –
Quote
A listening test based on "easy" examples might provide some interesting results then.
Such a test would overlook the exact opposite thing as the current one did. Hence, it would be 'as bad'.

However, a test with a fair balance of "easy", "complex" and "average" samples might provide good 'all purpose' results.

I am astonished at what I am saying. My original post did say the exact same thing.   

Listening tests sample selection

Reply #19 –
The issue that is raised here is an interesting one, and to restate, it's basically saying that when a CBR setting is tested against a VBR setting which comes out, on average, about at the bitrate of the CBR setting, then the VBR encode has an advantage of using more bits on complex stuff, while the CBR encode should actually sound better (assuming that the golden ears can make out slight variations) on part that are less difficult to encode.

Which is to say, if I encode some very-low-complexity music with Lame -V3, and the average bitrate is 90 kbps, then a Lame CBR 128 kbps encode is probably actually higher-quality for that sample.

Which is just to restate (or iterate out the ramifications) that CBR encodes at a constant bitrate, whereas VBR encodes at a constant quality level. Hence, if your hearing is good enough that you start noticing a difference at a certain level of quality, VBR makes more sense than CBR because it sticks about at that level of quality.
Stated another way, VBR should be noticeably better than CBR 128 when VBR jumps to 170 kbps on a complex part of the music. CBR at 128 will be better than VBR at 90 kbps, but the difference is going to be a lot more subtle.
God kills a kitten every time you encode with CBR 320

Listening tests sample selection

Reply #20 –
Quote
Stated another way, VBR should be noticeably better than CBR 128 when VBR jumps to 170 kbps on a complex part of the music. CBR at 128 will be better than VBR at 90 kbps, but the difference is going to be a lot more subtle.

You got it just right, but for the fact that this is just theory. A huge number of listening tests have been proving the first part, while a fewer number have been testing the second one.

Listening tests sample selection

Reply #21 –
I have actually been noticing the same idea with video. I used to back up video content from DVD to SVCD using TMPGEnc, and at some very low bitrate, VBR-CQ would be just deceiving. The point is that you cannot tell a few macroblocks when in a battle scene, but you can easily when in a huge closeup where nothing is moving. Because the brain will be less sensible to raw image quality when bombarded with lots of stuff, and more when nothing else is disturbing the viewing experience.

So Constant Quality is not an absolute thing in video, because the brain won't see details on an action frame 10x times bigger than the one on the closeup.

Of course, video and audio perception has little to do with each other, other than the fact that it is highly subjective. That was just for the story

Listening tests sample selection

Reply #22 –
Quote
Obviously the VBR version of Song A will sound better: More bits. Obviously the CBR version of Song B will sound better: More bits, same encoder!

I think you are making a fundamental mistake here. As guruboolez posted before, you are confusing quality with bitrate. More bits doesn't necessarily mean that quality is better and fewer bits doesn't necessarily mean that quality is lower. So you can't really say that a 96kbps VBR encoding is worse than a 128kbps CBR encoding. You would have to do a listening test to see if this is true. Although, I understand your scepticism about having a VBR encoder which comes out too strong (allocates more bits than needed) on difficult parts and then comes out too weak (allocates less bits than needed) on easier parts. But if the easier parts are harder to ABX anyway then does it really matter? It all comes down to how well tuned is a VBR setting and in my experience most codecs do a pretty good job at the task.

Quote
Now, the idea of VBR is to say that the difference between both versions of song B will be far less noticeable than the diff btw both versions of song A. This is why the VBR encoder chose to 'sacrifice' bits on song B in favor of song A.

No. The encoder does not sacrifice bits from one song to give them to another song. Neither from a certain part of a song to give them to another part of the same song. I think you are thinking in terms of 2-pass video encoding. A 1-pass audio encoder (as most) allocates bits making decisions for each frame without taking into account how many bits were spent on previous frames or how many it will spent on next frames. It just makes decisions to keep quality at a constant level. Other encoders are more aggresive (I've seen vorbis go up to 500kbps for a certain part with -q5) and others are less aggresive.

Now regarding the selection of VBR settings to use in a listening test you should try to think in a more practical way rather than a technical way. What is the question you are trying to answer with a listening test? Is it "Which encoder encodes samples A and B at bitrate X with better quality?" or is it "Which setting should I use to encode my music collection with optimal quality while taking into account storage space considerations?"

Listening tests sample selection

Reply #23 –
Quote
But if the easier parts are harder to ABX anyway then does it really matter?

But are they? clearly, the listening test I was linking to shows otherwise. And while I agree that the two samples in there were carefully chosen, they clearly highlighted that not all VBR codecs do this fine all the time. Your argument is opposed to the results of the test! 

Quote
No. The encoder does not sacrifice bits from one song to give them to another song.

Totally agreed, very very poor wording on my part.  No, I am not confusing this with a two pass encoding... It just came out wrong.

Quote
Now regarding the selection of VBR settings to use in a listening test you should try to think in a more practical way rather than a technical way. What is the question you are trying to answer with a listening test? Is it "Which encoder encodes samples A and B at bitrate X with better quality?" or is it "Which setting should I use to encode my music collection with optimal quality while taking into account storage space considerations?"

While I do agree with that, you also need to compare apples with apples. If the test is titled '128kbps listening test', you need to make sure you are comparing stuff at 128kbps. If the test is entitled 'recommended transparency settings listening test' then you don't have to make sure bitrate is the same, and you will compare quality AND bitrate. It all boils down to make sure you don't confuse people and call your listening test by its right label.

Listening tests sample selection

Reply #24 –
Quote
I think you are making a fundamental mistake here. As guruboolez posted before, you are confusing quality with bitrate. More bits doesn't necessarily mean that quality is better and fewer bits doesn't necessarily mean that quality is lower.

I think we just have a terminology problem here, but I strongly disagree with that. In my terminology, more bits will mean better quality (assuming the same encoder). The ABX test will be here to prove that this difference in quality is audible or not.

It's the same old idea of asking: If I leave the room and close the door, it the light still on even though there is noone inside? My answer to that is yes (unless the bulb burned of course )

So if you throw more bits, you will have a better quality (in theory). Is it audible or not is a different problem.

Also, don't forget that even though the difference btw A and B might be non-audible to most under standard listening conditions, the simple fact of having a bass/trebble button, an equaliser or just doing a recoding (of an MP3 in to Ogg for example) will degrade the quality of the file further. Starting with more bits will help, even though these bits looked overkill in an ABX test. More bits into the original will mean fewer distortions/artefacts/frequencies cut out, hence a more versatile file.