What's the problem with double-blind testing? Options

[mirrorsawlljk]

I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?

[duff]

Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?

Lossless and lossy compression schemes are just that, compression. It is in this sense that I mean that an mp3 file, for example, is compressed..

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.

[stephanV]

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.

This post has been edited by stephanV: Oct 21 2005, 17:33

[duff]

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.

It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

I will post studies that support a number of things I have claimed, including quite notably the neural dissociation of sensory and decision processes. The "assumptions" I'm making are rooted in my training as a cognitive psychologist who studies speech processing. Many of the things I've said are well supported in experimental studies...I'm not just making it up. Also, what I'm saying is testable. I'm just trying to explain the logic of why I believe a different methodology might be necessary to rescue at least some of the claims made by audiophiles. I'm not an audiophile.

Why does it not necessarily apply? Because the purpose of lossy audio compression is to throw away unperceivable parts of the audio. So if the sensors aren't missing anything, then why should it make any difference?

Again, the main point here is that fatigue effects caused by processing issues (not accessible to systems that verbally report differences in ABX listening tests) could have residual effects on listeners in more long term ways. Reaction time experiments could verify this processing claim I'm making, and a positive result (where an ABX test shows nothing) could indicate a relationship between perceptual processing and vague ideas of discomfort experienced by some listeners of compressed audio.

This post has been edited by duff: Oct 21 2005, 20:19

[stephanV]

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.

It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.

This post has been edited by stephanV: Oct 21 2005, 20:24

[bryant]

Why? How can our brain even tell there is something missing? And why MUST it interpolate this data?

These are again merely assumptions on your part.

It's not that our brains "know" there is something missing, it's that relative to a less noisy signal, the brain has to do more work in order to generate the best representation of the sound it can.

Now you are saying something completely different. I don't see how our brain would try to represent anything else than what it is perceiving. I don't see how a wave form produced by an mp3 decoder is conceptually different from one might encounter in other situations. But ok, I will wait for studies.

You have to keep in mind how MP3 encoders (and other lossy encoders) work. To say they "throw away" information is true, but it doesn't make it clear what they are really doing. What they do is reduce the resolution at certain frequencies and times (which is how they save space) and this has the effect of adding quantization noise to the signal. The idea is to essentially add noise to the signal as close as you can get to (but still below) the threshold where it's directly, consciously, audible. What's surprising is that this can be a relatively high level of noise (which is why these lossy codecs work so well). You can easily do a test where you subtract an original music signal from its encoded version. You might be surprised how much noise is added.

So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

What is in question is whether or not noise that is below the threshold where you can actually hear it still has an effect on processing, and what research is showing over and over is that whether or not something is directly perceived has very little to do with the underlying processing. The brain is processing the sounds into a "reality" for your conscious mind to act on, and does not want to bother you with unimportant details that might distract you. What if every stimuli registered by your senses was consciously perceived!!

Imagine walking blindfolded into a large room with sound reflective walls. You instantly become aware of the rough dimensions of the room, but you don't hear every echo of every footstep. A blind person would be able to "hear" in much greater detail, and in fact claim to be able to almost "see" the room. This is the kind of thing that the subconscious mind is constantly doing for us in the background, and what finally gets presented to our conscious mind is rather independent of what processing was required to achieve it. I admit that without at least a working knowledge of the literature, this may seem counterintuitive.

Another argument that was made is that if this extra processing automatic and we are not aware of it, then maybe it has no other effect. The problem with this is that the brain (and all parts of the body) don't work that way. The processing consumes resources (they actually measure this activity by measuring the radiation it generates) and all the body's systems try to minimize resource use. If I ask to you perform a math calculation in your head you would have to concentrate to do it (and you might refuse). If I asked you to do calculations over and over you would probably get tired and annoyed. Just because other processing is unconscious does not mean it's any less taxing, and the brain will always attempt to minimize it.

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.

The reason that I linked to the first of those two papers is that I find this field fascinating from a scientific viewpoint and I suspect that some HA members might be less aware of the richness of research discoveries in this area (compared to some other "harder" scientific realms).

The reason I linked to the second paper is that it directly relates to the auditory system, and points out a case where subconscious perception of a certain auditory stimuli turns out to be much more sensitive than what is consciously perceived. What could be more closely related to this discussion? I can imagine a version of the experiment where levels of noise might be added that, even when well below the threshold of conscious audibility, would have an effect on the measured subconscious perception of the timing. That would be the end of the discussion! That would prove that distortions below the threshold of conscious perception can effect how we hear music (at least to the somewhat open-minded).

Finally, and this was another motivation for me, there seems to be a tendancy here to oversimplify things that really are not that well understood. I am not trying to push any agenda and I am not an audiophile, but I am familiar enough with the scientific research to know that some of the claims often made here and taken as fact (at times in very condescending language) are simply not well supported at this time, and may in some cases turn out to be flatly wrong. I think it's fine if people have different opinions, and the free and lively interchange of ideas is what this board is all about. But I don't think it would hurt for a slightly more flexible attitude to prevail (and it would certainly reduce the possibility of many egg covered faces in the future).

Have a good weekend!

edit: grammer

This post has been edited by bryant: Oct 21 2005, 22:57

[Woodinville]

So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

Well, actually, it is.

First, reverberation is way, way above threshold.

A decent perceptual coder (AAC, WMA Pro for instance) is very near, or under, threshold at a reasonable rate.

The two issues are not in any fashion comparable. If there is no way to detect the difference in two signals coming down the auditory nerve, there is noting for the CNS to work on.

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.

No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.

[bryant]

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.

No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.

Sorry, you're the one who's wrong.

The only relevant measure of information here is the losslessly compressed wav. If we assume, like you did, 50% compression, then the bitrate would be 705 kbps (1411 kbps / 2). To get that down to 128 kbps requires eliminating 82%.

As for your other points, you may be taking the examples a little too specifically. If we knew exactly what effect was going to be discovered we would design an experiment and get it over with. The point I am trying to make is much more general, and is simply that listening tests involving only conscious subject choice may not be sufficient to measure all aspects of auditory perception, and that those areas it cannot properly measure may be involved with music enjoyment. If you think that's a preposterous stretch, good for you.