What's the problem with double-blind testing? Options

[mirrorsawlljk]

I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?

[duff]

I don't understand why you would want to maintain a definition of filtering that does not include the action of neurons that differentially respond to frequency-time information in an encoded signal. This is hardly unusual, and psychophysical researchers have been using "filter" in this way for over a century.

Kowalski, N., Depireux,D. A. & Shamma, S. A. (1996). Analysis of dynamic spectra in ferret primary auditory cortex. II. Prediction of unit responses to arbitrary dynamic spectra. J. Neurophysiol. 76, 3524±3534.

The reason I pointed that out at all is because you made a blanket statement about the auditory cortex and filtering. Just as you made a blanket statement about privileging "non events" in signal detection. You later said you only meant that in a restricted way, but you originally said something much more general. Anyway, onwards...

A paragraph from here speaks at least indirectly to an important issue:

Measurements carried out in living anesthetized lab animals revealed that the waves of maximal displacement produced by moderate sound levels in the living cochlea are actually much, much sharper than can be accounted for by the physical resonant properties of the basilar membrane in cadavers. That work led to an objective method for screening for hearing disorders in newborns that has become widely adopted in the past 5 years. It turns out that cochlear hair cells can be both receptors and effectors. In response to signals from their efferent innervation outer hair cells exhibit slow contractions and elongations. In addition, outer hair cells exhibit fast contractions in response to sound. Those contractions occur when the cells are depolarized and hyperpolarized. Currently, the molecular mechanisms that underlie this motility are under intense investigation. Two years ago the apparent motor protein that powers outer hair cell motility was identified and named prestin. Physiological experiments have established that the local resonance of the basilar membrane is directly modulated as a result of active changes in the length of outer hair cells, but just how that results in highly-tuned mammalian hearing performance is not yet understood.

Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal. But auditory processing doesn't work that way. I'd like to see any empirical support for your argument surrounding effort and noise even at that level. But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The articles I provided show that even when discrimination cannot be made between two stimuli that differ in how much noise they contain, neural systems are treating them differently. That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes. All we have to show to make our basic case, like the articles I presented do show, is that people can be unable to discriminate between two stimuli (rendering them equivalent in ABX terms), while their brains act differently during the presentation of them. The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.

Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.

This post has been edited by duff: Nov 11 2005, 04:04

[Pio2001]

The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level.

An ideal ABX test should be run under the exact same conditions under which the difference tested was first observed. This way, only differences relying on a decision process will be tested.

This works if ABX tests are used to check the validity of people's claims about sonic differences.

I can understand your criticism of ABX if ABX tests are used in order to evaluate a threshold of hearing, for example. In this case, there is no pre-existing decision process that the ABX test is supposed to validate. Differences smaller and smaller are just submitted to the listener.
I find this method rather flawed. The listener must have the possibility to train himself. For example, it becomes easy to pass ABX tests about MP3 compression once you have heard MP3 artifacts, while you can fail the same test if you don't know what to listen to. It happened to me with the Smashing Pumkins sample provided by Xerophase. I could hear no differences, but since he managed to get a successful score quite easily, I listened and listened to the samples again, until I finally found a small difference.

Once the listener is trained, it all becomes a decision process, and the ABX test can be applied. If you apply the ABX test without first training the listeners (and ITU-R BS.1116-1 recommendation for blind listening tests says that It is important that data from listening tests assessing small impairments in audio systems should come exclusively from subjects who have expertise in detecting these small impairments), then you take the risk to get a failure while your subjects are actually capable of passing your test. This could indeed be a situation where the reaction time detects a difference that the decision process doesn't detect (yet).