IPB

Welcome Guest ( Log In | Register )

What's the problem with double-blind testing?
mirrorsawlljk
post Oct 19 2005, 03:03
Post #1





Group: Members
Posts: 11
Joined: 15-October 05
Member No.: 25138



I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?
Go to the top of the page
+Quote Post
 
Start new topic
Replies
duff
post Nov 11 2005, 00:41
Post #2





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



I don't understand why you would want to maintain a definition of filtering that does not include the action of neurons that differentially respond to frequency-time information in an encoded signal. This is hardly unusual, and psychophysical researchers have been using "filter" in this way for over a century.

Kowalski, N., Depireux,D. A. & Shamma, S. A. (1996). Analysis of dynamic spectra in ferret primary auditory cortex. II. Prediction of unit responses to arbitrary dynamic spectra. J. Neurophysiol. 76, 35243534.

The reason I pointed that out at all is because you made a blanket statement about the auditory cortex and filtering. Just as you made a blanket statement about privileging "non events" in signal detection. You later said you only meant that in a restricted way, but you originally said something much more general. Anyway, onwards...

A paragraph from here speaks at least indirectly to an important issue:

Measurements carried out in living anesthetized lab animals revealed that the waves of maximal displacement produced by moderate sound levels in the living cochlea are actually much, much sharper than can be accounted for by the physical resonant properties of the basilar membrane in cadavers. That work led to an objective method for screening for hearing disorders in newborns that has become widely adopted in the past 5 years. It turns out that cochlear hair cells can be both receptors and effectors. In response to signals from their efferent innervation outer hair cells exhibit slow contractions and elongations. In addition, outer hair cells exhibit fast contractions in response to sound. Those contractions occur when the cells are depolarized and hyperpolarized. Currently, the molecular mechanisms that underlie this motility are under intense investigation. Two years ago the apparent motor protein that powers outer hair cell motility was identified and named prestin. Physiological experiments have established that the local resonance of the basilar membrane is directly modulated as a result of active changes in the length of outer hair cells, but just how that results in highly-tuned mammalian hearing performance is not yet understood.

Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal. But auditory processing doesn't work that way. I'd like to see any empirical support for your argument surrounding effort and noise even at that level. But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The articles I provided show that even when discrimination cannot be made between two stimuli that differ in how much noise they contain, neural systems are treating them differently. That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes. All we have to show to make our basic case, like the articles I presented do show, is that people can be unable to discriminate between two stimuli (rendering them equivalent in ABX terms), while their brains act differently during the presentation of them. The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

QUOTE
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.

This post has been edited by duff: Nov 11 2005, 04:04
Go to the top of the page
+Quote Post
Woodinville
post Nov 11 2005, 21:52
Post #3





Group: Members
Posts: 1401
Joined: 9-January 05
From: JJ's office.
Member No.: 18957



QUOTE (duff @ Nov 10 2005, 03:41 PM)
Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal.

I propose nothing whatsoever of that sort. There is lots of processing. My point is simple, if there is no detectable remains of an original signal at the auditory nerve, then nothing the cortex does can recover what isn't there in the first place.

And that's what it seems quite clear that basic auditory masking is, the limits of the cochlea to encode the stimulus in the first place.
QUOTE
But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The fact that their firing is graded with level is irrelevant. We're talking about the situation in which the noise from the PPM encoding (that's what neural transmission is, after all, PPM) is large enough that a change in firing rates or first-arrivals is not detectable given the known integration times of the auditory system.

First arrival is indeed the most sensitive, and if you think a minute, it's obvious that a rapid increase in firing rate after a period of low rate is quite easily detectable. The converse not so easily detectable.
QUOTE
That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes.

I have done no such thing. Please do not presume to lecture me or to put elementary, obvious fallacies in my lap.

I am ignoring nothing.

The signals in the experiments you cite are way, way over both absolute and masked thresholds. They are audible. They absolutely create detectable information in the auditory nerve.

A signal below masking threshold does not.

Your experiments are, I will presume, correct for what they measure, but they are not in any fashion relevant to this discussion.
QUOTE
The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

Now, exactly what is your concern with, DBT's, or ABX tests, and for what use? Detecting thresholds or something else?
QUOTE
QUOTE
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.
*



You can not say "will not be reflected", you haven't shown that. What you've shown is a discrimination test between two impaired signals with easily audible impairments, not a test of JND's.


--------------------
-----
J. D. (jj) Johnston
Go to the top of the page
+Quote Post

Posts in this topic
- mirrorsawlljk   What's the problem with double-blind testing?   Oct 19 2005, 03:03
- - bubka   some people can actually detect specific codecs by...   Oct 19 2005, 03:07
- - TheQat   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 03:27
|- - singaiya   QUOTE (TheQat @ Oct 18 2005, 06:27 PM)QUOTE (...   Oct 19 2005, 04:31
|- - Yaztromo   QUOTE (TheQat @ Oct 19 2005, 03:27 AM)Edit: H...   Oct 19 2005, 23:05
- - Tahnru   The closest thing I have seen to a legitimate crit...   Oct 19 2005, 04:29
- - Axon   The Audio Critic is notably pro-DBT.   Oct 19 2005, 07:06
|- - PoisonDan   QUOTE (Axon @ Oct 19 2005, 08:06 AM)The Audio...   Oct 19 2005, 12:18
- - onthejazz   Interesting publication. I like it, too bad its no...   Oct 19 2005, 10:03
|- - Donunus   The writer of the letter in stereophile is incorre...   Oct 19 2005, 11:00
|- - Danimal   QUOTE (Donunus @ Oct 19 2005, 05:00 AM)The wr...   Oct 19 2005, 20:20
|- - Donunus   QUOTE (Danimal @ Oct 20 2005, 03:20 AM)QUOTE ...   Oct 20 2005, 01:51
|- - stephanV   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)Well, ...   Oct 20 2005, 08:48
||- - bryant   QUOTE (stephanV @ Oct 19 2005, 11:48 PM)QUOTE...   Oct 20 2005, 18:11
|||- - Pio2001   QUOTE (bryant @ Oct 20 2005, 07:11 PM)To tell...   Oct 20 2005, 20:48
||||- - singaiya   Thanks for the papers, David. The first one is a g...   Oct 20 2005, 22:13
||||- - duff   QUOTE (Pio2001 @ Oct 20 2005, 07:48 PM)QUOTE ...   Oct 20 2005, 22:52
||||- - stephanV   QUOTE (duff @ Oct 20 2005, 11:52 PM)It's ...   Oct 20 2005, 23:11
|||||- - duff   QUOTE Might be... or might be not. You can't c...   Oct 20 2005, 23:36
|||||- - KikeG   QUOTE (duff @ Oct 20 2005, 11:36 PM)I think i...   Oct 21 2005, 00:00
||||||- - rjamorim   QUOTE (KikeG @ Oct 20 2005, 09:00 PM)Also, ps...   Oct 21 2005, 00:15
||||||- - duff   QUOTE (KikeG @ Oct 20 2005, 11:00 PM)QUOTE (d...   Oct 21 2005, 00:29
||||||- - rjamorim   QUOTE (duff @ Oct 20 2005, 09:29 PM)I'm n...   Oct 21 2005, 01:03
||||||- - Mike Giacomelli   QUOTE (duff @ Oct 20 2005, 04:29 PM)The audit...   Oct 21 2005, 05:56
||||||- - KikeG   QUOTE (duff @ Oct 21 2005, 12:29 AM)From the ...   Oct 21 2005, 08:10
|||||- - stephanV   QUOTE (duff @ Oct 21 2005, 12:36 AM)I think i...   Oct 21 2005, 08:34
|||||- - Woodinville   QUOTE (stephanV @ Oct 20 2005, 11:34 PM)Sorry...   Oct 21 2005, 21:19
||||- - krabapple   QUOTE (duff @ Oct 20 2005, 04:52 PM)QUOTE (Pi...   Oct 21 2005, 05:25
||||- - antz   QUOTE (duff @ Oct 20 2005, 10:52 PM)QUOTE (Pi...   Oct 21 2005, 13:39
||||- - Woodinville   QUOTE (antz @ Oct 21 2005, 04:39 AM)Seems to ...   Oct 21 2005, 21:20
|||- - stephanV   QUOTE (bryant @ Oct 20 2005, 07:11 PM)It turn...   Oct 20 2005, 21:43
||- - Donunus   QUOTE (stephanV @ Oct 20 2005, 03:48 PM)QUOTE...   Oct 21 2005, 04:10
||- - stephanV   QUOTE (Donunus @ Oct 21 2005, 05:10 AM)I do n...   Oct 21 2005, 08:43
||- - user   I recall, we have had successful abx tests even he...   Oct 21 2005, 09:20
||- - Pio2001   QUOTE (user @ Oct 21 2005, 10:20 AM)I recall,...   Oct 21 2005, 12:18
|||- - ff123   QUOTE (Pio2001 @ Oct 21 2005, 03:18 AM)QUOTE ...   Oct 21 2005, 17:43
||- - krabapple   QUOTE (user @ Oct 21 2005, 03:20 AM)I recall,...   Oct 21 2005, 21:54
|- - Cartoon   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)I do h...   Nov 12 2005, 13:40
- - KikeG   The "goosebump", emotional factor can be...   Oct 19 2005, 11:54
|- - Donunus   QUOTE (KikeG @ Oct 19 2005, 06:54 PM)The ...   Oct 19 2005, 12:06
|- - kjoonlee   QUOTE (Donunus @ Oct 19 2005, 08:06 PM)actual...   Oct 19 2005, 18:31
||- - Woodinville   QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)QUOTE...   Oct 21 2005, 01:27
||- - duff   QUOTE If you consider that we are wired to detect ...   Oct 21 2005, 01:38
||- - Woodinville   QUOTE (duff @ Oct 20 2005, 04:38 PM)So we are...   Oct 21 2005, 21:15
||- - duff   QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)QU...   Oct 21 2005, 23:43
||- - Woodinville   QUOTE (duff @ Oct 21 2005, 02:43 PM)When I sa...   Oct 22 2005, 08:00
|- - Lyx   QUOTE (Donunus @ Oct 19 2005, 01:06 PM)QUOTE ...   Oct 20 2005, 06:07
|- - Donunus   QUOTE (Lyx @ Oct 20 2005, 01:07 PM)QUOTE (Don...   Oct 20 2005, 16:51
|- - Lyx   QUOTE (Donunus @ Oct 20 2005, 05:51 PM)QUOTE ...   Oct 20 2005, 21:10
- - KikeG   Well, I think expectation effects in listening tes...   Oct 19 2005, 19:33
|- - krabapple   QUOTE (KikeG @ Oct 19 2005, 10:33 AM)Well, I ...   Oct 19 2005, 20:12
- - krabapple   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 20:06
- - Halcyon   There is nothing inherently bad about double blind...   Oct 19 2005, 20:29
- - ff123   Tests that try to distinguish very small effects *...   Oct 19 2005, 20:53
|- - krabapple   QUOTE (ff123 @ Oct 19 2005, 11:53 AM)Tests th...   Oct 19 2005, 23:40
|- - ff123   QUOTE (krabapple @ Oct 19 2005, 02:40 PM)Inte...   Oct 20 2005, 05:38
- - fcmts   There is another problem similar to wine blind tes...   Oct 19 2005, 22:15
- - stephanV   I don't see how that is a problem. The goal of...   Oct 19 2005, 22:22
- - duff   One distinction relevant to this issue is the diff...   Oct 20 2005, 23:00
- - duff   QUOTE The opposite of "blind", I suppose...   Oct 21 2005, 01:27
- - Jun-Dai   I think it's pretty clear that double-blind te...   Oct 21 2005, 07:22
- - KikeG   For helping doing long-term, casual listening-like...   Oct 21 2005, 08:30
- - duff   QUOTE Seems to be a lack of understanding of somet...   Oct 21 2005, 16:48
|- - stephanV   QUOTE (duff @ Oct 21 2005, 05:48 PM)The reaso...   Oct 21 2005, 17:29
||- - duff   QUOTE (stephanV @ Oct 21 2005, 04:29 PM)QUOTE...   Oct 21 2005, 20:06
||- - stephanV   QUOTE (duff @ Oct 21 2005, 09:06 PM)QUOTE Wh...   Oct 21 2005, 20:21
||- - bryant   QUOTE (stephanV @ Oct 21 2005, 11:21 AM)QUOTE...   Oct 21 2005, 22:53
||- - rjamorim   QUOTE (bryant @ Oct 21 2005, 07:53 PM)But I d...   Oct 21 2005, 23:12
||- - stephanV   QUOTE (bryant @ Oct 21 2005, 11:53 PM)So, if ...   Oct 22 2005, 00:00
||- - ChiGung   QUOTE (bryant @ Oct 21 2005, 10:53 PM)What...   Oct 22 2005, 00:25
|||- - bryant   QUOTE (ChiGung @ Oct 21 2005, 03:25 PM)I have...   Oct 22 2005, 23:39
|||- - ChiGung   just thinking -i do get all sorts of weird sensati...   Oct 23 2005, 00:34
||- - Woodinville   QUOTE (bryant @ Oct 21 2005, 01:53 PM)So, if ...   Oct 22 2005, 07:52
||- - bryant   QUOTE (Woodinville @ Oct 21 2005, 10:52 PM)QU...   Oct 22 2005, 23:32
||- - Woodinville   QUOTE (bryant @ Oct 22 2005, 02:32 PM)The onl...   Oct 24 2005, 21:51
|- - Lyx   QUOTE (duff @ Oct 21 2005, 05:48 PM)QUOTE See...   Oct 21 2005, 17:35
|- - krabapple   QUOTE (duff @ Oct 21 2005, 10:48 AM)QUOTE See...   Oct 21 2005, 22:01
- - duff   QUOTE It sounds counterintuitive that my brain wou...   Oct 22 2005, 00:30
|- - ChiGung   QUOTE (duff @ Oct 22 2005, 12:30 AM)Frankly, ...   Oct 22 2005, 00:47
||- - duff   QUOTE (ChiGung @ Oct 21 2005, 11:47 PM)QUOTE ...   Oct 22 2005, 01:14
||- - ChiGung   QUOTE (duff @ Oct 22 2005, 01:14 AM)The reduc...   Oct 22 2005, 01:26
|- - Woodinville   QUOTE (duff @ Oct 21 2005, 03:30 PM)Filtering...   Oct 22 2005, 08:07
- - ChiGung   A simplistic example - a sine wave of exactly 3122...   Oct 22 2005, 00:59
|- - Woodinville   QUOTE (ChiGung @ Oct 21 2005, 03:59 PM)A simp...   Oct 22 2005, 08:08
|- - ChiGung   QUOTE (Woodinville @ Oct 22 2005, 08:08 AM)QU...   Oct 22 2005, 14:09
- - ChiGung   -Sorry for flooding a bit there. Tasty subject   Oct 22 2005, 01:01
- - duff   So sorry for the long delay. I didn't forget a...   Nov 10 2005, 20:10
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 11:10 AM)Woodinsid...   Nov 10 2005, 20:23
- - duff   I don't understand why you would want to maint...   Nov 11 2005, 00:41
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 03:41 PM)Somehow, ...   Nov 11 2005, 21:52
|- - Pio2001   QUOTE (duff @ Nov 11 2005, 01:41 AM)The ABX p...   Nov 12 2005, 03:00
|- - Woodinville   QUOTE (Pio2001 @ Nov 11 2005, 06:00 PM)I can ...   Nov 12 2005, 09:33
|- - user   There aren't theoretical flaws with abx or DB ...   Nov 12 2005, 10:52
- - KikeG   Sorry, you still have provided no evidence that wh...   Nov 11 2005, 14:34
|- - Woodinville   QUOTE (KikeG @ Nov 11 2005, 05:34 AM)Sorry, y...   Nov 11 2005, 21:56
|- - Pio2001   QUOTE (Woodinville @ Nov 11 2005, 10:56 PM)Su...   Nov 12 2005, 02:32
- - duff   Let's back up a moment... There are at least ...   Nov 11 2005, 18:37
- - KikeG   About the first point, your degraded examples are ...   Nov 11 2005, 19:57
- - duff   QUOTE Well, if it's inaudible, there's a g...   Nov 11 2005, 21:19
- - Woodinville   QUOTE (duff @ Nov 11 2005, 12:19 PM)In other ...   Nov 11 2005, 21:59
4 Pages V   1 2 3 > » 


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 16th April 2014 - 09:02