IPB

Welcome Guest ( Log In | Register )

What's the problem with double-blind testing?
mirrorsawlljk
post Oct 19 2005, 03:03
Post #1





Group: Members
Posts: 11
Joined: 15-October 05
Member No.: 25138



I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?
Go to the top of the page
+Quote Post
 
Start new topic
Replies
duff
post Oct 21 2005, 16:48
Post #2





Group: Members
Posts: 38
Joined: 18-July 02
From: Los Angeles, CA
Member No.: 2666



QUOTE
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression. It is in this sense that I mean that an mp3 file, for example, is compressed..

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
Go to the top of the page
+Quote Post
Lyx
post Oct 21 2005, 17:35
Post #3





Group: Members
Posts: 3353
Joined: 6-July 03
From: Sachsen (DE)
Member No.: 7609



QUOTE (duff @ Oct 21 2005, 05:48 PM)
QUOTE
Seems to be a lack of understanding of something fundamental here: if you encode something to a lossy format it *does* lose information, it's not "compressed" at all. You've actually thrown away a lot of information! How do you know that the auditory system doesn't find it *easier* to decode since there's less "data"?


Lossless and lossy compression schemes are just that, compression. It is in this sense that I mean that an mp3 file, for example, is compressed..

The question of whether perceptual systems have an easier or harder time resolving stimuli that are impoverished (relative to a complementary uncompressed stimulus) is an empirical one that has been addressed in a variety of ways in perception research (mostly vision). I've been talking about ways to address this specifically with audio lossy schemes, and I'm presenting an idea regarding how this effect might not be addressed fully with an ABX paradigm that relies on decision tasks. A reaction time paradigm might be better suited to study the processing differences listeners likely experience.

The reason compressed data requires more effort on the part of whatever computational system is perceiving it, is because more inferential processing is required. If you are missing data in a signal, some mechanism in your brain must essentially interpolate, from the signal, the missing information.
*


This still does not show any evidence that this applies to lossy audiocompression which makes use of psychoacoustic models. It may apply, or it may not apply.

Why does it not necessarily apply? Because the purpose of lossy audio compression is to throw away unperceivable parts of the audio. So if the sensors aren't missing anything, then why should it make any difference? Anyways, discussing this wont really solve it because we simply lack DATA..... we need evidence and analysis..... not extrapolated from other fields..... else we are just speculating.

I definatelly dont think that current test-methods are the end of the road. And i dont think that current DBT-implementations are perfect. But i do think that they're the best ones we have available right now. I am interested in better methods and flaws in current methods, but only if its backed by hard evidence - not extrapolated speculations ala "well, it could be possible that....". Why the hard line? Well, i've seen too many of those "speculations" which in the past ended up being exposed as flawed. Show me something significant and applicable and i'll become interested.

This post has been edited by Lyx: Oct 21 2005, 17:43


--------------------
I am arrogant and I can afford it because I deliver.
Go to the top of the page
+Quote Post

Posts in this topic
- mirrorsawlljk   What's the problem with double-blind testing?   Oct 19 2005, 03:03
- - bubka   some people can actually detect specific codecs by...   Oct 19 2005, 03:07
- - TheQat   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 03:27
|- - singaiya   QUOTE (TheQat @ Oct 18 2005, 06:27 PM)QUOTE (...   Oct 19 2005, 04:31
|- - Yaztromo   QUOTE (TheQat @ Oct 19 2005, 03:27 AM)Edit: H...   Oct 19 2005, 23:05
- - Tahnru   The closest thing I have seen to a legitimate crit...   Oct 19 2005, 04:29
- - Axon   The Audio Critic is notably pro-DBT.   Oct 19 2005, 07:06
|- - PoisonDan   QUOTE (Axon @ Oct 19 2005, 08:06 AM)The Audio...   Oct 19 2005, 12:18
- - onthejazz   Interesting publication. I like it, too bad its no...   Oct 19 2005, 10:03
|- - Donunus   The writer of the letter in stereophile is incorre...   Oct 19 2005, 11:00
|- - Danimal   QUOTE (Donunus @ Oct 19 2005, 05:00 AM)The wr...   Oct 19 2005, 20:20
|- - Donunus   QUOTE (Danimal @ Oct 20 2005, 03:20 AM)QUOTE ...   Oct 20 2005, 01:51
|- - stephanV   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)Well, ...   Oct 20 2005, 08:48
||- - bryant   QUOTE (stephanV @ Oct 19 2005, 11:48 PM)QUOTE...   Oct 20 2005, 18:11
|||- - Pio2001   QUOTE (bryant @ Oct 20 2005, 07:11 PM)To tell...   Oct 20 2005, 20:48
||||- - singaiya   Thanks for the papers, David. The first one is a g...   Oct 20 2005, 22:13
||||- - duff   QUOTE (Pio2001 @ Oct 20 2005, 07:48 PM)QUOTE ...   Oct 20 2005, 22:52
||||- - stephanV   QUOTE (duff @ Oct 20 2005, 11:52 PM)It's ...   Oct 20 2005, 23:11
|||||- - duff   QUOTE Might be... or might be not. You can't c...   Oct 20 2005, 23:36
|||||- - KikeG   QUOTE (duff @ Oct 20 2005, 11:36 PM)I think i...   Oct 21 2005, 00:00
||||||- - rjamorim   QUOTE (KikeG @ Oct 20 2005, 09:00 PM)Also, ps...   Oct 21 2005, 00:15
||||||- - duff   QUOTE (KikeG @ Oct 20 2005, 11:00 PM)QUOTE (d...   Oct 21 2005, 00:29
||||||- - rjamorim   QUOTE (duff @ Oct 20 2005, 09:29 PM)I'm n...   Oct 21 2005, 01:03
||||||- - Mike Giacomelli   QUOTE (duff @ Oct 20 2005, 04:29 PM)The audit...   Oct 21 2005, 05:56
||||||- - KikeG   QUOTE (duff @ Oct 21 2005, 12:29 AM)From the ...   Oct 21 2005, 08:10
|||||- - stephanV   QUOTE (duff @ Oct 21 2005, 12:36 AM)I think i...   Oct 21 2005, 08:34
|||||- - Woodinville   QUOTE (stephanV @ Oct 20 2005, 11:34 PM)Sorry...   Oct 21 2005, 21:19
||||- - krabapple   QUOTE (duff @ Oct 20 2005, 04:52 PM)QUOTE (Pi...   Oct 21 2005, 05:25
||||- - antz   QUOTE (duff @ Oct 20 2005, 10:52 PM)QUOTE (Pi...   Oct 21 2005, 13:39
||||- - Woodinville   QUOTE (antz @ Oct 21 2005, 04:39 AM)Seems to ...   Oct 21 2005, 21:20
|||- - stephanV   QUOTE (bryant @ Oct 20 2005, 07:11 PM)It turn...   Oct 20 2005, 21:43
||- - Donunus   QUOTE (stephanV @ Oct 20 2005, 03:48 PM)QUOTE...   Oct 21 2005, 04:10
||- - stephanV   QUOTE (Donunus @ Oct 21 2005, 05:10 AM)I do n...   Oct 21 2005, 08:43
||- - user   I recall, we have had successful abx tests even he...   Oct 21 2005, 09:20
||- - Pio2001   QUOTE (user @ Oct 21 2005, 10:20 AM)I recall,...   Oct 21 2005, 12:18
|||- - ff123   QUOTE (Pio2001 @ Oct 21 2005, 03:18 AM)QUOTE ...   Oct 21 2005, 17:43
||- - krabapple   QUOTE (user @ Oct 21 2005, 03:20 AM)I recall,...   Oct 21 2005, 21:54
|- - Cartoon   QUOTE (Donunus @ Oct 20 2005, 02:51 AM)I do h...   Nov 12 2005, 13:40
- - KikeG   The "goosebump", emotional factor can be...   Oct 19 2005, 11:54
|- - Donunus   QUOTE (KikeG @ Oct 19 2005, 06:54 PM)The ...   Oct 19 2005, 12:06
|- - kjoonlee   QUOTE (Donunus @ Oct 19 2005, 08:06 PM)actual...   Oct 19 2005, 18:31
||- - Woodinville   QUOTE (kjoonlee @ Oct 19 2005, 09:31 AM)QUOTE...   Oct 21 2005, 01:27
||- - duff   QUOTE If you consider that we are wired to detect ...   Oct 21 2005, 01:38
||- - Woodinville   QUOTE (duff @ Oct 20 2005, 04:38 PM)So we are...   Oct 21 2005, 21:15
||- - duff   QUOTE (Woodinville @ Oct 21 2005, 08:15 PM)QU...   Oct 21 2005, 23:43
||- - Woodinville   QUOTE (duff @ Oct 21 2005, 02:43 PM)When I sa...   Oct 22 2005, 08:00
|- - Lyx   QUOTE (Donunus @ Oct 19 2005, 01:06 PM)QUOTE ...   Oct 20 2005, 06:07
|- - Donunus   QUOTE (Lyx @ Oct 20 2005, 01:07 PM)QUOTE (Don...   Oct 20 2005, 16:51
|- - Lyx   QUOTE (Donunus @ Oct 20 2005, 05:51 PM)QUOTE ...   Oct 20 2005, 21:10
- - KikeG   Well, I think expectation effects in listening tes...   Oct 19 2005, 19:33
|- - krabapple   QUOTE (KikeG @ Oct 19 2005, 10:33 AM)Well, I ...   Oct 19 2005, 20:12
- - krabapple   QUOTE (mirrorsawlljk @ Oct 18 2005, 06:03 PM)...   Oct 19 2005, 20:06
- - Halcyon   There is nothing inherently bad about double blind...   Oct 19 2005, 20:29
- - ff123   Tests that try to distinguish very small effects *...   Oct 19 2005, 20:53
|- - krabapple   QUOTE (ff123 @ Oct 19 2005, 11:53 AM)Tests th...   Oct 19 2005, 23:40
|- - ff123   QUOTE (krabapple @ Oct 19 2005, 02:40 PM)Inte...   Oct 20 2005, 05:38
- - fcmts   There is another problem similar to wine blind tes...   Oct 19 2005, 22:15
- - stephanV   I don't see how that is a problem. The goal of...   Oct 19 2005, 22:22
- - duff   One distinction relevant to this issue is the diff...   Oct 20 2005, 23:00
- - duff   QUOTE The opposite of "blind", I suppose...   Oct 21 2005, 01:27
- - Jun-Dai   I think it's pretty clear that double-blind te...   Oct 21 2005, 07:22
- - KikeG   For helping doing long-term, casual listening-like...   Oct 21 2005, 08:30
- - duff   QUOTE Seems to be a lack of understanding of somet...   Oct 21 2005, 16:48
|- - stephanV   QUOTE (duff @ Oct 21 2005, 05:48 PM)The reaso...   Oct 21 2005, 17:29
||- - duff   QUOTE (stephanV @ Oct 21 2005, 04:29 PM)QUOTE...   Oct 21 2005, 20:06
||- - stephanV   QUOTE (duff @ Oct 21 2005, 09:06 PM)QUOTE Wh...   Oct 21 2005, 20:21
||- - bryant   QUOTE (stephanV @ Oct 21 2005, 11:21 AM)QUOTE...   Oct 21 2005, 22:53
||- - rjamorim   QUOTE (bryant @ Oct 21 2005, 07:53 PM)But I d...   Oct 21 2005, 23:12
||- - stephanV   QUOTE (bryant @ Oct 21 2005, 11:53 PM)So, if ...   Oct 22 2005, 00:00
||- - ChiGung   QUOTE (bryant @ Oct 21 2005, 10:53 PM)What...   Oct 22 2005, 00:25
|||- - bryant   QUOTE (ChiGung @ Oct 21 2005, 03:25 PM)I have...   Oct 22 2005, 23:39
|||- - ChiGung   just thinking -i do get all sorts of weird sensati...   Oct 23 2005, 00:34
||- - Woodinville   QUOTE (bryant @ Oct 21 2005, 01:53 PM)So, if ...   Oct 22 2005, 07:52
||- - bryant   QUOTE (Woodinville @ Oct 21 2005, 10:52 PM)QU...   Oct 22 2005, 23:32
||- - Woodinville   QUOTE (bryant @ Oct 22 2005, 02:32 PM)The onl...   Oct 24 2005, 21:51
|- - Lyx   QUOTE (duff @ Oct 21 2005, 05:48 PM)QUOTE See...   Oct 21 2005, 17:35
|- - krabapple   QUOTE (duff @ Oct 21 2005, 10:48 AM)QUOTE See...   Oct 21 2005, 22:01
- - duff   QUOTE It sounds counterintuitive that my brain wou...   Oct 22 2005, 00:30
|- - ChiGung   QUOTE (duff @ Oct 22 2005, 12:30 AM)Frankly, ...   Oct 22 2005, 00:47
||- - duff   QUOTE (ChiGung @ Oct 21 2005, 11:47 PM)QUOTE ...   Oct 22 2005, 01:14
||- - ChiGung   QUOTE (duff @ Oct 22 2005, 01:14 AM)The reduc...   Oct 22 2005, 01:26
|- - Woodinville   QUOTE (duff @ Oct 21 2005, 03:30 PM)Filtering...   Oct 22 2005, 08:07
- - ChiGung   A simplistic example - a sine wave of exactly 3122...   Oct 22 2005, 00:59
|- - Woodinville   QUOTE (ChiGung @ Oct 21 2005, 03:59 PM)A simp...   Oct 22 2005, 08:08
|- - ChiGung   QUOTE (Woodinville @ Oct 22 2005, 08:08 AM)QU...   Oct 22 2005, 14:09
- - ChiGung   -Sorry for flooding a bit there. Tasty subject   Oct 22 2005, 01:01
- - duff   So sorry for the long delay. I didn't forget a...   Nov 10 2005, 20:10
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 11:10 AM)Woodinsid...   Nov 10 2005, 20:23
- - duff   I don't understand why you would want to maint...   Nov 11 2005, 00:41
|- - Woodinville   QUOTE (duff @ Nov 10 2005, 03:41 PM)Somehow, ...   Nov 11 2005, 21:52
|- - Pio2001   QUOTE (duff @ Nov 11 2005, 01:41 AM)The ABX p...   Nov 12 2005, 03:00
|- - Woodinville   QUOTE (Pio2001 @ Nov 11 2005, 06:00 PM)I can ...   Nov 12 2005, 09:33
|- - user   There aren't theoretical flaws with abx or DB ...   Nov 12 2005, 10:52
- - KikeG   Sorry, you still have provided no evidence that wh...   Nov 11 2005, 14:34
|- - Woodinville   QUOTE (KikeG @ Nov 11 2005, 05:34 AM)Sorry, y...   Nov 11 2005, 21:56
|- - Pio2001   QUOTE (Woodinville @ Nov 11 2005, 10:56 PM)Su...   Nov 12 2005, 02:32
- - duff   Let's back up a moment... There are at least ...   Nov 11 2005, 18:37
- - KikeG   About the first point, your degraded examples are ...   Nov 11 2005, 19:57
- - duff   QUOTE Well, if it's inaudible, there's a g...   Nov 11 2005, 21:19
- - Woodinville   QUOTE (duff @ Nov 11 2005, 12:19 PM)In other ...   Nov 11 2005, 21:59
4 Pages V   1 2 3 > » 


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 19th April 2014 - 09:12