IPB

Welcome Guest ( Log In | Register )

Overcoming the Perception Problem
item
post Oct 8 2012, 19:21
Post #1





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



I've noticed an increase in forum debate about the validity of transferring the credibility of ABX from the physical domain to perception testing. I'm wondering if anyone has found a way past this issue?

The purpose of blind testing is to subtract subjectivity from the effect of - for instance - a drug trial: to assess a medication's impact on a subject's physiology with interference from their psychology. But what about when the purpose of a test is subjective perception? How do we then subtract the effect of the method to arrive at a meaningful outcome?

While we would like to remove expectation bias from the equation, if the conditions under which this is done also change the perceptive state of the listener, the test is invalidated as surely as they would be by tissue sample contamination.

Recent large scale public experiments by Lotto Labs (http://www.lottolab.org/) demonstrated that perception acuity is dramatically altered by test conditions: for instance, that time contraction/dilation effects are experienced when exposed to colour fields. In one experiment, two groups were asked to perform an identical fine-grained visual acuity test. One group was pre-emptively 'manipulated' by filling in a questionnaire designed to lower their self-esteem. This 'less confident' group consistently performed worse on the test that the unmanipulated one: their acuity was significantly impaired by a subtle psychological 'tweak' that wasn't even in effect during the test.

It seems undeniable that the much grosser differences between the mental states of sighted and 'blind' listening - considered generously - cast serious doubt on the results thus obtained.

The harder line is that blind perception tests are a fundamental misappropriation of methodology. In psychology it's axiomatic that for many experiments the subject must be unaware of the nature of the test (see Milgram). If a normalised state is not cunningly contrived, results are at best only indicative of what a subject thinks they should do; at worst, entirely invalid.

Probing hearing, the point is that a test must not change the mental state of the listener.

The contrast between outcomes of sighted and listening tests is as stark as those demonstrating suggestibility (see McGurk), but giving too much credence to such an intrinsically unsound experimental approach (not spotting this difficulty) does no favours to our credibility at all.

The only way past the dilemma seems to be direct mechanical examination of the mind during 'normal' listening to explore why the experiences of sighted and unsighted listening differ. This seems to be an interesting question.

In the meantime, the idea that - despite the method problem - results from blind ABX are valid is at least supported by the majority of data derived from home testing, Audio DiffMaker et al, so we needn't get hung up on it.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
item
post Oct 8 2012, 19:52
Post #2





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



To rephrase: what, in any trial, is blind testing designed to filter out?

Not an attack - no need to be combative! - it's self-evident that psych evaluations often depend crucially on the subject not being aware of the purpose of the test. Why?

I think you may have misunderstood the Lotto Labs experiments I referred to: maybe check them out. They aren't about manipulating the tester's state: they explore how (surprisingly) easily perceptual states are changed by environmental conditions: changing the subject's mind changes the subject's mind . . . .

The degree to which perception can be differentiated from subconscious neural activity is a whole different (and tangential) question.

This post has been edited by item: Oct 8 2012, 19:55
Go to the top of the page
+Quote Post
Soap
post Oct 8 2012, 19:59
Post #3





Group: Members
Posts: 1001
Joined: 19-November 06
Member No.: 37767



QUOTE (item @ Oct 8 2012, 14:52) *
To rephrase then: what, in any trial, is blind testing designed to filter out?

Bias
QUOTE (item @ Oct 8 2012, 14:52) *
What doesn't need attacking, because it's self-evident, is that psych evaluations often depend crucially on the subject not being aware of the purpose of the test. Why is that?

So a test of perception is now a "psych" test? (Whatever that lump is.)

QUOTE (item @ Oct 8 2012, 14:52) *
I think you may have misunderstood the Lotto Labs experiments I referred to: maybe check them out. They aren't about manipulating the tester's state: they explore how (surprisingly) easily perceptual states are changed by environmental conditions: changing the subject's mind changes the subject's mind . . . .

Not in the slightest. They aren't about manipulating the tester's state, but they are dependent upon manipulation of tester's state.

Regardless, you dodged the question. Call it "tester's state" call it "environmental conditions", call it what you will. Where is your argument, much less your evidence, that the manner of ABX testing practiced creates a systematic bias in "environmental conditions"? The Lotto experiments were dependent on such a systematic influence. If you can't demonstrate one They Are Not Relevant.

QUOTE (item @ Oct 8 2012, 14:52) *
The degree to which perception can be differentiated from subconscious neural acticity is a whole different (and tangential) question.


Agreed, it is totally off topic and undefended. But it is one you brought up.

This post has been edited by Soap: Oct 8 2012, 20:00


--------------------
Creature of habit.
Go to the top of the page
+Quote Post
item
post Oct 8 2012, 23:32
Post #4





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



QUOTE (Soap @ Oct 8 2012, 19:59) *
QUOTE (item @ Oct 8 2012, 14:52) *
To rephrase then: what, in any trial, is blind testing designed to filter out?

Bias

Partly, yes: more specifically, in the clinical domain 'blindness' separates the psychological from the physiological. From the perspective of a drugs trial, psychological factors are generally extraneous and need to be excised from the process. From the perspective of an auditory trial, 'psychological factors' are the subject of the test.

QUOTE (Soap @ Oct 8 2012, 19:59) *
QUOTE (item @ Oct 8 2012, 14:52) *
I think you may have misunderstood the Lotto Labs experiments I referred to: maybe check them out. They aren't about manipulating the tester's state: they explore how (surprisingly) easily perceptual states are changed by environmental conditions: changing the subject's mind changes the subject's mind . . . .


Not in the slightest. They aren't about manipulating the tester's state, but they are dependent upon manipulation of tester's state. Regardless, you dodged the question. Call it "tester's state" call it "environmental conditions", call it what you will. Where is your argument, much less your evidence, that the manner of ABX testing practiced creates a systematic bias in "environmental conditions"? The Lotto experiments were dependent on such a systematic influence. If you can't demonstrate one They Are Not Relevant.


Again, the state of the tester is irrelevant: it's the subject we're interested in - the testees, if you will - and the test environment. You're not making yourself clear: are you saying you don't like the conclusions of the experiments I referred to? Or that environment makes no difference to perception? Or are you claiming that listening to music for pleasure, through known equipment, is materially the same as listening analytically under test conditions, 'blind' to what you're hearing?

QUOTE (Soap @ Oct 8 2012, 19:59) *
QUOTE (item @ Oct 8 2012, 14:52) *
The degree to which perception can be differentiated from subconscious neural acticity is a whole different (and tangential) question.


Agreed, it is totally off topic and undefended. But it is one you brought up.

As it seems to me that you introduced this irrelevant topic - and apparently vice versa - can we agree to move on?! I don't want to get bogged down redefining the problem, but would like to begin discussing solutions . . .

This post has been edited by item: Oct 8 2012, 23:38
Go to the top of the page
+Quote Post
item
post Oct 9 2012, 00:30
Post #5





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



Part of Beau Lotto's 'Public Perception project was documented by the BBC Horizon programme in 2011: http://www.bbc.co.uk/programmes/b013c8tb
http://www.lottolab.org/programmes-article...nperception.asp

He expressed surprise that such a subtle a priori manipulation of the subject's self-esteem (of all things!) would depress visual acuity to the extent it did. Particularly germane to this discussion is that the test required subjects to distinguish two subtly different colours.

That particular experiment neatly illustrates the problem, but the issue doesn't hinge on any single experiment: the point is general to all such tests. If the method in any way modifies the 'normal' state of the listener, the data will be invalid, and subsequent statistical analysis is a fool's errand. Attempts to borrow the credibility of drug trials for the purpose of a perception test fatally misunderstands the purpose of such testing and is (at best) sloppy.

Although it's tempting to reach for the conclusion that almost everything is identical, there is an equally valid interpretation of results generated by DBT tests which invariably demonstrate a diminution of differences (oranges become more like lemons, Stradivari become more like toys, speakers become more like speakers) that seem apparent when sighted: namely that the method itself results in a diminution of differences. The more 'blunt tool' results emerge from blind perception tests, the less credible they look.
Go to the top of the page
+Quote Post

Posts in this topic
- item   Overcoming the Perception Problem   Oct 8 2012, 19:21
- - Soap   You appear to be confused. Despite common shortha...   Oct 8 2012, 19:38
- - item   To rephrase: what, in any trial, is blind testing ...   Oct 8 2012, 19:52
|- - Soap   QUOTE (item @ Oct 8 2012, 14:52) To rephr...   Oct 8 2012, 19:59
|- - item   QUOTE (Soap @ Oct 8 2012, 19:59) QUOTE (i...   Oct 8 2012, 23:32
|- - item   Part of Beau Lotto's 'Public Perception pr...   Oct 9 2012, 00:30
||- - Porcus   QUOTE (item @ Oct 9 2012, 01:30) Part of ...   Oct 9 2012, 10:53
||- - item   QUOTE (Porcus @ Oct 9 2012, 10:53) You ca...   Oct 9 2012, 12:23
||- - Porcus   QUOTE (item @ Oct 9 2012, 13:23) This for...   Oct 9 2012, 16:04
|- - krabapple   QUOTE (item @ Oct 8 2012, 18:32) Partly, ...   Oct 9 2012, 03:51
|- - item   QUOTE (item @ Oct 8 2012, 18:32) Although...   Oct 9 2012, 11:14
- - DVDdoug   I just don't see how making a good scientific ...   Oct 9 2012, 00:59
- - greynol   @item: Perhaps you could share with us a little ab...   Oct 9 2012, 04:54
|- - Porcus   QUOTE (greynol @ Oct 9 2012, 05:54) a fai...   Oct 9 2012, 09:57
|- - item   QUOTE (greynol @ Oct 9 2012, 04:54) @item...   Oct 9 2012, 11:42
|- - 2Bdecided   QUOTE (item @ Oct 9 2012, 11:42) Positive...   Oct 9 2012, 14:15
||- - item   QUOTE (2Bdecided @ Oct 9 2012, 14:15) QUO...   Oct 9 2012, 14:37
||- - 2Bdecided   QUOTE (item @ Oct 9 2012, 14:37) Deprived...   Oct 9 2012, 17:43
|||- - item   Sorry - been away, but lots of noise (most - fasci...   Oct 19 2012, 17:34
||- - dhromed   QUOTE (item @ Oct 9 2012, 15:37) 'Kno...   Oct 9 2012, 19:11
|- - greynol   QUOTE (item @ Oct 9 2012, 03:42) QUOTE (g...   Oct 9 2012, 22:07
|- - item   QUOTE (greynol @ Oct 9 2012, 22:07) @item...   Oct 19 2012, 18:33
- - hlloyge   The only people I know of to shun DBT method of te...   Oct 9 2012, 12:35
|- - item   QUOTE (hlloyge @ Oct 9 2012, 12:35) The o...   Oct 9 2012, 13:25
|- - hlloyge   QUOTE (item @ Oct 9 2012, 14:25) The sole...   Oct 9 2012, 13:54
||- - item   QUOTE (hlloyge @ Oct 9 2012, 13:54) The o...   Oct 9 2012, 14:09
|- - Canar   QUOTE (item @ Oct 9 2012, 05:25) I think ...   Oct 9 2012, 17:09
|- - krabapple   QUOTE (item @ Oct 9 2012, 08:25) The sole...   Oct 10 2012, 01:25
|- - item   QUOTE (krabapple @ Oct 10 2012, 01:25) QU...   Oct 19 2012, 18:45
|- - Porcus   QUOTE (item @ Oct 19 2012, 19:45) QUOTE (...   Oct 19 2012, 19:22
|- - 2Bdecided   QUOTE (item @ Oct 19 2012, 18:45) Certain...   Oct 22 2012, 12:53
- - aethelberht   What I can perhaps maybe possibly gather from your...   Oct 9 2012, 13:17
|- - item   QUOTE (aethelberht @ Oct 9 2012, 13:17) W...   Oct 9 2012, 14:01
|- - aethelberht   "A negative means - equally - either a) the t...   Oct 9 2012, 14:14
|- - Arnold B. Krueger   QUOTE (item @ Oct 9 2012, 09:01) DBT is d...   Oct 9 2012, 14:53
|- - hlloyge   QUOTE (item @ Oct 9 2012, 15:01) Abstract...   Oct 10 2012, 21:04
|- - Porcus   QUOTE (hlloyge @ Oct 10 2012, 22:04) If y...   Oct 11 2012, 14:20
|- - hlloyge   QUOTE (Porcus @ Oct 11 2012, 15:20) Ehem ...   Oct 12 2012, 14:26
|- - Porcus   QUOTE (hlloyge @ Oct 12 2012, 15:26) QUOT...   Oct 12 2012, 15:04
|- - hlloyge   QUOTE (Porcus @ Oct 12 2012, 16:04) Not n...   Oct 12 2012, 22:14
|- - greynol   QUOTE (hlloyge @ Oct 12 2012, 14:14) I am...   Oct 13 2012, 14:25
|- - hlloyge   QUOTE (greynol @ Oct 13 2012, 15:25) QUOT...   Oct 14 2012, 13:35
|- - Porcus   QUOTE (hlloyge @ Oct 14 2012, 14:35) Yes,...   Oct 14 2012, 23:31
|- - hlloyge   QUOTE (Porcus @ Oct 15 2012, 00:31) QUOTE...   Oct 16 2012, 09:54
- - Porcus   While I certainly agree that putting humans in a l...   Oct 10 2012, 13:09
- - 2Bdecided   It's like The Princess and the Pea. Audiophile...   Oct 11 2012, 11:58
- - skamp   If ABXing negatively alters one's ability to h...   Oct 11 2012, 14:28
|- - krabapple   QUOTE (skamp @ Oct 11 2012, 09:28) If ABX...   Oct 11 2012, 18:24
|- - mzil   QUOTE (krabapple @ Oct 11 2012, 13:24) QU...   Oct 15 2012, 17:14
|- - krabapple   You're right that different terms apply when w...   Oct 15 2012, 20:51
- - greynol   I believe our skeptic has flown the coop.   Oct 11 2012, 16:17
- - dhromed   Not everyone is as much a netizen as most of us. P...   Oct 11 2012, 17:05
- - googlebot   While the OP's reasoning and claimed inference...   Oct 11 2012, 21:44
|- - [JAZ]   QUOTE (googlebot @ Oct 11 2012, 22:44) Do...   Oct 11 2012, 22:37
||- - bandpass   QUOTE ([JAZ] @ Oct 11 2012, 22:37...   Oct 12 2012, 06:44
|- - sld   QUOTE (googlebot @ Oct 12 2012, 04:44) Lo...   Oct 12 2012, 03:38
|- - krabapple   QUOTE (googlebot @ Oct 11 2012, 16:44) Wh...   Oct 12 2012, 06:59
||- - googlebot   QUOTE (krabapple @ Oct 12 2012, 07:59) Ex...   Oct 12 2012, 21:44
||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:44) Th...   Oct 12 2012, 21:51
|||- - googlebot   QUOTE (Nick.C @ Oct 12 2012, 22:51) Is th...   Oct 12 2012, 21:58
|||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:58) Ye...   Oct 12 2012, 22:21
|||- - googlebot   I do not see how calling the phenomenon "prec...   Oct 12 2012, 23:43
||- - krabapple   QUOTE (googlebot @ Oct 12 2012, 16:44) QU...   Oct 13 2012, 22:05
|- - 2Bdecided   @Porcus: QUOTE (googlebot @ Oct 11 2012, 21...   Oct 12 2012, 10:08
|- - Arnold B. Krueger   QUOTE (googlebot @ Oct 11 2012, 16:44) Co...   Oct 16 2012, 15:51
- - greynol   I would be careful not to limit the word perceive....   Oct 11 2012, 22:58
- - greynol   I extract all the joy I could ever need from simpl...   Oct 12 2012, 06:11
- - Porcus   Among my friends, we have been blind testing ... h...   Oct 12 2012, 09:30
- - dhromed   But is there a problem?   Oct 12 2012, 22:12
- - AndyH-ha   The sighted test difference is not coming from the...   Oct 13 2012, 03:32
|- - greynol   QUOTE (AndyH-ha @ Oct 12 2012, 19:32...   Oct 13 2012, 14:16
- - Nick.C   @googlebot: You are now allowing the results to be...   Oct 13 2012, 09:43
- - 2Bdecided   I think Googlebot is making a valid philosophical ...   Oct 15 2012, 12:20
|- - pisymbol   QUOTE (2Bdecided @ Oct 15 2012, 07:20) I ...   Oct 15 2012, 13:48
- - skamp   QUOTE (mzil @ Oct 15 2012, 18:14) B. The ...   Oct 15 2012, 22:49
|- - krabapple   QUOTE (skamp @ Oct 15 2012, 17:49) What g...   Oct 16 2012, 03:58
- - greynol   There are ways of cheating to get positive ABX res...   Oct 15 2012, 22:57
- - mzil   [Trying to bring this back on topic] There is not...   Oct 16 2012, 04:32
|- - Woodinville   QUOTE (mzil @ Oct 15 2012, 20:32) [Trying...   Oct 16 2012, 04:33
|- - Porcus   QUOTE (mzil @ Oct 16 2012, 05:32) Here...   Oct 16 2012, 06:19
- - mzil   Please enlighten me. I am not a scientist nor have...   Oct 16 2012, 04:39
|- - krabapple   QUOTE (mzil @ Oct 15 2012, 23:39) Please ...   Oct 16 2012, 22:28
|- - mzil   QUOTE (krabapple @ Oct 16 2012, 17:28) QU...   Oct 17 2012, 00:30
|- - saratoga   QUOTE (mzil @ Oct 16 2012, 19:30) So even...   Oct 17 2012, 03:03
||- - mzil   QUOTE (saratoga @ Oct 16 2012, 22:03) QUO...   Oct 17 2012, 05:35
||- - saratoga   QUOTE (mzil @ Oct 17 2012, 00:35) Sorry, ...   Oct 17 2012, 05:43
|- - krabapple   QUOTE (mzil @ Oct 16 2012, 19:30) QUOTE (...   Oct 17 2012, 04:53
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) QU...   Oct 17 2012, 06:36
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 17 2012, 07:22
|||- - mzil   QUOTE (mzil @ Oct 17 2012, 02:22) QUOTE (...   Oct 18 2012, 19:16
||- - Arnold B. Krueger   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 18 2012, 14:04
|- - Porcus   QUOTE (mzil @ Oct 17 2012, 01:30) QUOTE R...   Oct 17 2012, 18:51
- - knutinh   Self-reporting about ones mental state surely carr...   Oct 16 2012, 21:02
- - greynol   Let me get this straight, the subconscious mind is...   Oct 17 2012, 05:28
- - mzil   Thanks, Porcus. I'll check it out.   Oct 17 2012, 20:14
- - krabapple   It's best to be careful drawing conclusions fr...   Oct 19 2012, 17:53
|- - 2Bdecided   QUOTE (krabapple @ Oct 19 2012, 17:53) It...   Oct 22 2012, 12:49
- - AndyH-ha   I could be missing the point, or dozens of them, b...   Oct 19 2012, 20:56
|- - Arnold B. Krueger   QUOTE (AndyH-ha @ Oct 19 2012, 15:56...   Oct 19 2012, 21:30
- - AndyH-ha   My point was not that many reported tests involve ...   Oct 19 2012, 22:31
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 18:18