IPB

Welcome Guest ( Log In | Register )

Overcoming the Perception Problem
item
post Oct 8 2012, 19:21
Post #1





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



I've noticed an increase in forum debate about the validity of transferring the credibility of ABX from the physical domain to perception testing. I'm wondering if anyone has found a way past this issue?

The purpose of blind testing is to subtract subjectivity from the effect of - for instance - a drug trial: to assess a medication's impact on a subject's physiology with interference from their psychology. But what about when the purpose of a test is subjective perception? How do we then subtract the effect of the method to arrive at a meaningful outcome?

While we would like to remove expectation bias from the equation, if the conditions under which this is done also change the perceptive state of the listener, the test is invalidated as surely as they would be by tissue sample contamination.

Recent large scale public experiments by Lotto Labs (http://www.lottolab.org/) demonstrated that perception acuity is dramatically altered by test conditions: for instance, that time contraction/dilation effects are experienced when exposed to colour fields. In one experiment, two groups were asked to perform an identical fine-grained visual acuity test. One group was pre-emptively 'manipulated' by filling in a questionnaire designed to lower their self-esteem. This 'less confident' group consistently performed worse on the test that the unmanipulated one: their acuity was significantly impaired by a subtle psychological 'tweak' that wasn't even in effect during the test.

It seems undeniable that the much grosser differences between the mental states of sighted and 'blind' listening - considered generously - cast serious doubt on the results thus obtained.

The harder line is that blind perception tests are a fundamental misappropriation of methodology. In psychology it's axiomatic that for many experiments the subject must be unaware of the nature of the test (see Milgram). If a normalised state is not cunningly contrived, results are at best only indicative of what a subject thinks they should do; at worst, entirely invalid.

Probing hearing, the point is that a test must not change the mental state of the listener.

The contrast between outcomes of sighted and listening tests is as stark as those demonstrating suggestibility (see McGurk), but giving too much credence to such an intrinsically unsound experimental approach (not spotting this difficulty) does no favours to our credibility at all.

The only way past the dilemma seems to be direct mechanical examination of the mind during 'normal' listening to explore why the experiences of sighted and unsighted listening differ. This seems to be an interesting question.

In the meantime, the idea that - despite the method problem - results from blind ABX are valid is at least supported by the majority of data derived from home testing, Audio DiffMaker et al, so we needn't get hung up on it.
Go to the top of the page
+Quote Post
 
Start new topic
Replies
hlloyge
post Oct 9 2012, 12:35
Post #2





Group: Members
Posts: 689
Joined: 10-January 06
From: Zagreb
Member No.: 27018



The only people I know of to shun DBT method of testing audio equipment are those that live off them in some way (editors/writers of hifi magazines, hifi salesmen) and people who themselves believe in their superiority over the common plebs in terms of hearing. The first kind knows that utilizing DBT in their reviews would result in sales plummet, and the loss of income through payed reviews and advertising. Second ones are often technologically disabled, and are more prone to explain things to themselves (and, more dangerously, to others) through pure magic and rituals, than to actually learn what is going on, because the bubble they live in would burst.
Now, you say DBT testing would somehow influence the listener and he wouldn't hear the difference because of reasons. Bear in mind that these people often claim the "sky-earth" difference between two DACs, for example, so I hardly believe that testing that difference over the course of time (a month, a year) would involve any stress, and that they wouldn't hear even the tiniest difference, if it really exists.
That argument is so invalid - if you are so easily affected by switching buttons form A to B, X to Y, than I am sure that every listening to the same song is a new experience, and it sounds different altogether smile.gif and that difference, either is there or is not, that difference does not exist only when we are casually listening to music. Humans can't telepathically effect the bitstream in DACs or optical cables yet. It doesn't care what are you feeling, it just - streams and decodes, over and over again, every time you play the song.
I really don't care how the ABX testing in medical research works - I'm not into medicine at all, and for the hydrogenaudio's sake, it shouldn't matter. Only thing that matters is audio ABX test, which serves to individuals to see if they really can hear difference between two codecs, or two DACs, if they have equipment to set this up. Individuals set up the testing environment as they prefer (I like drinking cocoa, for example), and the test is straightforward in the results - either you can hear the difference, or you can't. If you can't, that doesn't mean there is none, it just means that you can't hear it. Someone else might.
So, why do you try so hard to convince us that ABX isn't valid method?
Go to the top of the page
+Quote Post
item
post Oct 9 2012, 13:25
Post #3





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



QUOTE (hlloyge @ Oct 9 2012, 12:35) *
The only people I know of to shun DBT method of testing audio equipment are those that live off them in some way (editors/writers of hifi magazines, hifi salesmen) and people who themselves believe in their superiority over the common plebs in terms of hearing. The first kind knows that utilizing DBT in their reviews would result in sales plummet, and the loss of income through payed reviews and advertising. Second ones are often technologically disabled, and are more prone to explain things to themselves (and, more dangerously, to others) through pure magic and rituals, than to actually learn what is going on, because the bubble they live in would burst.
Now, you say DBT testing would somehow influence the listener and he wouldn't hear the difference because of reasons. Bear in mind that these people often claim the "sky-earth" difference between two DACs, for example, so I hardly believe that testing that difference over the course of time (a month, a year) would involve any stress, and that they wouldn't hear even the tiniest difference, if it really exists.
That argument is so invalid - if you are so easily affected by switching buttons form A to B, X to Y, than I am sure that every listening to the same song is a new experience, and it sounds different altogether smile.gif and that difference, either is there or is not, that difference does not exist only when we are casually listening to music. Humans can't telepathically effect the bitstream in DACs or optical cables yet. It doesn't care what are you feeling, it just - streams and decodes, over and over again, every time you play the song.
I really don't care how the ABX testing in medical research works - I'm not into medicine at all, and for the hydrogenaudio's sake, it shouldn't matter. Only thing that matters is audio ABX test, which serves to individuals to see if they really can hear difference between two codecs, or two DACs, if they have equipment to set this up. Individuals set up the testing environment as they prefer (I like drinking cocoa, for example), and the test is straightforward in the results - either you can hear the difference, or you can't. If you can't, that doesn't mean there is none, it just means that you can't hear it. Someone else might.
So, why do you try so hard to convince us that ABX isn't valid method?

I think perhaps you underestimate the general level of the public's intelligence. Anyone who buys a piece of audio equipment knows that most visitors to their house will point out that this system sounds pretty similar to the last one they were excited about.

At the back of their mind, most buyers know that past a certain basic level of competence, expensive equipment is all counting the number of angels dancing on a pinhead. But people go see illusionists because the illusion is fun. People buy fancy boxes because there is pride of ownership - and - away from the hype, alone in their living room - for whatever reason - there is a absolutely real, fundamental sense of pleasure in music reproduction that may - or may not - derive from measured performance of the boxes. There is also the unshakeable fact that humans are status-driven, and that audio equipment is a status symbol, just like a car.

This may all be wrong, but it will persist. It is unaffected by our little erudite discussions about what ultimately can be measured or perceived.

I also know personally a number of editors, writers, salesmen and manufacturers: some do think they are superior to the plebs, but not usually because of their hearing: it's just their character. Similarly, I know a number of judges and car salesmen and postman who feel exactly the same way. It's also completely untrue that DBT for audio is unused in the audio industry: it's a standard tool for many makers and reviewers.

The sole, specific point I'm making is that DBT is rarely used in perception testing for obvious reasons outlined above, and attempting to smear its credibility from the physiological domain is intellectually dishonest. And that the abundance of negative results indicates coarse granularity in the test method as much as it supports any particular paradigm.
Go to the top of the page
+Quote Post
krabapple
post Oct 10 2012, 01:25
Post #4





Group: Members
Posts: 2157
Joined: 18-December 03
Member No.: 10538



QUOTE (item @ Oct 9 2012, 08:25) *
The sole, specific point I'm making is that DBT is rarely used in perception testing for obvious reasons outlined above, and attempting to smear its credibility from the physiological domain is intellectually dishonest.


And as I told you before, double blind protocols are common in pain perception studies, where self-reports of perception are the outputs. You going to address all that, or punt again?

QUOTE
And that the abundance of negative results indicates coarse granularity in the test method as much as it supports any particular paradigm.


It 'indicates' that to you because you believe it should, not because it's necessarily true. And as I said before, this amounts to nothing more than an argument from incredulity, if not ignorance.

As 2bdecided already noted, DBTs can support audio signal level differences down to the physical limits of human hearing. Their *granularity* is quite good, if the protocol is good, your windy assertions notwithstanding.

And now, Mr Joined-in-August, I trust you won't mind if I sit back and await your predictable departure (back?) to more woozy audio forums, where you'll claim you were driven out of HA for 'unorthodoxy' or 'thinking outside the box'.

This post has been edited by krabapple: Oct 10 2012, 01:34
Go to the top of the page
+Quote Post
item
post Oct 19 2012, 18:45
Post #5





Group: Members
Posts: 33
Joined: 7-August 12
Member No.: 102085



QUOTE (krabapple @ Oct 10 2012, 01:25) *
QUOTE (item @ Oct 9 2012, 08:25) *
The sole, specific point I'm making is that DBT is rarely used in perception testing for obvious reasons outlined above, and attempting to smear its credibility from the physiological domain is intellectually dishonest.


And as I told you before, double blind protocols are common in pain perception studies, where self-reports of perception are the outputs. You going to address all that, or punt again?


And you posit an equation between pain and hearing?!

QUOTE
And that the abundance of negative results indicates coarse granularity in the test method as much as it supports any particular paradigm.


QUOTE
It 'indicates' that to you because you believe it should, not because it's necessarily true. And as I said before, this amounts to nothing more than an argument from incredulity, if not ignorance.


Certainly not: the point explicitly made is that the results are open to two interpretations. You are insisting on one interpretation so dogmatically that the other, equally valid one, is being dismissed. Consideration of alternatives is what I like to call argument by method.

QUOTE (krabapple @ Oct 10 2012, 01:25) *
And now, Mr Joined-in-August, I trust you won't mind if I sit back and await your predictable departure (back?) to more woozy audio forums, where you'll claim you were driven out of HA for 'unorthodoxy' or 'thinking outside the box'.

Let's not snipe!

This post has been edited by item: Oct 19 2012, 18:46
Go to the top of the page
+Quote Post
2Bdecided
post Oct 22 2012, 12:53
Post #6


ReplayGain developer


Group: Developer
Posts: 4945
Joined: 5-November 01
From: Yorkshire, UK
Member No.: 409



QUOTE (item @ Oct 19 2012, 18:45) *
Certainly not: the point explicitly made is that the results are open to two interpretations. You are insisting on one interpretation so dogmatically that the other, equally valid one, is being dismissed.
I'm not sure they can be equally valid. One has been proven to be true sometimes (people swearing they hear a difference when nothing has been changed), the other cannot be tested.

Does an untestable hypothesis even have a place in science?

Cheers,
David.
Go to the top of the page
+Quote Post

Posts in this topic
- item   Overcoming the Perception Problem   Oct 8 2012, 19:21
- - Soap   You appear to be confused. Despite common shortha...   Oct 8 2012, 19:38
- - item   To rephrase: what, in any trial, is blind testing ...   Oct 8 2012, 19:52
|- - Soap   QUOTE (item @ Oct 8 2012, 14:52) To rephr...   Oct 8 2012, 19:59
|- - item   QUOTE (Soap @ Oct 8 2012, 19:59) QUOTE (i...   Oct 8 2012, 23:32
|- - item   Part of Beau Lotto's 'Public Perception pr...   Oct 9 2012, 00:30
||- - Porcus   QUOTE (item @ Oct 9 2012, 01:30) Part of ...   Oct 9 2012, 10:53
||- - item   QUOTE (Porcus @ Oct 9 2012, 10:53) You ca...   Oct 9 2012, 12:23
||- - Porcus   QUOTE (item @ Oct 9 2012, 13:23) This for...   Oct 9 2012, 16:04
|- - krabapple   QUOTE (item @ Oct 8 2012, 18:32) Partly, ...   Oct 9 2012, 03:51
|- - item   QUOTE (item @ Oct 8 2012, 18:32) Although...   Oct 9 2012, 11:14
- - DVDdoug   I just don't see how making a good scientific ...   Oct 9 2012, 00:59
- - greynol   @item: Perhaps you could share with us a little ab...   Oct 9 2012, 04:54
|- - Porcus   QUOTE (greynol @ Oct 9 2012, 05:54) a fai...   Oct 9 2012, 09:57
|- - item   QUOTE (greynol @ Oct 9 2012, 04:54) @item...   Oct 9 2012, 11:42
|- - 2Bdecided   QUOTE (item @ Oct 9 2012, 11:42) Positive...   Oct 9 2012, 14:15
||- - item   QUOTE (2Bdecided @ Oct 9 2012, 14:15) QUO...   Oct 9 2012, 14:37
||- - 2Bdecided   QUOTE (item @ Oct 9 2012, 14:37) Deprived...   Oct 9 2012, 17:43
|||- - item   Sorry - been away, but lots of noise (most - fasci...   Oct 19 2012, 17:34
||- - dhromed   QUOTE (item @ Oct 9 2012, 15:37) 'Kno...   Oct 9 2012, 19:11
|- - greynol   QUOTE (item @ Oct 9 2012, 03:42) QUOTE (g...   Oct 9 2012, 22:07
|- - item   QUOTE (greynol @ Oct 9 2012, 22:07) @item...   Oct 19 2012, 18:33
- - hlloyge   The only people I know of to shun DBT method of te...   Oct 9 2012, 12:35
|- - item   QUOTE (hlloyge @ Oct 9 2012, 12:35) The o...   Oct 9 2012, 13:25
|- - hlloyge   QUOTE (item @ Oct 9 2012, 14:25) The sole...   Oct 9 2012, 13:54
||- - item   QUOTE (hlloyge @ Oct 9 2012, 13:54) The o...   Oct 9 2012, 14:09
|- - Canar   QUOTE (item @ Oct 9 2012, 05:25) I think ...   Oct 9 2012, 17:09
|- - krabapple   QUOTE (item @ Oct 9 2012, 08:25) The sole...   Oct 10 2012, 01:25
|- - item   QUOTE (krabapple @ Oct 10 2012, 01:25) QU...   Oct 19 2012, 18:45
|- - Porcus   QUOTE (item @ Oct 19 2012, 19:45) QUOTE (...   Oct 19 2012, 19:22
|- - 2Bdecided   QUOTE (item @ Oct 19 2012, 18:45) Certain...   Oct 22 2012, 12:53
- - aethelberht   What I can perhaps maybe possibly gather from your...   Oct 9 2012, 13:17
|- - item   QUOTE (aethelberht @ Oct 9 2012, 13:17) W...   Oct 9 2012, 14:01
|- - aethelberht   "A negative means - equally - either a) the t...   Oct 9 2012, 14:14
|- - Arnold B. Krueger   QUOTE (item @ Oct 9 2012, 09:01) DBT is d...   Oct 9 2012, 14:53
|- - hlloyge   QUOTE (item @ Oct 9 2012, 15:01) Abstract...   Oct 10 2012, 21:04
|- - Porcus   QUOTE (hlloyge @ Oct 10 2012, 22:04) If y...   Oct 11 2012, 14:20
|- - hlloyge   QUOTE (Porcus @ Oct 11 2012, 15:20) Ehem ...   Oct 12 2012, 14:26
|- - Porcus   QUOTE (hlloyge @ Oct 12 2012, 15:26) QUOT...   Oct 12 2012, 15:04
|- - hlloyge   QUOTE (Porcus @ Oct 12 2012, 16:04) Not n...   Oct 12 2012, 22:14
|- - greynol   QUOTE (hlloyge @ Oct 12 2012, 14:14) I am...   Oct 13 2012, 14:25
|- - hlloyge   QUOTE (greynol @ Oct 13 2012, 15:25) QUOT...   Oct 14 2012, 13:35
|- - Porcus   QUOTE (hlloyge @ Oct 14 2012, 14:35) Yes,...   Oct 14 2012, 23:31
|- - hlloyge   QUOTE (Porcus @ Oct 15 2012, 00:31) QUOTE...   Oct 16 2012, 09:54
- - Porcus   While I certainly agree that putting humans in a l...   Oct 10 2012, 13:09
- - 2Bdecided   It's like The Princess and the Pea. Audiophile...   Oct 11 2012, 11:58
- - skamp   If ABXing negatively alters one's ability to h...   Oct 11 2012, 14:28
|- - krabapple   QUOTE (skamp @ Oct 11 2012, 09:28) If ABX...   Oct 11 2012, 18:24
|- - mzil   QUOTE (krabapple @ Oct 11 2012, 13:24) QU...   Oct 15 2012, 17:14
|- - krabapple   You're right that different terms apply when w...   Oct 15 2012, 20:51
- - greynol   I believe our skeptic has flown the coop.   Oct 11 2012, 16:17
- - dhromed   Not everyone is as much a netizen as most of us. P...   Oct 11 2012, 17:05
- - googlebot   While the OP's reasoning and claimed inference...   Oct 11 2012, 21:44
|- - [JAZ]   QUOTE (googlebot @ Oct 11 2012, 22:44) Do...   Oct 11 2012, 22:37
||- - bandpass   QUOTE ([JAZ] @ Oct 11 2012, 22:37...   Oct 12 2012, 06:44
|- - sld   QUOTE (googlebot @ Oct 12 2012, 04:44) Lo...   Oct 12 2012, 03:38
|- - krabapple   QUOTE (googlebot @ Oct 11 2012, 16:44) Wh...   Oct 12 2012, 06:59
||- - googlebot   QUOTE (krabapple @ Oct 12 2012, 07:59) Ex...   Oct 12 2012, 21:44
||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:44) Th...   Oct 12 2012, 21:51
|||- - googlebot   QUOTE (Nick.C @ Oct 12 2012, 22:51) Is th...   Oct 12 2012, 21:58
|||- - Nick.C   QUOTE (googlebot @ Oct 12 2012, 21:58) Ye...   Oct 12 2012, 22:21
|||- - googlebot   I do not see how calling the phenomenon "prec...   Oct 12 2012, 23:43
||- - krabapple   QUOTE (googlebot @ Oct 12 2012, 16:44) QU...   Oct 13 2012, 22:05
|- - 2Bdecided   @Porcus: QUOTE (googlebot @ Oct 11 2012, 21...   Oct 12 2012, 10:08
|- - Arnold B. Krueger   QUOTE (googlebot @ Oct 11 2012, 16:44) Co...   Oct 16 2012, 15:51
- - greynol   I would be careful not to limit the word perceive....   Oct 11 2012, 22:58
- - greynol   I extract all the joy I could ever need from simpl...   Oct 12 2012, 06:11
- - Porcus   Among my friends, we have been blind testing ... h...   Oct 12 2012, 09:30
- - dhromed   But is there a problem?   Oct 12 2012, 22:12
- - AndyH-ha   The sighted test difference is not coming from the...   Oct 13 2012, 03:32
|- - greynol   QUOTE (AndyH-ha @ Oct 12 2012, 19:32...   Oct 13 2012, 14:16
- - Nick.C   @googlebot: You are now allowing the results to be...   Oct 13 2012, 09:43
- - 2Bdecided   I think Googlebot is making a valid philosophical ...   Oct 15 2012, 12:20
|- - pisymbol   QUOTE (2Bdecided @ Oct 15 2012, 07:20) I ...   Oct 15 2012, 13:48
- - skamp   QUOTE (mzil @ Oct 15 2012, 18:14) B. The ...   Oct 15 2012, 22:49
|- - krabapple   QUOTE (skamp @ Oct 15 2012, 17:49) What g...   Oct 16 2012, 03:58
- - greynol   There are ways of cheating to get positive ABX res...   Oct 15 2012, 22:57
- - mzil   [Trying to bring this back on topic] There is not...   Oct 16 2012, 04:32
|- - Woodinville   QUOTE (mzil @ Oct 15 2012, 20:32) [Trying...   Oct 16 2012, 04:33
|- - Porcus   QUOTE (mzil @ Oct 16 2012, 05:32) Here...   Oct 16 2012, 06:19
- - mzil   Please enlighten me. I am not a scientist nor have...   Oct 16 2012, 04:39
|- - krabapple   QUOTE (mzil @ Oct 15 2012, 23:39) Please ...   Oct 16 2012, 22:28
|- - mzil   QUOTE (krabapple @ Oct 16 2012, 17:28) QU...   Oct 17 2012, 00:30
|- - saratoga   QUOTE (mzil @ Oct 16 2012, 19:30) So even...   Oct 17 2012, 03:03
||- - mzil   QUOTE (saratoga @ Oct 16 2012, 22:03) QUO...   Oct 17 2012, 05:35
||- - saratoga   QUOTE (mzil @ Oct 17 2012, 00:35) Sorry, ...   Oct 17 2012, 05:43
|- - krabapple   QUOTE (mzil @ Oct 16 2012, 19:30) QUOTE (...   Oct 17 2012, 04:53
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) QU...   Oct 17 2012, 06:36
||- - mzil   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 17 2012, 07:22
|||- - mzil   QUOTE (mzil @ Oct 17 2012, 02:22) QUOTE (...   Oct 18 2012, 19:16
||- - Arnold B. Krueger   QUOTE (krabapple @ Oct 16 2012, 23:53) Do...   Oct 18 2012, 14:04
|- - Porcus   QUOTE (mzil @ Oct 17 2012, 01:30) QUOTE R...   Oct 17 2012, 18:51
- - knutinh   Self-reporting about ones mental state surely carr...   Oct 16 2012, 21:02
- - greynol   Let me get this straight, the subconscious mind is...   Oct 17 2012, 05:28
- - mzil   Thanks, Porcus. I'll check it out.   Oct 17 2012, 20:14
- - krabapple   It's best to be careful drawing conclusions fr...   Oct 19 2012, 17:53
|- - 2Bdecided   QUOTE (krabapple @ Oct 19 2012, 17:53) It...   Oct 22 2012, 12:49
- - AndyH-ha   I could be missing the point, or dozens of them, b...   Oct 19 2012, 20:56
|- - Arnold B. Krueger   QUOTE (AndyH-ha @ Oct 19 2012, 15:56...   Oct 19 2012, 21:30
- - AndyH-ha   My point was not that many reported tests involve ...   Oct 19 2012, 22:31
2 Pages V   1 2 >


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 11:29