Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 243910 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #25
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
"We cannot win against obsession. They care, we don't. They win."

What's the problem with double-blind testing?

Reply #26
Quote
Quote
Quote
The "goosebump", emotional factor can be caused by placebo effect, you can't be sure that is not the cause. People has used same principle (emotional response and the like) to say cables sound different, for example.

You can do long-term blind tests too, not just the quick-switch, short snippet, typical abx test.
[a href="index.php?act=findpost&pid=335627"][{POST_SNAPBACK}][/a]

What do you mean by long term abx? how do I do this? Play entire songs?
[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

Well, this is a two-sided sword, because long-term DBTs are indeed theoretically no problem - but in practice, most ABX-tools aren't designed for this purpose, which makes a bit more difficult to setup.

It basically goes like this - any abx-test may take as long as you want - there is no limit on how long the timespan has to be between trials - there is no limit in how long the entire test takes - and there is no limit on the number of trials.

What this means is, that you could for example do a 6months long DBT, where you just pick one of your fav-albums, and just go your normal daily schedule.... BUT the application which you use for playback should make it invisible to you if the original or the encoded version is currently played. Then, during those 6 months, when you just normally listen to that fav-album, and think "this is the encoded version!", then you just click a button, and afterwards continue your normal daily life. Over the course of the whole 6 months, you repeat that a dozen times.... and afterwards you end up with a longterm DBT which has so many trials, that its accuracy is incredibly high.

The only thing to keep in mind in that case is that there should be no way for you during those 6 months to see the intermediate results. Another way would be to not limit the test by time, but instead define beforehand on the number of trials..... i.e. say "this test will end after 100 trials were done". If you go with the predefined number of trials approach, then you are not allowed to end the test before you complete the 100 trials, else the results will be invalid.
[a href="index.php?act=findpost&pid=335853"][{POST_SNAPBACK}][/a]


Damn! What regular consumer audio equipment supports this? There should be car decks for example that have this voting button

What's the problem with double-blind testing?

Reply #27
Quote
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[{POST_SNAPBACK}][/a]

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
[a href="index.php?act=findpost&pid=335879"][{POST_SNAPBACK}][/a]


It turns out that the science of perception is rich with results that run counter to common sense. Think how wrong common sense is when trying to understand quantum mechanics! Well, it is equally bad when trying to understand cognitive psychology and perception. To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

[a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=36467&view=findpost&p=321782]http://www.hydrogenaudio.org/forums/index....ndpost&p=321782[/url]

What's the problem with double-blind testing?

Reply #28
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?

What's the problem with double-blind testing?

Reply #29
Quote
Quote

...
The only thing to keep in mind in that case is that there should be no way for you during those 6 months to see the intermediate results. Another way would be to not limit the test by time, but instead define beforehand on the number of trials..... i.e. say "this test will end after 100 trials were done". If you go with the predefined number of trials approach, then you are not allowed to end the test before you complete the 100 trials, else the results will be invalid.
[a href="index.php?act=findpost&pid=335853"][{POST_SNAPBACK}][/a]


Damn! What regular consumer audio equipment supports this? There should be car decks for example that have this voting button :P
[a href="index.php?act=findpost&pid=335972"][{POST_SNAPBACK}][/a]

Well, yeah - i think its actually sad that most DBT-tools(be it player-plugins, hardware or dedicated apps) aren't designed with doing longterm "relaxed" tests... there are a handful of people who went through the hassle to setup 'em, but they're really a minority... :-/

Oh, to clarify about the "dont see the intermediate results" and "limit number of trials beforehand"-stuff....... this is because of the following:

If you are able to see the intermediate results AND are able to decide about when to end the test whenever you want, then you can just manipulate the test by ending it when you like the current results. Therefore, there are just two valid approaches to do an ABX:

1. You may end the test whenever you feel like it, but you may not see the intermediate results in this case.

2. You decide about the number of trials beforehand and may see the intermediate results. If you end the test before the defined number of trials were completed, then the test is invalid and must be discarded.
I am arrogant and I can afford it because I deliver.

What's the problem with double-blind testing?

Reply #30
Quote
It turns out that the science of perception is rich with results that run counter to common sense. Think how wrong common sense is when trying to understand quantum mechanics!

Not really, it is just that in the area where quantum mechanics is applicable we have no emperical knowledge/experience at all.

Quote
Well, it is equally bad when trying to understand cognitive psychology and perception. To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]

I couldn't read in those articles that our subconcious is turned off during an ABX test... in fact, it seemed to me the tests in the second article were performed under conditions ABX-test worthy.
"We cannot win against obsession. They care, we don't. They win."

What's the problem with double-blind testing?

Reply #31
Thanks for the papers, David. The first one is a good read, the second one was way over my head so I skipped that one.

But I understand Pio's question: given that humans respond differently to conscious & subconscious stimuli, how is casual listening, aka the sighted listening setting where WAVs give Donunus goosebumps, a subconscious stimulus while a DBT setting is not?

Donunus, maybe you are not getting goosebumps from mp3 because you can pass ABX tests between original vs 320. I would be more concerned if you were failing the ABX, because in that case you should get goosebumps from either format. Seems like your problem is not with DBT, but the curse of your golden ears 

 

What's the problem with double-blind testing?

Reply #32
Quote
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
[a href="index.php?act=findpost&pid=336039"][{POST_SNAPBACK}][/a]


It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be. Because of the increased effort required of perceptual processes, due to compressed information that must be resolved, listening experiences might have an uncomfortable element associated with them that uncompressed stimulus perception doesn't. An ABX paradigm might not be able to tap into that very well, if at all (because the effort is perceptual, not in a decision processs). But there are other methodologies that could measure whatever processing differences might exist.

What's the problem with double-blind testing?

Reply #33
One distinction relevant to this issue is the difference between perceptual and decision processes. Whenever you use a judgment paradigm requiring a subject to provide a verbal report, you invoke decision processes that necessitate additional processing beyond perceptual processing. In other words, in an ABX task when you ask a person to match two stimuli (presumably based on auditory quality), the person essentially relies on memory, which must be processed through a language system. There are other methodologies that attempt to tap into perceptual processes more directly, such as reaction time paradigms, or other body movements like jabbing. These methodologies often elicit data at odds with verbal report data for the same stimuli.

A good example of this is provided by research showing how early visual information gets parsed and differentially distributed to other information processing systems. One finding is called the Roelof’s effect. This is a phenomenon where visual distracters affect location judgments of objects for one system (a “cognitive” system you access by asking someone to verbally describe a location), but the distracters do not affect another system (a “motor” system where you ask subjects to point to the location). Imagine two vertical lines and a dot in between them. These stimuli appear for a couple of seconds, then disappear. The subject is then asked to identify the location either verbally or manually. Judgments of the dot’s position become biased by the relative placement of the vertical lines (e.g., if the dot is closer to the left line, then the location is judged to the right of its actual location), but this bias only happens to the cognitive system, not the motor system.

This illustrates that decision processes later in the information processing chain are often subject to top down influences, or bottom up deficits, that other systems, like motor systems, are not. Point being, decision processes rely on particular pre-processed information, and are often not a reliable source for determining differences in perceptual processing, whereas motor systems often are.

In terms of audio, this could mean that DBT might fail to distinguish auditory stimuli that in fact do have relevant differences in their acoustic properties. For example, perhaps processing auditory stimuli with degraded resolution might be more difficult, and result in subtle fatigue or other effects that aren’t reflected in verbal reports, but might be revealed in reaction time experiments. There are principled reasons why different systems have access to different information, and the details of this are obviously beyond the scope of the discussion. However, perceptual processes involved in the judgment of sound quality are not designed to be sensitive to fine distinctions between different digitized stimuli, so such stimuli might not affect subjective perception. That is, whatever distinctions that can be made will be byproducts of auditory system design features that might or might not be affected by digitization manipulations. But this doesn’t rule out other possible effects on processing that compressed stimuli might have on a listening experience.

The overall message from cognitive science is that our conscious notions are quite often deceiving, and sometimes just outright made up. So while DBT is methodologically sound, its implementation in perception research assumes incorrectly that subjective experience has access to the processes underlying it. That said, I am not arguing this perspective to justify bizarre audiophile notions, instead David and I are proposing that some of the elusive effects described by audiophiles might be based in reality. It’s a testable idea that is on the agenda, by the way.

What's the problem with double-blind testing?

Reply #34
Quote
It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be.

Might be... or might be not. You can't conclude or even suggest that based on the articles mentioned above. They only prove that subconcious stimuli can influence our behaviour. But when and where that happens is not mentioned. There is no reason to assume that we are more sensitive to subconsious stimuli during casual listening than during an ABX test.

Quote
Because of the increased effort required of perceptual processes, due to compressed information that must be resolved,

Is that so? Does listening to MP3 require more effort than listening to PCM?
"We cannot win against obsession. They care, we don't. They win."

What's the problem with double-blind testing?

Reply #35
Quote
Might be... or might be not. You can't conclude or even suggest that based on the articles mentioned above. They only prove that subconcious stimuli can influence our behaviour. But when and where that happens is not mentioned. There is no reason to assume that we are more sensitive to subconsious stimuli during casual listening than during an ABX test.


I'm not really referring to implications of just those articles. But the articles are quite specific about when and where unconscious information affects our behavior. In fact, there are five experiments in that second paper that show quite clearly how phase correction occurred in synchronized motor movements as a function of unconsciously perceived elements in an auditory stimulus. Anyway, as I described in my last post, the idea is that the ABX task relies on processing that has no direct access to the relevant parameters that are manipulated in some compression schemes.

Quote
Is that so? Does listening to MP3 require more effort than listening to PCM?


I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.

You might be confusing conscious effort from processing effort (the latter of which you have no conscious access to, and might only be revealed through implicit measurements like reaction time data to auditory stimuli).

What's the problem with double-blind testing?

Reply #36
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.
[a href="index.php?act=findpost&pid=336092"][{POST_SNAPBACK}][/a]

Our ear doesn't work that way. Our auditory processes don't derive any kind of waveform. They are more like a realtime frequency analyzer, from which only part of the data is sent to the brain. Lossy compression just tries to discard that information that is not audible, just that.

As to the process of having to take a decision being different from just listening, well, that should apply too to sighted listening tests, don't you think so?

Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.

What's the problem with double-blind testing?

Reply #37
Quote
Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.[a href="index.php?act=findpost&pid=336097"][{POST_SNAPBACK}][/a]


Right, more sensitive to certain artifacts, but maybe less sensitive to others?

What's the problem with double-blind testing?

Reply #38
Quote
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems. In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.
[a href="index.php?act=findpost&pid=336092"][{POST_SNAPBACK}][/a]

Our ear doesn't work that way. Our auditory processes don't derive any kind of waveform. They are more like a realtime frequency analyzer, from which only part of the data is sent to the brain. Lossy compression just tries to discard that information that is not audible, just that.

As to the process of having to take a decision being different from just listening, well, that should apply too to sighted listening tests, don't you think so?

Also, psychoacoustic research already found that quick switch, short length stimuli, blind  listening tests have proved to be more much more sensitive that casual listening.
[a href="index.php?act=findpost&pid=336097"][{POST_SNAPBACK}][/a]


The auditory system must derive, from a waveform, the characteristics that are relevant to any number of functions. In speech and music, the generated representation has all the characteristics of a waveform. From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform. Lossy compression tries to discard information that is not audible, but that doesn’t speak to the acoustic deficits in the original signal that the system must handle.

Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept. For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing). But the percept is there because the system can make an unconscious inference using limited information. Of course to the listener, it just seems like there’s a pitch. Degraded signals require more effort to process. I think that is a rather uncontroversial point.

I'm not sure what you mean by a sighted listening test?

What's the problem with double-blind testing?

Reply #39
Quote
I'm not sure what you mean by a sighted listening test?[a href="index.php?act=findpost&pid=336106"][{POST_SNAPBACK}][/a]


The opposite of "blind", I suppose. I.E, a test where placebo can take effect.

What's the problem with double-blind testing?

Reply #40
Quote
Quote
actually can you give me a link that explains placebo. thx[a href="index.php?act=findpost&pid=335628"][{POST_SNAPBACK}][/a]

The word placebo, when used in audio circles, is a little different from the medical term, where people can actually benefit a little from dummy treatments.

According to ff123, it's closer to expectation bias, where a person's expectation can influence his perception, if I understand correctly.
[a href="index.php?act=findpost&pid=335725"][{POST_SNAPBACK}][/a]


While I don't disagree, I'd go farther.

If you look at what the auditory system detects as an "event", it appears from the work done to be set to detect many 'false events' at a much lower risk of missing a real event.

This kind of performance (tradeoff between misses and false detections) is obligatory for any probabilistic system, of which the auditory system is quite.

I have heard this explained to others (somebody said this at a tutorial at the NY aes, for instance) as being for a simple reason:

If you hear the tiger coming, and hide, you live.

If you don't hear the tiger coming, you are out of the gene pool.

If you hear the tiger coming, and it isn't, all you do is look a bit silly.

If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause  "differences".
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #41
Quote
The opposite of "blind", I suppose. I.E, a test where placebo can take effect.


Haha. Of course. Audiophile jargon. 

Well, the distinction between decision and perception applies regardless of the presentation conditions, but I don't see any reason to have any other ABX paradigm besides a double-blind design.

What's the problem with double-blind testing?

Reply #42
Quote
If you consider that we are wired to detect nonevents in favor of missing any events, then that would explain quite nicely why changes cause "differences".


One has to be careful about generalizing this principle across all domains. Signal detection applies to many perceptual phenomena, but the costs and benefits associated with the four possible outcomes (hit, miss, false alarm, correct rejection) vary depending on what you are attempting to detect. So we aren't wired to detect "non-events" over events in general. For predator detection, yes...but not for something like, for example, detecting cheaters in a card game!

What's the problem with double-blind testing?

Reply #43
Quote
Quote
Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]

What you are saying is contradictional. You say that when you are paying close attention to the music you hear less difference than when you do casual listening. That is very odd and seems to be more related to expectation.
[a href="index.php?act=findpost&pid=335879"][{POST_SNAPBACK}][/a]

What I'm saying is not contradictory. All I'm saying is one cannot abx enjoyment because the enjoyment factor can only be realized when listening to the music casually. How can one enjoy the music as a whole when abx testing when youre busy listening to different aspects of the sound of a clip. Its a clinical procedure, not an emotional one.

I do not mean that  I "hear" less of a difference. What I mean is that the differences that I hear during analytical mode don't mean as much to me at that moment compared to when I listen to the music for enjoyment. When enjoying the music sometimes I think that I am tired of the song that I am listening to or feel bored but get more emotional or even cry when listening to the original. The analogy is exactly like listening to music in a different mental state. try listening when you are drunk and sober. The experience is different. Just like listening when you are analyzing isnt as  emotional as listening deeply to the soul of the music. At the time you are listening emotionally, you are not pinpointing the artifacts. The music speaks for itself. Sort of like placebo but this time it is not expectation bias, Its just the way it is whether or not I know which source cd is playing. To ramble even more, Its just like listening to music with different stereo systems. one is a more emotional experience, one is dull and boring. I hope you got what I meant. Its audiophile hocus pocus at work

What's the problem with double-blind testing?

Reply #44
Quote
Quote
Quote
To tell you the truth, I cannot understand how someone could read and understand the two papers I linked to in the following thread and still be convinced that DBT can be the final word on music reproduction. In any event, it is a very interesting field of science that I suspect is not well known to many HA members:

http://www.hydrogenaudio.org/forums/index....ndpost&p=321782
[a href="index.php?act=findpost&pid=335992"][{POST_SNAPBACK}][/a]


I have not read them. It seems that they deal with the effect of unconcious stimuli. Your point would be that sounds unconciously perceived might affect our perception of music.

But I don't see how this could explain the disparition of all "audiophile effects" under blind listening conditions. The unconcious stimuli don't disappear when the test is blind, so why should their effects disappear ?
[a href="index.php?act=findpost&pid=336039"][{POST_SNAPBACK}][/a]


It's not that any information disappears. The problem is that the ABX paradigm relies on cognitive processes that might not be directly affected by subtle differences between two presented stimuli, but other aspects of the listening experience might be. Because of the increased effort required of perceptual processes, due to compressed information that must be resolved, listening experiences might have an uncomfortable element associated with them that uncompressed stimulus perception doesn't. An ABX paradigm might not be able to tap into that very well, if at all (because the effort is perceptual, not in a decision processs). But there are other methodologies that could measure whatever processing differences might exist.
[a href="index.php?act=findpost&pid=336072"][{POST_SNAPBACK}][/a]



Might .  Could.  If you suppose X.

How about, when evaluating the endless number of claims of difference between audio,  considering what we *know* DOES happen when people hear two things they think are different  : various forms of 'unconscious' bias that can and DO lead to such absurdities as subjectively perceiving the same sound the be different.  Of *course* unconscious stimuli affect our behaviour -- particularly our *beliefs* and our *claims* about what we hear and why.  These 'unconscious' stimuli are things like: stuff we've heard or read about the gear, their brands, their cost, their appearance.
They easily lead us to believe, and claim, that A sounds much different (better/worse) than B -- even if A and B are actually the same

This really happens.  It's not suppositional. 

Guy buys a new CD player.  Oooh, ahhh, so much better than the previous one!  But is it?  How does he know?  Well, he paid more for it.  Or it does 'upsampling'.  Or he read that it's better.  Gosh darn it, it *SOUNDS* better to him.  That;s proof enough.

Was this guy listening in 'uncompressed' fashion?  Where's the evidence that this overrides the nuisance factors that are ALWAYS present when listening 'sighted'?y

What's the problem with double-blind testing?

Reply #45
Quote
The auditory system must derive, from a waveform, the characteristics that are relevant to any number of functions. In speech and music, the generated representation has all the characteristics of a waveform.


All sounds are waveforms.

Quote
From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform. Lossy compression tries to discard information that is not audible, but that doesn’t speak to the acoustic deficits in the original signal that the system must handle.


Its still a waveform, so whats the difference.  Its not like you're reading the samples.  From your ears perspective its all just sound.


Quote
Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept. For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing). But the percept is there because the system can make an unconscious inference using limited information. Of course to the listener, it just seems like there’s a pitch. Degraded signals require more effort to process. I think that is a rather uncontroversial point.


More effort to recover meaning from a distorted signal, sure.  But to say that processed audio is intrinsically harder to percieve is something entirely different.  What exactly are you basing this conclusion on?  I find it extremely unlikely that a signal that has been so marginally degraded that you cannot detect it via ABX would pose such a problem for one's senses as to consistantly fool a listener.

What's the problem with double-blind testing?

Reply #46
I think it's pretty clear that double-blind test cannot "disprove" anything--they can only disprove something beyond reasonable doubt and, more importantly, there's really no better option out there.  They can prove the presence of something, at least up to the point at which one begins to doubt whether the test was performed correctly. 

The very fact that one is taking a double-blind test is going to affect one's perception.  Whether that difference is significant enough to make something audible inaudible (or something inaudible audible) isn't really knowable, as far as I know.  The mere fact that you are scrutinizing your perception of the music is going to affect your findings.  Because of this, there's no way that you can prove that something that causes a very minor change in the sound waves hitting your ear doesn't have any audible effect.  You can prove, via a double-blind test, that someone trying to hear the difference cannot, which for me, at any rate, is proof enough that any change in the sound is negligible.

As for mp3-decoded sound waves causing your brain to work harder to interpret the signal:  it's certainly possible that an mp3-decoded sound wave would require a different amount of effort to interpret on your brain's part, without you knowing it.  It's not reasonable to conclude, however, that an mp3-decoded sound would be harder; it could as well be easier for your brain to interpret.  That said, the only way to disprove this would be to monitor brain activity.  Proving it, on the other hand, might be possible through some carefully controlled double-blind tests (polling tiredness after extensive sample-listening).  Whatever the difference is, if there is any, is going to be pretty damn subtle.

What's the problem with double-blind testing?

Reply #47
Quote
From the stimulus, the auditory cortex represents spectral and temporal features, so for all intents and purposes, this is a waveform.

OK, for a wider meaning of "waveform" than the one I was thinking of.

Quote
Take for instance your pitch perception system. It generates a pitch percept based on information in a waveform. The system sends a percept to your conscious awareness, but the effort required to derive that information might not be revealed in the percept.

You are assuming that it takes some kind of "effort" to "derive" such information, as if our auditory system had to struggle just to perceive sound. I don't know if that is true. Even if it was to some extent, which I guess can be... see my next replies.

Quote
For example, there is a phenomenon called the ‘missing fundamental’ where people perceive pitch (fundamental frequency, or F0) in a high band pass filtered segment that does not contain enough information to calculate a proper F0 through Fourier analysis (e.g., F1-F4 is missing).

True, and also true that this seems happen due to brain processing, not the ear.

Quote
But the percept is there because the system can make an unconscious inference using limited information.

I guess it can be called unconscious, because we have no conscience that is happening. But it can also be an automatic process that requires no extra effort, it's just the way the brain processes all sound.

Quote
Degraded signals require more effort to process. I think that is a rather uncontroversial point.

I'm no expert, but can't see why it's uncontroversial. Is there any actual evidence that supports this? Edit: I mean, in case of degraded signals where degradation is not audible. Because in case of missing fundamental, the difference is clearly audible.

Quote
I'm not sure what you mean by a sighted listening test?
[a href="index.php?act=findpost&pid=336106"][{POST_SNAPBACK}][/a]

The usual evaluation of audio equipment people do at home, not blind. The one with which many ones are able to perceive such great differences.

As Jun-Dai said, even if there were some problems with blind testing, it's still the least bad method.

What's the problem with double-blind testing?

Reply #48
For helping doing long-term, casual listening-like listening tests, it can be useful the utility fileABX, that makes "blind" copies of .wav test files. Those copies can be burn to CD, or copied to a DAP, in order to perform such a test.

It's also possible to generate copies of different sets of test files, and use all the sets to perfom the test, in order to avoid listening over just one pair of test samples. As long as the identification of each file of a set of files is unknown, and a proper statistical analysis of the final result of the listening evaluation is done, it will be a reliable blind test.

What's the problem with double-blind testing?

Reply #49
Quote
I think it goes without saying that compressed audio information is more difficult to resolve for auditory perceptual systems.

Present me good articles that describe this. I'm not interested in dogmas.

Quote
In the case of a 128kbps mp3, your auditory processes must derive a waveform from a signal that has 90% less information than its 16bit/44.1 counterpart.

Sorry, this is not even true. I would like you to show me how you calculate that an 128 kbps mp3 has 90% less information (define information too please) than the original PCM data. Then show me that our auditory system must derive more from a decoded mp3 than from the original PCM AND explain exactly what it is that is being derived. If anything, it seems to me the opposite must be true and working with less data is also is less fatiguing

Quote
You might be confusing conscious effort from processing effort (the latter of which you have no conscious access to, and might only be revealed through implicit measurements like reaction time data to auditory stimuli).

And I think you are making too many assumptions and speculations that you just can't reasonably back up right now.
"We cannot win against obsession. They care, we don't. They win."