Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: What's the problem with double-blind testing? (Read 243392 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

What's the problem with double-blind testing?

Reply #75
Quote
The reduction of resolution and introduction of noise into a signal, to my mind, qualifies as a degradation. That's what lossy encoders do. I use this in a technical sense, so I don't mean to disparage any lossy format with "pejorative" intentions! Bad mp3 file! Bad!

I know you dont mean that, but you are placing the change in sound in an troublesome context with the terms, when the change is virtualy sculpted as best it can to result in no perceptible troubles. I think it is only an added noise in one abstract sense,  fundamentaly its just a change.

When we listen to a difference track, we hear a jumbled up compound of lots of individual differences, and it sounds bad, but we are not percieving them in context, in their context they were (at least consciously) imperceptable. In their context they are ratios, they are the details of a relationship between one sound and another. Relating to them as the medium which they reference is misleading. 
no conscience > no custom

 

What's the problem with double-blind testing?

Reply #76
Quote
So, if we're adding noise to a signal that someone is trying to interpret, it's pretty obvious that it's going to require more processing. Do you find it easier to understand speech in a crowded bar or a quiet room? This is why they put up acoustic panels in auditoriums; too much reverberation acts like noise and makes speech more difficult to understand. This isn't really debatable.

Well, actually, it is.

First, reverberation is way, way above threshold.

A decent perceptual coder (AAC, WMA Pro for instance) is very near, or under, threshold at a reasonable rate.

The two issues are not in any fashion comparable. If there is no way to detect the difference in two signals coming down the auditory nerve, there is noting for the CNS to work on.

Quote
BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.
[a href="index.php?act=findpost&pid=336381"][{POST_SNAPBACK}][/a]


No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #77
Quote
When I said we aren't "wired" to detect "non-events" in general, I was referring to any phenomena that signal detection theory can be applied. This is a fundamental tenet of the theory; that is, the criterion location depends on the associated costs of different sorts of errors. How that manifests in various auditory contexts is quite variable depending what the task is. Also, it relates to attention and what the sound source is. 

And I am talking about what the auditory system actually seems to do.

Yes, you can change detection thresholds, in theory, but you know what?  The auditory system at its basic level, i.e. partial loudness, seems to clearly overdetect.
Quote
What I am proposing is a result of processing in the auditory cortex, not the ears, and is consistent with what we know about auditory perception. Your account of partial loudnesses doesn't really seem particularly relevant to the issue of whether processing differences between compressed and uncompressed audio could be measured in a way that ABX testing misses.

You argue that there must be a way to filter out overdetections (that always happen in the partial loudness domain) upstream. This is a de-noising question. While I can't prove it does not happen, my experiences (extensive) with psychometric testing is that auditory overdetection is the rule, not the exception.

Given any straw to grab, even null tests (i.e. same stimulus) will show "results". It's been done. Over and over.
Quote
My example of the missing fundamental phenomenon illustrates nicely how unconscious inference works.

It is irrelvant and shows nothing relating to what we are talking about. The partials are way, way above threshold.  A good perceptual coder does not inject noise at that level, period.

They are two entirely different scenarios, perceptual coding and phantom pitch.
Quote
There are many other examples as well. Our brains construct a good deal of what we "perceive" based on what is often quite degraded information. For example, your visual experience is radically different than the information on your retina. Perception has a huge filling in component.

And, again, you talk about information way, way above threshold.
Quote
The threshold theory is just another version of the idea discussed earlier concerning why it is more labor intensive, so to speak, to perceive a degraded stimulus rather than a less noisy stimulus. David's example of listening to speech in a noisy bar is a good example. But I will throw up some references soon about neural processing and stimulus quality. As one should expect, degraded signals are more difficult to process.
[a href="index.php?act=findpost&pid=336391"][{POST_SNAPBACK}][/a]


Before you argue that, explain how something for which the information to enable detection at all (the SNR and integration constants of the auditory system are, after all, pretty well known from phenominon) does not exist can make a difference.

In a noisy bar, the noise, again, is far, far above threshold.

Every one of your examples is using noise that is far, far above threshold. If your coded signals fit that bill, please get a new encoder.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #78
Quote
Filtering and unconscious inference processes take place in the brain on all sinusoidal waves (light and sound). These processes are taxed by noise.

Whoa there.  "Filtering" takes place in the organ of Corti, not in the brain, in the sense of frequency filtering like would be involved for a sine wave.

The eye is not a frequency analyzer, all frequency analysis in the eye is secondary in the brain.

The ear is, fundamentally, a frequency analyzer at the periphery. There's no point in going further with eye/ear comparisons, they aren't the same.
Quote
Frankly, I find it surprising that I need to convince people that degraded signals require more effort to process.

May I politely suggest, then, that you reconsider your idea?

A signal below threshold in the auditory system is GONE.

Noise below threshold is GONE.

AT least 80 years of research shows that it's gone in the PERIPHERY, in the organ of Corti, in the basic transduction.

Quote
This is not the speculative element of the idea. But I understand that our use of the term "effort" is not one people ordinarily consider. I will provide examples of research demonstrating this soon.
[a href="index.php?act=findpost&pid=336405"][{POST_SNAPBACK}][/a]


Your idea is unsupported and beyond speculative.

You will have to show that frequency analysis and information loss takes place after the cochlea if you wish to proceed successfullly.

Greenwood, Zwislocki, Fletcher, Zwicker, etc, all conclude (and I think accurately so) that this happens on the basilar membrane, and that things that are below the basic 30dB SNR of a given neuron in the auditory nerve are gone.

This is supported by a host of evidence, from BLMD to high-frequency unmasking on envelopes, to tone masking noise in ERB's ...

You're pushing way upstream here, and your suggestions would require a complete revamping of both the understanding and, frankly, the evidence of how the human auditory system works.

Again, for signals ABOVE threshold, especially quite above, I agree that you're right. I think, however, that you don't quite grasp what "under threshold" can mean in the auditory system.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #79
Quote
A simplistic example - a sine wave of exactly 3122.4873 Hz might be encoded as a sine of 3121 Hz.
[a href="index.php?act=findpost&pid=336409"][{POST_SNAPBACK}][/a]


Um, no, that's not what happens. The short-term frequency may be varied due to noise (phase noise, FM noise, whatever), but that's a different issue.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #80
Quote
Quote
A simplistic example - a sine wave of exactly 3122.4873 Hz might be encoded as a sine of 3121 Hz.

Um, no, that's not what happens. The short-term frequency may be varied due to noise (phase noise, FM noise, whatever), but that's a different issue.

The key phrase was 'simplistic example'
(I want duff to understand the 'noise' he is refering to, not run him out of town  )
I know that wasnt your intention woodnville, but..

..wait till we get the links

Lossy digital processes affect the waveform at least with digital rounding error of phase resolutions / frequencies & powers.
The difference which rounding error makes is a very tiny noise, containing only hazy information about the bulk density of the position over the waveform, or not even that perhaps.
Thats a noise that if it was loud would be very noisy*
The 'noise' of psychoacoustic 'info reduction' -attempts to be (or , mostly is) generated in sync with the major details of the source.
On top of the noisy syncronisationInfo loss which is minimised by encoders,
much stronger info losses can be applied to power - a reason for this could be that power signatures can change without causing as big kinks in the waveform as phase/freq signatures
And there are linear charts of masking&pinpointing perception levels used to hack into the power info even more.
Those economisations do not interfere with the waveform, they fit into, when they are removed the reduced waveform carries no details of the economisation any more.
They leave no discernable pattern on the waveform other than the very quiet rounding error chaos.
That would be the ideal, but it might not be a securely achieved idea by some formats & encoders.
no conscience > no custom

What's the problem with double-blind testing?

Reply #81
Quote
Quote

BTW, I'll clear up one minor error Greg made in his information comparison of a 128 kbps MP3 and the original wav. To compare the amount of real information stored you have to compare the size of the MP3 with a losslessly compressed wav, so it's really more like throwing away 80% of the information instead of 90%. This is more correct from an information theory standpoint.
[a href="index.php?act=findpost&pid=336381"][{POST_SNAPBACK}][/a]


No that's wrong, too.

With a decent lossy compressor, you get about 50% compression. So your 90% would become 45%, even with the odd metric you're using.

A better way is to point out that lossless compression gets you to 50%. THAT is "no loss at all" in the real signal. Ergo, something that runs at 88% is throwing away 38%.
[a href="index.php?act=findpost&pid=336466"][{POST_SNAPBACK}][/a]

Sorry, you're the one who's wrong.

The only relevant measure of information here is the losslessly compressed wav. If we assume, like you did, 50% compression, then the bitrate would be 705 kbps (1411 kbps / 2). To get that down to 128 kbps requires eliminating 82%.

As for your other points, you may be taking the examples a little too specifically. If we knew exactly what effect was going to be discovered we would design an experiment and get it over with. The point I am trying to make is much more general, and is simply that listening tests involving only conscious subject choice may not be sufficient to measure all aspects of auditory perception, and that those areas it cannot properly measure may be involved with music enjoyment. If you think that's a preposterous stretch, good for you.

What's the problem with double-blind testing?

Reply #82
Quote
I have this weird sensation sometimes when Im ~half asleep, where I feel sharp sounds from unexpected events like clangs and ticks as a wave of light travelling through my body. Ive wondered if that could be tranquil nerves in the body feeling the sound as it passes through them, or if it is just a kind of hallucination. Perception and conciousness are surely still bewildering and mysterious things.
[a href="index.php?act=findpost&pid=336402"][{POST_SNAPBACK}][/a]

I have these too! Sometimes I'll wake up and be certain that the doorbell just rang, and sometimes I feel like something popped inside my head. Very unpleasant! Fortunately these sensations don't happen when I'm fully awake... 

What's the problem with double-blind testing?

Reply #83
just thinking -i do get all sorts of weird sensations when im half awake....
wont go there now,//
no conscience > no custom

What's the problem with double-blind testing?

Reply #84
Quote
The only relevant measure of information here is the losslessly compressed wav. If we assume, like you did, 50% compression, then the bitrate would be 705 kbps (1411 kbps / 2). To get that down to 128 kbps requires eliminating 82%.[a href="index.php?act=findpost&pid=336612"][{POST_SNAPBACK}][/a]



Well, I think first you need to define what you're measuring.

Now, as to listening to differences, are you aware that DBT's with forced-choice and short stimulii have shown listener sensitivity down to the levels mathematics and science just barely predict are possible?

Look at the sensitivity of the hearing system at absolute threshold, and compare it to the ambient noise due to shot noise of air, for instance. The absolute thresholds are gotten via DBT's.

Look at the in-band sensitivity to tone-masking-noise, compare it to the resolution of a nerve fiber (considering firing rate and the 200 millisecond integration time of the auditory system).
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #85
So sorry for the long delay. I didn't forget about this thread....

I have compiled a number of articles that provide anybody here resources and empirical support for the various claims David and I have been making. I claimed that there is a distinction between sensory and decision processes. This recent Nature article provides empirical support for this. Cognitive scientists have understood this distinction for decades, but here is recent evidence demonstrating the neurological basis for it. This is a fundamental issue in our criticism of the validity of ABX testing as the last word on ultimate differences in the auditory processing of lossy audio. Acoustic differences that could matter for a listener’s overall experience might not be audible.

Here are two articles that show the difference between decision processes linked to the discrimination of stimuli versus sensory processes that are affected when there is no discrimination. There is clear enhanced brain activity that occurs during the presentation of noisier stimuli even when it cannot be distinguished from a less noisy counterpart.

Effects of Low-Pass Noise Masking on Auditory Event-Related Potentials to Speech
The Effects of Decreased Audibility Produced by High-Pass Noise

Woodinside claimed that no filtering processes happen after the ear. This is definitely incorrect, as filtering processes do happen in the auditory cortex (one of the many analogues between auditory and visual processing), but I’ll assume he meant that the relevant filtering for audio codecs involves processes that happen only in the ear. This is fine, as there’s no reason to distinguish peripheral (inner and outer ear) processes from central processes (auditory cortex) when talking about metabolic costs of neural effort. The neurons must work harder (wherever this happens) to resolve the signal and encode it for later processing. And this does not deal with the likely possibility that the resulting representations might differ depending on stimulus quality (not a necessary feature of our argument).

There has been quite a bit of skepticism regarding the claim that compressed or degraded stimuli require more effort to process. I could cite a list of papers a mile long showing reaction time increases as a function of stimulus complexity (which heavily imply that processing difficulty increases), but there is also work showing that neurons respond to increased attention demands. Additionally, there seems to be specialized neural systems for resolving degraded signals, which in turn contribute to learning. These processes require effort, and have associated metabolic costs. This is exactly the sort of thing we have been arguing.


What's the problem with double-blind testing?

Reply #86
Quote
Woodinside claimed that no filtering processes happen after the ear. This is definitely incorrect, as filtering processes do happen in the auditory cortex (one of the many analogues between auditory and visual processing), but I’ll assume he meant that the relevant filtering for audio codecs involves processes that happen only in the ear. This is fine, as there’s no reason to distinguish peripheral (inner and outer ear) processes from central processes (auditory cortex) when talking about metabolic costs of neural effort. The neurons must work harder (wherever this happens) to resolve the signal and encode it for later processing. And this does not deal with the likely possibility that the resulting representations might differ depending on stimulus quality (not a necessary feature of our argument).[a href="index.php?act=findpost&pid=340974"][{POST_SNAPBACK}][/a]


I'm sorry, but you're simply misusing the term "filtering" from the point of view of a signal-processing person. You are not, I trust, suggesting that the "filtering" that goes on in the CNS results in anything like the outcome of a linear filter, are you?

Psychologists often take words with specific meanings and add new meanings. Personally, I wish they'd use a different word, but in any case, the spatial capture is not a filter in any sense that I would normally use the word. I might use a filter to do spatial capture, that's a different problem altogether.

What you are, once again, ignoring, is that if there is NO INFORMATION AVAILABLE about a given distortion at the auditory nerve, you're arguing for something paranormal to happen later.

The evidence is in for outer hair cells, they depolarize at higher levels. So, there is no more effort going on in the basilar membrane, either.  So, missing information means "no stimulus", ergo if anything it's "less effort" in the periphery.  I think it's safe to say that the "cochlear amplifier" mechanism has gone FORD. (That's "found on road dead" for non-'mercans)

There is simply no evidence, no mechanism, and no non-sighted listener responses that in any fashion support this contention of "extra effort". None.  That's what this all boils down to. One one side we see a bunch of suppositions, hypotheses, and speculation, based on the idea that there must be something wrong with DBT's because they don't observe some things that people think exist.

You show two sets of evidence for noise WELL OVER THRESHOLD. You're using data from a completely different, and irrelevant, situation in order to make your argument.

On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.

That is really what I see on the table here, supposition vs. a century of evidence. If there is something here, some extraordinary evidence for its existance, that addresses the known flaws of human perception, has to come forth.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #87
I don't understand why you would want to maintain a definition of filtering that does not include the action of neurons that differentially respond to frequency-time information in an encoded signal. This is hardly unusual, and psychophysical researchers have been using "filter" in this way for over a century.

[span style='font-size:8pt;line-height:100%']Kowalski, N., Depireux,D. A. & Shamma, S. A. (1996). Analysis of dynamic spectra in ferret primary auditory cortex. II. Prediction of unit responses to arbitrary dynamic spectra. J. Neurophysiol. 76, 3524±3534.[/span]

The reason I pointed that out at all is because you made a blanket statement about the auditory cortex and filtering. Just as you made a blanket statement about privileging "non events" in signal detection. You later said you only meant that in a restricted way, but you originally said something much more general. Anyway, onwards...

A paragraph from here speaks at least indirectly to an important issue:

[span style='font-size:8pt;line-height:100%']Measurements carried out in living anesthetized lab animals revealed that the waves of maximal displacement produced by moderate sound levels in the living cochlea are actually much, much sharper than can be accounted for by the physical resonant properties of the basilar membrane in cadavers. That work led to an objective method for screening for hearing disorders in newborns that has become widely adopted in the past 5 years. It turns out that cochlear hair cells can be both receptors and effectors. In response to signals from their efferent innervation outer hair cells exhibit slow contractions and elongations. In addition, outer hair cells exhibit fast contractions in response to sound. Those contractions occur when the cells are depolarized and hyperpolarized.  Currently, the molecular mechanisms that underlie this motility are under intense investigation. Two years ago the apparent motor protein that powers outer hair cell motility was identified and named prestin. Physiological experiments have established that the local resonance of the basilar membrane is directly modulated as a result of active changes in the length of outer hair cells, but just how that results in highly-tuned mammalian hearing performance is not yet understood.  [/span]

Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal. But auditory processing doesn't work that way. I'd like to see any empirical support for your argument surrounding effort and noise even at that level. But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The articles I provided show that even when discrimination cannot be made between two stimuli that differ in how much noise they contain, neural systems are treating them differently. That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes. All we have to show to make our basic case, like the articles I presented do show, is that people can be unable to discriminate between two stimuli (rendering them equivalent in ABX terms), while their brains act differently during the presentation of them. The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

Quote
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.

What's the problem with double-blind testing?

Reply #88
Sorry, you still have provided no evidence that what you say applies to inaudible differences. So, you have no basis to say that an ABX test (which is just about audible or inaudible differences) may not be adequate because of these reasons.

Related to this, I believe that very subtle audible differences have been detected in DBTs even when the subjects at test couldn't consciously hear a clear difference.

What's the problem with double-blind testing?

Reply #89
Let's back up a moment...

There are at least two different arguments going on here. One is whether there is any evidence that degraded stimuli, or lossy compressed stimuli require greater effort to resolve by perceptual systems. I think I have provided evidence that is true. There is no principled reason why this would not apply to stimuli that we are not consciously aware of (e.g., inaudible). Earlier I discussed examples in vision where reaction times were longer in a reading task on a computer screen as a function of flicker rate, but subjects had no conscious awareness the stimuli were different. This is a perfect example of increased effort affecting processing that is not consciously accessible.

The other point relates to whether aspects of stimuli that we cannot see or hear affect our perception, and this is obviously true. There is a huge literature on the attentional blink, there are studies like the flicker rate study I just mentioned, as well as various other phenomena where subjects do not consciously notice some stimuli, but the perception of that stimuli is revealed by paradigms that tap into unconscious processes. For example, people look at a cross-hatch and are asked to judge whether the vertical line is longer than the horizontal line, and a small shape appears very close to the cross-hatch. Subjects claim they do not see the shape but in a yes/no recognition task, they take longer to say no to shapes they saw, than shapes they didn't see.

And the list goes on. Point being, conscious reports (which ABX tests rely on) are not totally reliable indicators of what affects perceptual systems.

What's the problem with double-blind testing?

Reply #90
About the first point, your degraded examples are *hugely* degraded, as opposed to what average quality lossy compression does. Then you say that there is no reason why the phenomena you describe would not apply to inaudible difference stimuli. Well, if it's inaudible, there's a great difference. I think it is too much to assume that it would apply equally, and I think I'm not being over-skeptical here.

About the second point, would those visual "degradations" (not consciously perceived) be perceived as some kind of difference when the subject is actually trying to see them, or when subjected to an A/B test? I mean, is it that those differences are not consciously perceived, but just under casual "looking", or without a reference? For example, subtle audible differences are sometimes not perceived when doing casual listening, but when paying attention, or in a close comparison with a reference (A/B test) they are easily distinguishable.

What's the problem with double-blind testing?

Reply #91
Quote
Well, if it's inaudible, there's a great difference


This represents a general feeling many people have, but in fact, I'm not so sure. In terms of information processing, there's no reason to invoke different principles to describe the relationship between stimulus characteristics and processing effort. The fact that it's "inaudible" in an ABX sort of way just means that the percept is not processed by a system that is consciously accessible. I can't stress enough how many processes happen that are NOT conscious.

Quote
About the second point, would those visual "degradations" (not consciously perceived) be perceived as some kind of difference when the subject is actually trying to see them, or when subjected to an A/B test? I mean, is it that those differences are not consciously perceived, but just under casual "looking", or without a reference? For example, subtle audible differences are sometimes not perceived when doing casual listening, but when paying attention, or in a close comparison with a reference (A/B test) they are easily distinguishable.


In phenomena like the attentional blink, attention has no impact on one's percept. Same with the flicker rate study. People cannot tell that the flicker rates are different, even when you draw their attention to it. The reason their reaction times are different is because certain relationships between their eye movement speeds and the flicker rates can cause people to repeat scans over text in extremely small ways (we're talking eye movements here, so these are tiny distances). There is no feeling associated with this other than rather long-term fatigue.

In other paradigms, like the cross-hatch/shape studies, of course when you direct someone's attention, they see the shape. The point of that work is to show that items not consciously perceived are actually processed. So, as I keep repeating, aspects of compressed stimuli might affect processing, and conscious reports might not reflect it.

Look, this is a testable hypothesis, and is not unreasonable given what we know. I keep trying to explain it because it's clear many people here are not very familiar with research in psychophysics and cognition. It is hardly a radical proposal that something like an ABX test does not capture all aspects of perceptual processing...in fact, it's a given. Whether or not this applies to lossy audio remains to be seen.

What's the problem with double-blind testing?

Reply #92
Quote
Somehow, you seem to suggest that most of what we perceive is "for free" after basilar membrane hair cells encode the signal.

I propose nothing whatsoever of that sort. There is lots of processing. My point is simple, if there is no detectable remains of an original signal at the auditory nerve, then nothing the cortex does can recover what isn't there in the first place.

And that's what it seems quite clear that basic auditory masking is, the limits of the cochlea to encode the stimulus in the first place.
Quote
But regardless, because the release of basilar membrane hair cells is graded, the next level of processing could be affected in a manner you are not acknowledging.

The fact that their firing is graded with level is irrelevant. We're talking about the situation in which the noise from the PPM encoding (that's what neural transmission is, after all, PPM) is large enough that a change in firing rates or first-arrivals is not detectable given the known integration times of the auditory system.

First arrival is indeed the most sensitive, and if you think a minute, it's obvious that a rapid increase in firing rate after a period of low rate is quite easily detectable.  The converse not so easily detectable.
Quote
That is the main point here, in terms of what this thread is about: you have ignored the difference between sensory processes and decision processes.

I have done no such thing. Please do not presume to lecture me or to put elementary, obvious fallacies in my lap.

I am ignoring nothing.

The signals in the experiments you cite are way, way over both absolute and masked thresholds.  They are audible.  They absolutely create detectable information in the auditory nerve.

A signal below masking threshold does not.

Your experiments are, I will presume, correct for what they measure, but they are not in any fashion relevant to this discussion.
Quote
The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. I don't see how you fail to see this distinction.

Now, exactly what is your concern with, DBT's, or ABX tests, and for what use? Detecting thresholds or something else?
Quote
Quote
On the other side we see a century of work that shows that humans are easily fooled into thinking things exist when they don't, especially through the auditory system.


Just to be clear, we agree on this point, for the most part. Clearly, many of the phenomena people report about different sounds are not there acoustically. What we are talking about though is that some of the consistent patterns in people's reported experiences can be at least partially explained as an unconscious effort problem that affects those experiences. This effort difference will not be reflected in a decison task such as an ABX test.
[a href="index.php?act=findpost&pid=341041"][{POST_SNAPBACK}][/a]


You can not say "will not be reflected", you haven't shown that.  What you've shown is a discrimination test between two impaired signals with easily audible impairments, not a test of JND's.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #93
Quote
Sorry, you still have provided no evidence that what you say applies to inaudible differences. So, you have no basis to say that an ABX test (which is just about audible or inaudible differences) may not be adequate because of these reasons.

Indeed.
Quote
Related to this, I believe that very subtle audible differences have been detected in DBTs even when the subjects at test couldn't consciously hear a clear difference.
[a href="index.php?act=findpost&pid=341106"][{POST_SNAPBACK}][/a]


Indeed they have. Subjects have more than once reported that they were "mostly guessing", could not articulate differences or sometimes even internalize the idea that they heard difference, but some of those subjects have still discriminated against a very slightly impaired signal at the 99.9% level. (18/20, for instance)

What's interesting is that in my experience, quite some of these subjects, who insisted that they were guessing, were right.  Quite a few more of the subjects who insisted that they heard "obvious differences" hadn't.

Trained listeners are very often humble, having been through those experiences, and will "sally on" even when convinced they didn't hear anything, only to find out that they did.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #94
Quote
In other paradigms, like the cross-hatch/shape studies, of course when you direct someone's attention, they see the shape. The point of that work is to show that items not consciously perceived are actually processed. So, as I keep repeating, aspects of compressed stimuli might affect processing, and conscious reports might not reflect it.[a href="index.php?act=findpost&pid=341166"][{POST_SNAPBACK}][/a]


You keep avoiding the obvious point.  The stimulus, crosshatch in this case, is way, way, way, way way above threshold, both absolute and masking.

The issue here is what happens to parts of a signal that are below absolute or masking thresholds.

And we are talking about the ear.

You must recall that unlike the eye, the periphery in the ear is a frequency analyzer. No such facility exists in any substantial form (lateral inhibition being the only thing to speak of) in the eye.  The eye, of course, is a 2D transducer, the ear is time/frequency, which you can argue is 2D, but with one dimension as time, you can't "hold" a stimulus there like you can for the eye.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #95
Quote
Subjects have more than once reported that they were "mostly guessing", could not articulate differences or sometimes even internalize the idea that they heard difference, but some of those subjects have still discriminated against a very slightly impaired signal at the 99.9% level. (18/20, for instance)[{POST_SNAPBACK}][/a]


Like Garf ? [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=6651&view=findpost&p=70284]http://www.hydrogenaudio.org/forums/index....indpost&p=70284[/url]

Getting 99.8 % just guessing. Fine...
But he was not only guessing, the stumulus was just not being presented to him  !

What's the problem with double-blind testing?

Reply #96
Quote
The ABX paradigm relies on a decision process, and does not take into account processing differences that might exist at any other level. [a href="index.php?act=findpost&pid=341041"][{POST_SNAPBACK}][/a]


An ideal ABX test should be run under the exact same conditions under which the difference tested was first observed. This way, only differences relying on a decision process will be tested.

This works if ABX tests are used to check the validity of people's claims about sonic differences.

I can understand your criticism of ABX if ABX tests are used in order to evaluate a threshold of hearing, for example. In this case, there is no pre-existing decision process that the ABX test is supposed to validate. Differences smaller and smaller are just submitted to the listener.
I find this method rather flawed. The listener must have the possibility to train himself. For example, it becomes easy to pass ABX tests about MP3 compression once you have heard MP3 artifacts, while you can fail the same test if you don't know what to listen to. It happened to me with the Smashing Pumkins sample provided by Xerophase. I could hear no differences, but since he managed to get a successful score quite easily, I listened and listened to the samples again, until I finally found a small difference.

Once the listener is trained, it all becomes a decision process, and the ABX test can be applied. If you apply the ABX test without first training the listeners (and ITU-R BS.1116-1 recommendation for blind listening tests says that It is important that data from listening tests assessing small impairments in audio systems should come exclusively from subjects who have expertise in detecting these small impairments), then you take the risk to get a failure while your subjects are actually capable of passing your test. This could indeed be a situation where the reaction time detects a difference that the decision process doesn't detect (yet).

What's the problem with double-blind testing?

Reply #97
Quote
I can understand your criticism of ABX if ABX tests are used in order to evaluate a threshold of hearing, for example. In this case, there is no pre-existing decision process that the ABX test is supposed to validate. Differences smaller and smaller are just submitted to the listener.
I find this method rather flawed. The listener must have the possibility to train himself.[a href="index.php?act=findpost&pid=341210"][{POST_SNAPBACK}][/a]



Your objection is invalid.

Curiously enough, it has been shown in very graphic ways (i.e. by doing it) that one of the most effective ways of training on simple signals is to just decrease the probe signal from obvious, slowly, trial after trial, until the subject fails to hear it at some given level, is given at least two tries after going up and back down a bit, and then taking that as threshold.

It's called "signal detection", it's old and well-established in the art.

Did it occur to you that what you, yourself, describe, when done in a sensible fashion, is a self-training test?

Now, nobody's ever argued the need for training that I know of, well, nobody who has done testing and observed subject performance, so I must have missed something about your point, of COURSE the () subject has to be trained one way or another.
-----
J. D. (jj) Johnston

What's the problem with double-blind testing?

Reply #98
There aren't theoretical flaws with abx or DB tests.
The practical and theoretical problems and limits of these tests are well known in various sciences.
I think, they were all mentioned already here at ha and also in this topic.

To the problem of abx-ing unconcious differences:
differences you cannot even listen by short A/B comparison,
you can vary the setup of an abx-test.
Extend the abx test to a long time period, listen each day for 30-xxx minutes, and rate eg. your fatigue.
I don't know now, if this has been tried already for good  (or even medium/low quality) lossy vs. CD.

of course, if such a test results to 50% , ie. pure guessing and same fatigue values for lossy and orignal,
it doesn't prove, that lossy eg. is not more fatiguing than original.
As known, abx test can only prove (with certain probability) one side of  a hypothesis (that was to be tested).

What's the problem with double-blind testing?

Reply #99
Quote
I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.
[a href="index.php?act=findpost&pid=335816"][{POST_SNAPBACK}][/a]


That is why, IMO, all this focus on lossy encoders is a bit outdated. Encode to lossless into your music archive/repository. Then play from there at home, stream to your stereo or connect a PC directly to the stereo. Filesize of lossless is not a big deal these days, very few have music collections that would require disk storage worth more than a few hundred to maybe a thousand dollars for a fully mirrored setup. Most could do with much less. The CDs are the backup anyway, but it takes time to rip, so some extra money into harddisk-crash protection is nice. The advantage is that you now also got a backup of your CDs, since you can recreate a bit-perfect(1) copy of the original CD if it dies from scratches of UV-exposure.

Then for portable use, transcode from the repository to mp3, aac, or whatnot.

In this context, what is "best" is moot. Not happy, just re-transcode to a different format or bitrate.

(1)Audio ripping is error prone,  but if using EAC in secure more and/or test&copy with AccuRip there it is quite likely that your copies are bit-perfect. There are also other factors, but please don't nit too much