Stereophile's article talks about the problems with ABX'ing. Now I'm a big fan of the ABX test and draw a healthy amount of skepticsm whenever such a test is NOT performed (as HA has taught me).

But on this page, they talk about Type 2 errors. Now the author doesn't explain how he derived these numbers and claims that Type 2 errors are the probability that an audible differerence existed but was not heard.

I don't understand what he's talking about, and I don't trust the fact that he references himself.

WARNING: This article is a pain to read, but I think all Stereophile articles are like that.

You can download a spreadsheet I made which shows how type 2 errors are calculated here:

http://ff123.net/export/TestSensitivityAnalyzer.xls

A type II error is the probability that, given an effect which is known to be audible to a certain percent of the population (or equivalently, audible a certain percent of the time), an ABX test will mistakenly declare that there is no audible difference. The type II error is dependent on the size of the effect, as well as on the type I error risk, and the number of listeners (or number of trials).

In short, it's the risk of declaring no difference when there actually is one.

I have created an ABX application which includes type II risk, but have not finished polishing it up (eliminating all the minor bugs) here:

http://ff123.net/export/abchr1.1beta2.zip

Setting up the various type I and type II risks prior to an ABX test greatly depends on the goals of that test. If one wants to really make sure that he doesn't falsely declare "no difference," he needs to make sure that he chooses a low-enough effect size and a low-enough beta (type II risk). The value chosen for alpha (type I risk) also makes a difference. Typically for very small effects, the required number of listeners or trials is very large. This has to be balanced against listener fatigue.

For codec testing, we don't really care about "no difference" errors. So the number of trials can be reasonable.

ff123

There is a Latin expression to the effect of "Cuit bonit?", I believe. My Latin is rusty. It translates as, "Who benefits?" When reading the Stereophile it is good to remember that they support high-end systems and declare that for $10,000 apiece some amplifiers are better than those for $500. There is little or no ABX'ing of these claims. They have no reason to persuade readers that differences are measureable or quantifiable. A less charitable way of saying this is that they are snake-oil salesmen.

Bob Carver was able to tweak a transistor amp he built well enough that not one person at that magazine was able to tell it apart from a high end, Conrad-Johnson, I think, tube amp. So much for their golden ears and objectivity. When they have to put up or shut up they fall flat.

That is just my opnion, FWIW.

Type II errors can be rather large in the criminal justice system. For instance, "society" generally considers it more upsetting to find an innocent person guilty (type I error) than to let a guilty person go free. That's why you'll hear things like "reasonable doubt". We bear type II errors because we want to protect the innocent, at the expense of letting some criminals slip through. I think the infamous O.J. Simpson case applies here.

Big time. The last thing Conrad Johnston, Mark Levinson etc etc want is objective analysis amongst reviewers.

They need to sell their product, bottom line.

I'd personally be more worried about alpha inflation with sequential ABX tests than I would with Beta (Type II) errors. People seem to forget that when you repeat the exact same test in sequence, the Type I error rate increases, so that two sequential tests with a significance cutoff of p<.05 will become p<.10, which is no longer statistically significant at the 95% significance level. I think this warrants more attention than the possibility of committing a Type II error

Just my statistical 2 cents.

ff123, that's a nice spreadsheet you have there. But in answering my question, you raised another one for me. What role does the proportion distinguisher (pd) play? How do we determine that value for a real life situation?

Sorry for bringing up all these questions that should have been answered if I had actually stayed awake in my probability distributions class (I got a C, but I maintain that the professor was confusing). The reason I'm asking is two-fold. First, in the interest and pursuit of the actual truth, a skeptic must be prepared to address all reasonable arguments, no matter how biased the source.

Second, I got into a cable quality debate with a friend online and I plan on testing his ears in the future (the date is not yet determined) on the difference between his silver cables and some thick copper cables that I'll pick up on the way to his place. But I want to ensure that I minimize the chance of error while simeoultaneously keeping the test(ing) reasonable. Beta-errors are something I had never heard of before and had not considered.

The rule of thumb given by Sensory Evaluation Techniques is:
pd < 25% represent small values;
25% < pd < 35% represent medium-sized values; and
pd > 35% represent large values

When you plug in small values for pd, you'll see that humongous amounts of listeners (or trials) can be required to achieve a low type II risk, even when the type I risk is allowed to rise.

Here is a real-world example from the food industry:

"A food company wants to test for similarity of the blended table syrup produced with corn syrups from the current and alternate suppliers. The sensory analyst and the project director note that to obtain maximum protection against falsely concluding similarity, for example by setting beta at 0.1% (i.e., beta = 0.001) relative to the alternative hypothesis, that the true proportion of the population able to detect a difference between the samples is at least 20% (i.e. pd = 0.20), then to preserve a modest alpha risk of 0.10 they need to have at least 260 assessors. They decide to compromise at alpha = 0.20, beta = 0.01, and pd = 30% which requires 64 assessors."



If your friend is up on statistics, he should insist on a large number of trials to keep the type II error down and to allow for small effects.

ff123

Edit: Note that proportion of distinguishers (i.e., pd) for a test involving multiple listeners is not exactly the same as the percent of the time that an effect can be heard by an individual listener using multiple trials. There is a conversion formula. But for the sake of simplicity, I would just assume that the two things are equivalent.

Cui bono, to whose good

Very educated discussion there at Stereophile

I agree in principle, but the reality is that 'golden ears' have often already claimed to be able to easily hear a difference between 'A' and 'B'. Just read any review in any high-end mag -- phrases like ' it was like a veil was lifted' or 'suddenly it all came into focus' or 'it was no contest' -- indicators of *big* subjective difference. -- are not uncommon.

So in this case, how many trials are necessary not to confidently *dispense with* the original claim, but to simply cast it into doubt...to shake his confidence? I predict that his friend will have no problem identifying his cables by 'sound' 100% of the time when he knows which ones are in the circuit. And that after a small number of trials under blinded conditions, it will be apparent to his friend that 'sightedness' has a biasing role in reports of audible difference.

Many 'objectivists' started out as 'subjectivists' who had a humbling experience. A common one is the 'phantom switch' -- it happened to me. I thought I had changed something (threw a switch); the difference was obvious to my ears; later I realized that I had actually accidentally *failed* to make the switch , and had been listening to the exact same thing all along.

For an intellectually honest person, that sort of experience should be enough to dash excessive confidence in sighted listening perceptions. I find it hard to believe that at least some of Stereophile's staff *hasn't* had such experiences.

In modern parlance: follow the money.

Not sure if I qualify...I'm an occassional freelance contributor to Stereophile, but I don't do component reviews, and I've never written that "it was like a veil was lifted" or anything like that. Never had the opportunity to.

Yes, I've had experiences like the ones you describe, and I'm willing to bet that most reviewers have, too. And, yes, such an experince should dash excessive confidence. But having one's excessive confidence dashed doesn't necessarily require jumping the subjectivist ship and swimming to the objectivist shore.The psychology of subjective experiences like listening is complex, but so is the science of audio. Quite possibly, there's a lot more going on there than we're able to explain with the simplest scientific models. Keeping an open mind can make you vulnerable to all sorts of psychological bullshit. But relying entirely on objective methods can close your mind to a very wide range of very real human experiences. This objectivist/subjectivist phenomenon is, I believe, more interesting than you make it out to be.

Yeah, I think the folks at Stereophile--the reviewers--ought to do a little ABXing every now and then, just to keep themselves honest. But, though ABXing may be necessary to PROVE what you hear, it isn't necessary to KNOW what you hear, at least not for a well-trained, adequately self-critical reviewer. Do the ones at Stereophile meet that criterion? I dunno. Some yes, some no, I suspect.

Jim

In my opinion you have understated the case.
Differences that are claimed to be stunningly obvious (like the "veil" comment, and a large number of others I've read at Stereophile, especially regarding speaker wire and high-end amplifiers) by reviewers of esoteric equipment need to be verified objectively if at all possible if a reviewer is to be taken seriously.

If a sound quality difference is claimed to be very substantial or dramatic, it should be
rather trivial to objectively verify via an A/B/X test. If such differences are not possible to ABX successfully those percieved differences should not be claimed to be dramatic as a consequence.

That is, if ostensibly dramatic differences cannot be detected in a statistically significant fashion by the reviewer sight unseen then such a reviewer has no business claiming that the differences they can "hear" only when they know which is which are somehow dramatic.

Any individual that claims such nonetheless is simply not "adequately self-critical". Being self-critical does not prevent one from thinking one hears something one does not. Being adequately self-critical in my view would be to test subjective impressions in any and every feasible objective manner, every single time.

And then report both the subjective impressions and the ABX results.

I cannot help but suspect that the manufacturers who make $1000 a meter speaker cable would be less inclined to send a publication sample product if such methods were rigorously implemented.

I think that's a completely reasonable opinion, but obviously it's one the editors and reviewers of Stereophile--and their readers--don't share. Clearly they won't convince you with their methods--haven't so far, anyway--and I'm sure they're okay with that.


Logistically, ABX tests on real-world components are far from trivial. Statistically valid tests are extremely time-consuming and a royal pain in the ass. When you're switching our wires you don't have the luxury of hearing two "samples", repeatedly, within seconds--or fractions of seconds--of each other. Apart from the time delay, with most components there's the issue of level matching, which typically requires an even longer delay between "samples." So it's not just a matter of logical difficulty--the time delay between "samples" presents serious methodological difficulties. And you have to have a friend. Even a relatively high-profile mag like Stereophile or TAS probably doesn't have the resources to support such tests. And even if they did, they probably wouldn't be able to get any reviewers---with audio knowledge and writing skill--to work for them.

ABX with real-world components is logically trivial, but it's a logistical nightmare. Most reviewers choose other methods because, given these real-world constraints, the other methods are (they believe) more effective. And, as proving the validity of their tests to skeptical readers isn't their goal, there's little to be said (in that context) for ABX tests.


Logically it's a fine idea, but practically it's prohibitive.

Cheers,
Jim

I think you have a fine career ahead of you as a politician should you choose it

I also think Stereophile's editors know where their advertising revenue comes from. Additionally I think they know their readership would drop precipitously if their objective measurements repeatedly did not support their subjective claims.



You have misinterpreted my meaning here.
I was not referring to the difficulty of setting up a physical ABX system for the components, but of the difficulty involved in actually detecting the differences which were previously subjectively "obvious" or "dramatic" or "like night and day". If the differences are obvious, dramatic or like night and day, then they should be readily ABXable. My experience shows that subjective impressions often disappear when subjected to ABXing, though this is probably not a result of the objective methodology.

When reviewers state the audible differences between products are obvious and dramatic, but then cannot detect those differences in ABX testing, the differences are simply not obvious and dramatic, and anyone claiming they are is selling smoke or smoking what may be illegal to sell.

Regarding logistical difficulty I have no doubt there are problems involved.

However, once assembled, the project ABX box does not appear to be a "logistical nightmare" to me, and I believe it features level matching capabilities:



I find the editors notes on the project ABX box to be particularly relevant:


Snake oil is a good term to use here given the historical context.
Does actively shielded speaker cable also solve Dropsy, the Gout and "feminine" and "male problems"?

Probably!

If the goal is to test modulation cable worth 1000 $ per meter, it rules out the use of the ABX box, unless all its component are themselves of much higher quality that 1000 $/m cable, in order for it to be transparent compared to the tested cable (or plugs, or other devices).

It would be the same thing as uploading all our test samples encoded in Vorbis 128 kbps, since it is transparent most of the time. However, we require lossless originals.
In the same way, the most basic rigor in modulation cable testing would be to plug the cables from the source to the ampli without any switch in the path of the signal.

The test should also be double blind, which increases the difficulty to an insane level. Basically, double blind means that there should be no operator in the room who knows what source is being played ! Which leaves two solutions. Getting the listeners out of the room while the sources are switched, and getting the oprators out when the listeners are in, or prepare enough complete identical listening systems (source + ampli + speakers) in one or more room, and leave the listeners with them to guess which one is which.

The discussion of type 1 and type 2 errors is interesting.

However, I wonder if it's strictly fair to apply them to ABX testing in the way described in stereophile.

Imagine that the difference between A and B is so subtle that it will only be picked out, on average, 60% of the time. That's only 10% better than chance. This gives a possibility of missing it (in a typical Hydrogen Audio ABX text) which sounds quite high, and apparently justifies why people miss subtle details in ABX tests that they would discover in non-blind tests.

However, if something is so subtle that, on average, you'd only pick it out 10% more than chance, anyone who has done a software ABX test knows that they actually do several mini ABX tests before deciding. You click A, you click B, you click X. You're not sure, so you click A and B again. You decide you've chosen a lousy part of the file for comparison so you move the slider. You try again. Then, when you finally feel sure, you click "X is A" or "X is B". That's 1 decision. Then you make the next 15 in a similar manner.

So your "60% chance" sample (p=.6) actually gives much better than 60% correct values in ABX. So it's not actually p=.6! The key question is that: for a sample which you can only give correct ABX results for 60% (i.e. where p really dos equal .6), what is the chance of hearing the difference during normal listening. That's the million dollar question. Objectivists would say it's very small, subjectivists would say it's larger!

Interesting read...

Cheers,
David.

No, it only requires that the ABX box itself be shown to be transparent when added to the signal chain.
Furthermore, as high-end has shown over and over that stuff like $1000/meter does NOT logically imply 'high quality'. There is no price/quality correlation at that level.



And this is because Vorbis 128 kbps *has* been experimentally shown not to be transparent in some cases. Moreover, from 'first principles' it's reasonable to assume that a lossy codec will introduce an audible artifact. Should we *assume* the same about a well-built ABX box? Audiophiles say yes, because they tend to believe that *everything* makes an audible difference. But is there reasoning from scientific principle or experimenta evidence to support the assumption?

I'd say no, and that the codec analogy is flawed for that reason. The assumption of audible artifact is reasonable for 128 VOrbis, but not for good solid-state switchers.



It can be done that way too, but the scientific evidence is that if anything you *lose* sensitivity to audible differennce doing things that way, by introducing latency between A and B. Results from such a trial could be attacked on those grounds. (though it would be hypocritical for a 'subjectivist' to do so, since quick-switching is rare in high-end reviews.)




Insane? No. Having the operator leave the room while the source is played is not much of a problem.
Audiophiles go to *far* more insance lengths to achiev ethe 'absolute sound'! Is doing proper DBT any harder than setting up a high-end turntable properly?

Btw, some audio magazines -- Sound & VIsion comes to mind -- *do* run the occasional DBT, and do feature writers who swear by them (e.g. Tom Nousaine, David Ranada).

? if the chance of hearing the difference 'for real' is 60%, then that chance is the same whether the listening is done blind or sighted. In proposing your htought experiment, you beg the question of how one knows that the chance of hearing a real difference is 60 % in the first place.

I think David was commenting that the size of the effect (or the chance of an individual hearing a difference) varies depending on how the test is conducted. If you just listen to A, and then to B, and then to X, in that exact order and to a set length of music, that's quite a different situation from being allowed to switch between A, B, or X at will, starting from and ending at arbitrary points inside a particular selection of music. The second situation is benefiting from a mini-training effect, and also from choosing the most difficult section of the music. And that doesn't happen in normal listening.

That line of reasoning implies that ABX'ing the way it's done around here is exaggerating the small defects out of proportion to their real-world relevance. But it can only help those who are trying to get at really small differences.

ff123

Oh, absolutely. But the Stereophile/TAS perspective is the opposite of that: it's that bias-controlled comparison *masks* differences that have real-world relevance (i.e., reports no differene when there is real difference). Ignoring decades of data on the fallibility of perception, and the need for independent verification of sense impressions, they've *equated* 'real-world relevance' with perception, so the conflict seems inevitable.

In other words, it requires the test result to be negative !
Testing an expensive cable, we are testing the hypothesis that normal cables and interconnects, like an ABX box, have an effect on the sound.
It is completely flawed to assume that they don't in order to prepare the test !

I finally had the time to read all the article.
It points out the importance of adding a recommendation to our guidelines for blind tests.

• I advise to never run sequencial ABX test, because the result is very difficult to calculate (I don't know if many people agree though, since in the last discussion about it, I standed alone for fixing the number of trials, and Guruboolez standed alone for using the sequencial method).
  
• We require to run tests only once (or to take all results into account).
  
• In addition, in order to minimize type II errors, we should advise people to give only answers of which they are absolutely certain, whatever time it takes to achieve.

This way, the "p" or "pd" parameter is equal to 1 and the risk of not finding a difference while there is one is null.
Note that, as pointed by Leventhal, in answer to Clarke in page 8, this is not the risk of concluding that no differences exist, but the risk of not concluding that a difference exists.

No -- *if* you assume that an audible effect from the ABX box is a reasonable concern, it requires that, *before* you compare the sound of cable A vs cable B when connected to system C, you compare the sound of the ABX box inserted between A and C, versus the sound of A connected directly to C.
(For thoroughness' sake I suppose you could also repeat the test with B and C).

Objective measurements of the signal coming into and out of the ABX box could, of course, also provide evidence for transparency (or not).



...or that the *expensive* cable does! The fact is that a very few expensive cables are patently not transparent -- they are designed to roll off high frequencies, though this isn't revealed in the advertising.



At what point do one's assumptions about test conditions stop being flawed? Is it flawed to discount the possible effect of day of the week, cycle of the moon, etc, on the test results? The problem with the 'audiophile' stance is that it's *infiinitely* skeptical when it comes to science, but quite the opposite when it comes to 'personal experience'.

If one wants to reinvent the wheel for every journey, I suppose one is free to do so. That doesn't make the assumption that it's necessary to do so, reasonable. I have indicated that one coudl test the trasnparency of the ABX box itself, both via listening and measurements, before moving on to test the difference between cables, using the ABX box. (The original designers of ABX boxes ran such tests, btw.)

It's interesting to note that for all the hundreds of claims af audible difference between cables, none has published controlled listening test evidence to back them up -- no company has ever offered such results in support of claims for their product. One might think that if their cables were as audibly superior as they claim, that they would. Meanwhile there is a large body of scientific literature (and a small body of controlled cable comparisons) that supports the assumption that audible differences between competently designed cables are more likely figments of the mind, than real.

For your consideration, here are the thoughts of John Dunlavy, noted speaker (and cable) designer, on this topic:

http://www.verber.com/mark/cables.html

I agree with you (that's why I'm rewriting ABC/hr), but practically speaking I'm not sure it makes all that much difference. The error only gets big for a large number of trials, where we can already infer that the difference, even if it is falsely gotten, must be pretty damn small.



Yes.



I think for codec testing, we're not really concerned with type II errors, but that's just my opinion. The presets I've put in the new ABC/hr reflect this assumption, however.

ff123

Certainly, assuming the measures of "quality" are in some way individually quantifiable with or without the use of an ABX-like system. Of course, I would be quite interested to discover interconnect that is indeed "worth 1000 $ per meter" rather than interconnect that is merely priced at that point

It seems to me that it would be well within the realm of possibility to build such a box using individual components that bench tested as perfectly as possible (in terms of electrical characteristics/tolerances), and then test the assembled result, modifying it if necessary to achieve the absolute best results.



As long as the hypothesis is falsifiable, of course!



The bolded section should be read and reread by every subjectivist "audiophile", five times daily. Mr. Dunlavy is absolutely not joking about the Nobel prize.

I'm going to read through the whole article again.

However, I have a feeling that these extra statistics (type 2 errors) are simply telling us what we already know - i.e. that the statistics aren't really proving anything. What's more, to use these extra statistics properly, it requires an a-priori knowledge of what we're trying to find out! The false proven statistic (type 1 error) is OK, but to accurately calculate the chance of a false unproven (type 2 error) you have to know the chance of hearing a difference given infinite trials. Well, if we knew that, we wouldn't be ABXing - because if it is more than 50% than that proves there is an audible difference!


So that means we should assume the worst. We should assume the difference is barely audible (i.e. 60% chance of picking it out). However, I think that makes the statistics even less useful. If the chance of hearing a difference is only 60%, then the gap between type 1 and type 2 errors - the region of useful results where you can be fairly certain to avoid both types of errors, is very small. But that's just common sense - if it's very hard to hear a difference, then it'll be very hard to ABX. On average, you won't get a statistically significant result until you've done a very large number of trials.


I'll read it again...

Cheers,
David.

I've read to the end (the print view makes life much easier!). The last letter is very sensible. It makes the point about probabilities better than I did.

Earlier in the debate, John Atkinson says how insensitive ABX tests are, and mentions some stereophile reviewers whose subjective reports we should trust more.

If I were to tell you the widely held beliefs about one of those trust-worthy reviewers, I'm sure I would be sued. All I can say is that many people in the industry choose not to trust him, allegedly with very good reason.


I can repeat an anecdote (probably legend!) about a different (anonymous) reviewer from the 1970s. This was an era where the "big" Japanese manufacturers were producing popular mass market equipment, but this equipment didn't always receive good reviews! Their latest amplifier was driven around to the reviewer's house in a Porsche. A note was attached: "Return the amplifier, keep the Porsche"

Cheers,
David.

Maybe he wrote a bad review, returned the amplifier and kept the Porsche

Good to know we have someone on the board that has pull with the Nobel committee.

As an erstwhile audio writer trained as a physicist, I agree that we probably have a solid theoretical grasp of the basic laws that govern the operation of cables (and to a lesser extent more complex audio devices) but I worry that the quote is being used here to imply more than it should. Of course we know the basic theories that apply to cables, component design, etc. But anyone who has ever done scientific research (and I suspect anyone who has ever done much engineering) knows that it's one thing to understand the basic theories, and quite another to properly apply them. Indeed, much of my research career was spent trying to figure out WHICH theory was the most appropriate for a certain (experimental) context, and how it should be applied. Often careful modeling is necessary. As far as measurement goes--yes, I think we can probably measure everything that's relevant; what's not possible, currently, is to directly relate what we measure to what we hear. These issues are far from trivial.

I would translate this quote, and the message that follows, as follows: "If a subjectivist audiophile's assertions don't correspond to expectations derived from a simple-minded, naive (mis)application of (poorly, by the writer) known laws of physics, then that person is dishonest and trying to rip you off." It's a mistake to slander those who have the good sense to believe more in their own perceptions than they do their own (inadequate) understanding of science. With respect, some people on this board would do well to take a similar approach.

It's true that there's little statistical proof of much of what the subjectivist audiophile camp asserts. Objectivists would do well to be measured in their response and stick to this lack of evidence in rejecting subjectivist's claims. When you get into the realm of theory you're on much less solid ground. I have never met an engineer who's insight into the laws of physics was deep enough to effectively rebut any but the most obviously dubious claims of the audio subjectivists. Furthermore, most experienced engineers know that much design--including electronic design--is based on phenomenology, convention, or tradition, and not on the direct application of theory. Theoretical understanding often lags engineering wisdom. That's not to say that there's any Nobel-inducing radical new theory involved. There's plenty of work to be done in figuring out the proper way to apply existing theories.

Jim

I think you're wrong about getting sued--they hear this stuff all the time, and are quite used to it, though I'm sure that doesn't take the sting out completely. Still, I trust that people who are sketpical and scientific enough to reject subjectivist claims made without statistical evidence will also be skeptical enough to reject "widely held beliefs" and urban legends presented without evidence.

Jim

I labelled the Porsche anecdote as a legend. I didn't label what I referred to in the earlier paragraph as such, for several very good reasons.

Cheers,
David.

jaustin,

You're right about the misapplication of theory (a little knowledge is a dangerous thing etc etc).

However, are you of the opinion that, if two devices display identical measured performance, then they must display identical subjective performance?

Or are you in the camp that believes there are differences that we can hear, but not measure?

I realise I'm making a grey issue black and white, but take it at face value: are there audible differences that are due to things that we cannot measure?

Cheers,
David.

It depends on whether you make the right measurements, and if anybody knows what the right measurements are, I haven't heard about it.

I believe that there are audible things that we do not yet know how to measure--or if we know how, that no one has yet made those measurements. Establishing a correspondence between perception and measurement isn't trivial. As an example, I think that we cannot rule out the possibility that particular distortion "signatures" are responsible for perceived differences between amplifiers and other audio equipment (including, eg, output stages of CD players). But the problem of relating a particular distortion signature to perception is very difficult because it's very hard to identify specific aspects of the sound that relate to those subjective perceptions. And, yes, whenever this is the case--that perceptions are difficult to specifically identify, it is appropriate to be skeptical--but not closed-minded--about whether those perceptions are real. (ABX testing is desirable under those circumstances, but can be difficult to perform with real-world equipment). Limiting yourself to statistically valid data is a legitimate, even essential, for many purposes, strategy, but in the interest of clarity and certainty such an approach limits the information that's considered.

Any statisticians out there? What are the chances of replacing the binary ABX test with a Bayesian, inductive approach that would be just as valid, statistically?

Cheers,
Jim

I'm not sure what is meant by this. Bayesian implies to me use of current knowledge as an aid in making a decision.

This can be applied, for example, in a sequential test, where we are allowed to see the results of each trial. There is a Bayesian method (Wald's ratio test) which can be used to determine an optimum stopping point for an ABX test given prior test parameters (type I and II error risks, probability of detection).

There are also several types of double-blind tests to determine whether or not a difference exists, of which ABX is just one. However, I think ABX is probably the most robust.

ff123

you could say exactly the same for atheism vs religion. i know of no scientists that can demonstrate successfully that God does not exist (there was an amusing Simpsons episode that dealt with this), but OTOH, what proof of God's existence can possibly be offered? certainly nothing but a few books of highly questionable origin.

but i wont get into that as i'll needlessly start offending people. i think it goes someting like this - you can't convert a hardcore sceptic, and you can't convert a hardcore audiophile, no matter what evidence you offer.

subjectivism and objectivism use completely different (and probably mutually unintelligible) languages. the issue will probably never be resolved.

but that "like a veil has been lifted" really gets up my objectivist nose (i can give proof of this too...just have a beer with me some time)

It does not require 'pull' to know what constitutes Nobel-quality discovery. It requires no pull for me to accurately assert that anyone who could prove the claims of creationists and thus overthrow the theory of evolution, would be awarded a Nobel prize.





Dunlavy himself has been advertised as an engineer/physicist, so it's possible his training has made him aware of these issues.

He has had much more, and in more detail, to say on cable design:

http://home.austin.rr.com/tnulla/dunlavy6.htm




It's a mistake to believe in your percerptions sans an awareness of the known flaws in perception.



And little scientific or engineering basis for it either. It may be unclear to subjectivists, but there are *two* pillars upon which skepticism of audiophile claim rests: 1) measurements which indicate only small differences between A and B, and 2) what we know about human psychology.



Please visit rec.audio.high-end, where I suspect you may be satisfied in that regard. Thena gain, I'm not sure which claims you are classifying as 'obviously' flawed.



If theoretical understanding lags behind engineering widom, and engineering wisdom tends to find audiophile claims absurd -- then what are we to conclude?

For cables, the measurements have been known for some time. If you assert otherwise, it up to you (or someone) to show that those measurments are inadequate for predicting audibility of cable difference....some nicely controlled listening tests would be a start.



I would ask, then on what basis you are skeptical of any reported phenomenon? Because everything from
lucky numbers to ghosts to God could be plugged into your paragraph above in place of 'audible things'.

To me, the 'I believe there are things we can hear but do not know how to measure' is a sort of special pleading. The belief exists to justify a view of anecdotal experience for which *other* , far more well-founded explanations -- psychological bias, in the case of audio -- exist. It never enough to say 'there's lots we don't know' -- it's important to show why what we *do* know is probably inadequate.
The audiophile argument seems to boils down to 'lots of us hear it'. Do you consider that sufficient reason to believe that current knowledge is inadequate?

I admit that I'm not big on testing methodologies and such. I don't fully understand ABX testing and so forth. At least, I don't understand how to read the results, though I do understand the purpose behind it.

So, it seems to me that "hearing" is just another form of "measurement". If you can really hear a real difference, then you should be able to hear that difference in a controlled situation, like an ABX test. The ABX test simply quantifies the difference that you are actually hearing and proves, to some desired statistical level, that you are actually hearing that difference.

In other words, the question seems flawed. If you can perceive a difference, then you can always measure that perceptual difference via some method or another. So there's no such thing as a difference that you can hear but not measure because the question is silly. Hearing is a form of perception, which can always be measured, really.

Or am I missing something here?

That's all well and good until the paradigm shifts. Our understandings of the physical world are merely models, and models, even long standing ones, are all subject to change. Remember Newtonian physics?

Still, if a validly conducted ABX test statistically demonstrates that someone can hear the difference between cables, doesn't that answer the question of whether there is a difference in sound between cables for that person, regardless of why they sound different?

Switching sides here, but just a little, and for only a moment. Newtonian physics is alive and well and does an outstanding job of describing the mechanics of the everyday world. I don't think we need to turn to new physics/paradigm shifts to explain why there are things about audio people don't understand.

There's a series of physics textbooks by a pair of Russian authors, Landau and Lifshitz. They described it as the "minimum of material" that a physicist must master to know his (or her) shit. I know more physics than most folks, but after ten years of training I probably knew ten percent of what's in those books.

Like I've said before, folks think they know theory, they think they know how to apply it, but much of what they think they know is freshmen-level stuff, zero-order approximations to reality. It's not that physics is just a model of reality--true enough, but irrelevant. The real issue is that the models most people have are simple-minded, even if they have technical training (like me). We've all got a long way to go in understanding before we can legitimately start claiming to be approaching the limits of known science.

Cheers,
Jim

The bold part also states :



Mind the two sides : to measure using available lab equipment, or to describe with known theory.

Paradigms shifts only affect the second side. Newtonian physics said (theoretical side) that masses generated a field force. In practice, it makes all objects fall toward the center of the earth with an acceleration of 9.8 m/s2.
Then Einstein published the Genreal Relativity. The theoretical side of Newton's theory was comlpletely removed. Masses generate no such thing as a field force, but their presence can be described as a curvature of space-time. In practice, it makes all objects fall toward the center of the earth with an acceleration of 9.8 m/s2.

So after a paradigm shift, maybe we can describe theoretically a phenomenon that is now unexplained. But we should already be able to measure it with available lab equipment, thus John Dunlavy won't give the Nobel so easily !

Pio (and JAustin) -

I believe we're all on the same page, really.

With regard to the "we should be able to measure it with existing lab equipment" thing, though, that's not always true. We measure only what we look for, and what we look for is governed by our model of the physical world. We generally don't build equipment to measure things we don't think exist.

Audio cables can be measured six ways to Sunday, but I bet no one has measured them for "pink interactions". What are "pink interactions"? Well, they don't exist in our current model, so we don't try to measure them. But when we start observing differing behaviors of different cable systems that shouldn't behave differently (because our models say they're functionally the same), we may need to change the model and start trying to measure something new (ok, so "pink interactions" is a lame one, but you get the point). That new measurement, of course, may require new equipment. Newton couldn't have measured E-M field strength or alpha-decay, but that doesn't mean those things didn't exist.

FWIW, I'm not saying that there is a noticible difference between different cables. I was just a little bored at work today, hence the James Burke stuff.

- Jeff

What an amusing comment, truly.
Please reread.



How very droll.
You have failed to understand the comment, and given your undoubtedly thoroughly prestigious background in physics I'm quite shocked.

Blind, preferably double blind controlled studies are the only effective way to circumvent bias effects, like the "placebo effect". This of course, has little to do with physics, but I'm still quite taken aback. If you knew a thing about drug trials you would know why it is necessary to have double blind trials in order to determine whether the experimental product is superior to the placebo. Otherwise you are prescribing Ginkco to cure cancer based on the "flows of chi".

The point is absolutely not in any way, shape or form that when assertions not derived from theory are made that they can be instantly rejected. Exactly the opposite is the case if the assertions carry with them statistically viable evidence.
Where is this evidence?

It is when assertions are made without a single, solitary shred of statistically viable evidence that they can be readily doubted until such evidence is presented.

As to my knowledge of "science", I would think any scientifically knowledgable individual would reserve comment until such time as an adequate sample set of my writings on scientific subjects became availible.
Apparently I am proven incorrect yet again. In point of fact I am a professional scientist in the chemical sector.


Of course.
The problem is, where are these tests?
This is the entire issue, the subjectivist crowd largely refuses to conduct them, and certainly the manufacturers of high end cable barring a few ("objectivist", you'll note, most of whom I've read of claim that their cable performs exceptionally when measured, but are largely if not totally inaudibly better than 12 Gauge zip) exceptions. Mr. Austin has suggested that bias effects can be somehow eliminated by ocasionally "calibrating" ones mind rather than always performing rigorous (and tedious) testing to check ones subjective impressions. I am aware of no evidence to suggest this is the case. The human mind is not a pipetteman or a sensor.


Precisely.

I know I won't have to remind Mr. Exhalted Professor of Great Science JAustin of this, but you must satisfy the correspondence principle in order to have a "paradigm shift". Hence Newtonian Mechanics being superseded due to the inaccuracy of prediction by the theory versus measured data.

Well, as you might guess, I disagree!

It's certainly very difficult (some would say impossible) to properly correlate measured performance with perceived performance: you can't always say what measured property causes such-and-such a perceived subjective effect?

However, if we just look at the "there's no measurable difference" vs "I can hear a difference", it doesn't necessarily mean that we failed to measure something. If you do specific measurements looking for X, Y, or Z then you might miss A, B or C. However, where ever it's possible to do a difference analysis between two devices, if that difference analysis reveals a difference that's lost in the noise of the experiment itself, and if the noise of the experiment is sufficiently low in sufficient dimensions to allow us to conclude that any difference that exists is just far too small to matter, then we can conclude that there is no difference worth mentioning between the two devices.

Subjectivists say "ah, but I can hear a difference". This is a difference that they can't ABX, and that can't be measured. I'd suggest that, in truth, there's no audible difference! The reason the pretty amplifier sounds nicer to them than the plain amplifier is the very well known phenomenon of what they see effecting what they hear.

When someone says "no, there is a difference, it's just that you can't measure it" (note, not "didn't measure it" but "can't measure it") then I simply don't believe them.


Of course, in the real world, the situations where you can carry out such a difference analysis are limited, and the equipment pairs which exhibit a null measured difference are rare indeed. However, you can do a fairly good job with loudspeaker cables.

With loudspeakers, you can measure faults more accurately than you can correct them, yet subjective reviewers often praise loudspeakers with measured 10dB frequency response faults!!! Yet I'm supposed to believe these same reviewers when they claim to hear a difference between devices that measure the same, and that they can't ABX.

I don't. The burden of proof clearly lies with them.

Cheers,
David.