Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Atkinson vs. Krueger on blind testing. FIGHT! (Read 28428 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Atkinson vs. Krueger on blind testing. FIGHT!

In a shockingly odd case of a Usenet fight getting completely out of control, John Atkinson of Stereophile and Arny Krueger of PCABX met at the Home Entertainment 2005 show and duked it out on stage.

I haven't been brave enough to listen to the MP3 of the debate yet (an hour long!), but the Stereophile summary is about as even-handed as it could possibly be, which isn't much. Sounds like not much new ground was really covered. In particular, Krueger is still adamant that N=16 is always sufficient.

EDIT: How I ever confused Arny Krueger with Todd Krieger is utterly beyond me. Thanks for catching that Robert.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #1 –
Quote
Todd Krieger of PCABX
[a href="index.php?act=findpost&pid=296893"][{POST_SNAPBACK}][/a]


It's Arny/Arnold Krueger

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #2 –
Quote
Quote
Todd Krieger of PCABX
[a href="index.php?act=findpost&pid=296893"][{POST_SNAPBACK}][/a]


It's Arny/Arnold Krueger
[a href="index.php?act=findpost&pid=296899"][{POST_SNAPBACK}][/a]



Fixed.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #3 –
"...and built his first ABX comparator for conducting blind tests in 1977."

just think - audiophiles have yet to understand the concept of ABX after it's existence for ~30 years.

they must be a smart bunch, eh?


later

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #4 –
Quote
"...and built his first ABX comparator for conducting blind tests in 1977."

just think - audiophiles have yet to understand the concept of ABX after it's existence for ~30 years.
never underestimate the power of imagination. for the good or the bad.

Quote
He subsequently found himself suffering through listening sessions until he finally acknowledged that the ABX protocol had failed him. He switched to a tube amplifier similar to the one that had "sounded the same" as the solid-state amp and spent the next two years in audio heaven (until he discovered another amp that played "Nearer My God to Thee" with greater fidelity).
OMG    what logic.
Nothing but a Heartache - Since I found my Baby ;)

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #5 –
I haven't listened to the debate, but if the write-up is to believed, it appears that people should judge audio equipment with their ears rather than with an ABX test that uses cold, unemotional equipment like people's ears.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #6 –
Quote
Quote
"...and built his first ABX comparator for conducting blind tests in 1977."

just think - audiophiles have yet to understand the concept of ABX after it's existence for ~30 years.
never underestimate the power of imagination. for the good or the bad.[a href="index.php?act=findpost&pid=296911"][{POST_SNAPBACK}][/a]


I think that the reason rather lies into the extreme difficulty to setup a double blind listening test on hardware. Actually it's quite impossible. The two last examples I know (Jason Victor Serinus with power cords, and myself with Cinch cables) were simple blind, not double. And having setup one of them, I can tell you about the work involved. Several monthes, people coming from different towns, the luck to know someone working in a hifi shop, tens of hours of work... just to get a simple bling test, which would be considered as completely meaningless if it passed, since not double blind !

Another reason is the knowledge needed to analyse such a test : maths, statistics, psychoacoustics...

It's very easy to hold an objectivist point of view. Running tests is another story.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #7 –
... or should, perhaps, the creator of the equipment desire to produce scientifically judged listening tests on the behalf of their product, rather than throwing it in the hands of an audiophile and waiting for them to grab at air (helped by obvious biases and/or brand-whoring) to deduce the value of the product?

anyways, i listened to the debate while at work.  it was somewhat interesting, and it felt a little like i was listening to someone's bad TOS#8 experience.  the argument/situation seems pretty straight forward:

1. audiophiles don't care about objectivity or acknowledge that it is attainable.
2. audiophiles can only be subjective because it would invalidate take more work to be objective and negate #3 and #4.
3. in avoiding a standard of deduction, audiophiles keep job/ego security.
4. in avoiding a standard of deduction, audiophiles avoid ever being wrong.

it's a market driven by ego and money, and no one wants to know that their <insert expensive thing here> is just as good as the crap someone bought at wal-mart.


...dunno
later

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #8 –
He is right that there *may* be some things which "slip through" in an ABX-test - since one cannot prove that something doesn't exist. However, a scientific double-blind listening test is still "the best of all evils", because subjective testing introduces too many error-sources and from experience with a high rate of 'em.

Thus, if the choice is "a limited test with high accuracy" or "an unlimited test with low accuracy"...... then only someone who isn't interested in the truth would choose the latter one. ABX is not the choice because it is perfect - but because its the best *reliable* possibility we have yet. Yeah, *maybe* there are some things which you notice with casual-listening, but which you don't notice while ABXing - but the degree of placebo with subjective listening makes it useless - except for boldening your ego and beliefs.

Thus, in simple words: its irrelevant if subjective listening catches something which isn't noticable in ABX - because subjective listening is too unreliable.

It's the same as with heart and head - we have both not because one of them is better, but because they serve different purposes: Want a guide in unknown terrain? Then choose heart/intuition. Want accuracy and robustness? Then choose head.

- Lyx
I am arrogant and I can afford it because I deliver.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #9 –
I am a bit concerned that in abx testing some people have trained themselves to listen for certain artifacts and are not making a judgment basedon overall quality, or overall satisfaction.

Please do not consider this to be a vindication of the autiophile position.  Think of it as a call to action for better abx testing and techniques.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #10 –
Quote
I am a bit concerned that in abx testing some people have trained themselves to listen for certain artifacts and are not making a judgment basedon overall quality, or overall satisfaction.
do we agree that overall satisfaction for the moment is when you fail an ABX test? if not please explain yourself.
the overall quality is very subjective. some encodes could have some very serious but few artifacts and some could have a constant disadvantage (like a to low lowpass).

but the goal of an ABX test is AFAIK to see if you can tell a difference, not overall quality. ABC-HR goes into that direction.

Quote
Please do not consider this to be a vindication of the autiophile position. Think of it as a call to action for better abx testing and techniques.
what do you suggest?
Nothing but a Heartache - Since I found my Baby ;)

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #11 –
Atkinson kind of summed it up during his opening speech:

Using science, he couldn't pick the difference between two amps.

Ignoring science, he decided he didn't like the sound of one amp.

The result is -  he doesn't like science.

The amps themselves sound the same - his own blind tests proved it.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #12 –
With ABX, you can only prove that the probability of you imagining a difference is lower than a certain threshhold.

In an ABX test, the null hypothesis is "There is no difference." The opposite, the alternative hypothesis is "There is a difference."

ABX success means that you can reject the null hypothesis, with a very high probability of the null hypothesis being false.

If you get really picky, ABX failure just means that you can't prove there's a difference. You can still argue that while you can't prove it, a difference still exists. Although your credibility will be at stake, in such a case.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #13 –
Quote
I haven't listened to the debate, but if the write-up is to believed, it appears that people should judge audio equipment with their ears rather than with an ABX test that uses cold, unemotional equipment like people's ears.
[a href="index.php?act=findpost&pid=296925"][{POST_SNAPBACK}][/a]


Very tersely stated, bleh.  I agree with this post.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #14 –
It seems to me that ABX testing is good to find a general trend about an encoder, like which one gives the overall most satisfying result (more transparency/less annoyance) at a specific bitrate.

To judge the sound produced by audio equipment objectively one would have to hide said equipment so that what is seen does not influence the conclusion. That's a bit tricky.

I think what magazines do is to rate audio equipment in their whole, and that probably include looks, biases and brand-whoring in the mix. But that doesn't matter as it is part of consumer satisfaction. Sound fidelity is only one component of it, and not necessarily the most important.

Code: [Select]
Audiophile ---> Money -.--------> Recordings -----.--> Listening experience
                         \                        /
                          "----> Audio system ---"

There are a lot of factors here, I would say Money is the root factor.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #15 –
The discernable quality between high end items is likely to be very small, in such cases then preference rubbish will always surface "been buying XYZ for years, this cheaper far-eastern amp cannot be as good", after all for $50,000 you are perceivably buying more than the amp, the same as luxury anything – it is something others do not have, and that makes you special.

To ABX the high end stuff (low end is easy, just abx against high end), you would have to reference it to a live performance!, not another similar amp.

Testing is important and has to be scientific - take pills, would you really trust a $10,000 pill that is said to increase your happiness by x10 that has not been blind tested?

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #16 –
Quote
There are a lot of factors here, I would say Money is the root factor.
[a href="index.php?act=findpost&pid=297007"][{POST_SNAPBACK}][/a]


This comment is right on the, er..., money!

As soon as you throw out the ability to actually objectively compare something then what is left? Exclusivity.

If it costs a fortune, or is a 'secret', or is very rare (either naturally or via a 'limited edition') then obviously it is better because you've got it and the plebs don't. QED

Note that new technology usually promises quality at lower prices, therefore to an audiophile (in the pejorative sense) most new technology is a bad thing, at least until someone cottons on and starts selling gold-plated optical cables and other expensive paraphanalia.

Similarly mass production means wider access and cheaper prices which lets the plebs in, obviously someone hand-making three speakers a year in his shed is going to be 'better' (by the audiophile measures given above)

You can see the same process in other subjective fields, the muso that likes underground bands that nobody has heard of  and then goes off them as soon as their song appears in an advert and they get a number one single, or the rich footballer who buys ridiculous 'designer' clothes that basically send one message "I have money but no taste", when the same designer goes mass-market via Wall-Mart then they won't have anything to do with it, or restaurants or clubs that everyone wants to go to because they are 'exclusive' and you can't get into them.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #17 –
Quote
Testing is important and has to be scientific - take pills, would you really trust a $10,000 pill that is said to increase your happiness by x10 that has not been blind tested?
[{POST_SNAPBACK}][/a]


The sad truth of course is that people do pay money for, and put their faith in, 'medicines' that have actually failed double-blind tests with placebos, even to the exclusion of 'traditional' (i.e. tested and proven succesful) cures.

[a href="http://en.wikipedia.org/wiki/Homeopathy]http://en.wikipedia.org/wiki/Homeopathy[/url]

I wonder if homeopaths think that audiophiles are kooks and vice versa. I'll bet they do. Everyone suffers from Magical Thinking (http://en.wikipedia.org/wiki/Magical_thinking) to some degree but it's so much easier to spot it in others.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #18 –
HighEnd components are the same like other luxuary stuff.

Owner of Cartier watches also miss planes like the Swatch guy.
.halverhahn

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #19 –
Quote
The sad truth of course is that people do pay money for, and put their faith in, 'medicines' that have actually failed double-blind tests with placebos, even to the exclusion of 'traditional' (i.e. tested and proven succesful) cures.

It's worth noting that placebos do in fact relieve disease, to a greater or lesser degree, in almost any medical condition.

Money is a factor, but it can't be everything. Audiophiles are also occasionally known to add "cheap tweaks" to their systems, like rubber legs, green pens and the occasional sandbox. PWB's near-legendary tweaks (Ranbow Foil in particular) are really not that expensive, he's been known to give them out for free, and people still claim a considerable improvement.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #20 –
Quote
You can see the same process in other subjective fields, the muso that likes underground bands that nobody has heard of  and then goes off them as soon as their song appears in an advert and they get a number one single...
[a href="index.php?act=findpost&pid=297016"][{POST_SNAPBACK}][/a]

While that may be the case with some people, there is indeed a connection between creativity and "deepness" - and popularity. Many bands tend to also change their style to something more average when they get wider mainstream access - which of course perfectly makes sense: it is called mainstream because it IS "the average" - also in terms of music-taste. So getting more mainstream-attention usually (but not always) comes together with a more average musicstyle.

However, i agree with the point you're trying to make - i just think that the above analogy is not a good one.

- Lyx
I am arrogant and I can afford it because I deliver.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #21 –
Quote
It's worth noting that placebos do in fact relieve disease, to a greater or lesser degree, in almost any medical condition.


Yes, but that's why you test cures against placebos in the first place. If there was no placebo effect then you'd just test against people who did nothing to treat their disease and any comparitave improvement in those actually taking the drug under test would be good evidence that it works. But because of the placebo effect random wonder cures would appear effective even if they were sugar water or other inert substances. Only if it is more effective than the improvement attributed to the placebo can you assume that it is actually having any effect.

Similarly, a shiny new cable will make the music you hear sound better if you believe that it works, and conversely a cheap cable will sound worse to you if you assume that cheap equals low quality. But these effects are entirely in your mind if you can't prove they exist using standard scientific methods. If it works for cancer cures, it's good enough for speaker cables.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #22 –
Thats brings up an interesting controversial idea concerning "efficiency": If you can get the same effect with imagination alone, then does that mean that placebo and imagination is more powerful than technology? It may not be the case in every scenario - however, entire spiritual philosophies are built on mindplay and -discipline - and they've shown to be quite effective in using placebo as a tool.

I'm not saying that we should from now on buy crappy equipment and use crappy encoders - what i'm asking myself instead is: could it be that humans are striving for better technology, but are completely forgettin that in some domains imagination and "feelings" are more powerful than technology?

- Lyx
I am arrogant and I can afford it because I deliver.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #23 –
Quote
just think - audiophiles have yet to understand the concept of ABX after it's existence for ~30 years


No, we understand the concept quite well. But it's like the argument I've made in here several times and have been shot down for not ABXing absolutely every piece of audio in question. I've been told by several of you that I cannot objectively say that an mp3 encoded at 128kbps sounds worse then a CD or DVD audio. This is absolute rubbish. Anyone with good hearing and a decent audio system can tell the difference. One has a nice clean tight base with crisp high ends and the other is muddy sounding with swishy phasing issues.

I do agree that in certain circumstances ABXing is important and is needed to hear a difference and make an objective decision. When two pieces of equipment or two audio tracks are very close to each other as far as sound is concerned IE comparing a AC3 file to a DTS it is necessary to ABX. However, most of the people who have replied seem to think it is needed in absolutely every instance. I see a lot of very intelligent book smart people in here who have no idea that audiophiles can hear the difference and don't need to ABX everything to hear the difference and distinguish between great, good, and mediocre quality.

Let me ask you folks this, do you really think when engineers master a song that they have to ABX every track or instrument to figure out the best sound. No, of course not.

Atkinson vs. Krueger on blind testing. FIGHT!

Reply #24 –
Quote
I do agree that in certain circumstances ABXing is important and is needed to hear a difference and make an objective decision. When two pieces of equipment or two audio tracks are very close to each other as far as sound is concerned IE comparing a AC3 file to a DTS it is necessary to ABX. However, most of the people who have replied seem to think it is needed in absolutely every instance. I see a lot of very intelligent book smart people in here who have no idea that audiophiles can hear the difference and don't need to ABX everything to hear the difference and distinguish between great, good, and mediocre quality..
no one doubts that one can hear the difference btw. a 64kbps encode and the original. audiophiles and and ppl who don't see themselves being an audiophile can easily spot this.
however, if the bitrate gets much higher, or other factors like cables etc are involved, the claim to hear a difference becomes much more doubtful.
can you distinguish btw 256kbps MP3 and the original?! that would be great quality, wouldn't it?
ABX is also there to eliminate the placebo effect, which makes one think one can spot a difference. do you deny the presence of this effect?
Nothing but a Heartache - Since I found my Baby ;)