What's the problem with double-blind testing? Options

[mirrorsawlljk]

I happened to pick up an issue of stereophile at a record store I visited and I was pretty shocked to see a seemingly intelligent person in the correspondence section bashing double blind testing as being unreliable. I'm afraid I don't understand his angle of attack. I don't see how anything could be a more reliable test of sound quality differences than a properly conducted double blind listening test.

I'm almost afraid to read the rest of the magazine if this is the kind of letter they think is worth publishing. Is there an audio magazine that isn't filled with this kind of thinking?

[onthejazz]

Interesting publication. I like it, too bad its not in full production anymore. Thanks much for the link. Anybody have any more quality sources?

[Donunus]

The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.

[Danimal]

The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.

The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear. The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode. That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.

[Donunus]

The writer of the letter in stereophile is incorrect in saying double blind tests are unreliable but he has a point. I do abx tests in foobar to compare different files a lot but there just are too many songs to abx to make sure all of one formats files are as good as the other ex. mp3 vs aac vs wav, etc... The thing I always notice though is that when using the analytical side of the brain(abx testing) only certain parts of the music is being focused and the emotional side of the music isn't being analyzed, therefore the goosebump factor of the sound the way the music delivers emotion cannot be analyzed! Its very hard to have the left and right side of the brain working at the same time.

My Example for this is when doing an abx of one particular song, I passed the test in foobar with flying colors(100% on abx test) but thought the differences were not really significant enough to make me keep the wav files from the cds and kept only the mp3s. After a few weeks of listening to the mp3s and really getting to know and love the songs better, I tried just popping the cd in for a listen... wow, goosebumps.... The parts I loved in the songs i was listening to gave me goosebumps for the first time. Now thats music! The feeling is lost on some parts of the mp3. And that was with my pc using sennheisers, not even my high end home rig.

Its not totally reliable to trust abx testing for determining the enjoyment one gets from his music because human memory can only remember only a certain number of seconds at a time like in an abx test. Its good for making codecs for eliminating artifacts but abx testing a few seconds of one song does not determine the accuracy of the dynamics of the whole piece. All the buildup of sound, the emotion! Bash me now but I have been an audiophile for more than 15 years and I am only 31, and have been a music lover since 5 years old. I know what I'm talking about. Listening to a bad stereo system, I can hear differences between different files for example but I don't really tend to care whether I am listening to the mp3 or the wav version. But with a very involving music system, the mp3 can sometimes put a damper on the fun factor of the song.

Also, stereophile loves posting "gray area" letters. I love that magazine. They exaggerate things sometimes but overall they are very accurate the way they subjectively describe the audio quality of equipment.

The point of double-blind testing is to make sure that you're getting goosebumps over something you actually hear, and not something you expect to hear. The whole point is to try to evaluate what you're hearing without knowing whether it's the original or the lossy encode. That shouldn't remove the "emotional" part of it, just make sure that you're reacting based upon something that is really there, and not what you want to hear or what you expect to hear.

Well, with mp3, even 320. I have passed foobar abx with many samples from my own cds compared to the original with flying colors. What I am saying is I cannot get goosebumps while I am in the analyzing mode cause I am not really listening to the music but to different aspects of the sound while doing the abx test. I do hear the differences and although they are not really that big from an analysis standpoint, they become bigger when listening for enjoyment. Some songs will just lose some life when encoded to mp3.