Are public listening tests constructed in such a way as to abstract the hardware setup?
Should we assume everyone at HA who gets involved with a public listening test has a "quality" setup, i.e. sound card with good DAC's, good speakers/headphones with good amplification, low-noise environment, etc. ?
Full Version: What about sound card hardware, drivers, speakers, etc?
From http://ff123.net/64test/criticism.html
I agree with the response.
QUOTE
2. Yes, I noticed that the test methodology involves having people send in subjective ratings based upon listening to some sound samples. Basing an opinion on something as enormously subjective as how people interpret sound, especially since they're doing so with non-standardized audio systems.
These surveys have absolutely zero scientific worth and are non-reproducable. If you want to read an article that uses good methodology and testing prodecures, go to http://www.r3mix.net and click on the analysis button.
Response:
[snip]...
Regarding non-standardized audio systems: The concern is that the results could be biased. The listening setups were not standardized, nor was any attempt made to perform a survey of what people were using to listen to the samples. While it might have been interesting to find out if people were using headphones or speakers, or if substandard (i.e., typical multimedia computer speakers) were being used, I think from a practical standpoint, it wouldn't have changed the test procedure or the analysis. How does one account for other things such as background noise in a test? This is as important a factor in being able to hear artifacting as the actual listening equipment. So is listener experience in hearing artifacting.
The most likely effect, in my opinion, of not having controls on the listening setup or on the listening environment, or on the actual listener, is not to add a bias, but rather to add statistical noise or to lessen the sensitivity of the comparison. In other words, I think that untrained listeners or people using cheap multimedia speakers would probably not be able to hear some of the artifacting that others hear, but their relative ratings of codecs would likely not differ radically from the experienced group of people using high quality headphones. Of course that's an unproven conjecture, but I think a reasonable one.
There's another reason why I think that the lack of controls on listener setup, environment, and training is not a fatal flaw: the actual results were statistically significant. I think the most reasonable interpretation of this is that the test was able to discriminate between codec quality in spite of, not because of, the lack of controls.
These surveys have absolutely zero scientific worth and are non-reproducable. If you want to read an article that uses good methodology and testing prodecures, go to http://www.r3mix.net and click on the analysis button.
Response:
[snip]...
Regarding non-standardized audio systems: The concern is that the results could be biased. The listening setups were not standardized, nor was any attempt made to perform a survey of what people were using to listen to the samples. While it might have been interesting to find out if people were using headphones or speakers, or if substandard (i.e., typical multimedia computer speakers) were being used, I think from a practical standpoint, it wouldn't have changed the test procedure or the analysis. How does one account for other things such as background noise in a test? This is as important a factor in being able to hear artifacting as the actual listening equipment. So is listener experience in hearing artifacting.
The most likely effect, in my opinion, of not having controls on the listening setup or on the listening environment, or on the actual listener, is not to add a bias, but rather to add statistical noise or to lessen the sensitivity of the comparison. In other words, I think that untrained listeners or people using cheap multimedia speakers would probably not be able to hear some of the artifacting that others hear, but their relative ratings of codecs would likely not differ radically from the experienced group of people using high quality headphones. Of course that's an unproven conjecture, but I think a reasonable one.
There's another reason why I think that the lack of controls on listener setup, environment, and training is not a fatal flaw: the actual results were statistically significant. I think the most reasonable interpretation of this is that the test was able to discriminate between codec quality in spite of, not because of, the lack of controls.
I agree with the response.
QUOTE(hellokeith @ Dec 29 2006, 04:51) 
Are public listening tests constructed in such a way as to abstract the hardware setup?
Short answer: no they aren't, at least as far as the tests conducted by HA users are concerned. Longer answer: that would be very difficult and expensive, but (AFAIK) some audio equipment manufacturers use ABX testing. I think I read that Infinity conducts ABX listening tests of their speakers against those of the competition in the same price class, in a controlled environment. Their listeners are trained beforehand.
QUOTE
Should we assume everyone at HA who gets involved with a public listening test has a "quality" setup, i.e. sound card with good DAC's, good speakers/headphones with good amplification, low-noise environment, etc. ?
As properly answered above, this assumption is not made, but that does not invalidate the tests.
Basically if you can detect an artifact in a statistically significant way, then it does not matter how good or bad your audio equipment is, or whether you have golden ears or not.
Can people with waaaaaay better equipment detect artifacts more reliably? I have no idea. I don't think this conjecture has been tested. I think I read that some tests have shown that people with trained ears can detect artifacts that others, untrained, cannot.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.