Jeff Atwood's "Great MP3 Bitrate Experiment", From the Coding Horror blog. Options

[kinnerful]

http://lifehacker.com/5920793/the-great-mp...rate-experiment
http://www.codinghorror.com/blog/2012/06/t...experiment.html

I guess lifehacker has a larger audience than hydrogenaudio... could be interesting

[mzil]

You shouldn't cherry pick raw data under any circumstances in a properly unbiased, double blind test. It makes the test suspect, regardless of the test conductor's intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then one needs to conduct a fundamentally new test, not discard raw data one "believes" to be compromised.

[In different circumstances I'd accept using one test in an attempt to find certain "gifted" test subjects ,who are then retested, however. This could be used, for instance, to find "golden-eared" listeners.]

This post has been edited by mzil: Jul 3 2012, 18:40

[db1989]

Disclaimer: meandering musings

You can't cherry pick raw data under any circumstances. It makes the test invalid regardless of your intentions, good or evil. If there was poor wording or a misunderstanding in the instructions, then you need to conduct a fundamentally new test, not discard raw data you "believe" to be compromised.

I don’t disagree in principle. Hail science! I was just pointing out that, however scientifically tenuous it might be, excluding data because they were submitted in the wrong format is not exactly equivalent to excluding data because they aren’t conducive to someone’s ulterior motive(s). At the very least, it’s not equivalent ethically: one is done in an effort to improve the reliability of a conclusion, whereas the other is done merely out of cynical self-interest.

Scientific ethics aside (just for a moment! ), is such filtering of incorrectly calibrated data even likely to be possible in any real-life study with any probability of preserving its objective reliability? I lack the experience to answer either way, and I suspect that it’s better avoided anyway due to the same concerns that you’ve raised – but in this case, I don’t think it’s very likely that one could do it. That was what I meant by my closing sentence, although I should have given it more consideration.

Of course, as you implied, this question should never arise: collection of data should be designed so as to preclude any of them being ‘incorrect’ or ambiguous. In this specific case, the take-home message is that instructions must be clear and unambiguous, so that respondents can provide useful data. It’s a shame how this test is somewhat marred by its shortcomings in that area and, as I said, how this confounding factor can’t be removed post hoc.

Disregarding all listeners who rated WAV as less than 5

Since you’ve just reminded me of something I wondered about earlier: how about disregarding all respondents whose data sets included each number only once? Or am I getting desperate here?