Jeff Atwood's "Great MP3 Bitrate Experiment", From the Coding Horror blog. Options

[kinnerful]

http://lifehacker.com/5920793/the-great-mp...rate-experiment
http://www.codinghorror.com/blog/2012/06/t...experiment.html

I guess lifehacker has a larger audience than hydrogenaudio... could be interesting

[Canar]

Data are data. If there has been some kind of procedural error and it's not feasible to re-run the experiment, it's entirely legit to restrict your data down to the valid subset, if there is some easy way to do so. If, due to some error, only 10% of your data are actually valid, and you can identify that 10% post hoc, there is no reason not to analyze that 10%. It might redeem the entire experiment.

Ideally, yes, you re-run the experiment and try ensure that 100% of your data are valid. This is not always feasible, nor should it be absolutely required.

[mzil]

If there has been some kind of procedural error and it's not feasible to re-run the experiment, it's entirely legit to restrict your data down to the valid subset, if there is some easy way to do so.
...

Huh? I suspect you don't really mean this, unless I am just completely mis-reading it. The feasibility or ease of re-running a test doesn't make a difference as to the legitimacy of the original test.

To paraphrase what you have written, one could say "If it is difficult to re-run a test, then we should accept at least the subset of the data that we believe wasn't compromised, due to the known error", [as long as we still have a large enough sample left over to make the results statistically significant, I guess]. "If it is easy to re-run the test, however, then the original data is suspect, should be ignored, and we should do the re-test."

The difficulty in re-running a test, but this time without the design flaw, doesn't change whether the original test data is legit or not. It either is or it isn't, regardless of the time needed/ease/difficulty in conducting a new test without the design flaw. Right?
---

"Cherry picking" is a type of confirmation bias, more accurately called a "fallacy of suppressed evidence" and may very well be unconscious in nature, despite its sinister sounding name. I wasn't, however, trying to speak poorly of anyone here or question their motives, but I seem to be alone here in thinking that claims of "pure and unbiased" motivation, which of course all scientists think applies to them , doesn't suddenly make cherry picking "acceptable". Everyone thinks their selection process is "sound, pure, and motivated only by the unbiased pursuit of truth".

As it says here, one's motivation may indeed be pure and honest, but the fallacy name, even if not a very good name, still applies:

"If the relevant information is not intentionally suppressed by rather inadvertently overlooked, the fallacy of suppressed evidence also is said to occur, although the fallacy’s name is misleading in this case. The fallacy is also called the Fallacy of Incomplete Evidence and Cherry-Picking the Evidence.."

I unfortunately don't have any more time on my hands to devote to this, so I'm outta here.
Happy July 4th everyone!

This post has been edited by mzil: Jul 3 2012, 23:35