http://online.wsj.com/article/SB1200446920...tml?mod=djemITP

Of course, it's a rather silly but fun article. It looks like for most of the comparison, each subject did just one trial. Three trials at most.

For the mp3 vs CD trial, for example, 18 our of 24 people preferred the CD in what seems to be a single trial each. How to measure the statistical significance of that?

No details in the test what encoder was used for those mp3's. Not even bitrate.

Well, it's good that he didn't post those results here, as he would have been considered quite incompetent
But it seems ok for the Wall Street Journal.

Already quite impressive that A-B was used.

Now since they have all the comfy setups they can create, they should increase the number of trials, yes. There shouldn't be any more claims of listening fatigue.

If the goal is to check what the average human can do, it does not matter if one person takes 10 trials or 10 persons take 1 trial. In fact the latter is better because it takes a bigger sample of humans.

18 out of 24 is a highly significant ABX. The question is of course if "three-quarters" really means 18
It is no surprise that poorly encoded or very low bitrate MP3's are distinguishable from a CD.

When talking about the test of speaker cables, he reports 61% of 39 people correctly identifying the cable. But that is actually not a statistically significant result.

Well, significant , yes -- my bino_dist table says 18/24 has a p = .011 -- but that score is 'green' (good) rather than 'yellow' (great) on the table.



Yes, but he wrote that they were 'high-quality' MP3s. Albeit with no explanation of what that means.
I've sent him an email asking what he meant.

(Also, 3/4 * 24 = 18, am I way off here?)




But two of them were were John Atkinson and Michael Fremer! How can you disbelieve?

That's great and everything, but how many listeners found cable A to be more danceable?

surely a sample size of 24 is too small to be able to draw any 'real' conclusions, even if the result is significant?