I made few first abx tests of mine on same track from differing codecs. encoded not by me. I used foobar2k for abx, and frankly wasn't able to distinguish between them at all. Yet I was getting upto solid 4/4. Then I was amused and thought heck lets try how far I can go that way - I took off my phones and simply tried to predict what the next A/B position would be. And guess what? I got to 11/11 before I made first mistake. uh-huh?

What does it mean? Luck? As it stands, I lost all confidence that ABX testing shows anything at all... or I should be making at least 30+ tests to rule out my ability to intuite next program generated random position?

IIRC Garf managed to do the same thing once.

Try the same e.g. 10 times. If you get 11/11 again you should seriously consider earning your money by playing roulette.

If you just click randomly, you'll get a 95% significant result 1/20 times (on average).

This is why having samples is important with ABX result - the results must be repeatable.

thats probaly why they call i 95% as there is a 5% chance you are guessing....

also, i notice that foobar2k tends to favour "x is a" more than "y is a".

but if you listen to the samples hard enough it's not a significant problem i think (at lease 1/20 times, but that's why you should keep a log of the test and say "i'm just guessing and can't hear a difference" for test where appropriate)

of course you could repeat the test and discard results, but that goes against statistical methodology.

No.