From Pio2001's very interesting post on designing a test to determine the audible effect of 24/96:

Is this really necessary? Sure, I agree with you completely that good quality random numbers are necessary to ensure a good test. Basically, my question is this: is using real random numbers (as opposed to PRNG generated "randomness") necessary to ensure that a test is valid?

I would expect that the output of a good PRNG, seeded with a decent seed, would be random enough that knowledge of the pattern of the random numbers would not bias a test. For example, the Mersenne Twister algorithm with the parameters set to decent values is well distributed in the first 623 dimensions. I haven't seen any evidence that a human would do better than chance at predicting the next value in the line. While lower quality PRNGs could be a problem - good quality PRNGs are widely available with several free implementations available. One good implementation is from Boost.

If there is a concern about PRNG derived values, then high quality randomness can still be obtained more easily than by throwing dice. www.random.org would be one source (and there are more on the web). A hardware RNG would also be easier to use in the test environment than dice throwing.

???
It must have been a joke. I mean, it's true that for good random you need a source from outside the computer... But dice? Talk about low tech, not to mention arduous. Anyways, if your random is good enough for crypto, it's good enough for an abx test.

Actually, I think his post was in the context of manually testing outside of a computer, with hifi equipment. So the person running the test would need have a random source... Though it still might be easier to pre-generate a random sequence ahead of time on the computer and print it out. Anything you can do to minimize the time between samples of the test is a good thing.

...
Random.org is pretty cool. There's also lavarnd, but they don't use a real lava lamp anymore so they're not nearly as cool as the old sgi lavarand was. That was pretty funny when I first saw it, back in the days when webcams were the in thing.

that was most certainly a joke (or he was drunk). you don't really need random numbers there, the only thing you'll have to do is reorder sample so that no one can figure out a pattern, a cheap pseudo random generator (even the craptacular rand in unix, which is totally predictable if you observe enough rolls) will do fine.

the only time you need real randomization in statistics is when you have a huge pool of samples, and you pick randomly out of that list, like 100 or whatever samples you'd like to test, this way you ensure that different characteristics of sound are captured evently (from classical to electic) even characteristics you are not aware of will be equally picked by this method. Nothing else needs real randomization.

once you picked the samples for your test, their ordering (though probably should change to avoid placebo) doesn't have to be random.

In hifi hardware tests, we don't have computers at hand.

Choosing ourselves a given sequence is something to avoid. The human thinking have some preconceptions about randomness that makes some choices more probable than other.

The use of dices was suggested in order to avoid coin flipping, that can be completely biased according to the way you flip it. For example, if you stand the coin slightly bend backwards and make it turn with a flick, most of the times, it will fall on the side that was in front of you. If then you just pull it from its prevous position, you get a mostly constant sequence of results from coin flipping.

This one depends a lot on how you use it. For example, some implementations of rand() use a generator from Numerical Recipes in C that alternate between odd and even (that is, the LSB alternates between on and off), the second least significant bit also has some pattern to it, with a period of about 30. A bad implementation that uses these bits (like X=(rand() & 0x1) would easily bias a test. Individual bits from these sorts of generators also exhibit a fair amount of skew (either 1 or 0 is more likely than the other), making either A or B more likely. While skew isn't likely to produce a false positive, it does mean you have to calculate p differently to get the right answer.

On an aside - Von Neumann's transition mapping algorithm for removing skew is one of the cleverest algorithms I have ever seen.
Ok, thanks, I understand now. I originally thought you were making a profound statement about the pattern prediction ability of test subjects

If you were really obsessive you could use casino dice and a new die for each roll.