Minimum number of required ABX trials, Split from from topic ID: 92851 Options

[sauvage78]

The minimal number of trials depends on how successfull you are.
How quickly you are sucessfull shows how confident in yourself you are.

The time & number of sucessfull trials are tied, you should never separate them when judging an ABX log.

With F2K ABX component, 8 sucessfull trials in a row (& if all successfull in a row, it usually means quick) trials is the minimum for me.

As soon as you begin to fail you can easyly increase to 10 or 12 to try to "erease" your failures.
In this case if you fail once or twice you can usually still get a signifiant result although it usually means the ABXing was hard, & by consequence longer as you begin to hesitate.

Usually if you begin to fail more than 3 times on 12 trials, it begins to be so hard & you have so much hesitation that it begins to take forever to ABX. At this stage I usually give up by myself & declare that I cannot ABX as in general it means I am not sure that the audio part I am focusing on actually contains any real artefact.

This post has been edited by sauvage78: Jan 12 2012, 00:42

[sauvage78]

Well there is the theory & there is real life practice. I agree that chosing the number of trials before the test & chosing a higher number of trials to avoid randomness is better ... but just try to organize a real test with several samples, several encoders (& maybe even several bitrates) ... you will quickly realize that the time it takes goes exponential very quickly specially as the part when you actually decide what to test also takes time & usually goes unoticed for beginner who read the test ...

To be short theorical perfection is indeed better, but time is the ennemy. If you cannot take the necessary time to ABX correctly, you'd better prevent yourself from making any quality claims.
That's why serious ABXing on a large scale is rare.

It''s been years that I wanted to re-ABX Apple AAC & CELT & so far I have always been unable to run the test as it would need 3 days in a row of work, which is discouraging as I use 100% lossless anyway.

This post has been edited by sauvage78: Jan 12 2012, 01:01