listening tests help, for an audio codec I developed Options

[legg]

How can I rate how good or bad will the audio codec be on average?

I'm thinking on doing at least 21 ABX trials on each file and if the subject managed to tell a difference and the guess probability is low I'd also take into account a 1-5 point scale to measure the perceived quality.

So far I have analized one subject, he did pretty well on most tests, except one, where the results where these:
15 out of 29, pval = 0.500

AFAIK, p<0.05 is good to be certain that he didn't know by chance, what about the rest?
How do I interpret that?, is the codec transparent for him at that test?
What are the ranges of the guess probabilities and how should I interpret them?

Thanks in advance.

This post has been edited by Luis G: Jun 18 2005, 19:52

[ff123]

If it were me, I'd just choose a whole bunch of different samples (say 30), and then rate each one against the reference using abc-hr. Then I'd plug the results into a statistical calculator (http://ff123.net/friedman/stats.html) to determine first if you found a significant difference from the reference and second how much that difference is. No ABX'ing is involved, plus you get a better indicator of codec quality by sampling a lot of different music.

BTW, I would also keep the samples where you rate the reference, rather than throw these cases out. So if you make some mistakes, the reference will average something less than 5.0.

ff123

[guruboolez]

If it were me, I'd just choose a whole bunch of different samples (say 30), and then rate each one against the reference using abc-hr.

I'm going off-topic, but I've an important question.
I'm trying to build a complete set of classical music sample, in order to replace the usual suit of 15 samples I'm using now for 18 months. My purpose is to obtain 100 samples, including many instruments, solo, chamber, orchestral, lyrical, noisy or not noisy, quiet and loud, etc... But I'm realizing that making ABX comparisons with so many samples would be a Herculean task.
What would be the best thing in your opinion:
- 100 samples rated in ABC/HR without ABX
- 15...20 samples rated in ABC/HR + ABX confirmation?