listening tests help, for an audio codec I developed Options

[legg]

How can I rate how good or bad will the audio codec be on average?

I'm thinking on doing at least 21 ABX trials on each file and if the subject managed to tell a difference and the guess probability is low I'd also take into account a 1-5 point scale to measure the perceived quality.

So far I have analized one subject, he did pretty well on most tests, except one, where the results where these:
15 out of 29, pval = 0.500

AFAIK, p<0.05 is good to be certain that he didn't know by chance, what about the rest?
How do I interpret that?, is the codec transparent for him at that test?
What are the ranges of the guess probabilities and how should I interpret them?

Thanks in advance.

This post has been edited by Luis G: Jun 18 2005, 19:52

[Digga]

So far I have analized one subject, he did pretty well on most tests, except one, where the results where these:
15 out of 29, pval = 0.500

AFAIK, p<0.05 is good to be certain that he didn't know by chance, what about the rest?
How do I interpret that?, is the codec transparent for him at that test?
What are the ranges of the guess probabilities and how should I interpret them?

if you say he did pretty well except this one, the means that he could tell the difference right? for the codec it would be more complimenting (i.e. better) if he couldn't tell the difference which means it's transparent for him and he's guessing if x is a or b.

anyway, either alpha=0,01 or alpha=0,05 are generally chosen.
btw, you are not certain, but this is just (given, very low) probability that one is guessing, i.e. considered not guessing.
a pval of 0.5 would be a reference for guessing.

for further more in depth info, look here.

edit: damn am I slow...

This post has been edited by Digga: Jun 18 2005, 20:29

[legg]

if you say he did pretty well except this one, the means that he could tell the difference right? for the codec it would be more complimenting (i.e. better) if he couldn't tell the difference which means it's transparent for him and he's guessing if x is a or b.

anyway, either alpha=0,01 or alpha=0,05 are generally chosen.
btw, you are not certain, but this is just (given, very low) probability that one is guessing, i.e. considered not guessing.
a pval of 0.5 would be a reference for guessing.

for further more in depth info, look here.

edit: damn am I slow...

Yes, I have read that thread, but it doesn't mention how to deal with the results when p>0.05, which is this case. An interval like 0.05<p<0.25 how is to be dealt with? It certainly doesn't say much of the codec, and I would hardly classify it into transparency. IMO p>0.5 means transparency, but I wanted to check with you gurus about this.

I thought that anyone could rate the quality, but what if the subject merely rated the correct file by chance, I need more certainty, and I'm using ABX as an indicator of the trustworthy of the subject ratings. Is this correct?

Btw, I'm more interested in rating the subjective quality of the codec.

Thanks again.

This post has been edited by Luis G: Jun 19 2005, 16:46