Hello.

Jan S. gave me the idea of running the AAC@128kbps listening test through Eaqual to see how close ODG results would come to the real results provided by listeners. So here they are:



The Pearson correlation to the two data sets is 0.698884. Quoting ff123:



If there is enough interest, I can do the same to a few other tests I conduced.

Regards;

Roberto.

For the uninitiated (like me . . .)

What is EAQUAL?

Interesting.

The correlation coefficient you calculated was for 60 values (12 samples x 5 codecs)?

What happens if you separate the data by codecs or by samples?

ff123

It was actually calculated on averages. (5 values)

Is it supposed to output a different coefficient if all the 60 values are used?



I will try that next.

I would think it's best to use the 60 values in calculating the Pearson r rather than the 5 averages.

For those who might not know, the Pearson r is a measure of how linear the relationship is between human ratings and EAQUAL ratings.

ff123

Also, I would expect the Pearson r to be somewhat better for the 64 kbit/s test because more of the 1-5 rating scale is covered. Correlation tends to be lower when the range of rating values are restricted.

ff123

Maybe it is a coincidence, but your data seems to indicate that for better codecs (according to the listening test) the correlation is lower.


Well, in fact it would be logical.

My humble guess is that it will be worse since Eaqual seems to be format biased AFAIC.
I recall vorbis giving positive values etc.

edit: that is why I suggested to test aac first.

Interesting Can you also post the results for the other contending codecs (of earlier test?)? If you have PEAQ and Opticom Opera, could you post their results as wel?

I can try to produce results later. It's quite a pain to do so because there's no way to automate the process - you must run Eaqual on each single test stream - and besides you have to detect offsets...

PEAQ is almost the same thing as Eaqual, with the difference that it is much slower.

And sure, I will use Opera if you buy it for me. Last time I checked, it cost several thousands of dollars

If you calculate the correlation based on the five averages, all you can say is that the averages have a correlation of 0.7 - it says nothing about the correlation between the underlying data sets.

For example, take the following two negatively correlated data sets, made up of 2 listening sessions:


so here the encoders are negatively correlated (approx -0.5) (ie a good sample for encoder A is a bad sample for encoder B), but if you work on the averages, they appear perfectly positively correlated.

Ok, I know, the data sets are totally contrived, but the point still stands that correlations (and many other statistics) are not valid if performed on summary data (like averages).

This is a minor statistical point, but so much of what we do here relies on stats that we might as well try to get it right

Anyway, love everything else you do,

regards,
Matt