Hey Ivan, just started doing your AAC vs. MP3 @128 Test #1. For those who dont know, below is the test announcement. But, anyway, it seems that 16 samples to evaluate (12 different) is a bit too much even for me to evaluate "correctly"..
Are you really going to include this many samples in test #2 also and 30sec samples?
How many results you have got so far?
Just did the first part of the test, second one to go...

Maybe I will reduce number of codecs to 6, with hidden reference and anchor. Not yet decided

I've god 3 reports so far - results differ for most codecs except for one codec that is ranked 'excellent' by all testers

Of course, these results can't be taken as accurate unless at least 30 people submit results. After that, we will perform statistical analysis and see what the real results are...

Hmm, pretty long way to go until 30 people has done it..

Program is working nicely, but it´s sad you had to drop the encryption of output files for now. Any idea when you could implement it again?

As soon as I figure out what went wrong with the encryption

Have you looked into using pgplib? It's portable and easy to implement. I know the libmcrypt works well, but I don't know how portable to Win32 it is.

I've done some integration of encryption in the past, you can talk to me if you need any pointers. I had once to write a non-standard blowfish implementation from scratch to talk to someone elses non-standard blowfish implementation (register.com). =)

I'll take a look at pgplib - actually, I used MD5 encryption from Win32 API (CryptoAPI) but it worked on my computer only...

Thanks,
-- Ivan

Well, I peered into the test a bit (after I took it and submitted my results, of course). I'm not exactly sure how this will be best analyzed. Probably the most conservative method will be to treat the two parts as two separate tests. Let's just say I was concerned right after I took the tests about the coherence of the results as a whole, but was encouraged some after I looked at the setup.

I may look into supplementing the Friedman analysis program I wrote with an ANOVA analysis option. ANOVA does make certain assumptions, such as normal distribtion of the listening panel, and equal-interval rating scale, but is more sensitive than the Friedman in return. BS. 1116-1 recommends ANOVA whenever possible over non-parametric methods like the Friedman.

ff123

Hmm... deviations between listeners are quite big.

For example, some of them are ranking codecs in range of 1.0 to 3.5 and others are ranking them in range of 3.0 - 5.0

I have collected 7 results up to date. Maybe we will have more accurate results after 30 results are submitted.

ff123 - what do you suggest for the statistical analysis?

Yeah. Some listeners are more sensitive than others.



You can run my friedman.exe program on it even now. The conclusions it pops out right now, though, will be the obvious ones.

ff123

I downloaded wavrate and tried the listening test, but it crashed when I opened either of the projects and kept doing so each time they were invoked. Sorry.

You have to unpac the files first. At least this was the reason why another person experienced a crash.

ff123

Yup, you are correct ff123. It seems to work fine now, only problem being I can hardly hear any difference whatsoever between the samples. Luckily my mother could hear a few but after 5 samples she also became tonedeaf and couldn't hear the differences anymore either. So I dunno if my results would be representative enough, at least if you're going for a "Is this audiophile transparent?" measurement. I'll try to get my results in though, I wanna do something in return for Ivan's excellent work.

Ok, I submitted my results also. Well, in the end that wasn't too much, since I evaluated part2 quite quickly.

So it's not so awful job.

But this number of clips and Archival Quality -test would take a long time...

Nice utilty. I had to figure out I had to decompress the files for it to work too though

Not having to ABX makes it much easier and faster to do tests
like this. On the other hand, it may mean that my results are essentially random (of course I don't _think_ they are).

Well, you'll see in the results I guess.

--
GCP

Ivan,

How is the test coming along?

ff123

Ok, castanets test is over and it seems that FhG AAC is #1 (as expected), PsyTEL AACEnc is #2, etc..

I will post results later today, and prepare test #2

...CASTANETS 128 TEST...

FRIEDMAN version 1.10 (Sept 29, 2001) http://fastforward.iwarp.com/
Friedman Analysis

t1 t2 t3 t4 t5 t6 t7 t8 person s1 s2 s3 s4 s5 s6 s7 s8
2 0 3 0 0 1 1 1 1 4.0 7.0 1.5 8.0 4.0 1.5 4.0 6.0
1 1 1 2 0 3 0 0 2 7.0 1.0 4.5 7.0 2.0 7.0 3.0 4.5
1 1 2 0 1 1 1 1 3 7.0 1.0 3.5 8.0 6.0 5.0 3.5 2.0
1 1 1 1 1 1 1 1 4 4.0 2.0 6.0 8.0 5.0 7.0 1.0 3.0
1 1 2 0 2 0 1 1 5 1.0 7.0 5.5 8.0 3.5 5.5 2.0 3.5
1 1 2 0 1 1 1 1 6 3.5 6.0 5.0 8.0 3.5 7.0 1.0 2.0
1 1 1 1 1 1 1 1 7 6.0 5.0 2.0 8.0 4.0 7.0 3.0 1.0
1 1 1 1 1 1 1 1 8 7.0 5.0 4.0 8.0 1.0 6.0 2.0 3.0
1 1 1 1 1 1 2 0 9 6.0 2.0 7.5 7.5 1.0 5.0 3.0 4.0

Input filename: results1.txt
Samples compared:

s1: Commercial AAC A, ranksum = 4.550E+01
s2: FhG IIS MP3Enc, ranksum = 3.600E+01
s3: PsyTEL FastAAC 2.0b, ranksum = 3.950E+01
s4: FhG IIS Reference AAC, ranksum = 7.050E+01
s5: LAME --nspsytune, ranksum = 3.000E+01
s6: PsyTEL AACEnc 1.2, ranksum = 5.100E+01
s7: FhG IIS FastENC, ranksum = 2.250E+01
s8: LAME 3.90, ranksum = 2.900E+01

Number of listeners: 9

Significance of data: 7.170E-05 (highly significant)
Critical significance of Fisher's LSD analysis: 5.000E-02
Fisher's LSD for rank sums: 2.037E+01

The following comparisons are each true with 95.0 percent confidence:
FhG IIS Reference AAC is better than Commercial AAC A
Commercial AAC A is better than FastENC
FhG IIS Reference AAC is better than FhG IIS MP3Enc
FhG IIS Reference AAC is better than PsyTEL FastAAC 2.0b
FhG IIS Reference AAC is better than LAME --nspsytune
FhG IIS Reference AAC is better than FastENC
FhG IIS Reference AAC is better than LAME 3.90
PsyTEL AACEnc 1.2 is better than LAME --nspsytune
PsyTEL AACEnc 1.2 is better than FastENC
PsyTEL AACEnc 1.2 is better than LAME 3.90


Ok, 96 kbits/s results along with ANOVA will follow-up shortly...

Too bad only nine listeners.

Which wav files corresponded to the various encoders?

ff123

Edit: here is the output as formatted by friedman.exe version 1.20:

http://ff123.net/export/aac128log.txt

Hmmm. How do I do pre-formatted editing on this forum? [pre][/pre] doesn't seem to work.

Edit by Dibrom: The correct tag is [ code] [/ code] of course without the spaces. Anyway, here you go:

Ah.. yes...

5006933F-DCCD-4c82-A7EF-E9E9BE22A78D is the WinZip password for the http://www.psytel-research.co.yu/downloads...s_codecinfo.zip

:idea: Ummm, where can I get a hit of that stuff? (Fisher's LSD) Is it blotter?

Hrmm, I'm not actually sure if it is possible or not. I'll have to check around, but if not I can just create a new tag which allows this behavior.

hi, ivan

when do you want to end the test?

now, do you only require the results for test1?


regard.
yan.