Alexander Lerch from HHI/ZPlane released perceptual quality measurement tool under GPL! Get it at:

http://sourceforge.net/projects/eaqual/


Here are some ODG (Objective Difference Grade) results, difference from the original (1 to 4), for castanets encoding:

Psytel AACEnc 2.0 @ 128 Kbits/s: -0.6833
Psytel AACEnc 2.0 @ 96 Kbits/s: -1.3516
Fraunhofer AACDemo @ 128 Kbits/s: -0.7383
LAME --nspsytune @128 Kbits/s: -0.9899
MP3Enc -qual 9 @128 Kbits/s: -0.7479

Remeber - this tool can't replace listening tests, but it could help in codec quality testing and development!

Wow, great news!
OGG RC3 and this, nice way to start the year !!!

Ivan,

Neat. What does psytel 1.2 score at 128 kbit/s? You could compare it against what your listening test said. For that matter, I could compare notes using dogies.wav and wayitis.wav to see if it matches up reasonably well.

ff123

I haven't tested it yet with PsyTEL 1.2, but others that wish to use this tool I must give some more information

- Offset value is essentially important, wrong offset leads to very wrong results

Some of the offsets:

Old FAAD + PsyTEL AACEnc (all versions): 2048 (1024 samples)

LAME --decode + LAME MP3: 64 (32 samples)

Lame --decode + FhG MP3Enc: 1172 (576 samples)

Old FAAD + FhG/Dolby AAC (hehe, if you have them 4392 (1024 + 1172 samples)

Etc...

I couldn't get eaqual to work with new FAAD - probably I wasn't too careful, offset might be different...

Anyway - a good way to find offset if you don't know encoder charateristics is:

- Encode castanets sample in very high quality VBR
- Open it in CoolEdit
- Setup the time slider to "samples"
- Identify first peak, and set the pointer right in the middle of the peak in both samples
- Subtract the sample values

If you were careful enough, the delay will be +/- 20-30 samples, and find first "expected" number (512, 576, 1024, 1172 and multiples) - multiply it by 2 and that is the sample offset.

Now, try it with the EAQUAL tool, and the resulting ODG should be in the range of -1.0 - 0.0 for high quality VBR presets.

Could anyone compile it for me...? I can´t seem to get it to work correctly with VC++ 6.0...

I get this with death2.wav:

c:ProgsEncodersEaqual>eaqual -fref d:testgeluidendeath2.wav -ftest d:death
2.wav -offset 2048 -srate 44100

EAQUAL - Evaluation of Audio Quality
Version: 0.1alpha
Author: Alexander Lerch, zplane.development
Copyright: Heinrich- Hertz-Institut (http://www.hhi.de)
zplane.development (http://www.zplane.de)
_______________________________________________________
Reference File: d:testgeluidendeath2.wav
Test File: d:death2.wav
Sample Rate: 44100
Number of Channels: 2

Press Escape to cancel...

Frame: 806
Time elapsed: 41.39

Resulting ODG: -3.9122
Resulting DIX: -4.1096

BandwidthRef 21521.9414
BandwidthTest 16249.1914
NMR 41.9918
WinModDiff1 163.0067
ADB 3.1234
EHS 1.4913
AvgModDiff1 200.9328
AvgModDiff2 573.1494
NoiseLoud 53.7488
MFPD 1.0000
RDF 0.9988

Is there something wrong in my command line?

Edit: I forgot to mention that the test file was encoded with Psytel aacenc 2.0 and decoded with a recent version of FAAD.

I've uploaded a Win32 binary here:
http://home.wanadoo.nl/~w.speek/download/Eaqual.zip

Thanks a lot...

Hmm... As the read-me file says, I thought the possible ODG values were from 0 to -4, so how do you explain a result like this? :confused:

Frame: 8339
Time elapsed: 340.78

Resulting ODG: 0.0655
Resulting DIX: 3.2649

BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000

(Encoded "Ace of base - Wave wet sand" with Lame 3.90 stable using "--abr 256 -q 0 -m j", and I also decoded it with Lame...)

Yes - I can't seem to find proper offset for latest FAAD! Please use older FAAD (I use build from September 2001) or ISO AACDec - their offests are 2048.

I will see with Menno to find proper value for latest faad...

Doesn't the readme say best ODG would be 0.0 which means identical file? I had -0.05 for MPC -xtreme, but I didn't checked for correct offset so it's unlikely it's a valid result.

Is there no easier way to find out the offset?


bye, Skeeve

I think that MPC has no delay, because it is using subband filtering without one frame look-ahead for block switching, but I will have to recheck..

I will do my best to find offsets for most popular codecs, and publish those values..

Ivan, I think that LAME you used is a bit old. I found no offset when comparing the original wav to the LAME decoded wav in CoolEdit (both had the first positive peak at sample 7905) and then I looked carefully at the decoding screen display where it mentions that the offset was removed:


c:lametest>lame3902.exe --decode castanets_test.mp3 castanets_test.wav
input: castanets_test.mp3 (44.1 kHz, 2 channels, MPEG-1 Layer III)
output: castanets_test.wav (16 bit, Microsoft WAVE)
skipping initial 1105 samples (encoder+decoder delay)


I think this offset stripping has been there for a while. I got the following ODGs with LAME 3.90.2 @128kbits/s on CASTANETS.WAV:


Lame3902 --abr 128 -h test.wav (136.4 kbps; qval=2) = -.5301
Lame3902 --abr 128 test.wav (135.6 kbps; qval=5) = -.5492
Lame3902 -q0 test.wav (qval=0) = -.7132
Lame3902 -q1 test.wav (qval=1) = -.7429
Lame3902 -nspsytune test.wav (qval=5) = -.7732
Lame3902 -h -nspsytune test.wav (qval=2) = -.7712
Lame3902 -h test.wav (qval=2) = -.7302
Lame3902 test.wav (qval=5) = -.7649
Lame3902 -f test.wav (qval=7) = -.7822


Some possible conclusions:
1. -q1 is inferior to -q2 (I don't think its use is recommended anymore).
2. If -nspsytune improves the perceived quality them this program penalises it.
3. ABR measures much better than CBR

Actually, the offset is still there... It doesn´t remove all of it...

Look:

[b]Offset 64:

Frame: 8339
Time elapsed: 340.78

Resulting ODG: 0.0655
Resulting DIX: 3.2649

BandwidthRef 21536.1094
BandwidthTest 20665.5117
NMR -20.0063
WinModDiff1 4.7231
ADB -0.3251
EHS 0.1606
AvgModDiff1 4.7216
AvgModDiff2 6.3346
NoiseLoud 0.0898
MFPD 1.0000
RDF 0.0000

Offset 0:

Frame: 8339
Time elapsed: 311.29

Resulting ODG: 0.0856
Resulting DIX: 3.4091

BandwidthRef 21536.2383
BandwidthTest 20665.4160
NMR -21.2534
WinModDiff1 2.6592
ADB -0.4966
EHS 0.1676
AvgModDiff1 2.5517
AvgModDiff2 3.5112
NoiseLoud 0.0434
MFPD 0.9642
RDF 0.0000

I also got a positive ODG with Ogg Vorbis RC3 -q 6 on ftb_samp.wav (ODG = 0.0228). I'm pretty sure that MPC and Ogg both have zero offset. I did the test with CoolEdit that Ivan described. These occasional positive ODG make me doubt the results of Eaqual. Would be nice if somebody had an explanation.

I'll talk with Alexander about that issue..

I've noticed that with PsyTEL VBR modes, too

Anyway, Menno fixed FAAD so now it decodes with zero offset, too! I am not sure when will he upload it to CVS, but expect that to happen soon..

-- Ivan

Here is what alexander told me regarding positive ODG:

Just for giggles, and because I love Ogg Vorbis , here's some more ODG values for castanets.wav:

OggEnc RC3 @ -q 3.00: -0.7350
OggEnc RC3 @ -q 3.50: -0.6381
OggEnc RC3 @ -q 4.00: -0.5918
OggEnc RC3 @ -q 4.50: -0.5131
OggEnc RC3 @ -q 5.00: -0.3051

Ogg should have a zero offset, IIRC. Decoding was done with WinAmp 2.78 w/ Ogg Vorbis Input Plugin 1.17 beta, and Nullsoft Disk Writer Plugin 2.0b.

Now we can see that ogg is the best

What kind of ODG values do you get using common settings/presets (standard to high quality) with LAME, MPC, AAC and OGG?

I posted some results here:
http://home.wanadoo.nl/~w.speek/eaqual.htm

hehe only ogg sounds better then the original
the only positive value in speeks tests

Try Lame with "--abr 256 -q 0 -m j" and you´ll see some more posivite values... So in essence, harmonic distortion should make the ODG values positive, or am I way wrong here? (I´m kinda´ new to this you know... )

Well, like I said - this tool could be used for codec debugging, and not for final marking of which codec sound better

For example - this codec ranks FAAC castanets @ 128 with ODG of -0.8 , and I am sure that even for ISO listener FAAC is still below "excellent" range at 128 kbps, etc..

I'm not sure if using --scale is such a good idea for this test (changes the amplitude vs. the original). However, it may not have a big impact on the results.

ff123

Here are the results with your 128 kbps command-line, but without --scale 0.93:
ftb_samp: -1.2156
sade__sweetest_taboo: -0.9003
iron: -0.8857
fools: -1.0177

ftb_samp and sade are slightly better with --scale 0.93. Iron and Fools are about the same. I didn't try the --alt-presets with --scale 1. Maybe later.

What is the offset for FhG. Fastenc (CoolEdit Pro filter)? With the test Ivan described I think it's about 96 (that's 192 for EAQUAL) when decode with LAME, but when I put an higher offset number in EAQUAL the ODG gets better. Wouldn't it be logical that the offset that gives best ODG is the right offset :confused:

Here's some more ODG values for Speek's samples, except one, because it's smarter than me.

(Speek, I can't decode sade_sweetest_taboo.pac. LPAC says that it's not a valid LPAC file. Did I pick the wrong decoder?)

These tests were done with lame 3.91 using --alt-preset 128 and oggenc rc3 -q 4.00. This was to test files that were nominally targeting 128 kbps, since that's the file size I want, and I don't want to futz with command lines until they're exactly 128 kbps (although that's useful information too). I just want to cram in the command line that gives me the best quality near 128 kbps. Quick and easy.

lame 3.91 --alt-preset 128
fools.wav : 126.4 kbps : ODG -0.9395
ftb_samp.wav : 121.1 kbps : ODG -1.3005
iron.wav : 126.9 kbps : ODG -0.8146

oggenc rc3 -q 4.00
fools.wav : 121.9 kbps : ODG -0.8032
ftb_samp.wav : 131.0 kbps : ODG -0.8292
iron.wav : 123.6 kbps : ODG -0.8476

Interesting results, to look at. Oggenc does a pretty good job at looking "consistent." I think I'm going to go play some Civ III now.

Here's some real world numbers with ogg:

Drowning Pool - Bodies
GT2 192.0 -.36184*
5 160.2 -.3340
5.5 180.8 -.0975
6 201.2 .0290

Primus - Mrs. Blaileen
5 145.0 -.3014
5.5 161.8 -.0781
6 177.7 .0332

Propellerheads - Bang On!
5 152.0 -.3271
5.5 170.0 -.0932
6 187.0 .0182

*Garf Tuned RC2 @ 160

Hmm, why do you use the "sade__sweetest_taboo" sample anyways? It was a sample i sent to Roel about a year ago, because of some lowpass issues. This sample isn't really a "hard" sample for encoders... or was it that what you wanted, an everyday-sample?

I tested my dogies.wav encodes using the analysis tool, and here are the results:

ODG

xing: -1.35
wma: -1.46
lame: -1.24
ogg: -1.48
aac: -1.15
mpc: -0.73

The listeners rated the files as follows:

xing: -2.61
wma: -1.77
lame: -1.75
ogg: -1.63
aac: -0.85
mpc: -0.74

In other words, xing was much worse than wma/lame/ogg, which in turn was much worse than aac/mpc. The analysis tool does not seem to rate the files the same way.

BTW, I went through a lot of trouble when I first made the WAV sample to time align all the files, and I just now double-checked to make sure that the samples were all aligned. The ogg file was shifted by 1 sample (23 microseconds), so I shifted it into alignment, but the numeric results didn't change.

I wouldn't necessarily declare results from this program infallible yet

ff123

To those using the w32 LAME 3.90.2 binary. I've noticed that
the decoded.wav generated by "lame --decode test.mp3 decoded.mp3" while not having a offset at the front may have added trailing silence so that the size of decoded.wav is slightly bigger than test.wav

This will cause incorrect ODGs particularly on smaller test files say 1MB. I've been improving accuracy by removing the trailing silence in CoolEdit. The extra size varies from 100 bytes to 3KB so far. So check those decoded wav file sizes.

I see, the file got truncated with uploading. I'm uploading it again now.


So that's were it came from! I was just picking some of the longer testfiles I have. Most are from the LAME test samples webpage, but I saw that this one wasn't. In the readme of EAQUAL it says that a test sample should be at least 10 to 20 sec., but the longer the better. Many test samples are to short.



No, I wouldn't to. I'm just getting some data. I don't know were it leads to. One of the things I'm thinking about is: Ogg seems to get the best ODG scores. Is it really the best encoder, or does it something better than other encoders that EAQUAL pays much attention to? Maybe Ogg does something else worse than some other encoders, but EAQUAL doesn't pay much attention to that factor? Just open questions, I don't have a clue

It is using 1024 point FFT window for the analysis - so it could miss some pre-echo events now and then.

It is ranking psytel velvet.aac (encoded with 2.01) better than Dolby Professional AAC encoding, and at least to my ears Dolby sounds better (not too much, but I can hear the difference)

Hi all.

Hope you enjoy EAQUAL.
Sorry about the confusion about the offset, it was a bug and now it is fixed (and available at sourceforge). Now the offset is really counted in samples. If you use the old version of EAQUAL, you have to multiply the number of samples with the number of channels.

Alexander

Hi!

You have to use any -scale switches very carefully. The algorithm is stable for small level differences, but it will suck for big level differences.
Scaling can be very useful because many decoders tend to clip full scale audio output - in this case the test signal would be distorted. Here you should use scaling to avoid wrong ratings with clipped audio signals.

Alexander

Hmmm... that's not what I see. From the results on your site it looks like mpc -xtreme is the best.


Jan.

My last message now

If you must use EAQUAL for codec ratings (which is not really its intention), please keep in mind that you have to average over _many_ audio files to get reasonable results - and with good reasons you may doubt those ratings too. You have to average results over least 20, 50 or 100 signals I would guess.

Alexander

Look again Most times Ogg Vorbis -q 6 gets a better score than MPC -xtreme. The times MPC gets a better score it has a much higher bitrate.

BTW. a positive ODG is very good according to Alexander Lerch (he is the author of EAQUAL).

BTW2. I'm not saying Ogg is better than MPC, I'm just saying it gets a better score from EAQUAL on the samples I tried.

Hehe... competition is now having to tune psymodels of their encoders to match EAQUAL values hehehe (just kidding)

Did anybody run EAQUAL on WMA? I know it sucks, but just for a quick check

What bitrate would you like?

96 and 128, but WMA is a VBR codec - it sometimes gives too large bitrate, greater than nominal...

I've justed added WMA8 results for 128 kb/s. It doesn't suck to hard according to EAQUAL, but the bitrates are a bit higher than the others.

Well, try psytel aacenc with -noshort switch (disables short blocks) - you'll notice that it also performs "good" but the results are unacceptable from the pre-echo point of view

If you use EAC's "Process File" function and remove leading & trailing silence, would that eliminate the offset problem for EAQUAL?
Offset will be the same for all alligned files?


bye, Skeeve

The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file. The offset is caused when it converted to an MP3 and then reconverted back to a wav. MP3 can have overlap of info between frames e.g. when you decode the first frame of a MP3 you can't be sure you have all the data for the very start of the music info. I believe there is feame overlap both in encoding and decoding but if you know what this delay is you can remove it. If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.

skipping initial 1105 samples (encoder+decoder delay)

Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.

The offset in question is not caused by leading or trailing digital silence in the original (ripped) wave file. The offset is caused when it converted to an MP3 and then reconverted back to a wav. MP3 can have overlap of info between frames i.e when you decode the first frame of a MP3 you can't be sure you have all the data for the start of the data stream. I believe there is overlap both in encoding and decoding but if you know what this delay is you can remove it. If you use LAME to decode a LAME encoded file these constant values are known to the program and it can compensate when decoding i.e.

skipping initial 1105 samples (encoder+decoder delay)

Not sure about the trailing digital silence that I'm seeing from the decoded file. This was not on the original and has been added after decoding so EAC could never strip it. The amount of added digital silence varies so it is not a constant number of samples to be removed.

I've added some more results to my EAQUAL page:
http://home.wanadoo.nl/~w.speek/eaqual.htm

Just out of curiosity, could someone also give --r3mix a shot.

You have to keep in mind, that it is difficult already to compare results from different listening tests, even if they were realized ITU-R BS.1116-compliant.
Perhaps you should think of EAQUAL as _one_ listener whose results are not depending on his mood and his concentration, who never gets tired and whose results are repeatable, though only his opinion.
BTW, what are the confidence intervals for your listeners?

The difference of one sample has indeed very little influence on the result. If you want to interpret this in the frequency domain, the big difference will only be there for frequencies near Nyquist. These frequencies will not influence the result much.

And it's true, no model can be infallible

Alexander

The results of the dogies listening test can be found at:

http://ff123.net/dogies/dogies_plots.html

With an experiment-wise confidence of 95%, MPC and AAC were deemed better than the others, and LAME/WMA8/OGG were deemed better than XING. N = 15.

The Fisher's protected LSD is known to not be "protected" if the number of samples exceeds 3, but I verified via resampling analysis that the results are indeed as given above with 95% confidence.

In other words, the listeners ranked the various codecs on this sample with clear and statistically significant preference.

ff123