A Signal Processing Approach to Codec Comparison
by Turing
2005 February 21

I have just finished a project to compare various audio compression codecs. My goal was to introduce some objectivity into codec comparisons using mathematics. I think the results will be interesting to other audio compression enthusiasts.

Summary

I found LAME, Ogg and RealAudio to be the best codecs. For my definition of the work “best” you will need to read the entire article. I also recommend that you use LAME 3.96.1 for higher quality (preset standard and above). If you have a need for more compression, the best program is LAME 3.90.3 (e.g. average bit rates fo 128 or 160 kbps).

The detailed analysis is contained in the attached PDF document. The source code for my comparison program is included.

You can't measure percieved quality by comparing wave forms. Theres not any meaningful relationship between the two.

Turing states that clearly in the paper already. "If it doesn't sound good, it isn't good. I inted the approach discussed here as a supplement to listening tests."

I agree with you. If you read the article you will see that I only intend to supplement listening tests. The human ear is the ultimate judge of quality (at present).

However, a computer based approach could be of some use to developers who want to test some ideas before waiting months for the results of proper listening tests.

The other thing that the approach outlined here does not address is the fact that a fraction of a second of bad encoding can ruin the whole listening experience. This single measure approach does not capture this kind of problem. I am working on a method to detect these transient problems.

Hello, and welcome to Hydrogenaudio.

You are reading this because you violated forum rule number 8.
Don't worry - you probably didn't know about it, or didn't
understand the implications, and we understand that. The
Hydrogenaudio Terms Of Service are here:

http://www.hydrogenaudio.org/forums/index.php?showtopic=3974

The gist of rule #8 is that if you make a claim, you must have
proper supporting evidence for it. This rule is the very core
of Hydrogenaudio, so it is very important that you follow it.

This is a generic post, and not all what follows may be
applicable to this situation. Read through it nevertheless,
it contains essential information and will help you understand
what to do (or not to do).

Why should I bother with all of this, I just want to report
a problem? (link)

For audio quality matters, 'proper supporting evidence' is
a blind listening test result demonstrating that you can hear a
difference, together with a test sample.
Graphs, non-blind tests, subtracting two files and so on are
definetely not!

A proper blind test serves several purposes: it shows that you
are serious towards our community, it proves to yourself that
you can indeed hear a difference, it provides an indication of
the seriousness of the issue at hand, and it helps pinpointing
the problem for differnent listeners.

The easiest and most common way to do a blind test is an ABX
test. There are several free utilities to do one:

http://www.pcabx.com/
http://ff123.net/abchr/abchr.html
http://www.kikeg.arrakis.es/winabx/winabx.zip
http://www.beryllium.net/~remco/linabx/

An ABX test requires you do identify an unknown (X) sample as
either the original (A) or the processed (B) sample. With some
statistics it can be figured out how likely it is that you were
actually hearing a difference instead of just guessing which was
which. Hydrogenaudio uses as a general guideline that < 5% change
of guessing is considered 'proof' that you are hearing a difference.
If you try the ABX test multiple tests, add up all attempts. You
can use http://www.ff123.net/abx/abx.html to calculate the p-value
( < 5% = < 0.05), though most ABX programs have it built in.

If you managed to get a significant score, congratulations, it
seems that the problem is real.

If applicable, you'll need to upload the test clip you used so
other people can verify it and developers can tinker with it.

Uploading copyrighted music is generally illegal, but fair use laws
generally permit short clips (< 30 secs) to be used for purposes
such as this. Your ISP probably has allocated you some webspace for
a webpage. You can upload the clip to it. If you're low on space,
compress it with a lossless encoder (e.g. FLAC http://flac.sf.net),
which will approximately halve the required space. If you don't have
your own webspace, you might want to try the IRC channel, ask a
friend, or simply post on HA and ask if someone wants to help you out.

If you finally make your post to Hydrogenaudio, try to include
as much information as is relevant, and be sure to explain exactly
what and where (important but often forgotten) you hear the problem
best.

Audio is to a large extent a subjective matter, and as such,
quality matters are prone to a few problems. The first is listener
preferences. Something that applies to you may not apply to the
majority of people. Maybe the clip is an exception or problem
case and not representative of general performance. This is why
being able to verify a result is imporant, as well as giving the
developers something concrete to work with.

The second is the mind. The human mind is powerfull, but has some
weaknesses. It is very vulnerable to suggestion and subconscious
influences, even for people experienced in these tests. No matter
how how 'sure' you are that a problem exists, verify that it's not
your mind playing tricks on you first, it'll save embarassement
later.

'Simply' reporting a problem generally doesn't tell us anything,
isn't indicative of anything, can be impossible to reproduce,
confuses people, and most importantly, wastes precious developer
time determining if the problem is real and serious or not.

You may have saved yourself 5 minutes, but you've cost other people
an hour. That's not very nice.

Sorry, but I disagree.

I'don't see any TOS#8 violation here

I see somebody taking a different approach to the evaluation of sound compression. And I see he doing in a very polite and respectfull way.

And even more he uses a scientific approch, that can be questioned but not laught at.

TOS#8 is very much important but cannot become a chain and ball to the scientific evolution. I bet Galileo Galilei would agree with me.

Sergio

I believe the rule does not apply. I am not making claims about subjective judgements. All of the results are quantitative.

If your objection is to the use of mathematics in the comparison, then perhaps I should post somewhere else. -- any suggestions?

Your mathematical supporting evidence seems quite adequate for your definition of the word 'best'. However, I believe the problem is with your comments about the version of LAME to use. It's probably not a good idea to recommend that a version of the LAME encoder should be retired in your first post at HA.

No offence Turing, but you wasted your precious time. Psychoacoustic audio compression cannot be caught in a simple cross correlation formula.
Just for fun try the test on some Wavpack hybrid compressed tracks. I bet that they score better than any of the codecs in your paper. Simply because it doesn't apply any form of spectral/temporal masking.

The wrapping is nice, but I fail to see how these results are any more important/valid than graph comparison. Introducing mathematics doesn't make the methology any more valid when comparing lossy codecs.

Maybe you're not breaking TOS#8, but giving advice based on such false conclusions isn't even close to what this forum is all about, IMHO.

Edit: Spelling

Some mods are just too fast waving that damned piece of text around. They even have a template of what to post!

The foundation of the study is based upon one person's perceived 'transparency' threshold. Regardless of whether the data and methodology supports your conclusions, you still can't use objective reasoning to support the subjective perception (opinion) of one person. (and present it as anything more than a personal opinion)

I disagree. The TOS#8 reminder was posted for a good reason.
Using cross-correlation as a measure for audio quality (and basing claims regarding transparency on it) is not meaningful in any way.

It might not be meaningful, but I guess he isn't as wise as you or he wouldn't even have tried his experience.

Turing politely came here with his proposal of a way to complement listening tests with objective data. He never claimed he wanted to replace listening tests, quite the opposite.

Now, some people are criticizing him with good arguments. Even your argument about cross-correlation being meaningless is probably a good one. But jumping on the thread ostensively rubbing TOS8 on his face (and doing only that) won't help him get educated, won't help him improve his methodology (if it is improvable) nor it will help anyone else.

This insane witch hunt here is effectively killing any possibility of discussion about quality assesment methodologies other than double blind listening tests.

I suggest you add his PDF to the index librorum prohibitum of HydrogenAudio.

You and UpNorth have good points.

Anyone following this thread or reading my article, PLEASE take these recommendations with a grain of salt.

LISTENING TESTS ARE THE BEST WAY TO COMPARE CODECS AND SETTINGS.

However, this forum is still recommending the 3.90.3 version of LAME. The LAME developers recommend 3.96.1. While eagerly awaiting the results of listening tests, I developed this method that can be used by an individual.

I also did my own listening tests, but I cannot tell the two version of LAME apart. The computer agreed with me.

I hate to quibble, but I disagree with your wording. The foundation of the study is the use of cross-correlation as a comparison tool. The foundation of my subjective recommendations is by 1/4% threshold.

Anyone with a more demanding threshold should be using at least --preset extreme. However, they could still attempt to us cross correlation to measure the difference.

The usefullness of cross-correlation is still open to debate in spite of my average hearing.

More like the only way if it's lossy (at least to my knowledge).

Anyway, is there any tool around, or would it make sense, to incorporate a psychoacoustic model in this kind of comparison (a non-listening test)? Applying a psychoacoustic model on the original, to remove the parts that wouldn't be perceivable, and then compare it to the decoded lossy file. It would still suffer from the same problem as the methology Turing used, namely that you can't tell if a possible deviation would be perceivable or not, but would it be useful as a guideline?

Haven't you ever heard of Eaqual, PEAQ and Opticom Opera?

They are definitely not as good as a listening test to assess quality, but they are wonderful solutions for developers that need to tweak countless parameters by hand and can't affort to hire hundreds of trained listeners.

Is simple cross-correlation in any way comparable to PEAQ or Eaqual?

Cross-correlation is completely worthless when it comes to transparency and turing's methodology is more than questionable:

Recommending to define a 'transparency threshold' based on cross-correlation results is IMHO worthy of reminding turing of TOS#8.

I am a member of this forum in order to learn. Could you point me to the tests that support your statement?

I am new here and I have probably missed the extensive comparisons of listening and cross-correlation that back up this statement.

quick question....did you try the Fhr, and if so, how did it compare?

Read his paper. He uses correletion interchangeably with quality, both explicitly when he compares quality (when he actually means correlation between wave forms) and implicitly when he goes on to give recommendations based on his analysis.

They are not the same thing, and this is a critical flaw.

Wouldn't be interesting to see how Turing model fit with experimental data? Maybe compare it with previous rjamorim's listening tests?

Sergio

I think you should be careful here. You're using the "lagged product estimation" for the cross-correlation. Note, that this is only an estimation of the real cross-correlation, and this estimation will have a variance itself for two given signals.

Given the fact that the signals your working with aren't totally stochastic, this variance will be very small, but so are the differences between the values for the cross-correlation you've found.

Furthermore, as said before, it's very hard to predict the perceptual quality of a signal in a mathematical way. There are a lot of trics to do so (estimating the cross-correlation isn't a popular method, by the way), but in the end listening tests are your final judgement.

I'm not that familiar with these kind of tools, and it's been some time since I last read about the topic. So, I'm afraid I don't remember the names, but AFAIR there were considerable shortcomings (but, as I don't even remember the names, I might be wrong about that). But, how good are the mentioned tools, and similar, at the moment? Do they serve only as development guidelines, or can they be used to compare codecs in a meningful way, like your listening tests does?

Anyway, I'll do some seaching and reading, as it's an interesting topic. The Listening Tests subforum, is probably a good place for me to start...

Oh, for sure, they are far from perfect (perfection would be a formal test conduced according to the ITU R-BS 1116-1 guidelines)



Depends on what you understand by meaningful

But no, I wouldn't choose my encoder of choice based on PEAQ results...

Anyway, check this out:
http://www.hydrogenaudio.org/forums/index....showtopic=20264

I think some of us is a bit harsh on dev0. I think he was very respectful and he did explain in his 1st post in a very detailed way what Turing needs to study more deeply. He took the trouble for detailed and friendly reply. In addition he (or she?) takes the trouble to moderate this forum day-by-day.

As for Turing-s achievement. Yes it is nice, but I could find 100+1 mathematical metrics and I could evaluate them one-by-one and release 100+1 whitepapers into this forum. This would not make a tiny bit forward anything unless you prove that the metric you used has direct connection with human perception.

Also it is pointless to go on and on about what sort if computers you have etc or even the kernel version you have, because everybody with the right mind would assume that you set up your conditions carefully etc and focus on your argument and on your message.

Triza

No trouble involved here, as rjamorim correctly pointed out, I just posted a ready-made template.
I still think it was a justified reply.

That's not the point, imo. Scientifically, the only _valid_ way to test a lossy audio codec is using blind tests. Those tools are of value for developers, but they can't be used to claim the quality of a codec. Furthermore, the forum rules clearly state that blind tests are the only way to support quality statements regarding sound quality (specially with codecs).

The article Turing looks good, but like Mike Giacomelli wrote, he makes claims about quality based on wave correlation. I never saw Ivan, Klemm, Garf, Dibrom or any other developer using PEAQ, Eaqual (which I assume to model the human ear somehow and because of that are much more complex and useful to *developers*), etc to make claims regarding quality on Hydrogenaudio.

Turing, don't take this in a wrong way. It just happens that one of the main reasons Hydrogenaudio was created for is related to only using scientific methods to assess the quality of sound codecs.

Cya

You might also want to check this thread.

I haven't tired Fraunhofer. I will do that.

I tended to concentrate on Linux versions, and I only have Fraunhofer on Windows.

I had hoped to generate some interesting discussion. There is some of that in this thread, but the rest seems to border on religious bigotry.

Some of that is my fault because I mixed up the word quality with high cross-correlation in my discussion. This is no proper and I should have used more careful wording. I apologize for this sloppy use of words.

However, a lot of comments indicate that any mathematical comparison of Codecs is comparable to thowing dice. I think there is more to it than that.

IMHO it is not a coincidence that the cross-correlation values vs. bit-rate graph matches (with one exception) exactly what one would expect. It also agrees with the comparisons made with listening tests that LAME --preset standard is superior to other codecs for comparable bit-rates.
these numbers are correct.

--

Let me state this again: it is not my intention to replace listening tests mathematical tests. One should not use cross-correlation as the sole basis for picking a codec. As stated before this method can supplement listening tests.

I had in mind a developer tweaking a program and then running a battery of tests to see if it was going in the right direction or not. I don't think this program is quite there yet, but the results are encouraging enough for me to think it can be further developed.

Again, I apologize for my poor choice of words.

I hate to be the asshole waving TOS#8 again, but it's up to you to verify the claim that LAME --preset standard is superior to other codecs for comparable bit-rates.

I only know one listening test performed at high bitrate. It's mine...
http://www.hydrogenaudio.org/forums/index....showtopic=23355

And --preset standard appeared to be worse than vorbis -q6 and mpc -q5.

Could you give some links to other listening tests at these bitrates? I'm interested



And how do you explain that you can recommand to yourself and to other people X or Y encoder based on your methodology - and only your methodology? It supposes that you're replacing listening experience by a mathematic formula. And I suppose that's precisely the reason why Dev0 and other people invoque TOS#8. Am I right?

You are right, it is not a coincidence.
For a given codec, if its internals do not drastically change between bitrates, it is perfectly normal that an higher bitrate is closer to the original, both in a perceptive way and in a mathematical way.
However, inter-codec comparison for a given bitrate is a matter of luck when using only cross-correlation.

TOS8:
8. All members that put forth a statement concerning subjective sound quality, must -- to the best of their ability -- provide objective support for their claims. Acceptable means of support are double blind listening tests (ABX or ABC/HR) demonstrating that the member can discern a difference perceptually, together with a test sample to allow others to reproduce their findings. Graphs, non-blind listening tests, waveform difference comparisons, and so on, are not acceptable means of providing support.

Turing PDF page 5/9:
• "I think the results are consitent enoughto draw some conclusions". [conclusions are all on page 6/9:]

• "I am sticking with LAME 3.96.1"
> it's not based on listening tests - but afterall it's not our problem. But Turing also recommand to everybody many things :

• "If for some reason you want higher compression I recommand LAME 3.90.3. This version out performs every competitor on ABR 128".
> it's well known that mp3 "outperforms" AAC, MPC, Vorbis at 128 kbps; isn't it And it's not a violation of TOS#8 to recommand on this board an encoder which performs good objectively but which is less enjoying on listening test....

• "this test results indicate that it is a very competitve codec".
> another statement which is based on "objective maesurment". The codec name is ATRAC3 (real audio, well known to outperform MPC and iTunes AAC at 200 kbps).

• "If you want to keep a clean system with only one encoder for all purpose this is the one to use".
> another recommandation coming from his methodology. And the fact that this memeber didn't test bitrates below 120 kbps doesn't forbid him to talk about "all purpose". Maybe low bitrate are totally useless?



As a consequence, we can't follow or defend Turing when he say that he consider listening tests as supreme "the ultimate judge of quality (at present)". His PDF is full of recommandations which are at the opposite of listening tests (personal and collective) done before.