A user test different codec using RMAA, first saw it HERE.

Here is the list of codecs tested and their parameter.

• WAV ( original 16/44.1 WAV test signal )
• MP3 128k ( Cool Edit 2000 1.1 build 2418 )
• Ogg Vorbis Q6 ( OggEnc v1.0.1, libvorbis 1.0.1, encoding -q6 )
• AC3 192k ( Vegas 4.0 192 kbps stereo, AC3ACM v0.7 decode )
• AAC 128k ( iTunes v4.5.0.31 AAC 128 kbps )
• WMA 128k (Vegas 4.0 128 kbps stereo, Window Media Audio ver.9 )

And HERE is the direct link to his page.

I wonder how Musepack will perform, too bad he didn't include in.

Without talking about the complete nonsense of this test, just one comment:



I wonder how, or maybe why, this guy is comparing MP3, AAC and WMA at 128 kbps with vorbis at -q6 (190 kbps)

WHY don't people understand that perceptual coders can't be evaluated for quality using graphs and numbers? We're not talking about cassette tapes here, people. I just don't get it!!

This has been discussed before: http://www.hydrogenaudio.org/forums/index....showtopic=4516&

Why not, mind to explain?

My thinking is straight forward. What is the objective of the codec? Compress and decompress and ensure the signal reproduce close to original, isn't it?

Not true. It's to compress and decompress and ensure that you can't tell the original and the compressed sample apart with your ears, not your eyes.

But hearing is subjective, measurement is objective. Thus the measurement comes in.

And what do you mean by can't tell with original and the compressed sample apart? Isn't that means to reproduce a signal close to original signal?

Sound quality is subjective, so you can't rely on somebody else's measurements.

No, you can distort the signal on purpose to get smaller filesizes, and it can still sound the same to your ears. That's lossy compression for you.

Sure. It's been demonstrated time and time again on this and other boards that the correlation between the way a signal looks on a graph and how it actually sounds is not as high as you might expect. For example, Microsoft did a campaign a few years back that compared it's Window media format to other formats by asserting that WMA "throws away" less of the .wav file, and even had difference files to show what the encoder had chucked. Well, guess what? WMA at 96kbps happened to sound the worst of all the formats tested once you listened to it, pretty graphs and difference files aside.

Another example is the infamous BLADE codec, which looks nice on a frequency graph because it lacked a lowpass, but sounded like a lawnmower running over a chipmunk once you listened to it. Things like SNR, crosstalk, and frequency response curves are NOT good indicators of a lossy codec's quality. Even MP3 at 96kbps probably has a high SNR, low crosstalk and THD, and (in the case of a bad codec like BLADE) a nice frequency response curve. Trouble is, since when has 96kbps MP3 sounded good? Trust you ears, not your eyes. If you want to be objective about sound quality, a double-blind ABX test is the way to go.

The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size. If you are looking for something that is absolutely true to the source, lossless is a better choice for you.

See for example JohnV's explanation in the 5th message of this page :
http://de.europe.creative.com/support/foru...&foru=12&page=2

Sound quality ain't subjective, personal preference is subjective. Please....

And this is not somebody else measurement, this is scientific measurement.

And, no, you don't distort the signal on purpose to get smaller filesizes, but you drop some bit and squeeze them to get smaller size, and during decoding you get the bit back by mathematic calculation. When there is a miss bit or bit error, that is when distortion and other symptoms occured. Or am I misunderstand how the codec works?


[Edited] Add last sentance.

He meant the same thing as you are saying.

A codec uses a psychoacoustic model to determine how "inaccurate" something can be stored (less accurate->less bits) and still sound the same. You can't judge how well this is done without having a perfect psychoacoustic model yourself (judging one model with another is flawed, too). The best thing you do have is your ears.

There's some great explanations posted why this kind of "scientific" analysis you're doing is useless and flawed, please read them.

1. Do you have the graph for the BLADE and WMA @ 96kbps? Appreciate if you can show them to me. May be a frequency sweep?

2. I trust my ears, and I trust the measurement too. Some people tend to prefer some harmonic coz it is easy on their ears, some are not. That is subjective. I may prefer something with higher harmonic, but that does not means the sound quality is the best. How we define sound quality here? Fidelity, right?

3. I agree with your statement:"The point of lossy compression is to obtain an accurate-sounding representation of the original signal at a fraction of the size." That said, a codec job is still doing sound reproduction job which close to original as much as possible in a decent file size.

4. No no, this is nothing to do with lossless. Just a discussion on lossy format.


[Edited] Remove unnecessary quoting.

That didnt sound very scientific.

"Dropping some bits" does mean adding distorsion.

Thanks for the link!!!

I promised you I will read it (I already bookmark it).

Why not scientific??? How you drop the bit? How you calculate back the bit? This is mathematic calculation and it is scientific. Am I wrong again?

Only during decoding when the calculation has a miss bit or bit error, then distortion occur. If you get the bit correct, no distortion should be happened.


[Edited] Remove unnecessary quoting.... again.

Just a remark after having read the link again : I think that JohnV uses improperly the term "quantization noise". "Encoding noise" or just "noise" should fit better. Besides this wording issue, the rest of the message is correct.

Mind to explain further on psychoacoustic model?

Personally, I think the best judgement is by Audio Precesion measurement equipment. What do you think?

And no, I am not the one who do the testing. I just happened to saw the link at RMAA Forum.


[Edited] Add last paragraph.

Example (that's not that far from what really happens):

I need to store 11, 52, 3, 93, 39, 87

and I store 1, 5, 0, 9, 4, 9 instead. (divide by 10 and only store the integer part)

Upon decoding I get 10, 50, 0, 90, 40, 90.

The difference between what I have and what I started with is the quantization noise (which is distortion).

The links above contain information on psychoacoustic models.

"Audio Precesion measurement equipment" cannot determine what is *audible* distortion as is hence completely useless.

Hmm.... I am afraid I cannot agree with you on this.

AP did measure distortion, it even measured THD+N and it measured those "audible" and "unaudible" (to human ears) distortion.


[Edited] Missing word.

Things like RMAA measure mathematical distortions of a sine wave.

Now:

1) Distortion in the audible range is *not* the same as (the amount of) audible distortion.
2) The numbers don't tell you how audible something is (want an IMD of 0.01 or 0.001? How bad is that?)
3) It could only mean something when you listen to pure sine signals only. Which no sane person does.

Again: completely useless.

The purpose of a psychoacoustic codec is to introduce as much (re)quantization noise as possible, while maintaining a good sound quality to your ears.

This is because reducing the number of bits to store numbers is equivalent to introducing noise.

Master's arguments are not illogical. Studies have been made that show how much THD people can hear (it must certainly be somewhere in the AES journal).
The point is elsewhere : a lossy codec removes the parts of the music that the psychoacoustic model finds the less audible.
Go to http://www.mp3-tech.org/ , in the menu, click "MP3", then "ovberview of the MP3 techniques".
This can be harmonics of an instruments, parts of sounds of other instruments after the attack of a percussion, some instruments that can't be heard during percussive sounds... The way lossy codec change the sound can appear completely chaotic...
More important, a codec begins its work of removing infos with its psychoacoustic model when very complex sound begin to appear. With RMAA test tones, only the roughest part of the model is at work (absolute threshold of hearing + frequency masking). Temporal masking, joint stereo tuning, choice between long and short blocks, and all the deep stuff, whose quality is capital for the result, is not at work on RMAA test tones. Thus the results can't give an idea of the real performances of the codec.

Ok, after reading the link, and some discussion with my friend, more or less I get the picture of it. Thanks again.

Now, the "inaudible" noise you mentioned here, from what I think, is truly subjective. Reason? Everyone has different ears.

When our ears are different from one to another, the ability to capture the quantization noise from the product of psychoacoustic model is also different.

Hmm.... so what is the best measurement tool here? While AP cannot measure how well the quantization noise handling by the codec, can it paint a overall behaviour picture by doing a frequency sweep?

Open for discussion, truly appreciate to those reply to me and clear my doubts.


[Edited] Typo in bold.

Blind listening tests.

But again, isn't that subjective?


[Edited] Remove unnecessary quoting.

Inter-subjective (collective) listening test is the answer to the problem.

Reading your post makes me think twice. You did have a valid points here: Collective. Hmm....


[Edited] Missing word in bold. Sigh.... why oh why I need to edit every of my posts...?

...but the masking curves of most people’s ears are fairly similar. If you code to please the best, then everyone else will be happy too.




Slightly - not greatly. Anyway, it's kind of irrelevant. Like every other piece of psychometric data, you work with averages and percentiles. The data is only for simple sounds (pure tones, bandpass noise) so extrapolating this to complex time-varying musical signals is difficult. The only way to judge if the codec is doing this properly is to listen to the result. If you try to second guess the human ear, then you may get close - but your test will be as much a test of the accuracy of your ear model as it is a test of the accuracy of the codec.




The frequency sweep show you which frequencies are passed by the codec when encoding a very easy signal. That reveals any fixed low and high pass filtering, but that's about all.

If you try a constant pure tone, you can usually see the shape of the band pass filters within the codec, because they'll be noise 20-40dB down from the tone, spread in the shape of the filter and the psychoacoustic model's masking prediction.

These are interesting features of the codec, but they don't tell you what it sounds like.




I think you need to go back and read those links again - I still don't think you've "got it" - sorry!

Cheers,
David.

P.S. the chapter of my thesis which discussed this is here:
http://www.hydrogenaudio.org/forums/index....showtopic=24632

The best measurement tool for this community are the listening tests ... a group of users compares different codecs with different music styles by double blind testing (with cheating being made impossible) ... a statistical analysis of the results is being performed afterwards to determine the overall codec ranking.

RMAA does have a use with codecs though - to test if a decoder was working correctly (when compared against other decoders of the same type) - it will not tell you if one decoder is sounding better than another, but rather if one decoder was way off and not working correctly (the iPod on high bit rate mp3s showed this).

Your basic assumption is that a human hearing is a linear measurement device (i.e. like a measurement instrument, such Audio Precision Cacade One, except less accurate) and that audibility/audible quality can be assessed through means of objective technical measurements.

However, human hearing is not a linear and "well-behaved" measurement instrument.

Human hearing is inherently non-linear (on signal to excitation level). This means that what you can measure doesn't always correlate well with what you can hear (under the best conditions) or vice versa.

What is even more problematic for making comparisons between non-linearities of human hearing and measurements is that there are known 100% accurate mapping between these two domains (i.e. measuring with tools and hearing with your "ears").

That is, there is no 100% (or anywhere near 99.95%) accurate way to map a measurement into a listening result: can you hear something or can you not. Please bear now in mind that I'm completely (for the sake of discussion) ignoring whether the listener "likes" or "prefers" something. I merely compare detection (detected by a measurement tool vs detected by a human hearing apparatus).

Because of these non-linearities and relatively rough mapping between measurements and hearing results, a lot of research has gone into trying to find how humans hear.

This field is a study of psychoacoustics and it measures sound in two domains: both as an acoustic signal (e.g. 1 kHz simple sinusoid at 40 dB) and as a human hearing event (i.e. 1000 mels).

The first measurement (acoustics) can be done with technological measurement instruments.

The second can only be done by listening with the human hearing apparatus.

Then it's a matter of combining these two slowly, by doing very low level small tests with hearing (first with simple clicks, tones and longer sounds and then further with complex sounds, multiple sound sources, etc).

However this process of combining the two (what is a sound as an acoustic event and what is that which the human hears) measurements is "only" roughly 130 years old as a scientific discipline.

As such, still much remains unknown about what we hear (compared to what we measure) and when do we hear it (under what acoustic conditions).

So, to make a long story short: what people have written about using your ear as a tool to measure "audible performance" (as opposed to signal quality measured performance) is true.

While there do exist some attempts at "objective measure" of sound quality (i.e. mimicking how humans hear through the use of algorithms), these are still very much at rudimentary stages and do not perform anywhere like human hearing for more complex signals (multisource complex tones or more interestingly, music and normal acoustic recordings).

I hope this illustrates the problem of hearing vs. measuring a little bit more.

Best regards,
Halcyon

PS A lot of what is true for hearing is true for vision as well. We may think that our sense portray an objective truth about the world we live in or that what we can see or not see, is easily measured. However, this is not the case. Seeing, as much as hearing, has things like masking, constancy, temporal and other effects which makes our vision very much a badly known and ill-behaving measurement device (even on very low level tasks which do not curry the burden of higher level semantic reasoning, such as meaning making about objects we build up in our visual array).

Just a quick question, when you all conducting a listening test, did you include the original wave file as reference?

I will going through David's document before I make any reply.

But one thing to clarify, Halcyon, I didn't make such assumption where a human hearing is a linear measurement device. If I do, kindly point out for me.

the whole thing can be explained without alot of mathematical terms:

lossy codecs are for listening - not for watching. It is about how the music sounds to you - not how it looks like.

So, mid-summary: it is unimportant how the music looks like - its only important how it sounds like.

That point being made, how to test how "transparent" a lossy audiofile >sounds< compared to the original source? Well, the most accurate tool to measure "perceived quality" are your ears. At the same time, your ears are the "target audience" - so its the direct way to measure perceived quality.

However, your ears can be tricked quite easily. The brain can make you imagine (and hear) things which aren't really there (placebo-effect) - so we need a counter-tool to exclude this error-source. The tool to do this is statistics and blind-listening tests.

So, you rank the transparency of a lossy codec with your ears - and minimize the possible danger of placebo-effects with a blind-listening tests. And by doing this multiple times you gain some statistics and therefore the "reliability" of the results increase.

Yes, a blind-listening test isn't 100% accurate and secure - but then again, nothing in the world is 100% for sure - "proven" in scientific terms only means "with such a high propability, that the risk of a false asumption is neglible".

- Lyx

...but if the two were correlated (as they were to some extent with conventional audio equipment) then the tests would be worth doing, and could tell you something. It's just that, with audio codecs, because of their design, the measurements aren't correlated with what you hear.

To suggest that measurements are useless because we listen to audio isn't fair or correct. Measurements can be useless for audio codecs because the codec is playing a trick on the human ear, and the measurement is measuring the wrong thing.

You could argue that every audio device is trying to trick the human ear, but only codecs try to give good subjective results and bad objective results. Without DSP to force the two to be different, they're actually closely linked! You can argue that the two aren't perfectly linked, but you know that a loudspeaker with a poor measured frequency response will sound like it subjectively has a poor frequency response. Compare a telephone speaker to your hi-fi - the measurement tells it like it is!


"We can't measure, we must only listen" is true with audio codecs. When you start to apply it to other audio equipment, you go down the rout of audiophile madness.

Cheers,
David.

Actually even the best of audio components have to be discerned with listening (that is, the relative superiority of the best of equipment).

Why?

Because we don't know what are all the objective technical measurements we should do in order to distinguish what is the best objectively.

Let me give you an example from headphones.

There exists no set of measurement data for headphones that tells how a perfect headphone should measure (if you do think it exists, please provide a reference, I'd be glad to revise my view on this).

In fact, listening tests have shown that headphones with less than ideal measurements from a set of professionally done dummy-head measurements can sound very good (to a set of trained analytical ears trying to assess accuracy, not personal sound preference).

Then again, a pair that measures much better according to some objective design ideals can sound really off, unnatural and even annoying [Ref: Finnish Hi-fi magazine headphone reviws 2002 (with dummy head measurements done the way they should be done)]

As such, listening is something we have to do even with audio equipment, although I do agree that completely forgetting measurements can quickly lead to absurdity.

regards,
Halcyon

Sorry for the late reply.

And David, I download your document but no time to print out for study. I am sorry for that. Thus, no comment by now.

And halycon, I understand where you come from, and I do agree with some of your opinion, but not all. Take an example, you bringing two high end set of amplifier, be it Mark Levinson or Krell, definately some will prefer ML and some prefer Krell, coz listening is subjective to preference. But in term of music reproduction, which simply means to reproduce the signal as close as original source, measurement still the way to go.

The problem with comparing audio codecs with audio electronics is that people judge electronics on many factors besides sound quality. I suspect that people who buy audio electronics rarely do a double-blind level-matched ABX test in their listening room before making a purchase. They're choosing Krell over Mark Levinson or vice versa as much based on their feelings about the gear's reputation, looks, or price as its sound.

A measurement graph of a lossy codec can't tell you how it sounds to your ears. What looks like the most accurate reproduction of the original file isn't necessarily the one that sounds the most accurate. The goal is to produce the most accurate sounding (to human ears) reproduction of the source possible for a given file size limitation, regardless of what a graph might indicate.

As an example, a sound that is far beyond the range of human hearing can be easily measured and displayed on a graph, yet its presence in a recording wouldn't affect how it sounds to you.

I am sorry that the example I made is to reply Halycon's latest post.

And yes, I know well what you say and I agree with you. And no, my example is not related to the codec but halycon's comment.

Hope that clear your doubt and any misunderstanding.


[Edited] Add last three words.