SoundExpert explained, Methodology issues Options

[Serge Smirnoff]

I found this thread among SoundExpert referals and was a bit surprised with almost complete misunderstanding of SE testing methodology and particularly how diff signal is used in SE audio quality metrics. Discussion on the topic from 2006 actually seems more meaningful. So I decided to post here some SE basics for reference purposes. I will use a thought experiment which is close to reality though.

Suppose we have two sound signals – the main and the side one. They could be for example a short piano passage and some noise. We can prepare several mixes of them in different proportions:

• equal levels of main and side signals (0dB RMS)
• half level of side signal (-6dB RMS)
• quarter level of side signal (-12dB RMS)
• 1/8 level of side signal (-18dB RMS)
• 1/16 level of side signal (-24dB RMS)

After normalization all mixes have equal levels and we can evaluate perceptibility of the side signal in the mixes. Here at SE we found that this perceptibility is a monotonous function of side signal level and looks like this:

Figure: Side signal perception

(1) In other words, there is a relationship between objectively measured level of side signal and its subjectively estimated perceptibility in the mix. And what is more:

(a) this relationship is well described by 2-nd order curve (assuming levels are in dB)
(b) the relationship holds for any sound signals whether they are correlated or not, the only differences are position and curvature of the curve.

(2) These side stimulus perceptibility curves are the core of SE rating mechanism. Each device under test has its own curve plotted on basis of SE online listening tests.
(3) Side signals are difference signals of devices being tested. Levels of side signals are expressed in dB of Difference level parameter which is exactly equal to RMS level of side signal in our case.
(4) Subjective grades of perceptibility are anchor points of 5-grade impairment scale.
(5) Audio metrics beyond threshold of audibility is determined by extrapolation of that 2-nd order curves. Virtual grades in extrapolated area could be considered as objective quality parameters regarding human auditory peculiarities.

So, yes, difference signal is used in SE testing. We take into account both its level and how human auditory system perceives it together with reference signal. Some difference signals having fairly high levels still remain almost imperceptible against the background of reference signal and vice versa; perceptibility curves reflect this.

This is the concept. Many parts of it still need thorough verification in carefully designed listening tests, which are beyond SE possibilities. All we can do is to analyze collected grades returned by SE visitors. This will be done for sure and yet this can't be a replacement of properly organized listening tests.

SE testing methodology is new and questionable, but all assumptions look reasonable and SE ratings – promising, at least to me. Time will show.

[Serge Smirnoff]

It's not hard to imagine the possibility of signal pairs (main,side) where you can't hear any difference between main and main+side but you can easily hear a difference between main and main+0.5*side.

In practice - never. In all cases perception of gradually unmasked artifacts is monotonous function. That was also confirmed by B. Feiten in already mentioned "Measuring the Coding Margin of Perceptual Codecs with the Difference Signal" (AES Preprint # 4417). This is the main point of SE metric that was stated in the first post (above the graph). Once again - not a single case where the curve was not monotonous and numerous cases of monotonous behavior. So I treat this as a fact.

Hint: phase is a bitch. ;-) Your implicit assumption is that both signals are independent. But this is not necessarily the case with perceptual audio coders. Take for example the MPEG4 tool called PNS (perceptual noise substitution). It just replaces some high frequency noise with synthetically generated noise of the same level. This is done by transmitting the noise level only. Obviously, we can use this tool in cases when the main perceptual feature is the energy level and anything else is not important. Then, we have the following properties: Noise level of original matches the noise level of the encoded result, so energy(main) = energy(main+side). Probability theory tells us that main and main+side are orthogonal. This implies a coherence between main and side of 0.7 -- ZERO POINT SEVEN. Hardly independent. This also implies that a 50/50 mix -- main+0.5*side -- would lose 3dB power. You can easily compute this via

CODE

main = [1 0];
side = [0 1] - main;
20*log10(norm(main+0.5*side))

(Matlab code)

So, by attenuating the sample-by-sample difference we actually amplify the perceived difference (since we lose power) in this case! What does that tell us? It tells us that you overrate sample-by-sample differences. Perceptual audio coders try to retain certain things so it sounds similar and tolerate other losses. And you're focussing on the "other losses" (as well). What you're doing is basically violating some of a perceptual encoder's principles (like keeping energy levels similar no matter how large the sample-by-sample difference will be). By amplifing the difference you could destroy some signal properties the encoder and our HAS cares about much more than you do. Sound perception is not as simple as you want us to believe. Sample-by-Sample differences are not important. And "extrapolating artefacts" this way is nothing but a big waste of time. Even testing with "attenuated artefacts" doesn't tell you anything. Your methodology breaks down because you're assuming that the difference is independent from the original. It is not.

I didn't make such assumption, quite the opposite - see 1b in the first post. Nevertheless, the case you discribe is realy interesting. If exaggerated and simplified a bit it will look like following:

We have a sound excerpt which has a time interval (between tonal parts) which consists purely of, say, white noise. Also we have a coder which can only substitute the noise with uncorrelated one whenever it detects that there are no tonal parts during that interval. Then diff. signal will consist of amplified noise portion (being uncorrelated they will be added not subtracted). So the version of our excerpt with amplified differences will have stronger noise part which can be detected in listening tests while in practice this is not important for HAS.

Is this the case you wanted to produce? If yes I will examine it more carefully. It is really interesting as it helps to determine the limits of the metric.

[2Bdecided]

In all cases perception of gradually unmasked artifacts is monotonous function.

How can you say this when SebG and Woodinville both gave you examples to the contrary?

I hit the exact problem Woodinville describes using the method I posted on the first page of this thread - a listener gets stuck in a "false" minima of audibility because double the difference gives you the original signal back (with the part "removed" by the codec being inverted, but that difference is not usually audible). Hardly monotonic - the chance of hearing the artefact becomes zero at a single gain setting (+6dB), and (with the specific audio I used - YMMV!) leaps back to the "expected" function very quickly either side of that.

Cheers,
David.

[Serge Smirnoff]

How can you say this when SebG and Woodinville both gave you examples to the contrary?

I hit the exact problem Woodinville describes using the method I posted on the first page of this thread - a listener gets stuck in a "false" minima of audibility because double the difference gives you the original signal back (with the part "removed" by the codec being inverted, but that difference is not usually audible). Hardly monotonic - the chance of hearing the artefact becomes zero at a single gain setting (+6dB), and (with the specific audio I used - YMMV!) leaps back to the "expected" function very quickly either side of that.

In many papers devoted to "coding margin" a special filtering is recommended to eliminate those "ghost" frequencies. We also use it.

[Woodinville]

How can you say this when SebG and Woodinville both gave you examples to the contrary?

I hit the exact problem Woodinville describes using the method I posted on the first page of this thread - a listener gets stuck in a "false" minima of audibility because double the difference gives you the original signal back (with the part "removed" by the codec being inverted, but that difference is not usually audible). Hardly monotonic - the chance of hearing the artefact becomes zero at a single gain setting (+6dB), and (with the specific audio I used - YMMV!) leaps back to the "expected" function very quickly either side of that.

In many papers devoted to "coding margin" a special filtering is recommended to eliminate those "ghost" frequencies. We also use it.

How do you know what "it" is? You have to work specifically to every bit rate, every bandwidth, every sampling rate, every different encoder?

This is not useful.

[Serge Smirnoff]

In many papers devoted to "coding margin" a special filtering is recommended to eliminate those "ghost" frequencies. We also use it.

How do you know what "it" is? You have to work specifically to every bit rate, every bandwidth, every sampling rate, every different encoder?

This is not useful.

Subtracting a portion of reference signal from output one it's not hard to figure out what frequencies are "ghosted' and remove them with FIR filter. So, yes, we do it for every test sample with amplified artifacts. This helps to get smoother perception curves. Every item tested at SE has its own unique curve plotted on results of SE listening tests. Extrapolating that curve we get resulting quality rating for each testing item.

[Woodinville]

In many papers devoted to "coding margin" a special filtering is recommended to eliminate those "ghost" frequencies. We also use it.

How do you know what "it" is? You have to work specifically to every bit rate, every bandwidth, every sampling rate, every different encoder?

This is not useful.

Subtracting a portion of reference signal from output one it's not hard to figure out what frequencies are "ghosted' and remove them with FIR filter. So, yes, we do it for every test sample with amplified artifacts. This helps to get smoother perception curves. Every item tested at SE has its own unique curve plotted on results of SE listening tests. Extrapolating that curve we get resulting quality rating for each testing item.

So, it's "by clip". This still seems useless.