SoundExpert explained, Methodology issues Options

[Serge Smirnoff]

I found this thread among SoundExpert referals and was a bit surprised with almost complete misunderstanding of SE testing methodology and particularly how diff signal is used in SE audio quality metrics. Discussion on the topic from 2006 actually seems more meaningful. So I decided to post here some SE basics for reference purposes. I will use a thought experiment which is close to reality though.

Suppose we have two sound signals – the main and the side one. They could be for example a short piano passage and some noise. We can prepare several mixes of them in different proportions:

• equal levels of main and side signals (0dB RMS)
• half level of side signal (-6dB RMS)
• quarter level of side signal (-12dB RMS)
• 1/8 level of side signal (-18dB RMS)
• 1/16 level of side signal (-24dB RMS)

After normalization all mixes have equal levels and we can evaluate perceptibility of the side signal in the mixes. Here at SE we found that this perceptibility is a monotonous function of side signal level and looks like this:

Figure: Side signal perception

(1) In other words, there is a relationship between objectively measured level of side signal and its subjectively estimated perceptibility in the mix. And what is more:

(a) this relationship is well described by 2-nd order curve (assuming levels are in dB)
(b) the relationship holds for any sound signals whether they are correlated or not, the only differences are position and curvature of the curve.

(2) These side stimulus perceptibility curves are the core of SE rating mechanism. Each device under test has its own curve plotted on basis of SE online listening tests.
(3) Side signals are difference signals of devices being tested. Levels of side signals are expressed in dB of Difference level parameter which is exactly equal to RMS level of side signal in our case.
(4) Subjective grades of perceptibility are anchor points of 5-grade impairment scale.
(5) Audio metrics beyond threshold of audibility is determined by extrapolation of that 2-nd order curves. Virtual grades in extrapolated area could be considered as objective quality parameters regarding human auditory peculiarities.

So, yes, difference signal is used in SE testing. We take into account both its level and how human auditory system perceives it together with reference signal. Some difference signals having fairly high levels still remain almost imperceptible against the background of reference signal and vice versa; perceptibility curves reflect this.

This is the concept. Many parts of it still need thorough verification in carefully designed listening tests, which are beyond SE possibilities. All we can do is to analyze collected grades returned by SE visitors. This will be done for sure and yet this can't be a replacement of properly organized listening tests.

SE testing methodology is new and questionable, but all assumptions look reasonable and SE ratings – promising, at least to me. Time will show.

[drewfx]

Just to be clear - I am not necessarily questioning your goals here.

The problem I see is that when you extrapolate to infinity a curve that attempts to quantify human perception, you are also implicitly implying that human perception itself extends infinitely. You are redefining "imperceptible" to mean "less perceptible". You need the curve (and math) to match the realities of human perception, or else any conclusions you attempt to draw from the extrapolations are essentially meaningless. What's the point of trying to create an objective measure that only applies to a hypothetical world?

I repeat my original assertion - the curve should be a flat line when it reaches the point labeled "imperceptible".

[knutinh]

I repeat my original assertion - the curve should be a flat line when it reaches the point labeled "imperceptible".

How wide is a Gaussian distribution?

If an encoder produce an audible flaw for 1/1000 people, for 1/1000 source materials for every 1/1000 times, that is a lot of tests to sort through blindly in order to find that one audible corner-case. And you never know if it is there until you find it.

If (and that is a big if) subjective score can be modelled as simple functions, then one could do simple, small-scale listening tests designed to extract those parameters, instead of determining the absolute threshold of audibility. If this extrapolation is sane (I have no idea if it is), then one could predict the outcome of exhaustive, expensive listening experiments from small ones, and say something clever about the likelihood of a given flaw ever being detected, right?

-k