Hydrogenaudio Forums > Potential Biases in MUSHRA Listening Tests ?

Full Version: Potential Biases in MUSHRA Listening Tests ?

Hydrogenaudio Forums > Hydrogenaudio Forum > Listening Tests

Kees de Visser

Dec 7 2007, 16:50

Did anyone attent the presentation or read this paper? I'm curious to hear comments. 20% seems a lot, but it might be good to know in which conditions that occurs.

QUOTE

P3-8 Potential Biases in MUSHRA Listening Tests—Slawomir Zielinski, Philip Hardisty, Christopher Hummersone, Francis Rumsey, University of Surrey - Guildford, Surrey, UK
The method described in the ITU-R BS.1534-1 standard, commonly known as MUSHRA (MUltiple Stimulus with Hidden Reference and Anchors), is widely used for the evaluation of systems exhibiting intermediate quality levels, in particular low-bit rate codecs. This paper demonstrates that this method, despite its popularity, is not immune to biases. In two different experiments designed to investigate potential biases in the MUSHRA test, systematic discrepancies in the results were observed with a magnitude up to 20 percent. The data indicates that these discrepancies could be attributed to the stimulus spacing and range equalizing biases.
Convention Paper 7179

ff123

Dec 7 2007, 23:55

I imagine it touches on some topics which can be found in this paper:

www.surrey.ac.uk/soundrec/ias/papers/Zielinski.pdf

Regarding one point in the above paper, I have often wondered how valid it is to average scores across listeners when we do group tests. The data is all there, I guess, for anybody who has an uncontrollable urge to look for multi-modal distributions and figure out where they might come from :-)

ff123

Woodinville

Dec 8 2007, 00:42

In short, what the paper pointed out is that anchoring is essential to the test, and that you can get rather surprising results when anchors are not in the positions expected.

Also, the scale was shown to stretch and contract with various qualitites of stimulii.

My problem is simpler.

It is well known and demonstrated that if one has 4 different signals, a may be preferred to b, b to c, c to d, and d to a. There is no "transitive" nature to the subjective response.

But MUSHRA ignores that.

This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.