Why Live-vs-Recorded Listening Tests Don't Work Options

[solive]

Thomas Edison was probably the greatest stereo salesman that ever lived. He believed that "listeners will hear what you tell them to hear", and he was pretty successful convincing thousands of listeners that his 1910 Edison Diamond Disc Phonograph reproduced recordings that sounded identical to a live performance. His secret weapon was an elaborate live-versus-recorded demonstration that managed to convince people that his phonograph sounded a lot better than it really was.

Several times over the past 10 years, I have been asked by live-versus-recorded apologists why I don't do these types of the tests since they claim they are the only true valid measures of loudspeaker fidelity or accuracy. That is what prompted me to write about why I believe live-versus-recorded listening tests don't work, in this month's blog article

Cheers
Sean Olive
Audio Musings

This post has been edited by solive: Jul 10 2010, 11:30

[kdo]

I have a couple of questions.

Live and Recorded Performances Must Be Identical

For live-versus-recorded tests to be valid, the live and recorded performance should be identical, having the same notes, intonation, tempo, dynamics, loudness, balance between instruments, and the same location and sense of space of the instruments. Otherwise, there are extraneous cues that allow listeners to readily identify the live and recorded performances. Midi-controlled instruments (e.g. player pianos) are but one example of how this problem could be resolved.

Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

I'm thinking of such scenario: suppose we need a test with 20 trials. Take 20 different singers with different voices, make 20 recordings. And then let some 20 more singers (again, all different voices) perform during the test, a sort of A/B test.
This way, I'm thinking, the singers don't even have to perform the same piece of music. It could be different music material every trial/performance.

Would it be possible to gather any statistically significant result from such a test?



And my 2nd question: can we consider our everyday practice of enjoying recorded music as the "ultimate" proof that such recordings are indeed capable of creating an illusion of live performance?
After several decades of such practical experience all across the globe, perhaps we already have enough evidence to draw some statistically valid conclusions? or still not?
I mean, okay, measuring the accuracy of a particular loudspeaker is one thing, but can't we say anything definitive of the technology in general?

This post has been edited by kdo: Jul 10 2010, 16:55

[kdo]

Would it be possible to design a valid test with the opposite approach? That is, instead of trying to reproduce a single performance identically, could we use various different performances and recordings every time?

A quick follow-up on my first question.

I did some googling and found that I was actually thinking of a kind of test called "Randomized controlled trial" (RCT).
The "explanatory" type of RCT with "parallel-group" design and "allocation concealment", in particular. The goal of such RCT is to test the 'efficacy' of a treatment or medicine given to a group of patients.

So, here goes my analogy:
* 'efficacy' = ability to create illusion of a live performance.
* Participants (patients) are the singers/performers.
* Half of the participants are allocated to receive the 'treatment' (record and playback via loudspeakers),
and the other half is allocated to receive no 'treatment' (perform live).

The problem, I guess, is that it might be not quite properly triple-blind, since our 'participants' (singers) know which 'treatment' they are receiving, obviously.

But maybe this bias could be eliminated, too: let's make all singers perform live, but let some of them ("control group") perform before dummy-listeners in one room, and the other group perform before the actual audience in another room. Or something like that. Mix them and confuse them.
Then the singers wouldn't know which of their performances would actually count, and the test becomes fully triple-blind, I hope.

So, on the surface it seems that it should be possible to eliminate the need for identical stimuli in a live-vs-recorded test - by using RCT method.

Any thoughts? Anybody?