Unfortunately I wasn't able to visit the june 2007 AES Conference in London about High Resolution Audio.
The paper/presentation about a high-res audio listening test seems interesting. I'm wondering if anyone on HA happens to have been there and can share some information.

preview of the paper session:

Without having read the paper, it's not clear to me whether the test was double-blind or not. Apparently it was not possible to replay sources, since the audio source was "live". How reliable would a test like this be ?

In any case it appears to be a test to see which of two digital systems sounds most like an analog one, which is considered 'technically best'. I'd be curious to see the reasoning behind such a claim. Are papers being presented, peer-reviewed?

So far though I haven't been able to find any details of the presentation. From the abstract , if nothing else it looks to me like M&M's sample pool was much deeper....

FWIW, John Atkinson is touting this presentation as evidence against the claims of Meyer and Moran's paper, in the current Stereophile (not online yet)

Here's what he wrote about it in September:

http://stereophile.com/asweseeit/907awsi/





And a month alter he reports a blind test he participated in with various sample rates and mp3 bitrates

http://stereophile.com/asweseeit/1007awsi/

I suspect he's full of shit, frankly, at least as far as getting the whole story, is concerned.

Finally the paper is online as pdf here. I've had a copy for quite some time but discussion is difficult until a paper becomes publicly available.
Enjoy. It's one of those tests the high-resolution proponents have been waiting for.

Kees de Visser

This paper is certainly worth reading for anybody interested in the subject.

This is extremely interesting. The authors write about tests done at NHK to identify the audibility of ultrasonics (above 21kHz), and seem to have chosen to concentrate only on the low frequencies:
Their second goal is even more interesting:


The results are extremely interesting. Some extracts:



In the conclusions, they hypothesize that the wide bandwidth system sounds less transparent because of what they call "noise-like artifacts" in the high frequencies (above 20kHz), which are not passed in the lower sampling rate version of the wide bandwidth system. I am not sure how they support this hypothesis, given that the analog reference system described also passed the ultrasonics.

They also claim that "Listeners judge high sampling conversion as sounding more like the analog reference when listening to standard audio bandwidth." If this effect can be reproduced in other papers using similar equipment, it suggests that the practical limitations in sampled data systems need to be addressed through oversampling (that is, sampling significantly above the Nyquist limit).

It's not very clear in the paper, but IMO it's possible that ultrasonic artifacts are introduced, added and/or modified in the AD/DA process.
It's also important to know that only two sample rates were used: 44.1 and 352.8 kHz, derived from a 128xFs 5 bit delta-sigma modulator. It would have been interesting to know if popular standards like 96 and 192 kHz would behave differently. I can imagine however that time and budget were limited.
Note also that only one brand of AD/DA converter has been used (Digital Audio Denmark). AFAIK it hasn't been tested if results remain identical with other brands.

Kees de Visser

So , how to reconcile these with Meyer and Moran's trials, which used several SACD players, and more subjects, with the conclusion that CD and SACD sound the same at normal volumes? Does it come down to matching digital formats against analog, versus against each other? Surround instead of two channel? THe use of a highly contrived 'musical scene' (an electric motor driving a bicylce wheel in an anaechoic chamber??!!) as probe signal, versus recorded music? THat some listeners may have been in fact judging ANALOG (the live feed) versus DIGITAL (both A/D conversions), in a nonblind comparison?

I found the choice of 352.8 kHz fairly interesting, but I suppose that deriving the signal from the same ADC with the same clock was the only way to keep the test fair. The paper is very light on the details of the design of the digital chain, which leads me to wonder what impact it could have had on the test results. DACs are known to change in performance (notably THD+N) with different sample rates (see the datasheets of most sigma-delta audio DACs for reference).
Absolutely. The fact that the paper doesn't make strong conclusions about audibility, however, means that it's less important to reproduce the results of the trials to support the conclusions.

One thing I see is that although the criterion is which digital system is more like the analogue, the discussion slips into using words like "preferred." Don't know if that makes a difference.

They also seem to be making heavy weather of the fact that their subjects found it hard to tell the difference -- they suggest redesigning the experiment, rather than accepting that the differences are small.

But it's interesting.

Thank you for the link, Kees de Visser !

All I can see is that the probabilities that the listeners heard differences are 40 % and 60 %.

This is inferior to the minimum 95 % of statistic relevance. Thus in both conditions, listeners failed to demonstrate that any difference was audible between the tested stimuli.

I don't understand the point in discussing the fact that 40 % can be said to be inferior to 60 % with 95 % certainty, since both are below the required threshold for statistical significance anyway.

Man, this article has the worst abstract I have ever seen! I had to plough through the whole rag-tag thing just to figure out what exactly they had done! Still not sure if I understand it though...

As I can see it, its even worse than Pio2001 states: The 95% CI intervals between C1 and C2 overlap ironically @ 0,5, which make them equal. Thus no conclusions can be drawn (as stated at conclusion 4). I won't even comment on conclusion 5 .

The whole thing about the lowpassed high resolution version being perceived as better than the non-lowpassed high resolution is very interesting. Their hypothesis that ultrasound frequencies actually cause more artefacts seems to go against their recommendation of using a non-lowpassed high resolution for archiving!

Some more interesting bits of information from the article:

- This appreciation must relate to a number of perceived characteristics other than high-frequency response partly because microphones and loudspeakers normally used in studios do not have a substantially wider response than the audible range.

- In double-blind tests, casual and professional listeners could not reliably identify high-bandwidth and high-resolution (192 kHz 24bit versus 48 kHz 24bit) conditions.

must've missed it when you posted it first.. odd.

anyway



and this quote struck me as very, very odd.
(read the article first, or at least the conclusions/test setup bits before posting.. the phrasing seems to be a bit awkward in places, but i can't really help that)

because unless i'm really, really reading this wrong they're saying that if you upsample to the highest SR your sound card can take before playing it back - no matter the quality of the input material - it will be perceived as 'more natural'.

(mind you, i don't really see why you'd need to do what they suggest next, namely: just turning on SSRC in your foobar dsp chain would seem to be enough)

the point, really, is that i almost can't believe that i'm understanding this right.
A second point: even if they are correct, it would seem that their conclusion that anything >20kHz is a waste of bw is somewhat hasty, and they imo should've added a 'middle' test - to say 40kHz - just to narrow down where the added bandwidth became overkill (more).



well, the topic of the paper is at least remarkably on-topic.. but i don't really see what you base your statement about the AD/DA chain on.
as i understand it, both the 20kHz recording equip and the 100kHz ones were attached to the same chain, so that doesn't really seem relevant here.
furthermore:


so what are they saying here then? that 20kHz cutoff mikes still produce overtones/noise in the 20-100kHz band that's relevant to the recording and is perceived as adding to the perceived 'naturalness' of said recording, even though they're not 'real'?

anyway, what i don't really see is why that would influence the archiving community.. i can see how this might be interesting to recording studios, but by the time it reaches the consumer, the format is already decided on. so why would it be relevant to him to save it in as high a sampling rate as possible?

Aha, perhaps I've found the source of confusion. AFAIK the paper assumes archiving to be the process of preventing (historical) analog source material from deterioration by transferring it to a new medium, which in these days usually means a digital format. On the first page of the paper:
Analog tapes don't age very well and at some day it will be impossible to play them back at all. If it is important to preserve the audio quality of the original, an as transparent as possible conversion process should be chosen. When the analog originals have become useless, all that's left is the copy, so the choice of archive format is very important. The paper gives some evidence that high sample-rate formats provide a copy that is sonically closer to the original compared to standard rate formats.
Since the cost of a higher rate format in the whole process of archiving is rather small, this could be important information for professional archiving institutions.

Until a double blind test can show that high resolution is sonically different than standard resolution, any listener can state anything according to his own taste.
All DBT that have succeeded were flawed one way or another.

This one is not flawed, it just failed.



It doesn't. The results are far below statistical confidence. The test failed to show any difference between the stimuli.

The p values are given in figure 1.
They are 0.4 and 0.6, while they should have been at most 0.01 or at least 0.99 for a type I error probability of 5 % (probability to have got the result by chance) because there are four possibilities of success : p1 <0.01 OR p1 > 0.99 OR p2 < 0.01 OR p2 > 0.99, which roughly multiplies the probability of getting a success by 4.

I'm having a hard time sorting through the math, someone tell me if I'm offbase...

They wanted to test whether sample rates of 44.1 kHz or 8 x 44.1 kHz sounded more like the original source. For those two choices there were two conditions tested: C1 (100 kHz audio bandwidth) and C2 (20 kHz audio bandwidth). (p.3, very last paragraph).

Under C1 conditions, the probability that the high sample rate (8 x 44.1 kHz) sounded more like the analog was 0.4 (p.8 table 3). Does that mean that the probability that the low sample rate (44.1 kHz) sounded more like the analog was 0.6?

And then for C2 (table 3 again), 0.6 probability that high sample rate sounded more like the analog. So does that mean 0.4 probability that low sample rate sounded more like analog?

Am I reading the numbers correctly?

If so, then I have to agree with Pio2001. This test failed to show that either of them sounded "more" like the analog. There was no statistical significance either way. I don't understand how conclusions 2 and 3 on p.9 can say "significantly smaller" and "significantly higher" when it seems like they really should say "a bit smaller" and "a bit higher".