"Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". E. Brad Meyer and David R. Moran. JAES 55(9) September 2007. It's worth noting that members of the BAS wrote the paper and performed in the tests. You'll need an AES membership to access the article, so no link.

Abstract:

Interesting, thanks!

Very interesting...people are still going to claim they hear a difference anyway, though...

Thanks for the info. Any details about the A/D/A loop ?
Discussing a publication based only on the abstract can be quite dangerous. However, complete AES articles are only available to members, or can be bought by non-members for $ 20 each. It is not allowed to share copies of AES documents with others. This makes public discussion rather difficult, although "It is permitted to quote from this Journal with customary credit to the source."
Interesting papers like this make me consider renewing my AES membership to have access to the online JAES and their On-line Electronic Library, despite the restrictions

Thanks for this

The test results show that the CD-quality A/D/A loop was undetectable at normal-to-loud listening levels, by any of the subjects, on any of the playback systems. The noise of the CD-quality loop was audible only at very elevated levels.

Sorry, does this mean they were testing whether this noise from the ADA conversion could be heard or not, or if there was an actual difference in sound quality between the hi-res and lowered-res sound? I.e. maybe there was a detectable difference in SQ but the question asked of the subjects explicitly referred to that noise only?

I'm confused.

Can smb remind me please if there was any AES article that would prove audibility of lowpass higher than 16 khz in music? Is there a consensus in the AES circles on this matter?

(The question is not directly related to the paper in the OP. Sorry)

It seems to me like they played hi-rez content downsampled to 16/44. That would be the fairest comparison, the high-rez master and the 16/44 version made directly from it.

And no one could tell the difference, what a huge surprise.

The title as well as the Abstract of this article seem only to refer to noise levels. No one should be making any inferences to sound quality. The conclusion of the article is that no one can distinguish a -96dB from a -120dB noise floor except at very high levers, which everyone in this forum knows already.

Manuel

@ hushypushy:
You've rush-read my post and haven't answered my query.

I suspect only the OP can, or anyone else who's read the paper, unless I'm being completely dim.

To clarify, I'm unsure how to interpret the abstract as quoted, because it only talks about detecting the presence of the ADA loop, not general sound quality.

@ madman:
Okay.

Did they noise shape the 16-bit audio? Noise shaping, as we know, decreases the perceptual noise floor. This should improve the perceived quality of the 16/44 content.

The several comments about lack of inferences on sound quality do not fit either the title or the quote. It says no one can tell whether straight SACD or resample audio is being played. When no one can correctly guess whether it is SACD or resample to CD spec audio, the only rational conclusion is that there is no audible difference in sound quality.

It says there is an audible difference is noise, but only when the volume is turned up very high. The only thing really unclear is “what noise?” Is it the CD noise floor, which cannot be heard in well mastered recordings except during very low level passages (played very loudly) , or is there some unusual (not to be expected in a normal CD) noise from their 16-bit/44.1-kHz "bottleneck."?

Sorry, I haven't had a whole lot of time to summarize the paper. I just wanted to be the first one to get the scoop

The paper is rather light on details of the setup. They don't list the speakers they use, or the DAC/ADC they use, or the analog gain stage they use to match levels. That's pretty disconcerting, but I'm unsure exactly how much that matters, given how many people were involved in testing. (They also tested at several different venues.) I'd suspect that nobody would be placated if they were more detailed in their equipment descriptions. They used an ABX CS-5 for switching. They tested the quantization level of 16/44 by ABXing the system with no music playing at +14db above the normal listening volume. When playing real music at that level, they described it as "unpleasantly (often unbearably) loud".

The paper concludes with the note that the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons. So it ends on a surprisingly pro-high-res note. I'm actually more psyched up to buy SACD and DVD-A now compared to before I read it. I certainly haven't given much attention to high res releases, partly due to the copy restrictions, but partly because I always figured it was all bollocks and I might as well stick with CD.

In summary, I don't think this test will convince any audiophiles about the uselessness of high res. Some people will always come up with a bullshit excuse to believe what they want to believe, and there are a lot of people who have blind faith in SACD/DVD-A. But it could give producers more leeway to release well-mastered stuff on CD instead of more obscure formats. Or it could convince more people to get high res players to listen to better mastered stuff. Either way, listeners win.

They tested the quantization level of 16/44 by ABXing the system with no music playing
Was this in addition to ABX tests on the music, or were they really just testing what people thought about the noise? Why would they bother with the noise? It is easy to demonstrate the difference between quantization noise under those conditions (no music, high volume level) and no quantization noise because of good noise shaped dither, but who is going to avoid dithering during mastering anyway? i.e. what is the point of the test? Is there any way it relates to differences between SACD and CD, or were they just exploring a difference from CDs and their setup i.e. they didn't (noise shape) dither the 16/44.1 reduced version?


the high res releases sounded much, much better than the same music on CD, for well-known mastering reasons
This is still not clear as far as these tests go. The quote you provided above seemed to say that when this SACD mastered music was "passed through a 16-bit/44.1-kHz "bottleneck."" it could not be distinguished from the original.

If so, then the tests say there is nothing at all about SACD per se that is better -- except the mastering standards, which are quite irrelevant to the tests. Those differences from CD are arbitrary processing decisions and have nothing per se to do with the different technologies. If the mastering differences were not eliminated from the test, the tests would be as useless as those dual layer disks that are purposefully doctored to convince the `gullible that SACD is inherently better.

I think your statements are pretty clear, but I don't think everyone will take them the same way.

OK, I finally got the paper (written by the Boston Audio Society). Test setup was a hi-res media player (DVD-A and SACD), connected directly to the ABX switcher. The same player's output was connected to a 16/44.1 HHB (professional) cd-recorder (not listed, but clearly visible on a picture). Analog level matching was done only in the "lo-res" path, which was also fed to the ABX switcher. They did listen to music material:
The high system gain was about 14 dB. At that level, the noise difference became noticable, but music at the same play-back level was found to be "unpleasantly (often unbearably) loud".

ps:

Sounds like more than just quantization noise was introduced with their CDDA "bottleneck". Because of this and beyond the fact that they jacked the volume to an unrealistic level to reveal the differences I'd say the evidence is even more damning that hi-res formats provide no tangible benefit when dealing with 2-channel audio.

It implies the best and most convenient content available to consumers would be the SACD / DVD-A versions, recorded onto CD-R!

(because of the "well-known mastering reasons" - i.e. commercial CD releases are trashed with too much DRC, SACD/DVD-A sometimes are not)

I like the implication (which I agree with entirely) that you can make a better CD at home (using a good source, and a CD recorder's analogue input) than you can buy in a shop. It's quite sad.

Cheers,
David.

I think, the result of this paper and tests are clear and not surprising due to previous experiences, knowledge and tests.

Maybe you wonder about their "final conclusion", that DVD-A/SACD offer quality benefits compared to corresponding CDs, due to mastering reasons.
Well, simple answer: Though the AES is more or less independent and working scientific,
it is with every science, science needs money from the industry (and vice versa the industry needs sometimes researches by scientists), so the performers of that study have printed this sentence to their paper, to have something compromising sounding towards their industry... (without changing facts, truth etc. !), though Joe Average and the mass-magazines might focus only on this sentence, and change it only a litlte bit to "DVD-A/SACD proven to be superior to CDs !"


Don't laugh, a writer for HiFi-magazines in Germany has done already this in past, he conducted (a very poor conducted) similar test, and though his results, which he published as some kind of diploma work, indicated that CD and high-rez are on par,
but he managed to cite his diploma thesis falsely in magazines he wrote later for, writing to the audiophile masses, "a" diploma work had shown, hi-rez is superior (without mentioning, it was his own crappy diploma work/thesis) lolololol.

In audiophile internet forums, this guy, his diploma thesis, and his magazine articles were torn "in the air", even by audiophile guys !!

At least, this episode has shown to me, that even "audiophiles" (nothing wrong with being audiophile, we all are!) know about the marketing crap written in certain HiFi-magazines, if they are informed eg. by internet. Not everybody buys, what's advertized in those magazines.

Theres really no need to move to an entirely new format though. Just more push for studios to release properly mastered CDs on labels like mofi or whatever.

Sorry, but hi rez is no guarantee of good mastering. I know for a fact that there are DVD-As out there that have the highly restricted dynamic range of modern CDs (Yes 'Fragile', anyone?). You're still taking your chances.

Agreed, but right now, the chances are not so bad. It all depends on how many producers do wind up remastering. Allegedly most of them have been doing it, but that could change in the future.

Moreover, if everybody really believed that the higher res is the ultimate reason of improved quality, then the perceived benefit of improved mastering (which is the real reason for improved quality) could be drastically reduced.

The myth of high res quality directly hurts consumers by increasing the risk of a shoddy mastering product.

So I've been monitoring Audio Asylum's thread on this (yeah, I know, bad idea). They do offer a few good criticisms.

• JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.
• Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.
• The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.
• Musical selections are not listed.
• Listeners are not described in terms of experience in any detail.

Too bad. I guess we'll have to wait for Atkinson & Co to perform the slam-dunk scientific demonstration that hi-rez really audibly matters

>> * JA asserted that the high res player they used, a Pioneer 563A, does not have any better of a dynamic range with high res recordings compared to CDs. Pioneer sez it's 108db, but hey, maybe it is lower in reality. It's asserted that this could obscure details in the high res listening, while still being low enough that the added quantization stage at 16/44 increases the noise level enough for audibility when listening to silence at loud levels.

These are not good criticisms. Just pro-forma nitpicking, so it appears something is wrong with the experiment. (There has to be, right?) There is nothing wrong with it. We used more than one hi-rez player. The noise floor of the venue was incredibly low. The loud levels were very loud. The source material --- lots of it, the widest range we could find, samplers, special demo cuts --- was extremely quiet in its noise floors.

>> * Like I said, virtually no details are provided on the exact equipment used. The BAS website mentions the 563A in passing in an old article describing the test.

It does not matter what we used, since we degraded a hi-rez system and no one heard any difference, ever, regardless. We could have used an even better-quality 16/44 loop. But perhaps a genuinely compelling reason will arise and we will list the gear. The experiment was expanded to several other venues, including serious tweak systems, recording studios, and the like. No difference in the ability of the listeners to hear the "degradation."


>> * The statistical analysis is rather shoddy. The test results are not broken down by listener, or listening location, or by almost anything else. Type A and B error is not defined. The null hypothesis is not defined.

What could this mean? Of course we spoke to statisticians, who were unanimous in saying it was a straightforward test of detectability, yes or no. Either some listeners heard a difference or not. No one did. No one came close. Any listener, any venue, any material. We did go over the results sorting by hearing bandwidth, sex, age, and experience. You will have to read the paper to be convinced, or not. If you really cannot afford to buy it, email me at drmoran@aol.com.


>> * Musical selections are not listed.

Made no difference. This kind of thing is just to try to poke holes in an ironclad coin-flip result. We got a very wide range of material, types, instruments, vocals, genres, etc. Special stuff made for hi-rez demos, etc. etc.


>> * Listeners are not described in terms of experience in any detail.

Not so --- just not enough detail for some who simply cannot believe the results.


David Moran

The genuinely compelling reason for a detailed methods section in any scientific paper, is so the work can at least theoretically be reproduced. And as you've seen, the subjectivist camp/Audio Asylum inmates will be all over you if you don't document everything scrupulously.





It suggest you did not publish the Type I and Type II error statistics associated with your tests. Most scientific papers that involve statistics at least include a Type I error (p value).




I'd be happy to buy it, but it appears to be not yet available for purchase. I'd love to get a copy.



Believe it or now, most of us here are very much on your 'side'. Hydrogenaudio even has advocacy of blind tests built into it Terms of Use.

It's just regrettable if a paper on this perennial hot topic failed to provide extensive method detail, because every omission will be used against it. And thus a topic that should have been laid to rest years ago, will remain 'hot'.

Perhaps you could set up a 'supplementary materials and methods' webpage to provide those details?
This is common in journals where printed space is at a premium (e.g., Science, Nature).

Thank you for replying (and hopefully you'll stick around!)

Well, duh! Of course it's nitpicking. That point isn't evidence of anything actually being wrong with the experiment - but it seems to me like it could potentially obscure real issues. Or, at the very least, the information is useful if another blind test would be attempted, in order to meet or exceed the technical specifications of the original test. (I'm not saying that I'd be doing it, but again, nice to know.)

Why would you wait for a reason to arise before listing the gear? That isn't a rhetorical question, and I'm not trying to be confrontational. I just don't understand. A lot of the individual ABX tests conducted here have pretty detailed gear descriptions associated with them, so I figure an AES paper would do the same thing...

I already emailed you (and Mr. Meyer) 3 days ago, actually. And I posted here when I first saw the complete article in the online JAES.

You're allowed to drag me back to the woodshed and beat me for calling the statistics "shoddy", if you want. Heck, I think I may have the least statistics experience of anybody commenting in this thread. What I meant to say was that the paper violated my expectations of how much detail is supposed to exist in the results/analysis. Based on other blind tests I've read, I would have expected to see clearly defined null hypotheses, the full listener responses, numerical analyses thereof, etc. Maybe some lip service to estimating the proportion of discriminators (which, honestly, really ought to be 1 anyway - but still, it's a subject worthy of debate, and I'd figure it's worth commenting over).

The results as they stand in the paper seemed to breeze through a lot of this very quickly, which surprised me. That doesn't diminish the results any - they are shockingly conclusive. They're just, well, short. And I think many audiophiles are using that as an excuse to ignore them completely. (Granted, though, most of them would probably ignore the paper anyway.)

Alas, you're confusing me for somebody who doesn't believe the results.

I really do. They match my personal evaluation of SACD/DVD-A, they match my understanding of audio engineering and psychoacoustics, and the obviously large amount of work that went into the setup and listening, combined with the large number of listeners, makes them extremely compelling. And as the rest of this thread indicates you'd be hard-pressed to find anybody else here who would not agree. Enough of us have done our own personal ABX tests to understand how much effort goes into them, and what kind of results one can draw from them.

But when such a compelling result is established, why be stingy with so much information? You state that the ball's now in the pro-highres camp to show any sort of proof of audibility, but it seems like if anybody actually attempted another test like this, they'd want to know all of these things, in order to try to strengthen the sensitivity of the test. You spend a considerable amount of effort in stating that so many of these factors were carefully eliminated with the best testing conditions, and yet you don't actually say how you do it. "Trust us" is not a valid argument!

Already a lot of audiophiles (not me!) are using the lack of information in this paper to justify some sort of AES cabal that does not actually engage in peer review. Like krabapple said - why try and settle the matter for good, when you've thrown them such a juicy bone to chew on at the same time? Unless the paper was meant only for AES readers, not for audiophiles.

As far as I'm personally concerned, all of those issues are really of omission than of actual error. I just want to know the information. Or, I just want to know why it's not there. It's mainly because I'm curious. I'm not expecting to find any flaws in it. And I had the expectation that asking for information like this is an extremely reasonable request.

I contacted the authors and they replied that they used most of the time a Yamaha DVD-S1500 (but not only that particular player) and the cd recorder was a HHB CDR-850.

Although it is a pity that they didn't specify exactly what they have done and so on, it is a strong indication that frequencies above the Nyquist frequency of the cd format are not needed in a strict sense. Also the increased resolution (dynamic range) of sacd or dvd-a is not a must have.

The burdon of proof is shifted I would say.

Regards,
Jacco

Okay. Thank you all for your thoughtful responses, and I apologize for misinterpreting or impugning anyone here. (Or indeed anywhere.)

Rightly or wrongly, the chief reason we did not provide details was to preclude, or try to preclude, being nitted to death over the details. Which surely would have happened. The hi-rez players sucked, your amp and preamp suck, your speakers suck, so of course you could not hear anything. Etc. etc. Quite apart from the other venues and the number of listeners and trials.

So, as I say, rightly or wrongly. Y'all will observe that none of the AESJ editors or reviewers felt the lack of specifics was of any import whatsoever. No one mentioned a thing, not even once. We were asked for some more breakout of the results, which we provided.

Nonetheless, since we are receiving almost exactly the same flak as if we had given all the specifics, I am sure at some point, soon, we will post it all somewhere, perhaps on the BAS website.

Thanks for your modulated reading and responding.

David Moran

Thanks David,

Good to talk to you this way

I think you did a great job by doing this experiment and if it was really true that there is a big difference between 44k1/16bits and high resolution, this should have been the outcome of your work. If it is that hard to differentiate, one could ask themselves the question what good it is in the first place to make such a big fuss about high resolution formats for consumer applications.

Regards,
Jacco

Good job. If you post the specific, then it's up to the naysayers to prove they weren't sufficient to the task.

IMHO it's a pity that the AES doesn't provide any standards or even guidelines regarding subjective listening tests. The AES technical committee on Perception and Subjective Evaluation of Audio Signals (PSEAS) doesn't seem to be of much help in this respect. Anecdotal evidence is still widely accepted in the pro audio community. A possible reason could be that most engineers don't know how to set up and conduct a proper listening test. An often used excuse is that it involves a huge budget and lots of time and therefore can only be done by large companies and institutions.
AFAIK Perceptual Coding is the only area where subjective evaluation is used methodically. This forum is a good example of that.
I do hope that the AES (or another organisation of equal importance) will provide standards or guidelines and make an effort in promoting them.

Kees de Visser

Thanks for your well thought out response and attention to this issue. Unfortunately the AESJ study would have been nitted to death on internet forums regardless. It's best to take things written on forums with a grain of salt. In the end the great majority of people believe what they want to believe. I hope to see more well designed and performed studies in the future.

What they said. Thanks for responding.

Clearly if there's a disagreement here, it's only in how much test detail in a paper is considered reasonable. And I can totally understand your position.

For everybody else here, Meyer has posted a number of extremely informative posts on AA, at least until the test details are released.

I like those responses. It will be interesting to hear which recordings were considered "the best commercial recordings we've ever heard" - and how they could easily sound just as good on CD, but apparently don't (presumably due to mastering, or non-availability).

Cheers,
David.