The Absolute Sound FLAC article Options

[SoAnIs]

My father subscribes to "The Absolute Sound," and in the latest issue they have an article about FLAC. Of course they have no (or very minimal) double-blind testing (something both my father and I believe is necessary for any review to be valid) and a history of endorsing bullshit, so we take their results with a truckload of salt.
They score their comparisons on a scale from 0-200 points, with CD audio as 100 points. The scores are generated by A/B comparison (not A/B/X) to a variety of reference qualities. The first issue crops up with this. 0 = Kazoo. 50 = 320kbps MP3. 100 = Red Book CD, 44.1khz/16 bit. 110 = 48khz/24 bit upconverted CD. 120 = 88/24 upconverted. 130 = 96/24 upconverted. 150 = 176/24 upconverted. 160 = 192/24 upconverted. 170 = 192/32 upconverted. 180 = SACD/HRx/High-res downloads. 200 = "Master Tape?". There are no references between 50 and 100. Furthermore, a "very large" difference of 50 points would "require only one A/B comparison and is so large that a 24-hour hiatus between listening sessions would still elicit the typical audiophile expression of 'Holy Cow! This is a night and day difference!'"
Both their reference systems used on-board sound cards capable of 192khz S/PDIF output into a PS Audio PWD DAC. Motherboard brand and model is not disclosed. Actual sound-card model is not disclosed.
This seems quite suspicious to me. I have been unable to successfully detect differences between 320kbps CBR MP3, wave ripped from CD using Exact Audio Copy, and CD Audio using Foobar2000's ABX comparator plugin. I used Epica's "Design Your Universe" tracks "White Waters" and "Burn to a Cinder" as my references this time. I used the onboard audio of my motherboard (an ASUS P6X58D Premium, Realtek ALC889 audio, with 24-bit/192khz default output set in Windows 7) and a set of Grado SR-60 stereo headphones. My CD drive is a Lite-On iHAS424. I don't have a fancy $4000 DAC. My ABX test results were no better than guessing, nowhere near a "night and day" difference audible with a 24-hour gap. I sincerely doubt that the lower quality of my headphones and DAC would be enough to mask such a difference. Thus, I doubt their measurement scale's accuracy.
They did their playback comparisons using JRMC (proprietary) with 192/32 realtime upsampling. They never once mention using the reference FLAC encoder/decoder, or even FLAC frontend.
The top 3 programs in sound quality were:
JRMC: 150 score for HRx 176.1khz/24-bit to FLAC
dBPowerAmp: 145
Foobar2000: 140
Original HRx wave file: 180
So they're claiming a large drop in sound quality.
Later they claim that converting from FLAC to WAV with Foobar can increase sound quality (where would the extra data be coming from?) and that other programs decrease sound quality. In all cases they claim a decrease in sound quality when compared to the original CD.
They also (via listening tests, no md5 hashing or such) claim that Wav->FLAC->Wav conversions create cumulative degradation. This is, of course, a very big claim.
So I took a wav file (a rip of "White Waters", mentioned above) and used the FLAC reference encoder to compress it. Then to decompress. I compared the MD5 hashes of the two .wav files, and they were the same. Just in case some strange miracle had occurred and I'd found an MD5 collision I compared the SHA-512 sums, and those were also the same. The two files were bit-for-bit identical. Just for fun, I repeated the wav->FLAC->wav process 10 times, and the files still had identical hashes. I then ABX tested the original and final files, and was unable to do better than guessing.

TL;DR: Bullshit detected.

[milosz]

I agree that two bit-identical files should sound the same.

However playing back two bit-identical files side-by-side or sequentially might engender some DAC or other digital-domain data handling difference. Just because the files on your drive are bit-identical doesn't mean your DAC is going to play them back exactly the same. The DAC should play them back exactly the same, but I think there's a possibility it might not. Things like jitter, LSB noise, etc can all come into play. Just a guess, but I would think that EVERY TIME you play a file back- whether from CD or hard drive or whatever- it will decode to analog a few parts per million (billion?) differently.

What I would like to see is a bit comparison in a pair of buffers to see if two bit-identical files present exactly the same bitstreams to the DAC. Does playing a file back happen without error? It might, I don't know. Does digital=>analog conversion happen exactly the same for each set of data, down to the last microvolt? I don't know.

But this is about file playback in general, not about some kind of file conversion loss or file-type-specific effect.

THAT said, I don't see that you could draw a conclusion that FLAC sounds worse than WAV given the experimental setup used by the authors of the Absolute Sound article. They could POSSIBLY conclude that the way their setup played these files back yielded some kind of differences- but you could not extrapolate this to say that FLAC just sounds worse than WAV in general. They failed to control for all the variables, by a wide margin. Very poor experimental design, and inexcusable hubris on the author's part in drawing their conclusions based on this flawed experiment.

It's clear that without double-blind testing these "quality tests" are just so much mental masturbation.

I subscribed to Absolute Sound in 2011, I wanted to check it out. I certainly wouldn't renew my subscription. If you examine the whole "subjectivist" argument in the light of consistent logic, you conclude that these golden-ear reviewers believe that their OPINION is somehow better or "more true" than actual objective reality. This notion is absurd on the face of it, and for anyone who is capable of rational thought the subjectivist approach can be clearly seen to be either the result of genuine delusion or cynical dishonesty.

[bug80]

What I would like to see is a bit comparison in a pair of buffers to see if two bit-identical files present exactly the same bitstreams to the DAC.

Question: are the contents of a Word-file different every time you open it on your computer? Do the words shuffle if you compress it using ZIP? If not, why would it be different for audio?

[DonP]

What I would like to see is a bit comparison in a pair of buffers to see if two bit-identical files present exactly the same bitstreams to the DAC.

Question: are the contents of a Word-file different every time you open it on your computer? Do the words shuffle if you compress it using ZIP? If not, why would it be different for audio?

Bad example... the same as long as it's the same computer and same version of Word. Change those and things are often not quite the same. Things I've seen changed are the appearance of the dots on a bullet list, and how variable width fonts render.