Extract HDCD, How I Can extract HDCD by software? Options

[SCIF]

How I Can extract HDCD by software(EAC, WMP9...)?

[SebastianG]

One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


Cheers!
SG

This post has been edited by SebastianG: Aug 27 2007, 18:23

[SiriusB]

One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!

Hi guys, great thread. I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs; the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD. You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?


Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe. FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example). I couldn't tell them apart in an ABX.

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs). THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction. see this post for examples.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them. Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed. I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.

From what I've read (e.g. John Marks' self-interview about his HDCD releases) it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion. So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

This post has been edited by SiriusB: Aug 30 2007, 03:40

[Christopher Key]

Hi guys, great thread. I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs; the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD. You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe. FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example). I couldn't tell them apart in an ABX.

That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC. (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly. (Not strictly true. A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested. I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you? I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs). THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction. see this post for examples.

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them. Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.

From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels. If you're so inclined, this can be fixed with a hex editor.

All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')

There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?

From what I've read (e.g. John Marks' self-interview about his HDCD releases) it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion. So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

Regards,

Chris

This post has been edited by Christopher Key: Aug 30 2007, 16:34

[krabapple]

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.

Theoretically very desirable, but practically not do-able in any easy way....so far. I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)

That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC. (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly. (Not strictly true. A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested. I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you? I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia

It was Joni Mitchell 'Mingus'. FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around. To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks. For hdcd.exe of course all I have to do is feed it the individual .wavs directly. I can do it all again and compare the files. Also I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.

Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.

Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'. The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)

CODE

undecoded
    Left    Right
Min Sample Value:    -32452    -32429
Max Sample Value:    32440    32415
Peak Amplitude:    -.08 dB    -.09 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -138.15 dB    -139.94 dB
Maximum RMS Power:    -2.58 dB    -4.66 dB
Average RMS Power:    -11.14 dB    -11.51 dB
Total RMS Power:    -10.54 dB    -10.9 dB
Actual Bit Depth:    16 Bits    16 Bits

Using RMS Window of 50 ms



decoded
    Left    Right
Min Sample Value:    -16226    -16214.5
Max Sample Value:    16220    16207.5
Peak Amplitude:    -6.1 dB    -6.11 dB
Possibly Clipped:    0    0
DC Offset:    0     0
Minimum RMS Power:    -144.17 dB    -145.97 dB
Maximum RMS Power:    -8.6 dB    -10.68 dB
Average RMS Power:    -17.16 dB    -17.54 dB
Total RMS Power:    -16.56 dB    -16.92 dB
Actual Bit Depth:    17 Bits    17 Bits

Using RMS Window of 50 ms

I can send this track to you too if you like.

From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels. If you're so inclined, this can be fixed with a hex editor.

Thanks. I'd much rather just use hdcd.exe, so long as it's creating files that are identical in all the ways that matter, to what the WMP WAV plugin generates.

There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.

Check out the link I made, to my Steve Hoffman forum post There I document a variety of hdcd behavior -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor). How can I tell if the 3 bits are in use?

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?

Beats me. I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters). Maybe there's more options?

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'. But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light.

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').

This post has been edited by krabapple: Aug 30 2007, 20:10