How I Can extract HDCD by software(EAC, WMP9...)?

HDCD just uses the standard 16 bit CD Audio encoding, so EAC will rip the "HDCD" part too.

For playback, WMP9 can decode the HDCD.

But only from CD, not from the ripped audio files, apparently (according to this post).

More about HDCD here.

Perhaps it could work when you mount the ripped wav with Deamontools.

I was thinking something like Virtual Audio Cable would let you get the HDCD decoded result, but I don't think those tools deal with >16 bit audio which makes them useless for this.

Is there programm for ripping HDCD in 24/44.1?

I don't think so, CD is 16bit.
More interesting:
Is there a program able to decode the HDCD info and to convert it to another (hIgher, 32bit float for example) format, and everything in digital way (no analogue step) ?

And what I'm always wondering:
Is HDCD a way to circumvent loudness; does a HDCD played as HDCD sound less clipressed compared to played as CD ?

I seriously doubt that this would be worth the effort. Having read the technical infos about HDCD which are publicly available (the AES paper) I currently believe HDCD to be just a technology with quite low benefit compared to plain noise shaping. It's just a thing to make a bit money.

They are claiming 20 bits dynamic resolution. This is actually 16+1+3 bits where approximately one bit is gained via this peak compression and three bits are gained via noise shaping. While playing an HDCD signal WITHOUT a special decoder you'll roughly experience the same signal to noise ratio compared to the output of an HDCD decoder. The peak decompression mechanism which is part of an HDCD decoder is fairly simple, but it won't increase the signal to noise ratio because most of the signal (except the peaks) will get attenuated by 6 dB.

IMHO the best thing would be to leave the format (44 kHz, 16 bit) the way it is. Even without a special decoder you will still benefit from the noise shaping part and get roughly 19 bits dynamic resolution.


SebastianG

edit: grammar

I was always under the impression that HDCD was simply a better dithering algorithm.

HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG

Yes,

EAC will extract the HDCD bit for bit,
you need a Lossless compression format (wavpack, flac, ape), or the original waves. (see for Lossless extraction www.high-quality.ch.vu - users' audio guides

eg. foobar2000 will play those files perfectly, if you set the bit depth to 16 bit, 24 bit will destroy the readability for an external HDCD decoder like built in amp.
(same for 5.1-dts-wav-CDDA).

If eg. foobar2000 plays the Lossless files, you need a soundcard, which ddoesn't upsample internally from 44.1 to 48 kHZ, and digital output to external hdcd decoder, like in amps.

I just noticed this thread, and given that I finally installed Windows XP last week I thought I would give try the HDCD playback in WMP XP.

I think the only HDCD I own is Pet Sounds by The Beach Boys, the 2001 remastered version.

I went into the devices menu and checked the box "play back audio CDs at 24 bit", is this all that is required to activate HDCD playback? I have a Chaintech 710 sound card set for 2 channel high sample rate mode.

I restarted WMP and when I started to playback the CD the first thing I noticed is that the volume was significantly quieter. Does this mean HDCD playback is working? Is there any other visible way to see that WMP is playing the CD back properly? I'm not really sure if playback sound better, but it is quieter than before I had checked the 24 bit playback box.

as written above, normally 24 bit playback of audioCds destroys headers, like for HDCD, or 5.1-dts-CD

the HDCD has all information inside its standard 16 bits, the additional informations are stored in least significants bits, maybe inside the 16th.

The wmp player should show "HDCD" if it recognized a HDCD.
iirc, previous wmp version showed hdcd, but didn't decode ?
is it safe now, that wmp decodes HDCD ?

A safer way is,
as written above, play HDCD by foobar2000, 16 bit, and transmit the digital output to an external amp/hdcd-decoder.
which will show u the hdcd logo, if it recognized the true hdcd content an decodes it.

I think, most soundcards don't have such a good analogue output, that u will benefit from wmp hdcd decoding, better to let do the D/A conversion and analogue site of the chain, by external amp.

Sounds very hopeful. Usually quieter means better.
My question still has not yet been answered. Could it be that mastering engineers 'abuse' HDCD technology to hide a less clipressed version of the audio inside the clipressed one ?

No, I don't think, that's possible.
because they have only 16 bits stereo
So, they have maybe 15 bits for the CDDA wave, and the 16th, least significant bit contains a digital compressed information, which can only be decoded by HDCD decoder.
So, if you play a HDCD via a normal CDplayer, the 16th bits would be converted by normal DDA-converter to pure noise. (quite silent noise of course, it is the 16th bit...)

bryant (wavpack developer) offered some time ago a tool, which was able to remove the 16th bit with hdcd information. This should enable to ABX hdcd version against the 15 bit CDDA version.
Dunno, where he offered it (at r3mix forum iirc), ask him,
though I should have it somewhere on my HD or a DVD backup.

It is possible. Why not ? Since they can play with the dynamics, and choose to have the player enhance it or not.

Then, again:
Is there a software (or any other way besides a digital-analogue-digital process), allowing me to convert HDCD to 20bit (or better) (or maybe only quieter 16bit) ?

To fully benefit from the HDCD encoded informations you have to convert the PCM stream to 88,2 kHz @ 24 bits. But I guess you only want to reverse the optional "peak compression" thingy which won't require upsampling. However, I don't think it'll be worth the effort. (!)

Is there a software ? Possibly. I don't know.
Is there a way ? Sure! Theoretically.
Does it make sense ? IMHO not.


SebastianG

No software, exept maybe the combination of Windows Media Player with something like Total Recorder.

Depends on your personal priorities...In my opinion, 'manual SBR' wasn't worth the effort... (I listened to those samples.)
Anyway, thank you for your help.

just to resurrect this from the dead (because I've been playing with HDCD + Total Recorder) TR allows a max PCM recording format of 48 kHz/16 bit. So if 88/24 is really required for proper PCM capture (as per SebastianG) , TR isn't the answer.

I've approached this repeatedly but am still not quite certain how to do a proper HDCD decoded vs nondecoded data (not listening) comparison. I'd like to be able to compare .wav data using soemthing like Audition. I've tried redigitizing the analog output from an HDCD-decoding player, and comparing it to the same file ripped, but in the end I'm not sure what I'm looking at. Flat-topped peaks seem to go away but I'm not sure that isn't just due to capturing an analog output.

And btw I can't for the life of me get WMP 11 to recognize an HDCD (i.e., show the HDCD logo). Anyone succeeded with this?

There has been a long running thread on the Slim Devices Audiophile forum where we have been working to decode HDCDs so we could play them back on our Squeezeboxen and Transporters. Best to read the thread backwards, its now 11 pages long!

We have reached the point where we definately have made progress. Using Windows Media Player 9, we can get the HDCD icon to light up, and we have successfully recorded bits using the Chronotron WAV Plugin for WMP9 (using a 24b audio card and the 24b option enabled).

The issue at hand is if the plugin is actually recording 20b and writing a 24b WAV (actually its WAVEX format) or just recording 16b and padding zeros to 24b.

The only other alternative I have found to date is to use Virtual Audio Cable (VAC) to see if it gets the same result as the plugin or something different. I have not yet tried this option.

If there is someone here who really understands HDCD, what is the expected result when comparing a 16b rip to a 24b WAV capture via WMP9? What should the waveforms look like if the decoder worked and the plugin actually captured 20b? Personally I would have expected increased dynamic range in accordance with the increased bit depth, and an unwinding of any range compression. So I should see spikier peaks and higher peak to average. Yes?

Here is my comparison of the 16b rip and the 24b capture:
The 16b rip is the trace on top and the 24b is the trace on the bottom. Closer inspection shows the peaks in the 24b capture are exactly 6dB down from the 16b capture. However, the peak to average seems about the same, thus the dynamic range did not increase. This suggests we don't actually have the HDCD decode captured.

Wow, I wish I knew about this plugin before.
Finally I can automatically extract HDCD info!

@Eric Carroll:
All HDCD titles don't use peak extension, try some other titles.

Glad to help on the plugin... It still means you have to play the whole song in real time...

But we have two users who seperately captured via the plugin and all we saw was a decrease in peak levels.

I only have one HDCD that I know of... I was doing this as an experiment, hoping to get to ABX the before 16b and after 24b to see if it was worth buying more. While we have put alot of work in on this, we haven't yet gotten to the "is the difference audible" stage yet...

If you have some HDCDs, could you try doing a 16b rip, then a 24b capture and compare them in Audacity to see what result you get?

Not if you tick "Disable audio output(fast recording)".

I will upload some waveforms as soon as I have the time.

Since I got too excited about this, here are the waveforms:

Sound Chaser from Yes - Relayer


The WMP output gave an extra 6 dB headroom, no more shaved peaks

Sister Andrea from Mahavishu Orchestra - Lost Trident Sessions


I amplified the waveforms for this song to the same relative volume.
This didn't get as much peak extension as the other title, but still a good 3 dB increase in dynamics.

Thanks for looking into this.



What exactly do you mean by this? You normalized them in Audacity? What do they look like without matching levels? Why was level matching required?

I didn't change my levels at all. If you could describe your procedure I will check my results against it.

For the sake of visual comparison I adjusted the waveforms of Sister Andrea to the same relative volume, the original would be 3 dB louder and the WMP output 3 dB quieter. I other words the WMP output for Sister Andrea was 6 dB quieter overall, but the peaks went up to -3 dB.

Well your results sure look like what I expected to see: clipped material becoming unclipped. Maybe my HDCD is just not encoded with the compression part enabled... Puzzling. I certainly see the 6dB quieter effect, but no enhancement on the peaks.

Just to confirm, you are using WMP9?

I used WMP 10.

I recall someone mentioning on the audioasylum forum that this peak compression isn't used on every HDCD disc. (Sorry, no reference) This should explain your observations.

Cheers!
SG

I have an alternative idea to capture the 2 streams (1. 15-bit+16th-HDCD-bit-as-noise --- 2. the-full-HDCD-decoded-wav).

If you have a soundcard with digital in and out, capable of 24 bit (and not resampling 44.1 to 48 kHz should be a basic, otherwise these tiny differences could be destroyed anyways),
plus an amp/receiver with HDCD decoder, you might be able to play the HDCD as normal CD and record it again,
and the next step:
Play digitally the HDCD, the amp decodes as true HDCD, and loops with some luck the decoded signals back to soundcard, which with some luck could record this.
But I am not sure, if soundcards and amps are capable of same time passing through digital in and out.

Or another alternative:
You have a DVD/CD player, which is able to decode HDCD to hdcd and give out it analouge.
Many cheap "no-name (lol, often better than well-known brand-name players, at least in features)" DVD-players should be able to do so nowadays.
You could record the analouge output of this DVDplayer, and get the HDCD content. If you have a good soundcard with some middle to good recording capabilities of 24bit/44.1/48/96 kHz or so, you could be successful.

edit addon:
I think, already my Cyberhome 635 DVD-A/V/SACD/CD player can decode HDCD and give out analouge. Together with terratec ewx 2496 I could record the wav. hm, but my time Sooner, but more likely later, I could carry out this experiment

Foobar 24 bit does not destroy the readability of an external decoder. Works just fine.

Still interested in this (mainly so I can archive my HDCDs as 'decoded' flacs)....so, what is actually 'done' during the decoding step? I presume the dither/noise shaping is already 'built into' the recording, and comes through regardless of whether an HDCD decoder is used. So I'm guessing the decoding is really just the 'decompression' step (acting on the optional peak compression) plus the choosing of an anti-aliasing filter. (And regarding that, can we assum,e that because WMP has an HDCD decoder, that it also has a variety of virtual anti-aliasing filters to choose from?)

I reviewed the papers about HDCD you can find on the net again recently. Here's my current view on it:

The dithering/noise shaping part they were advocating is nothing special. In fact, it is inferior to what other programs do (like Foobar or my requant.jar tool) and not provably safe in the sense that it avoids nonlinear distortions. You can't even call it "noise shaping" because it only introduces a bit of colored dither which has more power above 16 kHz -- it doesn't shape quantization noise.

If you decode the peak compression thingy (assuming it has been used) you can restore the upper 5 dB of the dynamic range which is usually rarely used (only for the occasional peaks, that's the idea).

Then there's the reconstruction filter selection. The whole idea of this is at least questionable. (I seriously doubt that this constitutes any subjective improvement!)

HDCD also supports some kind of dynamic range compression for the lower levels. It works like this: Quiet signals can be amplified in steps of 0.5 dB (0.0-7.5 dB = 4 bit side information) smoothly before quantization.

The side informations (whether peak compression is used, which reconstruction filter to use and the compressor gain) is transmitted via the least significant bits with the help of a linear feedback shift register. The only information you can't find on the net is how they actually code this side information. So, you can't implement a proper decoder.

Since the "benefits" of HDCD seem rather small it's -- in my humble opinion -- not worth to decode the stream to something like 24/44 or 24/88. One should rather keep it like it is and/or use a HDCD capable post processing algorithm within your player of choice (which only is supported by Microsoft's Mediaplayer IIRC).

In terms of signal-to-noise ratio (I'm not referring to peak signal to noise ratio), proper HDCD decoding won't help you a thing. The only thing you can improve is restoring the slightly compressed dynamics.

Cheers!
SG

edit: fixed typos

I've had a go at understanding hdcd; see,

http://forum.doom9.org/showthread.php?t=129136




If range expansion is enabled then samples less than -3.11dBFS (<= 0x5980) are attenuated by 6dB, samples greater than -3.11dBFS are attentuation progressively less, giving the end mapping of 0dBFS to 0dBFS.

If range expansion is not enabled (including non hdcd's) all samples are simply attenuated by 6dB.



I'm not sure how this works exactly. There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.



Exactly so.



Just needed a little bit of cryptanalysis.

On a candidate hdcd, during the silent intro at the start, the left and right channels matched exactly with a delay of a few hundred samples. This delay was rather fortunate, as it was hence, possible to identify where an hdcd code had been inserted by looking for a section where the correlation between the two channels stopped briefly. The locations of these sections were such that it was clear that the hdcd codes had been inserted into the left and right channels at the same time, and it was hence possible to assemble a set of encoded hdcd codes. Guessing that hdcd codes would contain some sort of magic number to mark them, and that a maximal length LFSR would have been used, it was simply a matter of trying all maximal length LFSRs on the data and looking for some correlation between them.

Hence, being able to extract hdcd codes, it was then fairly straighforward to use wmp as a reference to determine exactly what each bit of the code did.

Chris

Nice work! ... The infos about the LFSR are available in one of the HDCD white papers, btw.

Cheers!
SG

Thanks Sebastian,

I don't suppose you have a reference for the paper do you. I worked from what I found freely available on the internet, which mentioned that LFSRs were used, but not what tap values. Two example lengths were given, but the actual length was neither of those.

Chris

You can find this link at Wikipedia.

Take a look at the last page.

BTW, when I play HDCD in WMP 11 without enabling 24-bit I see the HDCD icon. When the 24-bit playback is enabled the icon disappears, but the audio is quieter (as it should be). I thought that icon should be visible when the decoder is decoding properly. Was it the same with previous WMP versions?

Huh... I never thought I'd see a command-line HDCD reference decoder. Congratulations!

Christopher, is there any reason something like this couldn't be implemented as a foobar2000 plugin, modified to silently monitor all 16-bit lossless sources and only trigger HDCD decoding when embedded HDCD information is detected?

- M.

I wish I'd seen that document before I attempted this, it would have made things quite a bit easier. I had seen the LFSR diagrams however.



From memory, the hdcd logo did come and go with (wmp 10?) for a bit, then spontaneously and permanently vanished. It hasn't reappeared with the upgrade to wmp 11.

Chris

I've just tried your command-line decoder on one track ripped from HDCD and the Audition's Amplitude Statistics tool reported that it's actual bit depth is 17 bits. The decoder printed out 'Detected HDCD' during decoding. Is that ok?

I'm afraid I know nothing about foobar2000, but I can't imagine any reason why such a plugin couldn't be written. Monitoring an audio stream and looking for an hdcd marker is not a particularly expensive operation. You would need some sort of look ahead to avoid playing the first samples of a track up to the first hdcd code without hdcd decoding however.

Chris



17 bits sounds about correct for a track just using peak expansion. The source is 16 bits (15 bits audio + 1 bit dithering), and the peak expansion expands the top 3.1dB to 9.1dB giving you an extra bit. The decoder will print out 'Detected HDCD' when it first sees a valid hdcd code in the audio.

Chris

Thanks!

Cheers

So is this literally replicating what an HDCD filter supposed to do??

I didn't try but I'd guess the answer is: Not exactly but probably close.

See, the exact inversion of the "soft dynamic range compression" requires an idea of how to interpolate between the gain values transmitted as 4bit side information each. This information gap is filled with common sense which might not match the exact specification.

Also, I only saw curves of what the peak compression thing is supposed to do, so one has to guess the parameters for the mapping according to the curve you can see in these docs.

Cheers!
SG

It's replicating exactly what wmp does when playing back an hdcd with 24-bit output enabled. I'd guess that that's pretty much identical to what happens in hardware, although I think there's may also be some upsampling done aswell.

Regards,

Chris





I calculated all my constants and the compression curve by passing test cases through wmp. The interpolation between gain values is done by multiplying the current gain by a constant each sample until it exceeds the target gain.

Reagrds,

Chris

It works for me. I think the real solution needs to be a foobar plugin because the converted files are huge!

Glad to hear it's working. The decoded files will be 50% larger when uncompressed, I'd be interested to know how their sizes differ when losslessly compressed with, e.g. FLAC.

I've never used foobar, and don't really have the time to start developing a plugin myself. However, if there are no objections from MS, I'll release the source code in a little while, and I'd hope it'd be pretty easy for someone to encorporate into a plugin.

Regards,

Chris

Oh, right. Sorry, I forgot that you tested the 'black box' WMP.


Let me just share my opinion on that with you all. I think it's not a good idea to decode HDCD "just for lossless storage" for the same reason you wouldn't download MP3 files and store them decoded as WAVs or FLACs on your harddisk. This decoding should only be done as preprocessing (prior lossy encoding) or as "postprocessing" (within a player, like the player decodes MP3). Otherweise you'll be wasting a lot of harddisk space.

I'm sure FLAC will have trouble catching the gaps (sample values not in use). For example: If you convert 16 bit samples to 24 bit samples and adjust the gain by -9dB to undo the dynamics compression only one out of 91 possible 24bit samples values is used (assuming you don't dither). Since 91 is not a power of two the least significant bits of the samples will look like random bits so that FLAC can't exploit the fact that the signal has an effective resolution of only 17.5 bits (16+1.5 due to the gain of -9 dB). So, approximately 573 kilobits/s (2*(24-17.5)*44100) are wasted (!!!).

Cheers!
SG

Interesting stuff, I have 20~ HDCDs around here.

Going to have a play

EDIT: How do you use this program, when I try and run it in the command prompt it just sits and does nothing. I tried specifying the wav file name and it still does nothing. What is the correct syntax? Even tried drag and drop of the wav, didn't work either