Follow strictly commandline from doom9.org including <, > and spaces.

hdcd.exe < 16bit.wav > 24bit.wav

Thanks guys.... Got it working.

I've started to run all my known HDCD encoded CDs through the HDCD decoder and I'm then compressing using WavPack 4.42 (-hhx6). I'll post more results once I get them.

WavPack compression results.

Buffalo Springfield, Last Time Around.

Original 16bit - 197mb - 806kbps
HDCD Decoded 24bit - 248mb - 1013kbps

Joni Mitchell, Clouds.

Original 16bit - 202mb - 749kbps
HDCD Decoded 24bit - 222mb - 827kbps

Joni Mitchell, Blue.

Original 16bit - 187mb - 724kbps
HDCD Decoded 24bit - 210mb - 811kbps

Joni Mitchell, Ladies of the Canyon.

Original 16bit - 259mb - 806kbps
HDCD Decoded 24bit - 282mb - 878kbps

Joni Mitchell, Song to a Seagull.

Original 16bit - 210mb - 771kbps
HDCD Decoded 24bit - 229mb - 841kbps

Van Halen, 1984.

Original 16bit - 246mb - 1031kbps
HDCD Decoded 24bit - 244mb - 1022kbps

Interesting result this, the 24bit HDCD decoded version compresses to a size even smaller than the original 16bit.

Van Halen, Van Halen I.

Original 16bit - 260mb - 1025kbps
HDCD Decoded 24bit - 258mb - 1018kbps

Same again here for Van Halen.

Neil Young, American Stars 'n Bars.

Original 16bit - 240mb - 879kbps
HDCD Decoded 24bit - 295mb - 1082kbps

Neil Young, On The Beach.

Original 16bit - 216mb - 763kbps
HDCD Decoded 24bit - 265mb - 936kbps

King Crimson, Islands.

Original 16bit - 208mb - 640kbps
HDCD Decoded 24bit - 214mb - 680kbps

King Crimson, In The Wake Of Poseidon.

Original 16bit - 222mb - 757kbps
HDCD Decoded 24bit - 233mb - 796kbps

King Crimson, Larks' Tongues In Aspic.

Original 16bit - 241mb - 724kbps
HDCD Decoded 24bit - 251mb - 754kbps

King Crimson, Starless and Bible Black.

Original 16bit - 246mb - 739kbps
HDCD Decoded 24bit - 255mb - 764kbps

FLAC seems to support 20 bit audio. I don't suppose there would be much of a compression gain from encoding 20 bit audio instead of 24 bit?

Would make no odds. This software HDCD decoder I'm using only outputs 24bit audio.

One could modify the decoder to output 24 bit samples where the least significant 4 bits are zero (effective resolution: 20 bits) so it'll compress better using FLAC and/or WavPack.

Since the HDCD decoder attenuates the signal at most by 13.5 dB which translates to 2.25 bits, you won't need more than 20 bits to represent the signal faithfully. So quantizing to 20 bits makes sense and won't hurt (won't effect the noise floor noticably).

Still, what's the point on decoding for lossless storage? Guys! Think about it. I'd keep the 16 bit signal as is and use a HDCD capable player instead!


...which can only mean that your CD doesn't make use of all HDCD features (no "soft dynamics compression").


Cheers!
SG

The point is easy to see. I want to be able to play these CDs' songs at the highest quality I can with the equipment I have. I don't have a HDCD decoder and I like to play all my music through my PC via Foobar. So to me it makes perfect sense. Don't be so narrow minded.



It could well be the reason. I might try re-ripping it in case I made a mistake somewhere.

I agree, HDCD content should be decoded on playback. But in the real world, I doubt I'll see a HDCD decoder plugin for my player of choice (Audacious) any time soon (and I don't own an audio receiver, let alone a HDCD-compatible one).

I'm with SebastianG. This decoder should be released as a player plug-in.

Cheap universal player

Can I just say I'm very excited about this tool!

Did a quick ABX on a beach boys HDCD disc, and couldn't tell the decoded one from the standard one, but I'll keep trying... perhaps with a more modern CD.

Wouldn't it be great if foobar was the second player (after WMP) to support decoding of HDCDs!

Ed

The analogy to mp3 decoding is a good one. Conceptually, hdcd is really just a lossy way of encoding 20 bit audio with (by design) the very useful property that the compressed output is very similar to a 16 bit version of the original.

I'm inclined to agree with you in principal that the decoding process should be done during playback or transcoding. In practice however, one doesn't always have the means to do this, and converting before hand is a good alternative. I would *strongly* recommend never deleting the original rip however, as this is the master copy. Hopefully plugins will become available, removing the need to store a preconverted version though.



I've not looked closely at how FLAC operates, but this certainly sounds highly likely. Huffman coding would work relatively well for this sort of situation, but would be fairly pointless for most audio, so is likely not used.

One point to note however is that wmp does truncate the output to 20 bits; maybe FLAC would be able to take advantage of that.

Regards,

Chris




I'll have a go tomorrow at adding some more diagnostics, maybe a --verbose option, which will list which features get used after the track's been processed. I'd guess:

Peak extend enabled at some point during the audio
Peak extend disabled at some point during the audio
Peak extend used (|sample|>0x5980 while peak extend used)
Range of gain adjustment values used.

The older foobar v0.8 series do allow Playback/Bit Depth to be specified using "Additional Options" which includes 20-bit padded to 24-bit. FLAC and Wavpack are supposed to take advantage if they detect that the bottom four bits are set to zero. If they aren't zeroed already using hdcd.exe then fb2k might save you a few kbps in your FLACs if you can be bothered to output to 20-bit padded to 24.

Well there's a new version available with a few more options and some more detailed information. For now it's available from,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd-new.zip

but it'll replace the existing version,

http://www.srcf.ucam.org/~cjk32/hdcd/hdcd.zip

fairly shortly.


To get usage information, run,

hdcd.exe -h

I won't explain here as I'd like the usage information to be self explanatory. If anything's unclear, let me know and I'll explain and update it.

Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris

Hi guys, great thread. I found it earlier and have been posting a lot to the Hoffman forum about my findings, using both the WMP plugin and hdcd.exe.

As for not using lossless storage, that means storing them as 16-bit WAVs; the only HDCD software player I know is player is WMP, which does not recognize wavs as HDCD. You either have to play an actual disc, or mount a virtual one, for WMP to recognize it at HDCD.

Or are you thinking of some other mechanism?


Regarding the two software decoders, I find that they produce almost identical results - I ripped a Joni Mitchell HDCD to my drive with EAC, then mounted it with Daemon tools for WMP (with the WAV plugin and 24-bit playback active), or else fed the .wavs to hdcd.exe. FOr a given track the two resulting .wav files almost null out, though there is tiny, residual signal at -88 dBFS or so, at a few spots (bit comparison with foobar says there are something like ~700 nonidentical samples, in one track, for example). I couldn't tell them apart in an ABX.

Regarding final bit depths, I find that the HDCDs which do NOT use peak extension, and which are highly compressed/limited, so far always decode to '17 bit' files , reduced by 6 dB (examples: Beck and Van Halen HDCDs). THose that use peak extend decode to 20 bit files (e.g. Yes, Joni Mitchell), though some that have no apparent peak extension, but are not super compressed dynamically, also decode to 20 bit files (e.g. King Crimson), with lesser amounts of level reduction. see this post for examples.

Lastly, I find that foobar2000 (v 0.9.4.4) cannot play decoded files made by the WMP plugin (which appear to be in 'wavex' format, and elicit a WAVEXTENSIBLE tag from flac.exe when I compress them) wheras it can play the 24 bit files produced by hdcd.exe, and flacs made from them. Unlike the older foobar, the newer one doesn't have options for setting playback bit depth, as best I can tell.


All feedback appreciated! I'm especially interested to know if the 'low resolution extension' and selectable filter options are actually used by either decoder (I've also got a hardware HDCD player and plan to compare its analog output to the digital 'rips')



From what I've read (e.g. John Marks' self-interview about his HDCD releases) it appears that the HDCD A/D converter adds the flag that will activate the HDCD logo even if NONE of the processing options are used during the mastering...in other words, if 'HDCD' technology us just used for its high-quality A/D conversion. So WMP would light up the HDCD logo, but simply attenuate these by 6 dB, as it does for all input when the 24 bit option is active (I recall this being a mandate from Pacific Microsonics originally).

You're right. I have the japanese HDCD release of Malmsteen's Concerto Suite for Electric Guitar and Orchestra. Here is the output of the hdcd.exe:


The Audition says that the actual bit depth of decoded wav is 17 bits, and the peak is -6.01 dB. This is the same for all tracks on the CD.


When I play this CD in WMP11 without 24 bit playback enabled, I see the HDCD logo. But when I enable 24 bit playback (as it should be), the logo doesn't show up.

Same here, for a while the logo wouldn't show up at all, but after switching the 24-bit option on and off a few times it finally appears while the 24-bit option is off

But I guess the 24-bit option is working because it sounds noticeably quieter.

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.



That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds. I've rechecked them all using the following method:

Rip CD using EAC. (Action -> Copy Image and Create Cue Sheet -> Uncompressed)
Manually edit the cue sheet to remove all but the first track entry (effectively create a single track that spans the entire CD)
Run hdcd on this file (hdcd.exe -o hdcd_hdcdN.wav hdcd.wav)
Run wmp on this file and capture the output with Chronotron.
Compare the two.

Apart from the fact that the wmp version is exactly one frame short, the two match exactly. (Not strictly true. A few times I've had the one or two samples not match, but they've always it's always been out by a power of 2, and have matched exactly if retested. I'm inclined not to trust the memory in the machine doing the testing.)

Which CD was it that wasn't matching for you? I've tested it with,

The Beach Boys - Pet Sounds
Mike Oldfield - Tubular Bells
Dire Straits - Sultans of Swing
Joni Mitchell - Blue
Mark Knopfler - Sailing to Philadelphia



Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.



From some very brief research, it appears that wavex (audio_format = FFFE) is simply a normal WAV, with an extra 24 bytes in the header that specify mapping from the channels in the file to playback channels. If you're so inclined, this can be fixed with a hex editor.



There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.

Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?



That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.

Regards,

Chris

Theoretically very desirable, but practically not do-able in any easy way....so far. I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)





It was Joni Mitchell 'Mingus'. FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around. To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks. For hdcd.exe of course all I have to do is feed it the individual .wavs directly. I can do it all again and compare the files. Also I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.






Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'. The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)



I can send this track to you too if you like.





Thanks. I'd much rather just use hdcd.exe, so long as it's creating files that are identical in all the ways that matter, to what the WMP WAV plugin generates.




Check out the link I made, to my Steve Hoffman forum post There I document a variety of hdcd behavior -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor). How can I tell if the 3 bits are in use?





Beats me. I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters). Maybe there's more options?







IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'. But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light.

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').

here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

King Crimson "Starless & Bible Black" 02 - Lament
HDCD Detected
Peak extend: Not enabled
Minimum gain: -4.0dB
Maximum gain : 0.0.dB

Joni Mitchell "Mingus" 09- The Dry Cleaner from Des Moines
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: -0.5 dB
Maximum gain : 0.0.dB

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

Yes "Big Generator" 04 - Almost Like Love
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB



(is there any way to get hdcd.exe to generate a log file? I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)

Thanks for posting your compression results, however I was very surprised that some albums would actually get smaller after the HDCD decoding. I can only imagine two reasons this could happen. One is if the original data was not compressed with the same settings (say, -hhx instead of -hhx6), and the other was if in some cases the HDCD program is actually throwing away a bit (but that wouldn't make any sense).

I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations!

BTW, I'm sure you have thought of this, but a very interesting complementary project would be an HDCD encoder. Then any band could release their own HDCD CDs, although Microsoft might have something to say about that!

Here's a possible explanation: The peak compression is heavily non-linear. It creates harmonic distortions which could be harder to predict than the original or decoded data.

Cheers!
SG

[quote name='krabapple' date='Aug 30 2007, 19:56' post='513502']
[quote name='Christopher Key' post='513457' date='Aug 30 2007, 11:33']

My view is that theoretically, you should only ever store the 16 bit master. That can then be played back as 16 bit or 24 bit dependent upon the listening environment. In practice however, until all my 'hifi' playback devices support hdcd, I'll probably have 16 bit and 24 bit versions in my library.
[/quote]

Theoretically very desirable, but practically not do-able in any easy way....so far. I'd love to see a foobar hdcd plugin. (Though 24-bit files of maybe a dozen HDCDs is not a big issue for me, storage-wise.)
[/quote]

Hopefully plugins will become available. Storage isn't really the problem, HDCDs will always be a very small percentage of one's music library, it's more the incovenience of having two versions available stored, keeping them both tagged correctly, making the correct one gets selected etc. It depends on your particular set up as to whether this is a problem

[quote]
[quote]
That's rather worrying. The software was designed to be bit identical to wmp, and is for my 5 test cds.

Which CD was it that wasn't matching for you?
[/quote]

It was Joni Mitchell 'Mingus'. FWIW I do my EAC rips as track by track, not album image + cue -- this was done long ago, before any of the hdcd--> wav tools were around. To mount the Mingus files in Daemon Tools for WMP, I used foobar2k to 'retrospectively' generate an album .wav + cue from the set of Mingus tracks. For hdcd.exe of course all I have to do is feed it the individual .wavs directly. I can do it all again and compare the files. Also I can send you the 16-bit master file of track or tracks from the HDCD, and you can check it, if you like.
[/quote]

If you get a chance, would you mind rechecking just one of the tracks. You may need to watch out for sample offsets, I did find that wmp rather bizarrely advanced to first 2m by 4 samples, then corrected itself. I've not been abl to get it to do it again however.

I'll see if I can acquire a copy of 'Mingus', along with a few others mentioned later to test this myself.

[quote]
[quote]
Tracks not using peak extend are simply right shifted by one bit, and are padded to 24bit with 7 zeroes. The peak extend table is specified with 20 bit precision, hence these tracks show up as 20 bit. Any tracks using gain adjustment (low level resolution extension?) will also show up as 20 bit.
[/quote]

Well, again, all I can report is that so far, the tracks that did NOT use peak extension, and show only a -6db crush, do NOT show up as 20 bit -- Audition calls them '17 bit'. The Beck 'Midnite Vultures' HDCD for example. Here's the Audition stats for track 1 , 'Sexx Laws', undecoded and hdcd.exe decoded. (please excuse the lousy formatting)


[/quote]

That's exactly as expected. If neither peak extend nor gain adjustment are used, the least significant seven bits of each sample will always be zero (as will the one most significant), hence the bit depth is detected as 17 bits. I don't know why the peak amplitude values aren't exactly 6dB lower in the decoded case, (0.09dB -> 6.11dB, not just the way the values are rounded), but the sample values are *exactly* halved.


[quote]
[quote]
There are 3 unidentified bits in the hdcd codes that might be used to switch filters. My test cd's had these permanently set to zero, and wmp didn't appear to do anything with them. I'd be very grateful to hear from anyone who finds a CD in which they're used.
[/quote]

Check out the link I made, to my Steve Hoffman forum post There I document a variety of hdcd behavior -- ranging from simple 6dB crush, all the way to ~6 dB of added 'dynamic range' (crest factor). How can I tell if the 3 bits are in use?
[/quote]

Thanks for the link, you initial post covers the behaviour nicely. Category 3 and 4 are essentially the same from a technical point of view, they both have peak extend enabled, category 4 has larger sample value before decoding that go further up the expansion curve. Given that I've got the data, I really should produce an accurate plot of the curve to demonstrate how it works.

[quote]
[quote]Both wmp and hdcd do allow the output level to be attenuated by up to 7.5dB (in .5dB steps). Is this what's referring to as 'low level extension'?[/quote]


Beats me. I've got some technical papers on HDCD waiting to be read. Sebastian might know already.
So far the HDCD options I can keep straight in my head are 1) peak extension 2) low resolution extension
3)selectable filters at A/D (I have yet to determine if these are applied at decoding too.. there was a huge flamewar about it on Audio Asylum between the maker of Ayre Cd players, who is very anti-HDCD, and Chris Tham, who isn't totally reliable on digital matters). Maybe there's more options?
[/quote]

http://web.archive.org/web/20030605091427/...o/overview.html

Mentions those three, and only those three, so I suspect that those are the options. I'll try to use their terminology from now on to avoid confusion.

Interestingly, the low level range extension referred to there (20dB in 4dB steps) is much different to and stronger that used by wmp (7.5dB in 0.5 dB steps).

I did catch the AA thread but got lost fairly quickly as a result of the strength of their convictions.


[quote]
[quote]

That certainly sounds sensible from a marketing point of view. As an end user, I'd probably be rather puzzled if the decoder didn't light up when playing material sold as hdcd encoded.
[/quote]

IMO, simply being converted from analog to digital, even by an admittedly high-quality converter, doesn't really qualify as 'hdcd encoding'. But it is good marketing for the converter, to brand these CDs as HDCD, and have them trigger the HDCD light.

Mandating that players licensed for HDCD lower all non-HDCD output by 6 dB is also good marketing, from a psychoacoustic POV -- people tend to think louder tracks sound better than quieter ones.

What's funny is that there are CDs out there that light up the HDCD decoder, but aren't marked as HDCD on their packaging. At least one of these even uses peak extension (Mahavishnu Orchestra 'Lost Trident Sessions').
[/quote]

Quite agree, particularly the loudness point. You can justify the behaviour from a technial point of view, if normal CDs weren't attenuated, and an hdcd code was spuriouusly detected during playback, you'd get a sudden jump in volume. On the other hand, when they are already attenuated by 6dB, a spuriusly detected hdcd code will just give you slightly more dynamic for a short period, which is much less offensive. I doubt this is the primary motivation however!


[quote name='krabapple' date='Aug 30 2007, 20:20' post='513505']
[quote name='Christopher Key' post='513258' date='Aug 29 2007, 14:12']
Finally, if anyone gets a message, 'Unknown codes seen:...', could you please let me know the exact output, as well as which CD produced it.

Regards,

Chris
[/quote]

here's that variety I mentioned, detected by hdcd.exe

Beck "Midnite Vultures', 02 - Nicotine & Gravy
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

Van Halen "1984" 02 - Jump
HDCD Detected
Peak extend: Not enabled
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

Yes "Yessongs" 05 - And You And I
HDCD Detected
Peak extend: Enabled permanently
Minimum gain: 0.0dB
Maximum gain : 0.0.dB
Unknown codes seen: 0x03

[/quote]

I'll try and get a copy of those that have the unknown codes. The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes. This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing. If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.

[quote]
(is there any way to get hdcd.exe to generate a log file? I can't select and paste from the cmd window. I'm also looking for a tool that would scan my flac collection for all HDCD tracks, to see if I have 'hidden' ones.)
[/quote]

Just redirect stderr to a file. Using the newer version, it should be as simple as,

hdcd.exe -o out.wav in.wav 2>mylog.txt

and with older version that only supports stdin / stdout,

hdcd.exe < in.wav > out.wav 2>mylog.txt


Also, take a look at the slimdevices thread, that's got some guidance on how to use this in a script,

http://forums.slimdevices.com/showthread.p...967&page=24


I did a quick scan of my library and was interested to find that all my Mark Knopfler CDs are HDCD, despite not being marked as such, as was *one* track from a Bond themes compilation CD. I'm guessing the compiler probably didn't realise either!!!

Ouch, trying to do it with only an analogue reference was going to be seriously hard!

I'd be very interested to know how you went about deriving the tap values used for the LFSR. There's a breif description of how I went about it earlier in this thread, but I can't believe that was the easiest way.




I would certainly be technically possible, although to do it well would require a lot more knowledge of audio processing than I have at present. The decoding is easy, there's no decisions to be made. Encoding isn't quite as straighforward. Peak extend is fine, low level range extend is fine with a bit of care to get a suitable level of hysteresis, but getting the dithering right is probably quite hard. I also have no idea yet quite how the filter switching works, given that wmp doesn't appear to do that.

It'd certainly be something to speak to Microsoft about, they may well not object. No HDCD encoders are being sold at present, so it's not really providing any competition, and electronic releases of HDCD music might well encourage some people to switch to wmp for playback.

Regards,

Chris

Thanks for addressing all the points in my (long) post, Christopher!

Regarding logs/scanning, before reading your reply I did some research and managed on my own to write two batch files that do the job for me...all of my files are .flacs, so one batch file decodes the first second of track 01 of each album to .wav, the second batch file runs hdcd.exe -a (i.e., .wav output suppressed) on those 1-sec wavs and directs the text output and stderr to a log file. Then I go back and delete all the tiny .wav files. Pretty kludgy but it worked fine!

Doing that I discovered some interesting things in my collection too...some things I'd either forgotten were HDCD, or never realized in the first place! (marked with asterisks). I've always thought the J. McLaughlin and J. Cash CDs sounded way loud, and maybe now I know why. Also interesting to find that some, but not all, King Crimson discs do employ peak extension.

(And because of your Bond result, I'm thinking maybe I need to do more than just 'sample' each album!)

You can pipe input to hdcd.exe if you want to avoid intermediate files. Something like,

flac [options required for wav to stdout] file.flac | hdcd.exe -a 2>>mylog.txt

should work. It's probably worth using -a rather than -i, as at least one hdcd track didn't have any hdcd information in the first second. I'll modify the code to default to ten seconds, but allow this to be changed from the command line, and will also have it exit as soon as it detects a valid code.

Regards,

Chris

Yes, piping makes more sense than the way I did it.

I too noticed that 1 second was cutting it too close -- as a value for the flac decoder, and as a result hdcd.exe -a missed a few known HDCD tracks in my collection (e.g., part one of 'Ommadawn'). When I changed the flac value to decode 5 sec, hdcd.exe -a detects ones it missed before. Do you think ten seconds will add still more?

FLAC --best compression results, 1.2.0 ICL compile

Brian Hughes - Along The Way (Audio CD)
HDCD Detected
Features Used:
Peak extend : Not enabled
Minimum gain : 0.0dB
Maximum gain : 0.0dB

Brian Hughes - Along The Way.wav: wrote 367162927 bytes, ratio=0,688
Brian Hughes - Along The Way 24.wav: wrote 368801906 bytes, ratio=0,461
(0.45% size increase)


Keiko Matsui - Dream Walk (Audio CD)
HDCD Detected
Features Used:
Peak extend : Enabled permanently
Minimum gain : -4.0dB
Maximum gain : 0.0dB

Keiko Matsui - Dream Walk.wav: wrote 281913513 bytes, ratio=0,585
Keiko Matsui - Dream Walk 24.wav: wrote 308084696 bytes, ratio=0,426
(9.28% size increase)

I've looked again more closely at exactly what wmp does. It does in fact extract and store the third bit from the hdcd codes, but then does absolutely nothing with it. I'm hence guessing that this single bit is used to switch between two alternate filters, which seems to fit in pretty closely with the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032


On the other hand, the AES paper,

http://www.paulita.lt/HDCD_AES_Paper.pdf

does refer to 'several' filters in a few places, and,

http://www.ednasia.com/article-511-destina...rtion-Asia.html

refers to four filters.

I'm not sure quite what the truth is, but am inclined to think there are only two filters, given that over the sample of tested CDs, the only currently unused bit that was ever set is the one extracted by wmp.


I'd be quite keen to add an option to do the correct upsampling as well. I've managed to extract a little bit of information; the AES paper specifically says that its a symmetric FIR filter, and another link that I've lost for now claims that it has less than 256 taps.

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs. With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.

I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris

I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac

18:05:31 : Test started.
18:09:54 : 01/01 50.0%
18:10:15 : 02/02 25.0%
18:10:40 : 03/03 12.5%
18:11:13 : 04/04 6.3%
18:11:49 : 05/05 3.1%
18:12:34 : 06/06 1.6%
18:13:49 : 07/07 0.8%
18:14:28 : 08/08 0.4%
18:15:12 : 09/09 0.2%
18:16:02 : 10/10 0.1%
18:16:15 : 11/11 0.0%
18:16:19 : Test finished.

----------
Total: 11/11 (0.0%)

Van Halen's 1984 yields some interesting results. I set up the decoder to output just the value of the third bit from each hdcd code to the left channel, and to leave the right channel untouched. Attached is a screenshot showing the start of track 3, Panama.



This shows bit3 being turned on for a brief period spanning each drum beat during the intro. Although much less clear, a similar pattern is visible throughout the rest of the cd. bit3 is also always turned on for a brief period covering the very start of each track.

My guess is that this is exactly as described in the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032

and that setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Chris

Edit: Just done the same experiment with Joni Mitchell - Both Sides Now, and this seems to be quite clearly what's going on. It's not used at all in tracks 4, 6, 12, which seem to feature much softer drum beats, and perfectly follows the percussion at about 2:22 into the first track.

Thanks for those results and to everyone else who's posted. They do seem to show that the lossless codecs are doing their job well, and getting rid of most of the redundancy in the files. The decoded files that compress to a smaller size are particularly interesting, and as Sebastian points out, are much closer to 'real' music.

Regards,

Chris

Can be deleted made a mistake, sorry.

Excellent work. Christopher. Well done.

Any chance for having the source released?

Triza

I have captured the analog output of a hardware HDCD decoder digitally if that's what you mean. I know of no consumer hardware decoder that digitally outputs the decoded stream.

Successfully ABXed but what was the 16-bit file? Was it the undecoded file straight from the CD, or was it the decoded file (20 bits of more) converted back to 16-bit with flat dither?

I'm interested in the latter, because I doubt it's ABXable without going insanely loud on the volume control.

Hi Sebastian!

Actually, I think it's a lot simpler than that. At least on the tracks I tried, the peak extension is triggered very infrequently, so most of the time you have only 16 bits and WavPack can encode that as 16 bits. But when the peak extension is triggered in a block, that entire block must be encoded in 20 bits (or whatever the worst sample is, bits-wise). When you're using the rather large default blocks (0.5 - 1.0 seconds) then a high percentage of the blocks have to go 20 bit, but when using smaller blocks (0.05 - 0.10 seconds) most can be just 16 bit.

David

I had read the patent and the AES paper, so I knew that it was [probably] based on the modulo-1 sum of 3 taps from a delay line of the audio MSB. So I wrote a simple program to scan entire tracks trying all possible combinations of [reasonably short] tap delays and then analyzing the resulting bitstream for patterns that repeat far more often than chance would predict. When I put in the correct taps, a 48-bit pattern jumped out!

However, as I mentioned, I never could find anything else. The 48-bit pattern was always the same on every HDCD track I tried, and the bits on either side (and everywhere else I looked) were always perfectly random. I knew I had found the right thing because if I changed a single bit of the 48 then my converter wouldn't light the HDCD indicator and if I changed other bits it still would (and I had to burn CDs for every experiment because I didn't have spdif output!)

My suspicion at the time was that there was some fixed dynamic range expansion implemented and that it was either on or off. Either I missed something, or the CD tracks I tried all happened to just use the same simple scheme.

I'll be interested to eventually see where my error was...

David

edit: added details

I believe that there are HDCD decoding chips that will output 88.2kHz 24 bit PCM. With any consumer device based upon one of these, it should be physically possible to tap into this signal. I doubt it's doable without specialised equipment in practice though.

If you've access to an decoder with an analogue out, it might be possible to apporiximate the correct behaviour. I'd propose creating a test file with a slow 10kHz - 22.5kHz sweep, an impulse train, and a low frequency square wave. This file would then be duplicated, and each version would have HDCD codes added such that one version played back through the normal upsampling filter, and the other, through the 'transient' filter.

By then looking at the captured output from the two filters, it should be possible to have a good at reproducing them.

I'm away all of next week, but I'll see if I can have a go a creating them when I get back.

Regards,

Chris




I do intend to release the source code when possible. At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Regards,

Chris

Isn't it because LAME is distributed only as source code, that it doesn't require paying royalties?

Compared: Grabbed with EAC 16-bit track VS this track converted to 24-bit using hdcd.exe. Both files were converted to FLAC and Replay Gain (Track Gain) was applied. ABXed with Headphones in quiet part in the middle of the song.

Thanks for the info, FotH. I hadn't noticed that Wag The Dog was HDCD, and that's an enjoyable track, so I might give it a go myself one day when I've got all my gear assembled again. I'd fancy comparing the 24-bit decoded output to the same output converted to 16-bit (flat dither) by fb2k, and perhaps converted to 14-bit padded (flat dither) also.

Perfectly. I guess this 'transient bit' can be ignored then.

Cheers!
SG

So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.

Cheers!
SG

for heavens sake!!! it´s finally happening... i´ve waited so long for this.

a year ago or so, i posted a wish here for someone to program a little tool, that would be capable of decoding hdcd´s. everyone except one was saying: why? it´s microsoft, it´s pointless, you can´t hear it... etc. etc.

sadly, i´m lacking the skills of programming something, so that i would be able to do it myself... but i can´t.

my statement always was, that a hdcd only sounds the way the artist or the engineer intended, when decoded.

with the help of the chronotron plug-in i found out, that a hdcd really CAN be 20 bit - at least, wavelab is showing this to me. i still don´t know how this works (i never completeley understood all posts here in this thread), but here comes a picture:



i opened one not-decoded wave-file & the decoded one. i reduced the volume of the not-decoded file about 6dB. i pressed play, then i simply made two screenshots when the play-marker reached a transient and merged them together with photoshop. you can see the peak extend (white) and the changing bit-depth in the bit-meter (dark pink). when transients are emerging, the resolution jumps from 17 to 20 bit. i don´t know, if this information is new to someone...

Sounds like you got to exactly the same point I did, albeit by a rather simpler approach. I too found that I has exactly the same repeating through all my 4 test cds, with about a very few occurences of a slight variation of the pattern on one cd. Fortunately, having wmp as a reference, it was possible to deduce what the patterns meant.

The 48bit patten is made from a 32bit marker, an 8 bit code and an 8 bit checksum. (Actually, it seems to be a 31bit marker, following by 1 bit that determines the format of the code and whether there is a checksum). The pattern I was seeing was the code for disabling the transient filter, enabling peak extend, and not performing any gain adjustment (0x10). The few slight variations were use of the gain adjustment feature.

Regards,

Chris

Thanks,

I wasn't aware of that. I'll have a chat with the LAME guys and see if their argument has been tested / accepted.

Regards,

Chris

Indeed, it probably isn't worth the bother of decoding, but it would be nice to know. I've created four wav files, an impulse train and a 10kHz -> 22.05kHz sweep, each set to use both the normal and transient filter:

http://www.srcf.ucam.org/~cjk32/hdcd/transient_filter/

Sirius, do you think you could pipe these four through your hardware decoder and capture the output. I'd strongly recommend against feeding the decoded signal into anything other than a capture device, as I doubt speakers (amp too???) would fare very well with them!

Regards,

Chris

I just downloaded your HDCD.EXE and gave it a whirl. Works great - THANKS. I think it's MUCH nicer to have all that stuff decoded to play in a non-HDCD player - a lower average signal level is a small price to pay for decoding (expanding) the dynamic compression, and most recordings don't use the full 90+ dB that redbook CD offers anyway.

On a couple of my HDCD recordings it said "Unknown codes seen: 0x03" These CDs are from Reference Recordings which apparently employs Johnson, one of the coinventors of HDCD.

Here's a link to a sample WAV file. I picked the shortest - only 5 MB in size - because my FTP server has limited bandwidth.
ftp://mclements.net/HDCDTest1.wav