Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Extract HDCD (Read 208695 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Extract HDCD

Reply #75
Regarding logs/scanning, before reading your reply I did some research and managed on my own to write two batch files that do the job for me...all of my files are .flacs, so one batch file decodes the first second of track 01 of each album to .wav, the second batch file runs hdcd.exe -a (i.e., .wav output suppressed) on those 1-sec wavs and directs the text output and stderr to a log file. Then I go back and delete all the tiny .wav files.  Pretty kludgy but it worked fine!


You can pipe input to hdcd.exe if you want to avoid intermediate files.  Something like,

flac [options required for wav to stdout] file.flac | hdcd.exe -a 2>>mylog.txt

should work.  It's probably worth using -a rather than -i, as at least one hdcd track didn't have any hdcd information in the first second.  I'll modify the code to default to ten seconds, but allow this to be changed from the command line, and will also have it exit as soon as it detects a valid code.

Regards,

Chris

Extract HDCD

Reply #76
Yes, piping makes more sense than the way I did it.

I too noticed that 1 second was cutting it too close -- as a value for the flac decoder, and as a result hdcd.exe -a missed a few known HDCD tracks in my collection (e.g., part one of 'Ommadawn').  When I changed the flac value to decode 5 sec, hdcd.exe -a  detects ones it missed before.  Do you think ten seconds will add still more?

Extract HDCD

Reply #77
FLAC --best compression results, 1.2.0 ICL compile

Brian Hughes - Along The Way (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

Brian Hughes - Along The Way.wav: wrote 367162927 bytes, ratio=0,688
Brian Hughes - Along The Way 24.wav: wrote 368801906 bytes, ratio=0,461
(0.45% size increase)


Keiko Matsui - Dream Walk (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Enabled permanently
  Minimum gain  : -4.0dB
  Maximum gain  : 0.0dB

Keiko Matsui - Dream Walk.wav: wrote 281913513 bytes, ratio=0,585
Keiko Matsui - Dream Walk 24.wav: wrote 308084696 bytes, ratio=0,426
(9.28% size increase)

Extract HDCD

Reply #78
I'll try and get a copy of those that have the unknown codes.  The 0x03 means that at some point, the decoder has seen both 0b000 and 0b001 in the upper three bits of the hdcd codes.  This could well be for filter switching, although I couldn't see wmp doing anything with these bits so there was no way of knowing.  If it does nothing with them, then there's really not a lot I can do, unless I can see the digital output of a hardware decoder that also upscales to 88.2kHz.


I've looked again more closely at exactly what wmp does.  It does in fact extract and store the third bit from the hdcd codes, but then does absolutely nothing with it.  I'm hence guessing that this single bit is used to switch between two alternate filters, which seems to fit in pretty closely with the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032


On the other hand, the AES paper,

http://www.paulita.lt/HDCD_AES_Paper.pdf

does refer to 'several' filters in a few places, and,

http://www.ednasia.com/article-511-destina...rtion-Asia.html

refers to four filters.

I'm not sure quite what the truth is, but am inclined to think there are only two filters, given that over the sample of tested CDs, the only currently unused bit that was ever set is the one extracted by wmp.


I'd be quite keen to add an option to do the correct upsampling as well.  I've managed to extract a little bit of information; the AES paper specifically says that its a symmetric FIR filter, and another link that I've lost for now claims that it has less than 256 taps.

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.

I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris

Extract HDCD

Reply #79
I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac

18:05:31 : Test started.
18:09:54 : 01/01  50.0%
18:10:15 : 02/02  25.0%
18:10:40 : 03/03  12.5%
18:11:13 : 04/04  6.3%
18:11:49 : 05/05  3.1%
18:12:34 : 06/06  1.6%
18:13:49 : 07/07  0.8%
18:14:28 : 08/08  0.4%
18:15:12 : 09/09  0.2%
18:16:02 : 10/10  0.1%
18:16:15 : 11/11  0.0%
18:16:19 : Test finished.

----------
Total: 11/11 (0.0%)

Extract HDCD

Reply #80
I'm going to go and try to buy some of the CDs known to use this bit to see if I can at least identify which filter is which, and what might cause the encoder to change between the two.

Regards,

Chris


Van Halen's 1984 yields some interesting results.  I set up the decoder to output just the value of the third bit from each hdcd code to the left channel, and to leave the right channel untouched.  Attached is a screenshot showing the start of track 3, Panama.



This shows bit3 being turned on for a brief period spanning each drum beat during the intro.  Although much less clear, a similar pattern is visible throughout the rest of the cd.  bit3 is also always turned on for a brief period covering the very start of each track.

My guess is that this is exactly as described in the last paragraph of,

http://www.hydrogenaudio.org/forums/index....mp;#entry333032

and that setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Chris

Edit:  Just done the same experiment with Joni Mitchell - Both Sides Now, and this seems to be quite clearly what's going on.  It's not used at all in tracks 4, 6, 12, which seem to feature much softer drum beats, and perfectly follows the percussion at about 2:22 into the first track.

Extract HDCD

Reply #81
FLAC --best compression results, 1.2.0 ICL compile

Brian Hughes - Along The Way (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Not enabled
  Minimum gain  : 0.0dB
  Maximum gain  : 0.0dB

Brian Hughes - Along The Way.wav: wrote 367162927 bytes, ratio=0,688
Brian Hughes - Along The Way 24.wav: wrote 368801906 bytes, ratio=0,461
(0.45% size increase)


Keiko Matsui - Dream Walk (Audio CD)
HDCD Detected
Features Used:
  Peak extend  : Enabled permanently
  Minimum gain  : -4.0dB
  Maximum gain  : 0.0dB

Keiko Matsui - Dream Walk.wav: wrote 281913513 bytes, ratio=0,585
Keiko Matsui - Dream Walk 24.wav: wrote 308084696 bytes, ratio=0,426
(9.28% size increase)


Thanks for those results and to everyone else who's posted.  They do seem to show that the lossless codecs are doing their job well, and getting rid of most of the redundancy in the files.  The decoded files that compress to a smaller size are particularly interesting, and as Sebastian points out, are much closer to 'real' music.

Regards,

Chris

Extract HDCD

Reply #82
Excellent work. Christopher. Well done.

Any chance for having the source released?

Triza

Extract HDCD

Reply #83
Quote
If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.


I have captured the analog output of a hardware HDCD decoder digitally if that's what you mean.  I know of no consumer hardware decoder that digitally outputs the decoded stream.

Extract HDCD

Reply #84
I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac


Successfully ABXed but what was the 16-bit file? Was it the undecoded file straight from the CD, or was it the decoded file (20 bits of more) converted back to 16-bit with flat dither?

I'm interested in the latter, because I doubt it's ABXable without going insanely loud on the volume control.
Dynamic – the artist formerly known as DickD

Extract HDCD

Reply #85

I also tried this with 3 of my HDCDs, and found that for tracks encoded using "peak extension" you can gain considerable compression improvement by using very small block sizes. I found that --blocksize=4410 worked nicely, and on some tracks --blocksize=2205 was even better. And I'm talking like 5-10% better compression!

Here's a possible explanation: The peak compression is heavily non-linear. It creates harmonic distortions which could be harder to predict than the original or decoded data.

Cheers!
SG

Hi Sebastian! 

Actually, I think it's a lot simpler than that. At least on the tracks I tried, the peak extension is triggered very infrequently, so most of the time you have only 16 bits and WavPack can encode that as 16 bits. But when the peak extension is triggered in a block, that entire block must be encoded in 20 bits (or whatever the worst sample is, bits-wise). When you're using the rather large default blocks (0.5 - 1.0 seconds) then a high percentage of the blocks have to go 20 bit, but when using smaller blocks (0.05 - 0.10 seconds) most can be just 16 bit.

David

Extract HDCD

Reply #86

Christopher, this is a very cool program! I actually was working on this many years ago (before WMP had this) and also found the data "key" that enables the HDCD functioning. However, I was never able to find any other information to control parameters (the signal was just either there or not, 10 times per second IIRC). Anyway, my next step was to start measuring the effect using an external HDCD decoder and a scope, but I never got that far. Anyway, congratulations! 


Ouch, trying to do it with only an analogue reference was going to be seriously hard!

I'd be very interested to know how you went about deriving the tap values used for the LFSR.  There's a breif description of how I went about it earlier in this thread, but I can't believe that was the easiest way.

I had read the patent and the AES paper, so I knew that it was [probably] based on the modulo-1 sum of 3 taps from a delay line of the audio MSB. So I wrote a simple program to scan entire tracks trying all possible combinations of [reasonably short] tap delays and then analyzing the resulting bitstream for patterns that repeat far more often than chance would predict. When I put in the correct taps, a 48-bit pattern jumped out!

However, as I mentioned, I never could find anything else. The 48-bit pattern was always the same on every HDCD track I tried, and the bits on either side (and everywhere else I looked) were always perfectly random. I knew I had found the right thing because if I changed a single bit of the 48 then my converter wouldn't light the HDCD indicator and if I changed other bits it still would (and I had to burn CDs for every experiment because I didn't have spdif output!)

My suspicion at the time was that there was some fixed dynamic range expansion implemented and that it was either on or off. Either I missed something, or the CD tracks I tried all happened to just use the same simple scheme.

I'll be interested to eventually see where my error was... 

David

edit: added details

Extract HDCD

Reply #87
Quote

If anyone has access to a hardware decoder with a digital output, it should be possible to calculate the exact tap values for each filter by analysing the output from various suitable inputs.  With an analogue decoder, it should be possible to approximate the filters' responses, but getting the exact values won't be possible.


I have captured the analog output of a hardware HDCD decoder digitally if that's what you mean.  I know of no consumer hardware decoder that digitally outputs the decoded stream.


I believe that there are HDCD decoding chips that will output 88.2kHz 24 bit PCM.  With any consumer device based upon one of these, it should be physically possible to tap into this signal.  I doubt it's doable without specialised equipment in practice though.

If you've access to an decoder with an analogue out, it might be possible to apporiximate the correct behaviour.  I'd propose creating a test file with a slow 10kHz - 22.5kHz sweep, an impulse train, and a low frequency square wave.  This file would then be duplicated, and each version would have HDCD codes added such that one version played back through the normal upsampling filter, and the other, through the 'transient' filter.

By then looking at the captured output from the two filters, it should be possible to have a good at reproducing them.

I'm away all of next week, but I'll see if I can have a go a creating them when I get back.

Regards,

Chris


Any chance for having the source released?


I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Regards,

Chris

Extract HDCD

Reply #88
I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Isn't it because LAME is distributed only as source code, that it doesn't require paying royalties?

Extract HDCD

Reply #89
Quote
Dynamic     Posted Yesterday, 18:13
     
QUOTE(Fool_on_the_hill @ Aug 31 2007, 21:19) *

I did ABX on track Wag The Dog from Mark Knopfler's album Wag The Dog:foo_abx 1.3.1 report
foobar2000 v0.9.4.4
2007/08/29 18:05:31

File A: C:\wagthedog24bit.flac
File B: C:\wagthedog16bit.flac

Successfully ABXed but what was the 16-bit file? Was it the undecoded file straight from the CD, or was it the decoded file (20 bits of more) converted back to 16-bit with flat dither?

I'm interested in the latter, because I doubt it's ABXable without going insanely loud on the volume control.


Compared: Grabbed with EAC 16-bit track VS this track converted to 24-bit using hdcd.exe. Both files were converted to FLAC and Replay Gain (Track Gain) was applied. ABXed with Headphones in quiet part in the middle of the song.

Extract HDCD

Reply #90
Thanks for the info, FotH. I hadn't noticed that Wag The Dog was HDCD, and that's an enjoyable track, so I might give it a go myself one day when I've got all my gear assembled again. I'd fancy comparing the 24-bit decoded output to the same output converted to 16-bit (flat dither) by fb2k, and perhaps converted to 14-bit padded (flat dither) also.
Dynamic – the artist formerly known as DickD

Extract HDCD

Reply #91
My guess is that (...) setting bit3 is supposed to switch the decoder from the standard, sharp recontruction filter to one with a more gentle roll off that better preserves transients.

Does this seem to make sense?

Perfectly. I guess this 'transient bit' can be ignored then.

Cheers!
SG

Extract HDCD

Reply #92
Perfectly. I guess this 'transient bit' can be ignored then.

Cheers!
SG



So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Extract HDCD

Reply #93
So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.

Cheers!
SG

Extract HDCD

Reply #94
for heavens sake!!! it´s finally happening... i´ve waited so long for this.

a year ago or so, i posted a wish here for someone to program a little tool, that would be capable of decoding hdcd´s. everyone except one was saying: why? it´s microsoft, it´s pointless, you can´t hear it... etc. etc.

sadly, i´m lacking the skills of programming something, so that i would be able to do it myself... but i can´t.

my statement always was, that a hdcd only sounds the way the artist or the engineer intended, when decoded.

with the help of the chronotron plug-in i found out, that a hdcd really CAN be 20 bit - at least, wavelab is showing this to me. i still don´t know how this works (i never completeley understood all posts here in this thread), but here comes a picture:



i opened one not-decoded wave-file & the decoded one. i reduced the volume of the not-decoded file about 6dB. i pressed play, then i simply made two screenshots when the play-marker reached a transient and merged them together with photoshop. you can see the peak extend (white) and the changing bit-depth in the bit-meter (dark pink). when transients are emerging, the resolution jumps from 17 to 20 bit. i don´t know, if this information is new to someone...
marlene-d.blogspot.com

Extract HDCD

Reply #95
I had read the patent and the AES paper, so I knew that it was [probably] based on the modulo-1 sum of 3 taps from a delay line of the audio MSB. So I wrote a simple program to scan entire tracks trying all possible combinations of [reasonably short] tap delays and then analyzing the resulting bitstream for patterns that repeat far more often than chance would predict. When I put in the correct taps, a 48-bit pattern jumped out!

However, as I mentioned, I never could find anything else. The 48-bit pattern was always the same on every HDCD track I tried, and the bits on either side (and everywhere else I looked) were always perfectly random. I knew I had found the right thing because if I changed a single bit of the 48 then my converter wouldn't light the HDCD indicator and if I changed other bits it still would (and I had to burn CDs for every experiment because I didn't have spdif output!)

My suspicion at the time was that there was some fixed dynamic range expansion implemented and that it was either on or off. Either I missed something, or the CD tracks I tried all happened to just use the same simple scheme.

I'll be interested to eventually see where my error was... 

David

edit: added details


Sounds like you got to exactly the same point I did, albeit by a rather simpler approach.  I too found that I has exactly the same repeating through all my 4 test cds, with about a very few occurences of a slight variation of the pattern on one cd.  Fortunately, having wmp as a reference, it was possible to deduce what the patterns meant.

The 48bit patten is made from a 32bit marker, an 8 bit code and an 8 bit checksum.  (Actually, it seems to be a 31bit marker, following by 1 bit that determines the format of the code and whether there is a checksum).  The pattern I was seeing was the code for disabling the transient filter, enabling peak extend, and not performing any gain adjustment (0x10).  The few slight variations were use of the gain adjustment feature.

Regards,

Chris

Extract HDCD

Reply #96
I do intend to release the source code when possible.  At the moment, Microsoft are charging royalties for use of HDCD on non MS platforms, and I rather fancy a source code distribution might fall foul of that.

Isn't it because LAME is distributed only as source code, that it doesn't require paying royalties?


Thanks,

I wasn't aware of that.  I'll have a chat with the LAME guys and see if their argument has been tested / accepted.

Regards,

Chris

 

Extract HDCD

Reply #97

So does this mean you think that the sound of either filter sampling to 88.2 is aren't that different from each other?

Although I don't know the exact specs of the reconstruction filters I feel I can safely say that decoding this (non-)feature isn't worth the trouble.


Indeed, it probably isn't worth the bother of decoding, but it would be nice to know.  I've created four wav files, an impulse train and a 10kHz -> 22.05kHz sweep, each set to use both the normal and transient filter:

http://www.srcf.ucam.org/~cjk32/hdcd/transient_filter/

Sirius, do you think you could pipe these four through your hardware decoder and capture the output.  I'd strongly recommend against feeding the decoded signal into anything other than a capture device, as I doubt speakers (amp too???) would fare very well with them!

Regards,

Chris

Extract HDCD

Reply #98
I'm not sure how this works exactly.  There are 3 bits left that could be used for controlling the reconstruction filters, although there were always zero in all my test cds, and wmp didn't seem to respond to them at all.

I just downloaded your HDCD.EXE and gave it a whirl. Works great - THANKS. I think it's MUCH nicer to have all that stuff decoded to play in a non-HDCD player - a lower average signal level is a small price to pay for decoding (expanding) the dynamic compression, and most recordings don't use the full 90+ dB that redbook CD offers anyway.

On a couple of my HDCD recordings it said "Unknown codes seen: 0x03" These CDs are from Reference Recordings which apparently employs Johnson, one of the coinventors of HDCD.

Here's a link to a sample WAV file. I picked the shortest - only 5 MB in size - because my FTP server has limited bandwidth.
ftp://mclements.net/HDCDTest1.wav

Extract HDCD

Reply #99
Quote
I was always under the impression that HDCD was simply a better dithering algorithm.
[a href="index.php?act=findpost&pid=268929"][{POST_SNAPBACK}][/a]


HDCD is:
1) dithering + noise shaping
2) peak compression (kind of a reversible dynamic compressor)
3) hiding commands in some least significant bits of the samples to tell an HDCD decoder which anti-alias lowpass filter it has to use for upsampling

...whereas most of the dynamic range "increase" comes from (1).
("increase" ? yes, it depends on how you measure it. If you perceptually weight the quantization noise the noise power will be lower compared to dithering without noise shaping)


SebastianG


I just came across this post and I had a question for.  #2 threw me off when you said "reversible dynamic compressor."  are the signal peaks actually compressed (or limited?) or is it a compressor acting as an expander (the "reversible" part is what lead me to believe you were describing a compressor reversed)?