lossyWAV 1.2.0 Development Thread, Added noise WAV bitdepth reduction method Options

[Nick.C]

lossyWAV doesn't support 32-bit float values (9 to 32-bit signed integers only) - and I don't think that FLAC does either, unfortunately.

It is *possible* to include 32-bit float compatibility in lossyWAV, however a target lossless codec would need to be found which could handle floats and also make use of the wasted-bits feature.

[edit] integer compatibility added. [/edit]

This post has been edited by Nick.C: Dec 9 2008, 08:24

[Canar]

Ah right, it's WavPack that supports 32-bit float...

[[JAZ]]

I have uploaded a new version of jLossyWav. This time it actually works


The compression is slow, around 0.5X realtime. I hope it can be improved some still, but probably by not much. As i said, this is more an educational port than anything else.


It is producing smaller files than the sharp version of lossywav, but this time the differences are small:

cd1_02.wav_sharp.lossy.wav: wrote 866856 bytes, ratio=0,298
cd1_02.wav_j.lossy.wav: wrote 865984 bytes, ratio=0,298

This probably comes from a different way of calculating the bitsToRemove value, and i've seen the difference file to be quite similar to the one of sharp, just removing more bits in some places. (i.e. what should be expected)

I still want to work the code a bit more, concretely subdivide the Analyzer.process() function, and make more obvious what it is doing.



Download was at :
http://www.hydrogenaudio.org/forums/index....=67697&st=0

[edit: Woops! corrected the link]

This post has been edited by [JAZ]: Dec 20 2008, 18:01

[Nick.C]

Excellent work, [JAZ] - it seems to be coming along nicely.

I've been thinking about removing the FACT chunk for some time now - reasons:

1) Is it really necessary?
2) If the output is piped then the FACT chunk is missing anyway;
3) If a file is re-processed then either no bits will be removed (new q > old q) or the appropriate number of bits (new q < old q) or nearly no bits at all (new q = old q);
4) It's a bit of a pain and requires much more WAV chunk processing than would otherwise be needed;
5) Using the --sampledist parameter the distribution of LSBs and MSBs for the input and output can be examined for signs of pre-processed audio.

Feedback gratefully received.

I have been using the shifted array index trick introduced by Gregory and have now applied it to all of the relevant arrays. This required the Window_Function to be slightly re-written to make it compatible.

Overall memory footprint has reduced quite a bit, but to reduce it further will require modifications to the WAV handling unit as it has quite a large memory requirement (currently....).

Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

[edit] Clarity regarding memory footprint [/edit]

This post has been edited by Nick.C: Dec 22 2008, 09:30

[Gregory S. Chudo...]

I agree that the FACT chunk is quite useless.

The whole point of lossyWAV is that we can compress the output with the lossless format of our choice, but with most formats the FACT chunk will be lost.

Some indication that the file is actually lossy is required though. Naming the files as *.lossy.* is not enough, because there's a lot of software out there which automaticly renames files according to the tags.

However, all lossless formats that i know of support some kind of tagging. Tags are usually preserved when transcoding from one format to another. So i think the best way to do this is to choose a tag, e.g. "LOSSY", and add it when compressing to lossless format - we need special options for each lossless format anyway to make sure the block size is right, etc.

For example, lossyFLAC command line would be:

CODE

c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - --tag=LOSSY=yes -b 512 -5 -f -o%d

Also we might want to store scaling_factor in correction file to be able to merge it with lossy part.

This post has been edited by Gregory S. Chudov: Dec 22 2008, 13:28

[botface]

I agree that the FACT chunk doesn't serve much purpose as it's easilly lost. I was religiously putting "LossyWav" in the comment tag to avoid accidental reprocessing until the realisation dawned that it really didn't matter anyway.

[[JAZ]]

About the fact chunk:

With a self compiled version of lossywavSharp, and my system settings, I get errors with that chunk and have to remove it in order for FLAC to encode it. I believe the reason is because the lossywavSharp implementation does not pad the chunk. (A chunk has to be always even. If it is odd, a zero has to be appended).
Probably it doesn't matter, since the compiled version seems to work.

About its use, it helps identify a .wav file, not a losslesly compressed counterpart.

I don't see wav chunks as being something difficult, from the software side of the fence. As for programs/end user, I believe it doesn't matter.



About the memory usage:

Since Java does arrays in a different way, I haven't worried about optimizing them. Yet, I saw several of those that can be changed as to reduce their size.
Most notably, these:

public int[] spreading_averages_int;
public float[] spreading_averages_rem;
public float[] spreading_averages_rec;
public float[] skewing_function;

Their size can be calculated when hi_bins and lo_bins are calculated. (After knowing the bitdepth and sampleRate of the input file)

spreading_averages_int = new int[hi_bins + 1];
spreading_averages_rem = new float[hi_bins + 1];
spreading_averages_rec = new float[hi_bins + 1];
skewing_function = new float[hi_bins + 2]

This post has been edited by [JAZ]: Dec 23 2008, 23:56

[Gregory S. Chudo...]

I believe the reason is because the lossywavSharp implementation does not pad the chunk.

This was a bug in the earliest version, fixed long time ago. Funny thing, this bug revealed itself only past 12AM - when the date string became one byte longer

This post has been edited by Gregory S. Chudov: Dec 23 2008, 23:57

[Hancoque]

Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

What about the checking for matrix surround content? I know, nobody has provided any test samples to analyse. I looked for a Pro Logic II encoder to create some material from 5.1 files but couldn't get one. I wonder how many music albums there are that feature matrix surround encoding. Do they justify any further investigation?

[Nick.C]

I have (sort of) implemented M/S checking - I'll go back to it and make sure that it makes sense....

This post has been edited by Nick.C: Dec 25 2008, 22:01

[Brent]

With enduring hope that ALAC somehow someday would be compatible with lossywav, I tried the ALAC encoder thats been newly added to ffmpeg a few months ago. Same results as with iTunes: no reduction in filesize...

Just thought I'd mention it.

[Nick.C]

Didn't find that. Thanks Nick.

Could sombody tell me the correct code to transcode to LossyWAV with correction file?

At the moment I'm using foobar200 with following comandlines:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c C:\Programme\"Audio Tools"\foobar2000\lossyWAV.exe - --standard --silent --stdout|C:\Programme\"Audio Tools"\foobar2000\Takc.exe -e -p4 -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

What do I have to add to get the correction file? It should have the same file name just with the ".lwcdf.tak" extension. I know it has to do with the Command "-C" but I don't get it to work.

thanks in advance.

lossyWAV correction file creation is incompatible with piped output. You would have to write a batch file and run it instead of the "/d /c .... etc...."

[sidewalking]

Well I have tried and failed...I haven't pinned down the right sequence yet...

What I have used so far...

CODE

/d /c C:\"Program Files (x86)"\foobar2000\codec\lossywav - --portable --silent --stdout | "C:\Program Files (x86)\Windows Media Components\Encoder\WMCmd.vbs" -silent -a_codec WMA9LSL -a_mode 2 -a_setting Q100_44_2_16 -input %s -output %d

Problem is it gives back a WSHShell / undefined variable...probably has to do something with the piping or some other little variable.

I will have to stick with the workaround method I have setup at the moment. Convert to Lossy Portable FLAC and then convert that to WMA-Lossless.

I will keep trying every once in a while to try to figure some tweak or some trick to get it to work as a command line.

Cscript.exe is required to run .vbs files. I believe it is in C:\Windows\System32\ and put the .vbs as an argument after it with full path in quotes.

I can't get it to work, either. I can get the original WAV to encode to WMA Lossless, but I can't get all three processes into the converter string to work. Maybe if there were a batch to call from foobar to handle the lossyWAV conversion and then input that file into the WMALS encoder for a lossyWMALSL file, not just a WWMA Lossless file.

The wiki code here doesn't work, even if you allow for a cscript.exe prompt first:

CODE

Encoder: c:\program files\windows media components\encoder\WMCmd.vbs
Extension : lossy.wma
Parameters: -input %s -output %d -a_codec WMA9LSL -a_mode 0 -a_setting Q100_44_2_16
Format is : lossless or hybrid
Highest BPS mode supported: 24

It addresses the encoding of a regular WAV file, but doesn't include the lossyWAV processing.

[Nick.C]

It addresses the encoding of a regular WAV file, but doesn't include the lossyWAV processing.

You're right, of course - I'll work up a batch file to carry out the lossyWAV processing then encode to WMALSL.

[UED77]

You're right, of course - I'll work up a batch file to carry out the lossyWAV processing then encode to WMALSL.

Actually, a batch file would be a good idea for the popular formats: FLAC, TAK, WavPack, and WMALSL. That would ensure that the compressors are called with the proper blocksizes, and could also place a custom tag in the file that would replace the current (and unwieldy) FACT chunk.

[sidewalking]

The only thing I'd like to see is a nice GUI front end - I've mentioned that a couple of times before. I don't mean to go on about it but it would make life a bit easier for me as a non-techie.

I am working on something similar to the Speek/Flac front ends right now. It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).

[Rigapada]

I am working on something similar to the Speek/Flac front ends right now. It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).

I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

--Rigapada

[Nick.C]

.... that would replace the current (and unwieldy) FACT chunk.

Unfortunately, the FACT chunk is a necessary evil - if the user creates correction files when processing and wishes to revert to lossless at any point. The FACT chunks of the .lossy.wav and .lwcdf.wav files are compared and, if identical, the audio data is merged to reinstate the lossless original.

However, when using piped encoding, the FACT chunk cannot be used (and neither can correction files be created) as FLAC (for one) does not accept additional chunks when piping input.

[botface]

The only thing I'd like to see is a nice GUI front end - I've mentioned that a couple of times before. I don't mean to go on about it but it would make life a bit easier for me as a non-techie.

I am working on something similar to the Speek/Flac front ends right now. It won't be fancy, but will relieve the CLI anxiety that some people get (and cause them to pass over or check out newer things).

I agree that Foobar is going to be the most useful, but if there is a need for a basic front end like the ones mentioned - I will throw something together.

(...and I will integrate Flac and probably WavPack, since I am a WavPack junkie).

That's jolly decent of you,old chap!

Actually, I've been using Foobar and am quite happy to do my conversions that way - or directly from EAC. But I think you're right in that it might encourage people even less techie than me to use LossyWav.

If you want any testing done just give me a shout

Thinking of releasing a 1.2.0 soon - although obviously without the adaptive noise shaping method (a very interesting, complex and baffling proposition that will remain firmly on the to-do list).

Nick, if the ideal noise-shaping is proving troublesome is there any mileage in introducing a simpler noise-shaping method as a short term option? I have absolutely no idea whether that's a viable proposition but if it were it might make LossyWav useable at lower "q" settings and hence a viable alternative to lossy codecs at the higher end of their range.

[Nick.C]

There is noise shaping already implemented in lossyWAV (at v1.1.0). This uses SG's published cofficients which are optimised for 44.1kHz and 48kHz.

As an aside, I've been continuing my development of the spreading function - I will post a beta v1.1.1f later today.

[botface]

There is noise shaping already implemented in lossyWAV (at v1.1.0). This uses SG's published cofficients which are optimised for 44.1kHz and 48kHz.

As an aside, I've been continuing my development of the spreading function - I will post a beta v1.1.1f later today.

Nick,
Sorry. Must've missed that in the change log

[sidewalking]

That's jolly decent of you,old chap!

Actually, I've been using Foobar and am quite happy to do my conversions that way - or directly from EAC. But I think you're right in that it might encourage people even less techie than me to use LossyWav.

If you want any testing done just give me a shout

No worries, it isn't much to write home about right now. I am just learning programming and am not that good with batch files, but using examples of the Flac/Speek frontend as inspiration, as well as the batch file examples here, I am able to get something basic that will write and execute the batch file for processing. I can handle a front end that will write the files, but I may need help from others on what the batch files should most efficiently contain.

You are more than welcome to help test - I would appreciate the help! Thanks for offering.

I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

I can handle the input/output, but so far haven't gotten to tagging. With some more time and research I am sure I can figure it out. Just give me some time and I will tackle it as soon as I can.


Nick.C: Is an option to specify output file name in the plans for the future, or do you want to stay with just the output directory option?

[Gregory S. Chudo...]

I have a small number of folders of Monkeys audio and Flac files, carefully ripped and tagged. I would like to convert them to Lossy Tak files with tags. Can your program help me do it in a single step?

--Rigapada

CUETools might help, although it's not exactly designed for that, so there are some limitations, e.g. files must be CDDA (44.1khz 16bit stereo). If your files are not accompanied by .cue sheets, use 'CUESheet creator' button first.

Then turn on the LossyWAV checkbox, select the desired codec, gaps appended cue style, press 'batch' button and browse to the desired folder.

This post has been edited by Gregory S. Chudov: Jan 23 2009, 23:36

[UED77]

Unfortunately, the FACT chunk is a necessary evil - if the user creates correction files when processing and wishes to revert to lossless at any point. The FACT chunks of the .lossy.wav and .lwcdf.wav files are compared and, if identical, the audio data is merged to reinstate the lossless original.

Yes, I understand the need for the FACT chunk (see this post of yours) but I'm thinking there could be a way of noting in some tag that the file was processed by LossyWAV.

The FACT chunk should still be added to the lossyWAV-processed WAV file.
But upon losslessly compressing the lossyWAV, perhaps add a tag to the output file that contains the same information as the FACT chunk would. Of course, there is nothing to stop the user from removing this tag later, but that's his fault, so to speak.

Of course, it could be more trouble than it's worth, and ultimately you never know how "lossless original" a PCM file is, but if the technical capability exists, we might decide to let the user know in good faith that his file is no longer "pristinely original".

[sauvage78]

Hi,

Is it possible yet to use lossywav with multichannel lossless Dolby True HD & DTS Master ?
I haven't tried personnaly but I am more & more interested in it ... I am specially scarred by this:

1.2.0: Checking of S (=L-R) channel for matrix surround content

should i wait for 1.2.0 to give it a try ?

This post has been edited by sauvage78: Jan 24 2009, 09:24