lossyWAV Development, WAV bit reduction by 2BDecided Options

[Nick.C]

If you're going to do that, you should also move the threshold shift down from where it is, into that calculation, otherwise the additional accuracy is pretty meaningless.

I will do that for the next rev.

Thank you. Just for clarity: this includes those thresholds, but hasn't fixed the "contamination" bug?

Yes - I didn't change the bit-removal procedure.

[Nick.C]

Comparison of "round" method vs "floor" method used when determining bits_to_remove from min_bin values. Noise_threshold_shift moved into this calculation rather than in threshold_index calculation.

Attached comparison for the 50 sample set. Removed.

Rounding or flooring is selected at present using a command line parameter -floor, however, it may be prudent to remove this option but always use floor.
[edit]I feel that due to the small change in bits_to_remove, and for less likelihood of added noise over the min_bin value, that the code should be changed to always floor this result. v0.3.1 will include this.[/edit]

@David, as an aside, I mentioned earlier about randomness in the bits_to_remove in the Matlab script - this would happen if the samples were dithered when bits were removed.

This post has been edited by Nick.C: Jan 19 2008, 16:30

[2Bdecided]

@David, as an aside, I mentioned earlier about randomness in the bits_to_remove in the Matlab script - this would happen if the samples were dithered when bits were removed.

Yes, it would. Where's the smiley for "hangs head in shame"!

Cheers,
David.

[Nick.C]

Where's the smiley for "hangs head in shame"!

Nope, no need - yours was a minor oversight. Floor implemented with noise_threshold_shift moved. v0.3.1 tonight, I think.

[Nick.C]

Revised LossyFLAC6_x.m - no pollution of audio data when removing bits, floor rather than round used in determining min_bin.

lossyWAV alpha v0.3.1 attached.

CODE

lossyWAV alpha v0.3.1 : WAV file bit depth reduction by 2Bdecided.
Transcoded by Nick.C & Halb27 from a script, www.hydrogenaudio.org

lossyWav usage: <input wav file> <options>

Options:
-1, -2 or -3  quality level (1:overkill, 2:default, 3:compact)
-cbs <n>      analysis codec_block_size (512<=n<=4608, default=576 bytes)
              (should match codec block size used in target compression codec)
-o <folder>   destination folder for the output file
-dither <n>   dither selection, 0<=n<=2, default=0. 0=no dither; 1=rectangular
              dither; 2=triangular dither.
-clipping <n> clipping prevention selection, 0<=n<=1, default=0. 0=none;
              1=fixed clipping prevention amplitude reduction, taking into
              account dither amplitude (if any).
-vsfl         select variable spreading function lengths
              (sfl = number of fft bins averaged during convolution of fft
              results. short fft = short sfl, long fft = long sfl)
-wsf          select weighted spreading functions.
              (weighted average of fft bins during convolution of fft results
              weighted towards lower frequency fft bins, 5/8:3/8)
-nts <n>      noise_threshold_shift=n (-15.0<=n<=0.0, default -1.5dB)
              (reduces overall bits to remove)
-overlap <n>  fft_overlap = fft_length/n (2<=n<=8, default=2)
              (increases number of fft analyses per codec block)
-skew <n>     skew results of fft analyses by n dB (0.0<=n<=12.0, default=0.0)
              with a (sin-1) shaping over the frequency range 20Hz to 3.7kHz.
              (artificially increase low frequency bins to take into account
              higher SNR requirements at low frequencies)
-quiet        significantly reduce screen output
-nowarn       suppress lossyWAV warnings
-detail       enable detailled output mode

Options not yet implemented:
-bitdepth <n> forced output bitdepth (16 or 24)
-flac         optimizations for use with FLAC
-wv           optimizations for use with wavPack
-tak          optimizations for use with TAK

Default noise_threshold_shift magnitude reduced (-3.0db to -1.5db) at the same time as the change from round to floor.

Output now *identical*(!) in terms of bits removed per block.

[edit]Default clipping prevention bug report acted upon - v0.3.1 removed, v0.3.1b below.[/edit]

This post has been edited by Nick.C: Sep 27 2007, 22:45

Attached File(s)

 LossyFLAC6_x.zip ( 5.29K ) Number of downloads: 193

 

[halb27]

Nick, you're the king, incredibly fast in bringing out the new versions with all the current items included, and wonderful quality judging so far:

I did a quick test with v0.3.1 using -cbs 1024 -skew 9.0 -cliipping 0 (an explicit setting of -clipping 0 was necessary - it seems like 0 isn't the default):

Atem-lied: Could not abx, which is very remarkable as more bits are removed than with say v0.2.0 - quality improvement due to not dithering?

bibilolo was fine to me as well.

Even with 2Bdecided's dither_noise_test.wav there was no audible issue to me, and this is a very artificial sample, and many bits are removed.

This post has been edited by halb27: Sep 27 2007, 22:11

[Nick.C]

I did a quick test with v0.3.1 using -cbs 1024 -skew 9.0 -cliipping 0 (an explicit setting of -clipping 0 was necessary - it seems like 0 isn't the default):

Atem-lied: Could not abx, which is very remarkable as more bits are removed than with say v0.2.0 - quality improvement due to not dithering?

bibilolo was fine to me as well.

Even with 2Bdecided's dither_noise_test.wav there was no audible issue to me, and this is a very artificial sample, and many bits are removed.

-clipping 0 *should * be a default setting, I'll investigate. [edit] Ahem...... mea culpa! Corrected in alpha v0.3.1b[/edit] Superseded by v0.3.2

My gut feeling is that the move from round to floor has made quite a difference where previously bits_to_remove would have been 1 more than the new version (all else being equal) for certain codec_blocks in the sample.

I've been listening to my 51 sample set (added a Black Sabbath extract from Iron Man ) at -3 -nts 0 -dither 2 -clipping 1 -skew 9 -wsf -vsfl and I'm *really* pleased at the result - so much so that this may become my DAP conversion setting.

This post has been edited by Nick.C: Sep 29 2007, 13:48

[Nick.C]

Just out of curiosity, will bit-reduction cause ALAC compress any better. I just bought an iPod and the idea of lossy flac is great but now I'm stuck without my trusty FLAC format if I want lossless music (and rockbox doesn't really appeal to me that much).

Thanks,
Bobby

Sorry for the late reply Bobby - I think in the original thread there's some discussion about the applicability of the method to various codecs and from memory only really FLAC, Wavpack & Tak will benefit. The other way of looking at is is to just give it a try and see if the compressed file is indeed smaller!

Regards,

Nick.

(or, you could Rockbox your iPod for FLAC support!)

This post has been edited by Nick.C: Sep 28 2007, 08:23

[Nick.C]

Performance results for v0.3.1b:

Guruboolez's 150 sample set:

WAV: 252.36MB; 1411.2kbps;
FLAC: 122.17MB; 683kbps;
lossy -1: 95.7MB; 535kbps;
lossy -2: 89.6MB; 501kbps;
lossy -3: 88.1MB; 492kbps.

My "working" 52 sample set:

WAV: 121.53MB; 1411.2kbps;
FLAC: 68.2MB; 792kbps;
lossy -1: 42.6MB; 495kbps;
lossy -2: 39.5MB; 458kbps;
lossy -3: 38.6MB; 449kbps.

I am currently processing a number of albums and will revert with "real-world" results:

AC/DC - Who Made Who :
WAV: 385.3MB; 1411.2kbps;
FLAC: 238.9MB; 875kbps;
lossy -3: 112.2MB; 411kbps;

Gerry Rafferty - City To City:
WAV: 541.8MB; 1411.2kbps;
FLAC: 307.9MB; 802kbps;
lossy -3: 161.2MB; 420kbps;

Jean Michel Jarre - Oxygene:
WAV: 400.7MB; 1411.2kbps;
FLAC: 219.5MB; 773kbps;
lossy -3: 134.8MB; 475kbps;

The Shamen - Boss Drum:
WAV: 663.3MB; 1411.2kbps;
FLAC: 433.4MB; 922kbps;
lossy -3: 170.0MB; 362kbps;

Van Morrison - Astral Weeks:
WAV: 477.0MB; 1411.2kbps;
FLAC: 255.9MB; 757kbps;
lossy -3: 142.6MB; 422kbps.

I carried out a multi-generational experiment and was pleasantly surprised to see that the output matches the input after surprisingly few generations. This was with no dither or clipping prevention. On clipping prevention, up to now there has been no check made to see if the new sample value lies in the range of permitted sample values - v0.3.2 will incorporate this and also issue warnings at the end of processing if samples have required to be clipped to the maximum permissible or minimum permissible value respectively.

This post has been edited by Nick.C: Jan 19 2008, 16:29

[user]

You yield in the end to ca. 50% of Lossless (eg. flac) bitrates, in numbers around 450 kbit/s (+-50).
The technical approach of removing bits reminds me to wavpack lossy, which has also great more or less tranparent results at these high - very high lossy bitrates.
Isn't the technical approach between wavpack lossy and this algorithmn very very similar, ie. (nearly) no psycho-acoustic model which might cause artifacts, but instead a higher noise floor, be it perceptual or not ?

(At a side note, both, the results of wavpack lossy and of this algorithmn could be also similar to DAT at 32 kHz, 12 bit, ie. the way a DAT recorder in longplay position downsamples from 44.1 kHz 16 bit to 32 kHz, 12 bit, to achieve the half of the data.
While for many people without possibility of X/Y or A/B, even not to speak of ABX, DAT Longplay 32 kHz 12 bit is also "transparent").

Can somebody of the inventors (of this algorithmn) explain in a few simple words, what is then the difference between wavpack lossy and this new algorithmn ?
(or link to that post, where that has been explained already )

This post has been edited by user: Sep 28 2007, 15:32

[bryant]

Isn't the technical approach between wavpack lossy and this algorithmn very very similar, ie. (nearly) no psycho-acoustic model which might cause artifacts, but instead a higher noise floor, be it perceptual or not ?

Can somebody of the inventors (of this algorithmn) explain in a few simple words, what is then the difference between wavpack lossy and this new algorithmn ?

There are several differences, but I'll mentioned what I think are the most important.

The first is that this is designed to work with FLAC, or any other lossless compressor that can detect and optimize away redundant zeros in the audio data words. The advantage of this is that it can be used with the many existing FLAC hardware (and software) players without modification or update.

The other major difference is that lossyFLAC is intelligent. WavPack lossy is CBR and simply gets as close as it can to the source given the allowed bitrate. LossyFLAC is VBR because it analyzes the source and tries to determine the most quantization noise it can add and still be completely transparent. In this way it's more like the WavPack lossy “quality” mode I worked on (and abandoned) several years ago.

Both methods rely on some psychoacoustic principles to achieve their results. In the WavPack case it chooses noise shaping and joint stereo encoding to make the noise less audible. LossyFLAC uses them to help determine how much white noise it can add (i.e., the fact that we are less sensitive to added noise at higher frequencies). However, neither have a psychoacoustic model in the strict sense and, of course, neither use any sort of frequency domain encoding or digital filtering in the signal path.

[Brent]

Just out of curiosity, will bit-reduction cause ALAC compress any better. I just bought an iPod and the idea of lossy flac is great but now I'm stuck without my trusty FLAC format if I want lossless music (and rockbox doesn't really appeal to me that much).

Thanks,
Bobby

I ran a test with .3.1b -3:


(I'm too lazy to type it out, I attached image incase host goes down)

So it has no effect on filesize, slightly negative in fact. LossyWav causes some bad clipping in the case of Ali's Here and Unbreakable btw.
Too bad actually, cause it seemed a good way to save some space on the iPod.

This post has been edited by Brent: Sep 28 2007, 21:11

Attached thumbnail(s)

 

[Nick.C]

I ran a test with .3.1b -3:

So it has no effect on filesize, slightly negative in fact. LossyWav causes some bad clipping in the case of Ali's Here and Unbreakable btw.
Too bad actually, cause it seemed a good way to save some space on the iPod.

Thanks for the testing. You can switch on fixed amplitude reduction clipping prevention by using the -clipping 1 parameter.

Attached lossyWAV alpha v0.3.2. Superseded by v0.3.3

Code tidied up further and will now require -force parameter to over-write an existing file. If clipping occurs, warning(s) are issued after processing. I say warning(s) because it counts clipping over maximum and under minimum separately.

This post has been edited by Nick.C: Sep 29 2007, 22:14

[Mitch 1 2]

Could you please add a command-line parameter to set the process priority to "low"?

[Nick.C]

Could you please add a command-line parameter to set the process priority to "low"?

Mitch,

I'll certainly look into it and see if I can - I've never tried it before, but there must be code available to implement it (somewhere....).

Thanks for your input.

Nick.

[guruboolez]

@Nick.C
Did the settings change since version ~0.20? I'm unable to make the last versions (>0.30) work with foobar2000.

What I'm using is:
encoder: C:\windows\system32\cmd.exe
extension: lossy.flac
Parameters: /d /c C:\CODEC\lossywav\LossyWav.bat %s %d
Format is: lossy
bps: 16 bits

my batch files is the following one:

CODE

@echo off
set lossyWAV_path="C:\CODEC\lossywav\lossyWav.exe"
set flac_path="C:\CODEC\lossywav\flac.exe"
rem echo %lossyWAV_path% %1 -c 576 >>flac_lossy.txt
rem echo %flac_path% -8 -f -b 576 -o"%~DPN2.flac" "%~DPN1.lossy.wav" >>flac_lossy.txt
rem echo del "%~DPN1.lossy.wav" >>flac_lossy.txt
%lossyWAV_path% %1 -c 576
%flac_path% -8 -f -b 576 "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"
set output_file=
set lossyWAV_path=
set flac_path=

[Nick.C]

@Nick.C
Did the settings change since version ~0.20? I'm unable to make the last versions (>0.30) work with foobar2000.

What I'm using is:
encoder: C:\windows\system32\cmd.exe
extension: lossy.flac
Parameters: /d /c C:\CODEC\lossywav\LossyWav.bat %s %d
Format is: lossy
bps: 16 bits

my batch files is the following one:

CODE

@echo off
set lossyWAV_path="C:\CODEC\lossywav\lossyWav.exe"
set flac_path="C:\CODEC\lossywav\flac.exe"
rem echo %lossyWAV_path% %1 -c 576 >>flac_lossy.txt
rem echo %flac_path% -8 -f -b 576 -o"%~DPN2.flac" "%~DPN1.lossy.wav" >>flac_lossy.txt
rem echo del "%~DPN1.lossy.wav" >>flac_lossy.txt
%lossyWAV_path% %1 -c 576
%flac_path% -8 -f -b 576 "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"
set output_file=
set lossyWAV_path=
set flac_path=

Settings did change a bit - "-c" is now "-cbs", but a codec_block_size of 576 is now the default setting, so you can remove "-c 576" altogether.

[guruboolez]

It works fine again. Thank you very much!

[guruboolez]

Lossywav is impressive on harpsichord recordings.
It brings a 1062 kbps flac encoding to a pretty nice 436 kbps: 60% reduction!!!

Sound quality was fine, but strangely (because I didn't heard something wrong at first sight and I didn't expect to hear something different from original) my ABX score isn't bad at all:

CODE

foo_abx 1.3.1 report
foobar2000 v0.9.4.5 beta 1
2007/09/29 16:36:04

File A: F:\Fiocco - [Demeyere] Pièces de Clavecin, Oeuvre Premier\CD 1\12-Vivace.flac
File B: N:\12-Vivace.lossy.flac

16:36:04 : Test started.
16:36:29 : 01/01  50.0%
16:36:35 : 02/02  25.0%
16:36:58 : 02/03  50.0%
16:37:04 : 03/04  31.3%
16:37:14 : 04/05  18.8%
16:37:31 : 04/06  34.4%
16:37:36 : 05/07  22.7%
16:37:41 : 06/08  14.5%
16:37:54 : 06/09  25.4%
16:37:59 : 07/10  17.2%
16:38:11 : 08/11  11.3%
16:38:18 : 08/12  19.4%
16:38:21 : 08/13  29.1%
16:38:24 : 09/14  21.2%
16:38:29 : 10/15  15.1%
16:38:40 : 11/16  10.5%
16:38:45 : 12/17  7.2%
16:38:51 : 13/18  4.8%
16:39:00 : 14/19  3.2%
16:39:08 : 15/20  2.1%
16:39:15 : 16/21  1.3%
16:39:23 : 16/22  2.6%
16:39:31 : 16/23  4.7%
16:39:36 : 17/24  3.2%
16:40:04 : 18/25  2.2%
16:40:18 : 19/26  1.4%
16:40:27 : 19/27  2.6%
16:40:40 : 20/28  1.8%
16:40:51 : 20/29  3.1%
16:41:02 : 21/30  2.1%
16:41:07 : 21/31  3.5%
16:41:11 : 22/32  2.5%
16:41:32 : Test finished.

----------
Total: 22/32 (2.5%)

Second half was better than first one (it's usually the opposite). There were no noise, no artefact, but something hard to define (audiophile would call it "lack of soundstage" or something similar).
I'm not completely sure that this ABX score is valid (since I didn't really fix a target: first 16 trials, then I decided to extend it to 24 and when I reached 24 I was so confident that I extend it again to 32 trials). Pio2001 explained once why such test need a better score to be valid.

I uploaded a short sample from this sample (the part I ABXed here):
http://rapidshare.com/files/59085975/fiocco.zip.html

[Nick.C]

From the look of your command line, you're not changing anything that isn't default in v0.3.2. If you're looking to maintain the same processing rate, use "-nts n" where n is less than -1.5 this will reduce the bits removed in the processing - at the rate of 1 bit per -6 of nts (noise_threshold_shift).

The other way to do it would be to use the "-1" quality option to see if that makes a difference to your sample.

[Nick.C]

@Mitch 1 2: I have found a method of lowering process priority, although the permissible priority settings vary with Windows version. Are you running Windows later than NT, ME, 98 & 95? If so then I can provide 3 settings: Normal, Below_Normal and Low priority. If not, Below_Normal vanishes as it is not supported by the O/S mentioned.

I will implement "-below" and "-low" ("-normal" unavailable, but default) parameters to lower the process priority - but below won't work on all versions of Windows.

This post has been edited by Nick.C: Sep 29 2007, 22:20

[Nick.C]

lossyWAV alpha v0.3.3 attached. Immediately superseded by v0.3.3b - output file access bug.

-below and -low parameters incorporated, setting below_normal process priority and low process priority respectively.

Now checks for read-only output files and will attempt to set read/write access, will also report if read/write access setting fails.

CODE

lossyWAV alpha v0.3.3 : WAV file bit depth reduction method by 2Bdecided.
Transcoded to Delphi by Nick.C & Halb27 from a script, www.hydrogenaudio.org

Usage: lossyWAV <input wav file> <options>

Options:

-1, -2 or -3  quality level (1:overkill, 2:default, 3:compact)
-o <folder>   destination folder for the output file
-force        forcibly over-write output file if it exists.

-cbs <n>      analysis codec_block_size (512<=n<=4608, default=576 samples)
              (should match codec block size used in target compression codec)
-nts <n>      noise_threshold_shift=n (-15.0<=n<=0.0, default -1.5dB)
              (reduces overall bits to remove)
-skew <n>     skew results of fft analyses by n dB (0.0<=n<=12.0, default=0.0)
              with a (sin-1) shaping over the frequency range 20Hz to 3.7kHz.
              (artificially decrease low frequency bins to take into account
              higher SNR requirements at low frequencies)

-dither <n>   dither selection, 0<=n<=2, default=0
              (0=no dither; 1=rectangular dither; 2=triangular dither)
-clipping <n> clipping prevention selection, 0<=n<=1, default=0. 0=none;
              1=fixed clipping prevention amplitude reduction, taking into
              account dither amplitude (if any).
-vsfl         select variable spreading function lengths (sfl)
              (sfl = number of fft bins averaged during convolution of fft
              results. short fft = short sfl, long fft = long sfl)
-wsf          select weighted spreading functions.
              (weighted average of fft bins during convolution of fft results
              weighted towards lower frequency fft bins, 5/8:3/8)
-overlap <n>  fft_overlap = fft_length/n (2<=n<=8, default=2)
              (increases number of fft analyses per codec block)

-quiet        significantly reduce screen output
-nowarn       suppress lossyWAV warnings
-detail       enable detailled output mode
-below        set process priority to below normal.
-low          set process priority to low.

Options not yet implemented:

-bitdepth <n> forced output bitdepth (16 or 24)
-flac         optimizations for use with FLAC
-wv           optimizations for use with wavPack
-tak          optimizations for use with TAK

This post has been edited by Nick.C: Sep 30 2007, 13:39

[Mitch 1 2]

I've tested lossyWAV 0.3.3 on Windows XP SP2, and there's a serious problem with output file permissions which is not present in the 0.3.2 version. lossyWAV fails to gain write access even when there is no existing file.

[Nick.C]

I've tested lossyWAV 0.3.3 on Windows XP SP2, and there's a serious problem with output file permissions which is not present in the 0.3.2 version. lossyWAV fails to gain write access even when there is no existing file.

That sounds like inadequate testing on my part prior to release. It *should* work if the output file already exists, however this is embarassing - I'll get v0.3.3b out as soon as possible.

This version should work - apologies for the error. - File removed, superseded.

Nick.

This post has been edited by Nick.C: Oct 5 2007, 09:12

[Mitch 1 2]

It's alpha software, so I forgive you.
Now lossyWAV works as expected, and it even handled my conflicting parameter tests!