lossyWAV 1.1.0 Development Thread., Added noise WAV bit reduction method. Options

[Nick.C]

Following the release of lossyWAV 1.0.0b, I feel it is time to kick off development of the next minor release.

Items currently on the list for inclusion in 1.x.0:

1.1.0: STDIN input;
1.1.0: STDOUT output;
1.1.0: Channel independent bit removal;
1.1.0: Reversion to same bits-to-remove for all channels;
1.1.0: Noise shaping;
1.2.0: Checking of S (=L-R) channel for matrix surround content;

If you have any ideas, suggestions, code optimisations, etc, please post them here.

CODE

lossyWAV 1.1.0b, Copyright (C) 2007,2008 Nick Currie. Copyleft.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.

Process Description:

lossyWAV adds white noise to the processed output. The amount of added noise is
based on analysis of the signal levels in the frequency range 20Hz to 16kHz.

If signals above the upper limiting frequency are at an even lower level, they
can be swamped by the added noise. This is usually inaudible, but the behaviour
can be changed by specifying a higher --limit (in the range 16kHz to 20kHz).

For many audio signals, there is little content at very high frequencies, and
forcing lossyWAV to keep the added noise level lower than the content at these
frequencies can increase the bitrate dramatically for no perceptible benefit.

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-I, --insane        highest quality output, suitable for transcoding;
-E, --extreme       high quality output, also suitable for transcoding;
-S, --standard      default quality output, considered to be transparent;
-P, --portable      good quality output for DAP use. Not considered to be fully
                    transparent, but considered fit for its intended purpose.

Standard Options:

-c, --check         check if WAV file has already been processed; default=off.
                    errorlevel=16 if already processed, 0 if not.
-C, --correction    write correction file for processed WAV file; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-h, --help          display help.
-L, --longhelp      display extended help.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-o, --outdir <t>    destination directory for the output file(s).
-v, --version       display the lossyWAV version number.

Advanced Options:

-                   if filename="-" then WAV input is taken from STDIN.
    --blockdist     show distribution of lowest significant bit of input
                    codec-blocks and bit-removed codec-blocks.
-D, --dither <n>    enable variable PDF dither of output; default=off;
                    0 = rectangular; 1 = triangular; 0.5 = half way between.
-l, --limit <n>     set upper frequency limit to be used in analyses to n Hz;
                    (16000<=n<=20000), default = 16000.
    --linkchannels  Revert to original single bits-to-remove value for all
                    channels rather than channel dependent bits-to-remove.
-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
                    default = --standard = 5; --insane = 10; --extreme = 7.5;
                    --portable = 2.5)
    --sampledist    show distribution of lowest significant bit of input
                    samples and bit-removed samples.
    --scale <n>     scaling factor from WaveGain, etc; (0.0<n<=8.0),default=1.0
-s, --shaping <n>   enable fixed noise shaping; (0.00<=n<=1.00); default=q/10;
                    0.00 = off, 1.00 = 100% effectiveness, 0.50 = 50%, etc.
    --stdinname <t> pseudo filename to use when input from STDIN.
    --stdout        write processed WAV output to STDOUT.
-w, --writetolog    create (or append to) lossyWAV.log in the output directory.

System Options:

-B, --below         set process priority to below normal.
-d, --detail        enable detailed bits-to-remove information output mode
    --low           set process priority to low.
-n, --nowarnings    suppress lossyWAV warnings.
-Q, --quiet         significantly reduce screen output.
    --silent        no screen output.

Special thanks:

David Robinson      for the publication of his lossyFLAC method, guidance, and
                    the motivation to implement the method as lossyWAV.
Horst Albrecht      for ABX testing, valuable support in tuning the internal
                    presets, constructive criticism and all the feedback.
Sebastian Gesemann  for the noise shaping coefficients and help in using them
                    in the lossyWAV noise shaping implementation.
Don Cross           for the Complex-FFT algorithm used.

Link to the hydrogenaudio wiki article

Suggested foobar2000 converter setup:

lossyFLAC:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyTAK:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyWV:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name

Change log 1.1.0c: 30/04/2009
Exactly as 1.1.0b except that the WINE incompatibility issue has been fixed.

Executable here.
Source here.

Change log 1.1.0b: 03/08/08
FFT lengths will now increase for higher bitrate audio, i.e. 88.2/96kHz, 176.4/192kHz and 352.8/384kHz;
improved logfile output and --detail output;
reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.

Change log 1.1.0: 12/07/08
Certain advanced parameters removed for final release.

Change log 1.0.1x RC4: 12/07/08
Final release candidate prior to release of 1.1.0

Change log 1.0.1w RC3: 02/07/08
Code tidied up a bit more (yet again....);
--wine parameter modified to stop the program using Windows API function calls when using piped input (should hopefully stop crashing under Wine).

Change log 1.0.1v RC2: 30/06/08
Code tidied up a bit more (again....);
--wine parameter implemented to stop the program using the GetLastError Windows API call when using piped input (should stop crashing under Wine).

Change log 1.0.1u RC1: 20/06/08
Code tidied up a bit more;
--bitdist parameter introduced to allow user to "examine" the distribution of lowest set bit on a codec-block by codec-block basis, channels treated separately.

Change log beta 1.0.1t: 11/06/08
Revision to STDIN handling - bug found where last codec-block read from foobar2000 using STDIN input was not being written to the output file.

Change log beta 1.0.1s: 09/06/08
Revision to STDIN handling. Now (fingers crossed) should work successfully inside Foobar2000;
Code and help tidied up;
Dither function fixed and augmented. Taking on board a statement by SG with respect to using a dither function somewhere between rectangular (rand - 0.5) and triangular (rand-0.5)+(rand-0.5), i.e. (rand-0.5)+s*(rand-0.5) {0<=s<=1}. s=0 = rectangular dither; s=1 = triangular dither. -D, --dither now requires a supplementary <n> in the range 0<=n<=1.

Change log beta 1.0.1r: 03/06/08
Implementation of fast square root function using lookup tables for fxtract(ed) exponent and mantissa of input value;
--scale parameter corrected to accepted values in the range 0<n<=8.

Change log beta 1.0.1q: 30/05/08
Codec-block overflow bug (when codec-block-size=4096) corrected;

Change log beta 1.0.1p: 29/05/08
Quality synonym automatic noise shaping bug corrected;

Change log beta 1.0.1o: 29/05/08
Spreading function spread-zones and spreading-function string modified to allow finer control of high frequency zones;
Code "recovered" from 1.0.1e after a minor hardware failure

Change log beta 1.0.1n: 26/05/08
Implementation of -H, --highskew <n> parameter. Functionally identical to the internal skewing applied to the FFT results (-36dB @ 20Hz to 0dB at 3.45kHz) except applied from 3.45kHz upwards. Valid in the range 0 to 36 (0=default=no high skew applied).

Change log beta 1.0.1m: 25/05/08
reintroduction of max-inter-block-change implementation limits increase in bits-to-remove between codec-blocks to 1 bit.

Change log beta 1.0.1k: 23/05/08
static maximum_bits_to_remove limitation re-applied in serial with dynamic maximum_bits_to_remove limitation;
Automatic noise shaping now applied using a shaping-factor of quality-level / 10.

Change log beta 1.0.1j: 23/05/08
-q <n> quality selection moved to advanced settings;
-E, --excessive changed to --extreme; -I, --insane added, equivalent to -q 10;
--lowpass changed to -l, --limit in keeping with discussion;
Process Description text added to --longhelp.

Change log beta 1.0.1i: 23/05/08
-q <n> quality selection moved to advanced settings;
-E, --excessive; -N, --normal; -P, --portable quality "names" introduced following discussion in the development thread. These equate to -q 7.5; -q 5.0 and -q 2.5 respectively.

Change log beta 1.0.1h: 20/05/08
minimum bits to keep values changed for -q 0 and -q 1 to 2.333 and 2.667 respectively.

Change log beta 1.0.1g: 22/05/08
Reference_threshold > threshold_index > bits_to_remove calculation refined;
spreading function string modified;
minimum bits to keep values changed for -q 0 and -q 1;
--writetolog (-w) parameter implemented to write minimal output to "lossyWAV.log". Appends to existing file if already exists;
--lowpass <n> parameter re-implemented to allow users to set upper frequency limit of the range that lossyWAV uses in its analyses (16000<=n<=24000).

Change log beta 1.0.1f: 20/05/08
Filenaming logic "improved" when STDIN and STDOUT used together.

Change log beta 1.0.1e: 19/05/08
STDIN / STDOUT mode tidied up. Use the following as a flossy.bat file for foobar conversion:

CODE

@echo off
z:\bin\lossyWAV %1 --low --nowarnings --quiet %3 %4 %5 %6 %7 %8 %9 --stdout|z:\bin\flac - -5 -f -b 512 -o%2

Unfortunately, due to the nature of piped input to FLAC, the lossyWAV 'fact' chunk is lost. This means no record is kept within the file that is has been processed with lossyWAV (however, the lower the quality setting of the processing, the more likely the bitrate will be an obvious indicator that the file has indeed been processed with lossyWAV);
Minor error found and amended in revised remove_bits procedure, no minimum_bits_to_keep value was being applied, although this has little impact at -q >= 2;
New parameter --linkchannels implemented to revert to old remove_bits method whereby all channels share the same bits_to_remove. Implementing this, I found an error in the original which was forcing more bits to be lost to clipping prevention than should have been (i.e. output was more conservative).

Change log beta 1.0.1d: 18/05/08
STDIN / STDOUT mode modified again (use '-' as a filename to enable STDIN input, --stdout to enable STDOUT output).
Console output has been redirected to 'con', rather than STDOUT.

Change log beta 1.0.1c: 16/05/08
STDIN / STDOUT mode modified again (use '-' as a filename to enable STDIN input).

Change log beta 1.0.1b: 15/05/08
Channel independent bit-removal implemented;
STDIN / STDOUT mode modified - still very much a work in progress.

Change log beta 1.0.1: 14/05/08
STDIN / STDOUT mode commenced.

This post has been edited by Nick.C: May 14 2009, 13:10

[SebastianG]

Also, the quantization/dithering part could be done jointly on the channels using a generalized metric in the spirit of M/S coding. You usually don't want the quantization noise's coherence (comparing left versus right) to differ greatly from the coherence between the left and right of your original signal, I suppose.

edit: This is probably overkill at the moment.

This post has been edited by SebastianG: May 15 2008, 14:46