lossyWAV 1.2.0 Development Thread, Added noise WAV bitdepth reduction method Options

[Nick.C]

Following the release of lossyWAV 1.1.0b, I feel it is (again) time to kick off development of the next minor release.

Items currently on the list for inclusion in 1.x.0:

1.32.0: Implementation of SG's new noise shaping method;
1.2.0: Checking of S (=L-R) channel for matrix surround content;
1.2.0: Revisit the spreading function;

If you have any ideas, suggestions, code optimisations, etc, please post them here.

Link to the hydrogenaudio wiki article

lossyFLAC resultant bitrates:

CODE

10 Album Test Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 854kbit/s| 626kbit/s| 539kbit/s| 452kbit/s| 365kbit/s| 295kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 854kbit/s| 632kbit/s| 548kbit/s| 463kbit/s| 376kbit/s| 285kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.2.0 | default  | 854kbit/s| 627kbit/s| 544kbit/s| 460kbit/s| 376kbit/s| 288kbit/s| -------- | -------- |
|v1.2.0 |    -t    | 854kbit/s| 582kbit/s| 514kbit/s| 450kbit/s| 385kbit/s| 341kbit/s| 310kbit/s| 283kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+

55 Problem Sample Set
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|Version| Settings | FLAC -5  |--insane  |--extreme |--standard|--portable|  --zero  | --nasty  | --awful  |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.0.0b| default  | 780kbit/s| 655kbit/s| 582kbit/s| 503kbit/s| 417kbit/s| 330kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.1.0c| default  | 780kbit/s| 654kbit/s| 583kbit/s| 508kbit/s| 425kbit/s| 321kbit/s| -------- | -------- |
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|v1.2.0 | default  | 780kbit/s| 654kbit/s| 585kbit/s| 510kbit/s| 427kbit/s| 325kbit/s| -------- | -------- |
|v1.2.0 |    -t    | 780kbit/s| 623kbit/s| 565kbit/s| 506kbit/s| 441kbit/s| 391kbit/s| 354kbit/s| 322kbit/s|
+-------+----------+----------+----------+----------+----------+----------+----------+----------+----------+

Suggested foobar2000 converter setup:

lossyFLAC:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyTAK:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyWV:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

There is a known problem within foobar2000 (although more likely to do with cmd.exe itself) when running an executable within the cmd.exe command line from a path which includes spaces. The suggested fix for this is to enclose the element of the path which contains spaces within double quotation marks ("), e.g. c:\"program files"\directory_where_executable_is\executable_name

Change log 1.2.0: 16/12/09
Code optimisation;
Removal of negative -q values in default mode. Quality range for --altpreset remains -4 to 10 (--quality -4 --altpreset == --quality 0 --limit 15159).

Change log 1.1.5c: 21/11/09
Minor revision to internal setting for --altpreset.

Change log 1.1.5b: 20/11/09
Major revision to internal setting for --altpreset.

Change log 1.1.5a: 18/11/09
Bugfix: Correction to high sample-rate processing.

Change log 1.1.4s: 07/11/09
Bugfix: manual --limit setting not working as it should.

Change log 1.1.4r: 03/11/09
Bugfix: shaping in altpreset mode was artificially limited to 50% (only affected -q 6.5 and above).

Change log 1.1.4q: 02/11/09
Reversion to use of previous noise pre-calculated constant;
Shaping now OFF by default. To enable shaping use -s or --shaping, without a parameter for automatic shaping or with a value 0<=n<=1 for user specified shaping.

Change log 1.1.4p: 22/10/09
Mutual exclusivity of shaping, hilimit and altpreset removed;
Added noise pre-calculated constant removed in favour of improved derived formula;
--altpreset parameter now also -t.

Change log 1.1.4n: 27/09/09
Mutual exclusivity of shaping, hilimit and altpreset corrected.

Change log 1.1.4m: 26/09/09
--postanalyse function removed;
--limit changed to --hilimit and --lolimit;
--altpreset parameter introduced which changes default behaviour for shaping and hilimit.
[shaping = 0.5*(max(0,q/10)+max(0,q/10)^2.584962)) -q 0 = 0; -q 5 = 0.3333; -q 10 = 1]
[hilimit = round(14000 + 2000 * max(0,q/10)) / samplerate * 64) * (64/samplerate)]

Change log 1.1.4k: 24/08/09
--postanalyse function modified to use existing spreading function.

Change log 1.1.4j: 23/08/09
--limit lower range changed to 10000Hz.

Change log 1.1.4h: 22/08/09
--limit lower range changed to 14500Hz.

Change log 1.1.4g: 20/08/09
--maxsnr removed. -p or --postanalyse parameter implemented. Using this parameter checks the noise level of the correction data and compares to the low value derived from the associated source audio. If the correction noise (i.e. that of the difference signal) is greater than the source audio low value then the bits_to_remove value is reduced for the codec-block until the added noise is lower. Code further tidied. -F or --fftw parameter removed as FFTW dll is now automatically used if found (slight speed-up makes this the fastest way to go). Stack error fixed which occurs when libfftw3-3.dll v3.2.2 is used (newly released).

Change log 1.1.4f: 24/07/09
Bug in --maxsnr parameter fixed. Bug in pure Delphi compile fixed.

Change log 1.1.4e: 22/07/09
Major code redevelopment - more units, hopefully clearer. New parameter: -Y, --maxsnr <n> which allows specification of difference between maximum FFT result and added noise. Maxsnr works with both default spreading and --sortspread. Link to FFTW Windows DLL download page.

Change log 1.1.4d: 07/06/09
Bug fixed whereby lossyWAV would crash if 'libfftw3-3.dll' could not be initialised. If --fftw parameter is used and the DLL cannot be found then lossyWAV will revert to the existing FFT routines and output a warning. Link to FFTW Windows DLL download page.

Change log 1.1.4c: 05/06/09
FFTW can now be optionally used for FFT analyses in lossyWAV. Use of FFTW requires the presence of "libfftw3-3.dll" on the host computer, somewhere on the path and the addition of -F or --fftw to the lossyWAV command line. FFT (Delphi and assembler) further optimised. General code tidy-up. Link to FFTW Windows DLL download page.

Change log 1.1.4b: 14/05/09
FFT (Delphi and assembler) further optimised. Radix-4 FFT implemented in assembler and Delphi and Radix-8 in Delphi. Significant speedup of Delphi FFT throughput.
General code tidy-up.

Change log 1.1.4a: 05/05/09
--sorspread parameter no longer takes an additional parameter, now on/off;
spreading function changed slightly - now properly computes old and new averages separately;
FFT Real routine corrected as was giving wrong signs of some complex output values (did not affect magnitude of results);

Change log 1.1.3k: 30/04/09
Fault-finding release #1 to attempt to determine cause of WINE incompatibility. (Successful!! )

Change log 1.1.3j: 15/04/09
--sortspread parameter modified (again), now takes a parameter between 0 and 7, 2 is equivalent to beta 1.1.3i.
--centre parameter removed.
Reference_threshold tables removed in favour of direct calculation of the level of added noise due to bitdepth reduction using derived formula.

Change log 1.1.3i: 07/04/09
New --sortspread parameter modified (again). Bitrate matched with default spreading for my 55 problem sample set. Will revise table for my 10 Album Test Set.

Change log 1.1.3h: 05/04/09
New --sortspread parameter modified. Removed - bug found.

Change log 1.1.3g: 02/04/09
New --sortspread parameter introduced for testing purposes.

Change log 1.1.3f: 31/03/09
New --centre and --underlap <n> parameters introduced for testing purposes; Revised source.

Change log 1.1.3e: 18/03/09
Removal of old and new spreading functions in favour of variant; Code tidy up - speed improvements for pure delphi compile; Revised source.

Change log 1.1.3d: 05/03/09
Bug fix (would crash with a range error sometimes); Speedup of --varspread code. Revised source.

Change log 1.1.3c: 24/02/09
Introduction of -V or --varspread parameter to enable variant spreading function - a hybrid between the old and the new. Revised source.

Change log 1.1.3b: 23/02/09
Bug-fix: high sample rates with 1.1.3 would cause a range-check error or random results. Revised source.

Change log 1.1.3: 22/02/09
Integration of data structures used in new and old spreading functions. Source release.

Change log 1.1.2j: 18/02/09
Implementation of -O or --oldspread parameter to enable the use of the spreading function used in v1.1.0b instead of the revised version currently under development. This gives very slightly different results to v1.1.0b as is to be expected due to the revision of the reference-threshold constants at beta v1.1.1d.

Change log 1.1.2i: 12/02/09
Addition of a -N or --nasty (-q -2.0) and -A or --awful (-q -4.0) to allow extremely low quality levels to be explored.

Change log 1.1.2h: 12/02/09
Addition of a -N or --nasty (-q -2.0) to allow extremely low quality levels to be explored. Removed: Bug to be fixed.

Change log 1.1.2g: 10/02/09
Addition of a -r or --randombits parameter to randomise the zeroed lsbs.

Change log 1.1.2f: 09/02/09
Further modification to the spreading_function.

Change log 1.1.2e: 06/02/09
Further modification to the noise shaping process - first attempt to attenuate noise-shaping where bits_to_remove is zero for a particular codec block.

Change log 1.1.2d: 05/02/09
Further modification to the noise shaping process - audio data now no longer scaled prior to noiseshaping.

Change log 1.1.2c: 04/02/09
Further modification to the noise shaping process - noise shaping performed even when no bits removed.

Change log 1.1.2b: 03/02/09
Repair of the noise shaping process - now continuous for each channel rather than treating each codec-block totally separately;

Change log 1.1.2: 28/01/09
Code optimisations and data optimisations;
Revisions to the spreading function;

Change log 1.1.1e: 30/09/08
Interim beta, with source as reversion to Delphi complete (with conditional define to re-enable all IA-32/x87 code).

Change log 1.1.1d: 10/09/08
Further revision to the simplified spreading function - slightly higher bitrates than 1.1.1c but I'm happier with the method;
Reference-threshold constants re-calculated using more iterations (2^(32-fft-bit-length) iterations, i.e. 512K iterations for 8192 sample FFT and 128M iterations for 32 sample FFT) and for the first time taking into account FFT-result values less than 1. This only really affects bits-to-remove values between 1 and 7, which is in line with my expectation when I made the change to the noise-calculation method;

Change log 1.1.1c: 02/09/08
Further revision to the simplified spreading function;
Dither removed;

Change log 1.1.1b: 26/08/08
Revision to the simplified spreading function. All bin "averages" now calculated taking into account a variable proportion of bins to either side, i.e. "average" = (fft_result[i]+(fft_result[i-1]+fft_result[i+1])*factor)/(1+2*factor), where factor = 0.0 at 20Hz and 1.0 at 16kHz, with linear interpolation for intermediate values.

Change log 1.1.1a: 25/08/08
Fundamental simplification of spreading function methodology put forward for comment. All bin "averages" now calculated taking into account a fixed proportion of bins to either side, i.e. "average" = (fft_result[i]+(fft_result[i-1]+fft_result[i+1])*factor)/(1+2*factor), where factor = 0.26 in this case;
FFT result overall averaging now carried out prior to the spreading function rather than at the same time;
Reference_threshold constants revised slightly.

Change log 1.1.0b: 03/08/08
FFT lengths will now increase for higher bitrate audio, i.e. 88.2/96kHz, 176.4/192kHz and 352.8/384kHz;
improved logfile output and --detail output;
reference threshold constants for rectangular dither and triangular dither have been calculated so added noise should be the same for dither off and any dither level between 0 and 1 - the number of bits-to-remove will however reduce with "increasing" dither.

This post has been edited by Nick.C: Dec 16 2009, 22:31

[Antonski]

Suggested foobar2000 converter setup:

lossyFLAC:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyTAK:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyWV:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

Just out of curiosity, why there is no example for lossyAPE (MonkeyAudio)?
Sorry if this has already been mentioned somewhere, the lossyWAV threads are a bit too long

~