lossyWAV 1.1.0 released., Added noise WAV bit reduction method. Options

[Nick.C]

lossyWAV 1.1.0c is now available for download.

CODE

lossyWAV 1.1.0c, Copyright (C) 2007,2008 Nick Currie. Copyleft.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.

This program is distributed in the hope that it will be useful,but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
this program.  If not, see <http://www.gnu.org/licenses/>.

Process Description:

lossyWAV adds white noise to the processed output. The amount of added noise is
based on analysis of the signal levels in the frequency range 20Hz to 16kHz.

If signals above the upper limiting frequency are at an even lower level, they
can be swamped by the added noise. This is usually inaudible, but the behaviour
can be changed by specifying a higher --limit (in the range 16kHz to 20kHz).

For many audio signals, there is little content at very high frequencies, and
forcing lossyWAV to keep the added noise level lower than the content at these
frequencies can increase the bitrate dramatically for no perceptible benefit.

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-I, --insane        highest quality output, suitable for transcoding;
-E, --extreme       high quality output, also suitable for transcoding;
-S, --standard      default quality output, considered to be transparent;
-P, --portable      good quality output for DAP use. Not considered to be fully
                    transparent, but considered fit for its intended purpose.

Standard Options:

-c, --check         check if WAV file has already been processed; default=off.
                    errorlevel=16 if already processed, 0 if not.
-C, --correction    write correction file for processed WAV file; default=off.
-f, --force         forcibly over-write output file if it exists; default=off.
-h, --help          display help.
-L, --longhelp      display extended help.
-M, --merge         merge existing lossy.wav and lwcdf.wav files.
-o, --outdir <t>    destination directory for the output file(s).
-v, --version       display the lossyWAV version number.

Advanced Options:

-                   if filename="-" then WAV input is taken from STDIN.
    --blockdist     show distribution of lowest significant bit of input
                    codec-blocks and bit-removed codec-blocks.
-D, --dither <n>    enable variable PDF dither of output; default=off;
                    0 = rectangular; 1 = triangular; 0.5 = half way between.
-l, --limit <n>     set upper frequency limit to be used in analyses to n Hz;
                    (16000<=n<=20000), default = 16000.
    --linkchannels  Revert to original single bits-to-remove value for all
                    channels rather than channel dependent bits-to-remove.
-q, --quality <n>   quality preset (10=highest quality, 0=lowest bitrate;
                    default = --standard = 5; --insane = 10; --extreme = 7.5;
                    --portable = 2.5)
    --sampledist    show distribution of lowest significant bit of input
                    samples and bit-removed samples.
    --scale <n>     scaling factor from WaveGain, etc; (0.0<n<=8.0),default=1.0
-s, --shaping <n>   enable fixed noise shaping; (0.00<=n<=1.00); default=q/10;
                    0.00 = off, 1.00 = 100% effectiveness, 0.50 = 50%, etc.
    --stdinname <t> pseudo filename to use when input from STDIN.
    --stdout        write processed WAV output to STDOUT.
-w, --writetolog    create (or append to) lossyWAV.log in the output directory.

System Options:

-B, --below         set process priority to below normal.
-d, --detail        enable detailed bits-to-remove information output mode
    --low           set process priority to low.
-n, --nowarnings    suppress lossyWAV warnings.
-Q, --quiet         significantly reduce screen output.
    --silent        no screen output.

Special thanks:

David Robinson      for the publication of his lossyFLAC method, guidance, and
                    the motivation to implement the method as lossyWAV.
Horst Albrecht      for ABX testing, valuable support in tuning the internal
                    presets, constructive criticism and all the feedback.
Sebastian Gesemann  for the noise shaping coefficients and help in using them
                    in the lossyWAV noise shaping implementation.
Don Cross           for the Complex-FFT algorithm used.

Link to the hydrogenaudio wiki article

Suggested foobar2000 converter setup:

lossyFLAC:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.flac
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\flac - -b 512 -5 -f -o%d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyTAK:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.tak
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\takc -e -p2m -fsl512 -ihs - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

lossyWV:

CODE

Encoder: c:\windows\system32\cmd.exe
Extension: lossy.wv
Parameters: /d /c c:\"program files"\bin\lossywav - --standard --silent --stdout|c:\"program files"\bin\wavpack -hm --blocksize=512 --merge-blocks -i - %d
Format is: lossless or hybrid
Highest BPS mode supported: 24

Enclose the element of the path containing spaces within double quotation marks ("), e.g. C:\"Program Files"\directory_where_executable_is\executable_name. This is a Windows limitation.
N.B.: lossyWAV 1.2.0 development thread here

This post has been edited by Nick.C: Apr 17 2012, 20:51

Attached File(s)

 lossyWAV_1.1.0c.zip ( 75.56K ) Number of downloads: 785
 lossyWAV_Source_20090430_2145_1.1.0c.zip ( 201.82K ) Number of downloads: 271

 

[botface]

lossyWAV 1.1.0 is now available for download.

Well done, Nick. Another hassle-free release

[Nick.C]

Well done, Nick. Another hassle-free release

Many thanks - glad to be of service!

[Axon]

Yayayay! Thanks a lot Nick.

Has there been a comparison of lossless encoder performance when combined with lossyWAV?

[Nick.C]

Has there been a comparison of lossless encoder performance when combined with lossyWAV?

Erm, no - but it would not be too difficult to set one up.

I would imagine Lame, ogg vorbis and neroAacEnc would be the targets at varying quality levels spanning the range for each, from both lossless FLAC and lossyFLAC (--portable, --standard, --extreme and --insane presets).

So, maybe 3 codecs x 6 levels per codec x (4 quality presets of lossyFLAC + lossless FLAC) = 90 average bitrates for an agreed test set of tracks / samples.

Comments / suggestions please! [edit] Can't count.... [/edit]

This post has been edited by Nick.C: Jul 14 2008, 20:36

[carpman]

Has there been a comparison of lossless encoder performance when combined with lossyWAV?

Though lossy would also be interesting, perhaps.

C.

[Axon]

Yeah, I'm mainly curious as to if comparing lossyTAK/lossyFLAC would yield an increase in, um, performance increase relative to a comparison of TAK/FLAC. Certainly some encoders seem to handle white noise better than others, so by reducing the bit depth in lossyWAV's fashion, that impact could be reduced.

[Nick.C]

Has there been a comparison of lossless encoder performance when combined with lossyWAV?

Though lossy would also be interesting, perhaps.

C.

Apparently I can't read and can't count....

I'll get to work on the lossyFLAC / lossyTAK / lossyWV comparison using recommended FLAC / TAK / Wavpack settings above at -q 0, --portable, --standard, --extreme and --insane.

[edit] After a quick batch file or two and making sure that my source FLAC set was encoded at -5. Lossless FLAC / TAK / Wavpack all encoded without using the 512 sample blocksize parameters. So, for my 55 problem sample set:

CODE

             +------------+------------+------------+
             |    TAK     |    FLAC    |  Wavpack   |
+------------+------------+------------+------------+
| lossless   | 749 kbit/s | 787 kbit/s | 772 kbit/s |
+------------+------------+------------+------------+
| --insane   | 634 kbit/s | 654 kbit/s | 661 kbit/s |
| --extreme  | 564 kbit/s | 583 kbit/s | 599 kbit/s |
| --standard | 488 kbit/s | 508 kbit/s | 529 kbit/s |
| --portable | 405 kbit/s | 425 kbit/s | 444 kbit/s |
| -q 0       | 299 kbit/s | 321 kbit/s | 342 kbit/s |
+------------+------------+------------+------------+

This post has been edited by Nick.C: Jul 14 2008, 22:15

[sauvage78]

thks Nick, now that noise shaping was done in 1.1.0, is there still room for improvement (lower bitrate) in 1.2.0, or are you reaching the end of what you wanted to do with lossyWAV ? I mean now the todo list for 1.2.0 seems rather small from my non-technical point of view ... shouldn't 1.2.0 be a 1.1.1 now ? will there be just cosmetic changes & bugfix from now on ???

This post has been edited by sauvage78: Jul 14 2008, 21:23

[Nick.C]

thks Nick, now that noise shaping was done in 1.1.0, is there still room for improvement (lower bitrate) in 1.2.0, or are you reaching the end of what you wanted to do with lossyWAV ? I mean now the todo list for 1.2.0 seems rather small from my non-technical point of view ... shouldn't 1.2.0 be a 1.1.1 now ? will there be just cosmetic changes & bugfix from now on ???

My intention is to understand and implement SebastianG's new noise shaping method, but for that I will also have to introduce / find a PSY model of some kind.

I would hope that by using the new noise shaping method some additional bits can be removed for the same apparent quality level of output, thereby further reducing the bitrate. However, this could be some time in happening as my basic understanding of the topic has to improve somewhat before I start implementation.

Unlike last time, I will not be starting the development of 1.2.0 a day or two after the release of the new version.

In a lot of ways, I've achieved what I set out to do (about a year ago) in accordance with David's original method (albeit with a few tweaks here and there).

I have an idea for a cosmetic change for the --detail parameter and may make modifications so that its output can be written to the logfile, although as more than one instance of lossyWAV could be running I want to keep the time that the logfile is open for output to an absolute minimum.

Cosmetically, I am happier with 1.1.0 than I was with 1.0.0b, but that has only come about by changing the way that the screen / log output is handled.

[Agent69]

Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?

[carpman]

Your question assumes that disk space is never an issue on HDDs.
I use lossyWAV but don't have a DAP.
Also it could be used to reduce bandwidth issues in terms of selling high quality transcodable files online (I have a big problem with buying audio that's trapped in a particular format, i.e MP3).
If you can save space without any loss in audible quality it seems sensible to do so. Space = money, so why not save it where you can, even if money is not an issue.

C.

[Synthetic Soul]

Yeah, I'm mainly curious as to if comparing lossyTAK/lossyFLAC would yield an increase in, um, performance increase relative to a comparison of TAK/FLAC. Certainly some encoders seem to handle white noise better than others, so by reducing the bit depth in lossyWAV's fashion, that impact could be reduced.

Using Nick's figures above, it seems it is much of a muchness, although WavPack seems to benefit a little less (as can be seen by FLAC actually producing smaller bitrates with most settings:

CODE

             +------------------+------------------+------------------+
             |    TAK           |    FLAC          |  Wavpack         |
+------------+------------------+------------------+------------------+
| lossless   | 749 kbit/s       | 787 kbit/s        | 772 kbit/s      |
+------------+------------------+------------------+------------------+
| --insane   | 634 kbit/s (85%) | 654 kbit/s (83%) | 661 kbit/s (86%) |
| --extreme  | 564 kbit/s (75%) | 583 kbit/s (74%) | 599 kbit/s (78%) |
| --standard | 488 kbit/s (65%) | 508 kbit/s (65%) | 529 kbit/s (69%) |
| --portable | 405 kbit/s (54%) | 425 kbit/s (54%) | 444 kbit/s (58%) |
| -q 0       | 299 kbit/s (40%) | 321 kbit/s (41%) | 342 kbit/s (44%) |
+------------+------------------+------------------+------------------+

IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.

[Nick.C]

IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.

He did indeed !

[2Bdecided]

Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?

Yes, most modern CDs don't deserve lossless (they're such bad quality) while requiring very high lossless bitrates. LossyWAV tames them with little to no side effects.

I can always find something better to do with the HDD space. A few TBs don't go far when you have video and photos, plus a lot of audio to back up.

Cheers,
David.

[Axon]

To say nothing about vinyl. Gawd! My needledrop FLACs are shrinking by 45% at --standard!

If there ever was a demonstration of the reduced dynamic range of vinyl, lossyWAV has to be it. But clearly, there's a silver lining to it.

Thanks for the new comparisons Synthetic. It's very interesting to see that TAK actually pulls ahead of FLAC by an additional 2% at -q0, although the gain is eliminated at --standard.

[Nick.C]

Out of curiousity, and with all due respect to its developer, is LossyWAV just for use with DAPs? Is there any reason to use it when disk space is not an issue?

Yes, most modern CDs don't deserve lossless (they're such bad quality) while requiring very high lossless bitrates. LossyWAV tames them with little to no side effects.

I can always find something better to do with the HDD space. A few TBs don't go far when you have video and photos, plus a lot of audio to back up.

Cheers,
David.

As David said (and lossyWAV is an implementation of his lossyFLAC proposal) there are little or no side effects - even lossyWAV --insane|FLAC yields a 15% (or more) saving over lossless FLAC.

Also, I am also noticing that newer albums seem to compress less well in FLAC - but compress as well as older material after both have been processed with lossyWAV.

[edit] I use lossyFLAC solely with CorePlayer on my iPAQ h2210 with 32GB CF card and 4GB SD card. My FLAC archive is safe on my server. As an aside, I processed 262 CD images (252h32m38s) from FLAC on my server to lossyWAV -q 1.25|FLAC -5 in 3h05m last night - so transcoding for immediate use is really quite feasible (Intel C2D @ 3.0GHz, 4GB RAM). My iPAQ would play lossless FLAC of course, but my hearing is not good enough to determine the difference and circa 335 kbit/s for lossyWAV -q 1.25|FLAC -5 is palatable from a size viewpoint. I believe that lossyFLAC requires less processor effort when decoding than, say, MP3 - so I should get longer battery life (although I installed a 3600mAh battery just to be sure.... ). [/edit]

This post has been edited by Nick.C: Jul 16 2008, 19:50

[Brent]

I searched the entire 1.1.0 development thread, but I couldn't get the answers I'm looking for: has anyone ever succesfully ABX'ed --portable? Or, if there just is no direct answer to that question, what's the lowest quality setting anyone has ever succesfully ABX'ed? I'm curious how paranoid I should be to use a higher setting (I myself am quite happy with --portable, but I obviously havn't had time to check my entire library for errors, let alone ABX'ing).

[Nick.C]

That's a good question - I believe that Mardel or Sauvauge78 managed to ABX at -q 2 with specific samples using an older beta of lossyWAV, which prompted a bit of tightening of internal preset parameters. However, as --portable == -q 2.5, I think it includes a suitable safety margin over -q 1 - but it's all down to your ears / equipment / listening environment really....

As my sig says, I am happy with -q 1.25 for my purposes (listening to it right now on my PC) - however my lossyFLAC collection is my lossy copy - I keep the lossless FLAC safely tucked away.

halb27 is, I believe, best placed to advise on archive quality lossyFLAC - his setting is --extreme for archiving.

This post has been edited by Nick.C: Jul 16 2008, 21:00

[halb27]

... has anyone ever succesfully ABX'ed --portable? ...

I just finished another listening test as we have a new final release and there were some changes since my last test. Moreover I have some CDs to encode which I didn't want to encode before this final release.

I decided to use the --portable quality as I don't remember having ever tested it since noise shaping is involved with it as a default. I didn't read your post before my test.

I used my usual set of potential problems Atem-lied, harp40_1, bruhns, badvilbel, trumpet, furious, keys_1644ds, triangle-2_1644ds, bibilolo, herding_calls, S37_OTHERS_MartenotWaves_A, dither_noise_test, Fiocco, Under The Boardwalk, utb, 00000_00595ms, Livin_In_The_Future, eig_essence, castanets.

I was not able to abx a problem. If there was a suspicion against transparency for me furious is a candidate for this, but it's beyond my ability to abx it (and I'm not sure if my weak suspicion is unconscious placebo cause using foobar I've seen furious' bitrate is relatively low compared to that of other problem samples).

I also tested a series of full length tracks and everything was fine to me.

To finish it up I encoded my regular music test set using --extreme and listened to the error files. In many - if not most situations - the error is inaudible even with no music to mask the noise. When noise is audible it's very low volume. I also looked at the added noise's spectrum and everything works as expected: noise is in the very high frequency region most of the time and seldom goes down to 10 kHz (judging from Nero wave editor display).

So everything's fine, and even --portable provides excellent results for my ears.

Thank you, Nick, for your great work.


As for your question, Brent: lossyWAV has had a long life as far as the various versions are concerned. With regard to the versions since the introduction of the quality naming scheme like --portable I'm pretty sure nobody abxed --portable. Moreover we can't but talk about nowadays quality of --portable since the default inclusion of noise shaping. That's not a long time ago.

This post has been edited by halb27: Jul 16 2008, 21:18

[Nick.C]

Thank you, Nick, for your great work.

A pleasure, Sir! Said before but worth repeating - lossyWAV would not have happened without David's proposal publication, Matlab script and guidance (& restraint ), our early collaborative efforts and all of your ABX testing (if the output was good enough to require to be ABX'ed ).

This post has been edited by Nick.C: Jul 16 2008, 21:27

[carpman]

IIRC Thomas mentioned something about a specific LossyWAV setting coming for TAK.

He did indeed !

Just out of interest, did he mention an approx. timescale for this?

C.

[Axon]

Now this is cute. Just for grins I tried lossyWAV --standard on a 24/44 needledrop I had. The bitrate dropped from 1504kbps compressed to 506kbps - a 65% reduction in file size? -q1.25 was 403kbps.

That makes a lot of sense though, most notably because my needledrops are not RIAA equalized, so the amplifier noise is largely flat.

[Brent]

... has anyone ever succesfully ABX'ed --portable? ...

As for your question, Brent: lossyWAV has had a long life as far as the various versions are concerned. With regard to the versions since the introduction of the quality naming scheme like --portable I'm pretty sure nobody abxed --portable. Moreover we can't but talk about nowadays quality of --portable since the default inclusion of noise shaping. That's not a long time ago.

As a lowly enduser not dedicated enough to evaluate every -q setting, the reason I enquired about --portable was that I supposed there was some rationale behind it being -q2.5. --standard is q5 and the commandline utility informs me that it is considered to be transparent, but from the looks of it --portable is transparent too. Because I read about a lot of testing in the q0-q2 range, I wondered wat exactly the reason could be to go any higher than --portable, because most reported no succes in ABX-ing in the topend of that range.

So, thank you very much for your results. And, if I may, what exactly is your reason to go with --extreme for your noise test? Are you considering moving your lib to that setting, and you've picked it over lower q settings just to be on the safe side?

[halb27]

.. what exactly is your reason to go with --extreme for your noise test?

For finding out whether --portable is transparent to me I did the listening test.
For testing the machinery not to have a bug I decided to listen to the error noise and look at it's spectrum.
I did this with --extreme on one hand because this is my usual encoding setting (I don't keep a losslesss archive any more, so I want extremely high quality with my lossyWAV archive). On the other hand --extreme is necessary to see most of the noise go to the extreme end of the spectrum. You can also expect only from a very high quality setting to hear only low volume noise or no noise at all when there's no masking by the music. This is totally irrelevant for judging about transparency (that's what ABX listening tests are for), I just wanted to make sure noise gets low volume in an absolute sense with an extremely high quality setting. Just a more formal, technical test whether everything's fine.

This post has been edited by halb27: Jul 17 2008, 10:29