lossyWAV Development, WAV bit reduction by 2BDecided Options

[halb27]

... I would prefer to keep -skew constant.

No problem for -1 for me. Though -skew 40 brought a little progress for differentiating good and bad, it was not very significant. Same goes for -snr 21/24/27. It doesn't really matter.
So we can use -skew 36 throughout all the quality levels, and maybe -snr 24 or 27 for -1 for the good of a certain systematics.

This post has been edited by halb27: Nov 22 2007, 10:11

[Nick.C]

... I would prefer to keep -skew constant.

No problem for -1 for me. Though -skew 40 brought a little progress for differentiating good and bad, it was not very significant. Same goes for -snr 21/24/27. It doesn't really matter.
So we can use -skew 36 throughout all the quality levels, and maybe -snr 24 or 27 for -1 for the good of a certain systematics.

Taking into account what you were saying and after some iteration:

-2 quality settings: -fft 10101 -snr 21 -skew 36 -nts 1.5 -spf 22224-22235-22336-12347-12358; Yields 42.22MB / 476.3kbps.

[halb27]

-2 quality settings: -fft 10101 -snr 21 -skew 36 -nts 1.5 -spf 22224-22235-22336-12347-12358; Yields 42.22MB / 476.3kbps.

I will try that out this evening to see what it means for regular/problem tracks.

-snr 21 -skew 36 -nts 1.5 is fine to me.

Not using 1s for the short blocks also perfectly matches my ideas.

Using a spreading of 22224 on the 64 sample FFT instead of 22235 as of my proposal will bring average bitrate up for regular music, but it is also more defensive and in a sense more appropriate for the rather crude frequency resolution of a 64 sample FFT even at higher frequencies. I did consider such a thing too but gave it away for efficiency. Just a matter of taste. For taking care of the restricted resolution of the 64 sample FFT an alternative is to use a 64 and 128 sample FFT with a spreading of 22235 for FFT64 and 22236 for FFT128. This doesn't push up average bitrate significantly but costs another analysis.
I don't care very much about these details but would prefer the 64 and 128 sample FFT solution because efficiency isn't seriously affected.

-fft 10101 looks nice being so symmetric, but what is the idea behind the 256 sample FFT?
I prefer a 512 sample FFT as an alternative addition to the 1024 sample FFT which reaches out pretty much into the neighboring blocks, and because of this might push up the decisive min values due to energy from the neighboring blocks. I don't know whether this can really be a problem, but I don't feel very secure using only a 1024 sample FFT.

A spreading of 12358 for the 1024 sample FFT looks a bit demanding to me for the high frequency regions especially as we do take good care of HF with the short FFT(s) with your proposal and with my 2 short FFTs alternative. But maybe it doesn't matter much as the demanding values for the short FFTs push up bitrate for the high frequencies already. In case we agree on my proposal for the two short FFTs my feeling is not to care much for the HF with the long FFTs, not to be very demanding here, and leave this up to the short FFTs.

Let's see what the statistics says for regular/problem tracks.

This post has been edited by halb27: Nov 22 2007, 13:03

[Nick.C]

I will try that out this evening to see what it means for regular/problem tracks.

-snr 21 -skew 36 -nts 1.5 is fine to me.

Not using 1s for the short blocks also perfectly matches my ideas.

Using a spreading of 22224 on the 64 sample FFT instead of 22235 as of my proposal will bring average bitrate up for regular music, but it is also more defensive and in a sense more appropriate for the rather crude frequency resolution of a 64 sample FFT even at higher frequencies. I did consider such a thing too but gave it away for efficiency. Just a matter of taste. For taking care of the restricted resolution of the 64 sample FFT an alternative is to use a 64 and 128 sample FFT with a spreading of 22235 for FFT64 and 22236 for FFT128. This doesn't push up average bitrate significantly but costs another analysis.
I don't care very much about these details but would prefer the 64 and 128 sample FFT solution because efficiency isn't seriously affected.

-fft 10101 looks nice being so symmetric, but what is the idea behind the 256 sample FFT?
I prefer a 512 sample FFT as an alternative addition to the 1024 sample FFT which reaches out pretty much into the neighboring blocks, and because of this might push up the decisive min values due to energy from the neighboring blocks. I don't know whether this can really be a problem, but I don't feel very secure using only a 1024 sample FFT.

A spreading of 12358 for the 1024 sample FFT looks a bit demanding to me for the high frequency regions especially as we do take good care of HF with the short FFT(s) with your proposal and with my 2 short FFTs alternative. But maybe it doesn't matter much as the demanding values for the short FFTs push up bitrate for the high frequencies already. In case we agree on my proposal for the two short FFTs my feeling is not to care much for the HF with the long FFTs, not to be very demanding here, and leave this up to the short FFTs.

Let's see what the statistics says for regular/problem tracks.

I tried to merge your idea for short fft's and my previous -2. 12358 at 1024 samples doesn't make a huge difference (if any) to my sample set.

As an aside, I finished my 1496 track processing at -3: 16.8GB / 348kbps output from 40.8GB / 859kbps input

[M]

This may have already been addressed somewhere in the previous twenty-two pages (I have not read them all!)... but has anyone done a listening comparison to high-bitrate MP3 or AAC, so average users will have a plain-English frame of reference?

- M.

[halb27]

I tried to merge your idea for short fft's and my previous -2. 12358 at 1024 samples doesn't make a huge difference (if any) to my sample set.

Yor sample set is made up of problem tracks to a large extend. This is welcome information, but it is good to match it against the average bitrate of a regular track set. I use a 12 sample full regular track set for this purpose.

As an aside, I finished my 1496 track processing at -3: 16.8GB / 348kbps output from 40.8GB / 859kbps input

So pretty much the same as the 343 kbps average of my 12 sample set.

This post has been edited by halb27: Nov 22 2007, 13:45

[halb27]

This may have already been addressed somewhere in the previous twenty-two pages (I have not read them all!)... but has anyone done a listening comparison to high-bitrate MP3 or AAC, so average users will have a plain-English frame of reference?

- M.

The usual thing: Everything is fine for high quality aac and mp3 users under nearly all circumstances.
For mp3 it's easy of course to find samples that show the superiority of lossyWav (try eig as a sample).

Roughly speaking lossyWav can be attractive to people who like lossless codecs but don't like file sizes ~2/3 that of the corresponding wav files which is immanent to losslessly encoding musical genres similar to rock music.

Using -1 it's for instance a disc space saving alternative to lossless archiving for people with a huge musical collection.
Using -3 it's for instance a high quality method for usage on a FLAC or wavPack enabled DAP.
Using -2 for instance allows for a universally usable compromise.

All acording to everybody's likings.

Sure usefulness only applies if the quality is really extremely good. Unfortunately we don't get a lot of listening feedback. Anyway all we know so far is that we can be very content with lossyWav's quality.

[Nick.C]

This may have already been addressed somewhere in the previous twenty-two pages (I have not read them all!)... but has anyone done a listening comparison to high-bitrate MP3 or AAC, so average users will have a plain-English frame of reference?

- M.

My ears aren't good enough...... Also, I don't have access to online storage (at all) so, I couldn't host one. If anyone else wishes to host a listening test, I would be only too happy to assist in the preparatory work.

Nick.

[Nick.C]

Yor sample set is made up of problem tracks to a large extend. This is welcome information, but it is good to match it against the average bitrate of a regular track set. I use a 12 sample full regular track set for this purpose.

I'm currently processing a selection of my 1496 track set.

Oh, one thing: when I refer to lossyFLAC I am referring to lossyWAV preprocessed WAV encoded to FLAC -8. Similarly, lossyTAK, lossyWAVPACK (as opposed to WAVPACK lossy ), etc.

Artist - Album / FLAC / lossyFLAC -2 / lossyFLAC-3;

CODE

AC/DC - Dirty Deeds Done Dirt Cheap    / 781kbps / 398kbps / 331kbps
B52's - Good Stuff                     / 993kbps / 408kbps / 361kbps
David Byrne - Uh-Oh                    / 937kbps / 398kbps / 344kbps
Fish - Songs From The Mirror           / 854kbps / 384kbps / 335kbps
Gerry Rafferty - City To City          / 802kbps / 400kbps / 338kbps
Iron Maiden - Can I Play With Madness  / 784kbps / 422kbps / 370kbps
Jean Michel Jarre - Oxygene            / 773kbps / 454kbps / 372kbps
Marillion - The Thieving Magpie        / 790kbps / 404kbps / 344kbps
Mike Oldfield - Tr3s Lunas             / 848kbps / 421kbps / 364kbps
Scorpions - Best Of Rockers N' Ballads / 922kbps / 421kbps / 353kbps

So, overall an average of 850kbps / 410kbps / 350kbps

This post has been edited by Nick.C: Dec 5 2007, 12:18

[halb27]

So, overall an average of 850kbps / 410kbps / 350kbps

Is the 410 kbps result for your new -2 proposal or for the current default setting?
With the -2 default I got 430 kbps (with v0.4.7) and my sample set is expected to yield a slightly lower bitrate than your set as can be seen for -3. Or was there a change for the -2 default since v0.4.7?

This post has been edited by halb27: Nov 22 2007, 15:19

[halb27]

Your -2 setting of -fft 10101 -snr 21 -skew 36 -nts 1.5 -spf 22224-22235-22336-12347-12358
yields 404/539 kbps on average with my regular/problem set.
As it is a bit more conservative than my setting I like this result and agree with this setting.

[Nick.C]

Your -2 setting of -fft 10101 -snr 21 -skew 36 -nts 1.5 -spf 22224-22235-22336-12347-12358
yields 404/539 kbps on average with my regular/problem set.
As it is a bit more conservative than my setting I like this result and agree with this setting.

That's great. With that in mind, I've been playing with Josef Pohm's excel morphing..... See attached. Assigning the "corners" creates some quite reasonable numbers. Attached.

[halb27]

Finally I managed to do my listening test for current -3 with my problem sample set as well as utb.flac I once had problems with.
Everything is fine, and I will use this setting for my DAP.

This post has been edited by halb27: Nov 22 2007, 19:50

[Nick.C]

Finally I managed to do my listening test for current -3 with my problem sample set as well as utb.flac I once had problems with.
Everything is fine, and I will use this setting for my DAP.

Great, -2 & -3 settings now fixed (subject to usual caveat that if anyone hears any artifacts, please let us know).

lossyWAV alpha v0.5.0 attached: Superseded.

CODE

lossyWAV alpha v0.5.0 : WAV file bit depth reduction method by 2Bdecided.
Delphi implementation by Nick.C from a Matlab script, www.hydrogenaudio.org

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-1            extreme quality [5xFFT] (-cbs 512 -nts -3.0 -skew 36 -snr 21
              -spf 12224-12225-12225-11226-11236 -fft 11111)
-2            default quality [4xFFT] (-cbs 512 -nts +1.5 -skew 36 -snr 21
              -spf 22224-22235-22346-12347-12358 -fft 10101)
-3            compact quality [2xFFT] (-cbs 512 -nts +6.0 -skew 36 -snr 21
              -spf 22235-22236-22347-22358-2246C -fft 10001)

-o <folder>   destination folder for the output file
-nts <n>      set noise_threshold_shift to n dB (-18.0dB<=n<=+6.0dB)
              (-ve values reduce bits to remove, +ve values increase)
-force        forcibly over-write output file if it exists; default=off

Codec Options:

-wmalsl       optimise internal settings for WMA Lossless codec; default=off

Advanced / System Options:

-snr <n>      set minimum average signal to added noise ratio to n dB;
              (0.0dB<=n<=48.0dB) Increasing value reduces bits to remove.
-skew <n>     skew fft analysis results by n dB (0.0db<=n<=48.0db) in the
              frequency range 20Hz to 3.45kHz
-cbs <n>      set codec block size to n samples (512<=n<=4608, n mod 32=0)
-fft <5xbin>  select fft lengths to use in analysis, using binary switching,
              from 64, 128, 256, 512 & 1024 samples, e.g. 01001 = 128,1024
-overlap      enable conservative fft overlap method; default=off

-spf <5x5hex> manually input the 5 spreading functions as 5 x 5 characters;
              These correspond to FFTs of 64, 128, 256, 512 & 1024 samples;
              e.g. 22235-22236-22347-22358-2246C (Characters must be one of
              1 to 9 and A to F (zero excluded).
-allowable    select allowable number of clipping samples per codec block
              before iterative clipping reduction; (0<=n<=64, default=0).

-window       select windowing function n (0<=n<=6, default=0); 0=Hanning
              1=Bartlett-Hann; 2=Blackman; 3=Nuttall; 4=Blackman-Harris;
              5=Blackman-Nuttall; 6=Flat-Top.
-clipping     disable clipping prevention by iteration; default=off
-dither       dither output using triangular dither; default=off

-quiet        significantly reduce screen output
-nowarn       suppress lossyWAV warnings
-detail       enable detailled output mode

-below        set process priority to below normal.
-low          set process priority to low.

Special thanks:

Dr. Jean Debord for the use of TPMAT036 uFFT & uTypes units for FFT analysis.
Halb27 @ www.hydrogenaudio.org for donation and maintenance of the wavIO unit.

Once -1 settings are fixed, then I'll remove excess options and we're probably ready to go beta.

This post has been edited by Nick.C: Nov 26 2007, 11:42

[GeSomeone]

Hi, regarding my remark some pages back about noise when encoding silence with lossyFlac. I updated my post as I think I know what must have happened. It had nothing to do with lossyWav.

dithering from the foobar2000 converter must have been to blame. Even though it was set to "only dither lossy sources" it seemed to have kicked in somewhere. A new test with setting to "Never Dither" (and latest lossyWav) was as expected. No more noise.

I liked to clear that up.

the result (that doesn't matter IMO):
silence.flac bit rate 3
silence.lossy.flac bit rate 12

This post has been edited by GeSomeone: Nov 23 2007, 17:35

[Nick.C]

Hi, regarding my remark some pages back about noise when encoding silence with lossyFlac. I updated my post as I think I know what must have happened. It had nothing to do with lossyWav.

dithering from the foobar2000 converter must have been to blame. Even though it was set to "only dither lossy sources" it seemed to have kicked in somewhere. A new test with setting to "Never Dither" (and latest lossyWav) was as expected. No more noise.

I liked to clear that up.

the result (that doesn't matter IMO):
silence.flac bit rate 3
silence.lossy.flac bit rate 12

Thanks for the clarification - I think that it just reinforces the opinion that dither should not be used, as David said some time ago (along with amplitude reduction, although that is now not required due to the iterative clipping prevention).

[edit] Oh, and I think that the four-fold increase in bitrate is due to FLAC encoding at -b 512 rather than default of -b 4096 (I think). [/edit]

This post has been edited by Nick.C: Nov 23 2007, 21:39

[halb27]

For -1:

My proposal some posts ago was

-1 -skew 40 -snr 21 -fft 11011 -spf 22224-22225-11235-11246-12358 -nts -1
which yields 452/576 kbps for my regular/problematic set.

Now that we want to have -skew 36 -snr 21 throughout the quality levels I suggest we trade -skew 40/36 for
-nts -1/-2.

So I suggest we use

-1 -skew 36 -snr 21 -fft 11011 -spf 22224-22225-11235-11246-12358 -nts -2

This post has been edited by halb27: Nov 25 2007, 10:06

[Nick.C]

For -1:

My proposal some posts ago was

-1 -skew 40 -snr 21 -fft 11011 -spf 22224-22225-11235-11246-12358 -nts -1
which yields 452/576 kbps for my regular/problematic set.

Now that we want to have -skew 36 -snr 21 throughout the quality levels I suggest we trade -skew 40/36 for
-nts -1/-2.

So I suggest we use

-1 -skew 36 -snr 21 -fft 11011 -spf 22224-22225-11235-11246-12358 -nts -2

Done, -1 quality settings fixed. Will post as alpha (beta?) v0.5.1 tonight.

[halb27]

IMO it's alright giving beta state to lossyWAV.

[Nick.C]

lossyWAV beta v0.5.1 attached. Superseded.

-1 quality settings : -skew 36 -snr 21 -fft 11011 -spf 22224-22225-11235-11246-12358 -nts -2;[size=1]

CODE

lossyWAV beta v0.5.1 : WAV file bit depth reduction method by 2Bdecided.
Delphi implementation by Nick.C from a Matlab script, www.hydrogenaudio.org

Usage   : lossyWAV <input wav file> <options>

Example : lossyWAV musicfile.wav

Quality Options:

-1            extreme quality [4xFFT] (-cbs 512 -nts -2.0 -skew 36 -snr 21
              -spf 22224-22225-11235-11246-12358 -fft 11011)
-2            default quality [3xFFT] (-cbs 512 -nts +1.5 -skew 36 -snr 21
              -spf 22224-22235-22346-12347-12358 -fft 10101)
-3            compact quality [2xFFT] (-cbs 512 -nts +6.0 -skew 36 -snr 21
              -spf 22235-22236-22347-22358-2246C -fft 10001)

-o <folder>   destination folder for the output file
-nts <n>      set noise_threshold_shift to n dB (-18.0dB<=n<=+6.0dB)
              (-ve values reduce bits to remove, +ve values increase)
-force        forcibly over-write output file if it exists; default=off

Codec Options:

-wmalsl       optimise internal settings for WMA Lossless codec; default=off

Advanced / System Options:

-snr <n>      set minimum average signal to added noise ratio to n dB;
              (0.0dB<=n<=48.0dB) Increasing value reduces bits to remove.
-skew <n>     skew fft analysis results by n dB (0.0db<=n<=48.0db) in the
              frequency range 20Hz to 3.45kHz
-cbs <n>      set codec block size to n samples (512<=n<=4608, n mod 32=0)
-fft <5xbin>  select fft lengths to use in analysis, using binary switching,
              from 64, 128, 256, 512 & 1024 samples, e.g. 01001 = 128,1024
-overlap      enable conservative fft overlap method; default=off

-spf <5x5hex> manually input the 5 spreading functions as 5 x 5 characters;
              These correspond to FFTs of 64, 128, 256, 512 & 1024 samples;
              e.g. 22235-22236-22347-22358-2246C (Characters must be one of
              1 to 9 and A to F (zero excluded).
-allowable    select allowable number of clipping samples per codec block
              before iterative clipping reduction; (0<=n<=64, default=0).

-window       select windowing function n (0<=n<=6, default=0); 0=Hanning
              1=Bartlett-Hann; 2=Blackman; 3=Nuttall; 4=Blackman-Harris;
              5=Blackman-Nuttall; 6=Flat-Top.
-clipping     disable clipping prevention by iteration; default=off
-dither       dither output using triangular dither; default=off

-quiet        significantly reduce screen output
-nowarn       suppress lossyWAV warnings
-detail       enable detailled output mode

-below        set process priority to below normal.
-low          set process priority to low.

Special thanks:

David Robinson for the method itself and motivation to implement it in Delphi.
Dr. Jean Debord for the use of TPMAT036 uFFT & uTypes units for FFT analysis.
Halb27 @ www.hydrogenaudio.org for donation and maintenance of the wavIO unit.

[edit] For Foobar2000 converter settings, see wiki article. [/edit]

This post has been edited by Nick.C: Dec 4 2007, 09:47

[halb27]

Thank you, Nick.

[GeSomeone]

IMO it's alright giving beta state to lossyWAV.

The program is stable enough I would think too, but a little "How to use blurb" should be added when you go Beta, (maybe the example foobar script + screenshot too). The info is in this thread, but you can't expect a test user to go through all that.

[Nick.C]

IMO it's alright giving beta state to lossyWAV.

The program is stable enough I would think too, but a little "How to use blurb" should be added when you go Beta, (maybe the example foobar script + screenshot too). The info is in this thread, but you can't expect a test user to go through all that.

Yes, that would be a sensible idea. A guide and a licence document have to be the next items on the agenda.

[jesseg]

new lFLCDrop version...

lFLC.bat Change Log:
v1.0.0.1
- added line to delete all wavs from temp directory before encoding

this should get rid of problems caused by closing the batch window in the middle of encoding.

It still works fine with lossyWAV as of v0.5.1.22 Beta.


re: the batch file for EAC, what I'm probably going to end up doing is creating an app to handle it. No gui or commandline is really needed, other than for a critical error message I guess, and most people have EAC setup to run the encoders hidden anyways. This will allow me to do some more fancy stuff than a batch allows for, and it should be able to handle unicode in the tagging (assuming EAC and tag.exe can handle it, I haven't had time to look into it yet)

[edit] removed, newer version posted later in the thread [/edit]

This post has been edited by jesseg: Nov 29 2007, 21:46

[Axon]

Forgive me for asking a fundamental (and admittedly critical) question; I'm very late to this particular party. Before I start, I must say this idea (and all the work that has gone into it) is incredible, and I would not hesitate to use it once the kinks are ironed out. From the original post by 2BDecided:

This isn't about psychoacoustics. What you can or can't hear doesn't come into it. Instead, you perform a spectrum analysis of the signal, note what the lowest spectrum level is, and throw away everything below it. (If this seems a little harsh, you can throw in an offset to this calculation, e.g. -6dB to make it more careful, or +6dB to make it more aggressive!).

I don't see a use for a quasi-lossy bitrate reduction of a lossless format, if the reduction is known to produce artifacts in reasonable configurations. If people are able to ABX this in a wide variety of different modes, that doesn't give me much confidence in using lossyWAV at all, no matter what the settings. If I can deal with a probabilistic chance of artifact audibility, why not stay with lossy?

This doesn't seem like the sort of algorithm that lends itself to tuning. If the technique is independent of psychoacoustics, then the only advanced setting that ought to exist is -skew.

Is that too harsh? Perhaps I'm being overly critical on beta code?

This post has been edited by Axon: Nov 27 2007, 07:56