lossyWAV 1.1.0 Development Thread., Added noise WAV bit reduction method. Options

[halb27]

Nothing new, just an observation for those who like to use lossyWAV in extremely high quality mode like me:
I tried v1.0.1b -q 7.0 --shaping 0.5 (instead of --shaping 1.0 which I did before).
This yields a bitrate of 503 kbps on average with my regular track set which is only 9 kbps more than when not using --shaping. That's more or less for free, and noise is still so much in the HF region that the most important frequency range of the fundamentals is more or less free of noise, and the overall noise perception when listening to the correction file is very low usually.
So I think this rather simple noise shaping which we have already is very favorable when using high quality settings.
As is known with low quality settings things are different: average bitrate when using -q 1.5 goes up from 312 kbps to 342 kbps when using --shaping 0.5.

This post has been edited by halb27: May 17 2008, 15:28

[Nick.C]

This release specifically made to see of collector's crash issue has been resolved....

Sorry, no changes. It still doesn't run.

I'll get round to tracking the issue this evening - sorry for the delay!

Nothing new, just an observation for those who like to use lossyWAV in extremely high quality mode like me:
I tried v1.0.1b -q 7.0 --shaping 0.5 (instead of --shaping 1.0 which I did before).
This yields a bitrate of 503 kbps on average with my regular track set which is only 9 kbps more than when not using --shaping. That's more or less for free, and noise is still so much in the HF region that the most important frequency range of the fundamentals is more or less free of noise, and the overall noise perception when listening to the correction file is very low usually.
So I think this rather simple noise shaping which we have already is very favorable when using high quality settings.
As is known with low quality settings things are different: average bitrate when using -q 1.5 goes up from 312 kbps to 342 kbps when using --shaping 0.5.

Sounds good - it could even be a standard part of the preset, i.e. quality_noise_shaping_factor : array[0..Quality_Presets] of double = (0,0,0,0.1,0.3,0.5,0.6,0.7,0.8,0.9,1);

This post has been edited by Nick.C: May 17 2008, 15:33

[halb27]

Sounds good - it could even be a standard part of the preset, i.e. quality_noise_shaping_factor : array[0..Quality_Presets] of double = (0,0,0,0.1,0.3,0.5,0.6,0.7,0.8,0.9,1);

I thank you very much for having implemented noiseshaping as I really like it at something like -q 7.0.
I'm not sure however with quality settings that aren't so high whether it's safe to use and which way to use.
One sorrow for instance: with a weak noise shift like 0.1: isn't there a risk that noise level is increased in the area around 6 kHz where we're very sensitive towards noise? Another one: roughly speaking the quality assuring machinery controls SNR in various ways, but isn't the SNR of certain frequency regions made worse by shaping the noise?
With -q 7.0 --shaping 0.5 I feel pretty safe as I think a) --shaping 0.5 shifts noise for the most part pretty much beyond 6 kHz, and b) with -q 7.0 there's a security margin that I expect to cover a certain decrease in HF SNR due to noise shifting.

With this understanding - hope it's correct - I would prefer not to default to current noise shifting other than with high quality settings.
With high quality settings >= 7.0 however it does make sense to me: something like --shaping 0.5 for -q 7.0, --shaping 0.6 for -q 8.0, --shaping 0.7 for -q 9.0, --shaping 0.8 for -q 10.0 (the exact details being a matter of taste).
The current noiseshaping may be favorable also for low bitrate (I listened to -q 1.5 --shaping 0.5 and was very content), but may be it's wise to leave it up to the user and not default to it.

This post has been edited by halb27: May 17 2008, 19:08

[botface]

Sounds good - it could even be a standard part of the preset, i.e. quality_noise_shaping_factor : array[0..Quality_Presets] of double = (0,0,0,0.1,0.3,0.5,0.6,0.7,0.8,0.9,1);

I thank you very much for having implemented noiseshaping as I really like it at something like -q 7.0.
I'm not sure however with quality settings that aren't so high whether it's safe to use and which way to use.
One sorrow for instance: with a weak noise shift like 0.1: isn't there a risk that noise level is increased in the area around 6 kHz where we're very sensitive towards noise? Another one: roughly speaking the quality assuring machinery controls SNR in various ways, but isn't the SNR of certain frequency regions made worse by shaping the noise?
With -q 7.0 --shaping 0.5 I feel pretty safe as I think a) --shaping 0.5 shifts noise for the most part pretty much beyond 6 kHz, and b) with -q 7.0 there's a security margin that I expect to cover a certain decrease in HF SNR due to noise shifting.

With this understanding - hope it's correct - I would prefer not to default to current noise shifting other than with high quality settings.
With high quality settings >= 7.0 however it does make sense to me: something like --shaping 0.5 for -q 7.0, --shaping 0.6 for -q 8.0, --shaping 0.7 for -q 9.0, --shaping 0.8 for -q 10.0 (the exact details being a matter of taste).
The current noiseshaping may be favorable also for low bitrate (I listened to -q 1.5 --shaping 0.5 and was very content), but may be it's wise to leave it up to the user and not default to it.

I have no knowledge of how the noise shaping is done in LossyWAV but if you have any control over it surely it should be possible to ensure that any noise is always shifted well out of harm's way

[Nick.C]

I have no knowledge of how the noise shaping is done in LossyWAV but if you have any control over it surely it should be possible to ensure that any noise is always shifted well out of harm's way

lossyWAV uses SebastianG's noise shaping method for 44.1kHz and 48kHz with thanks.

Speaking about it with SG, using any "factor" applied to the coefficients (factor to the power of (the coefficient index -1)) will work for any value of factor in the range 0.0 to 1.0. In this way I presume that even using 0.1 will tend to move some of the white noise added by the lossyWAV bit reduction method into the high frequency area.

[SebastianG]

[...] using any "factor" applied to the coefficients (factor to the power of (the coefficient index -1)) will work for any value of factor in the range 0.0 to 1.0. In this way I presume that even using 0.1 will tend to move some of the white noise added by the lossyWAV bit reduction method into the high frequency area.

Yes. This is a simple trick you can do with minimum phase filters. The "factor" actually scales the poles and zeros of the filter's transfer function. As they move closer to the origin (factor going from 1.0 down to 0.0) the filter's response becomes more and more flat. Setting this parameter to 0.0 is equivalent to disabling noise shaping.

I'm currently trying to get something fancier to work: Adaptive filters that quickly respond well to what the psychoacoustic model "decides" to be irrelevant.

Cheers,
SG

[Nick.C]

There has been a request for a DLL of lossyWAV. I have no experience of how this may be achieved, let alone for a DSP like lossyWAV.

Any thoughts, hints, tips, standard interfaces, etc would be extremely well received.

Nick.

[Nick.C]

lossyWAV beta 1.0.1d attached to post #1 in this thread.

[collector]

lossyWAV beta 1.0.1d attached to post #1 in this thread.

Up and running again in win98 Thanks Nick.

[Nick.C]

lossyWAV beta 1.0.1d attached to post #1 in this thread.

Up and running again in win98 Thanks Nick.

About time too - I was getting worried about that one....

[Nick.C]

lossyWAV beta 1.0.1f attached to post #1 in this thread.

[edit] problem with file-naming logic when stdin & stdout used in conjunction.... [/edit]

This post has been edited by Nick.C: May 20 2008, 10:02

[collector]

lossyWAV beta 1.0.1f attached to post #1 in this thread.
[edit] problem with file-naming logic when stdin & stdout used in conjunction.... [/edit]

Can't the flossy.bat-command line not end in something like -o"%2 -q3" ? I'm not into Foobar. From the start I rename my files like <musicfile -lq3.wav> since only one file in a testphase can be called musicfile.lossy.wav. That's an indication of lossy but no quality label. And to compare them I need to know the q that was used.
The --stdout works fine though

This post has been edited by collector: May 20 2008, 12:45

[Nick.C]

lossyWAV beta 1.0.1f attached to post #1 in this thread.
[edit] problem with file-naming logic when stdin & stdout used in conjunction.... [/edit]

Can't the flossy.bat-command line not end in something like -o"%2 -q3" ? I'm not into Foobar. From the start I rename my files like <musicfile -lq3.wav> since only one file in a testphase can be called musicfile.lossy.wav. That's an indication of lossy but no quality label. And to compare them I need to know the q that was used.
The --stdout works fine though

Glad to hear that the STDOUT output is working for you.

Why not use:

CODE

lossywav %1 -q %2 %3 %4 %5 %6 %7 %8 %9 --stdout|flac - -5 -b 512 -f -o"%~n1.-q%2.lossy.flac"

It should work in the way you intend.

[shadowking]

...I've attached an example. a..._MS_done.flac is a critical sample for this issue, that you might choose to encode with lossyWAV.

David.

I don't have the right setup to check these files ATM. You may want to check Dualstream or even wavpack.

Ghido said this regarding Optimfrog DS:

"- independent quantization levels for each channel (some TC use
joined channel quantization, reducing spatial sound imaging)"

[Mardel]

Lossywav why cant work with *.wav??? (lossywav *.wav -q2 in command line) %lossyWAV Error% : Input file: *.wav does not exist.

I prefer batch files, but i cant do this and i wont like to convert it one bye one.

This post has been edited by Mardel: May 20 2008, 17:48

[Nick.C]

Lossywav why cant work with *.wav???

I prefer batch files, but i cant do this and i wont like to convert it one bye one.

CODE

for %a in (*.wav) do lossywav "%a" -q 2

would work perfectly well from the command line or

CODE

@for %%a in (*.wav) do lossywav "%%a" -q 2

in a batch file.

This post has been edited by Nick.C: May 20 2008, 17:55

[Mardel]

CODE

@for %%a in (*.wav) do lossywav "%%a" -q 2

in a batch file.

Thx. This work correctly now.

[Nick.C]

CODE

@for %%a in (*.wav) do lossywav "%%a" -q 2

in a batch file.

Thx. This work correctly now.

I would use:

CODE

@for %%a in (*.wav) do lossywav "%a" %1 %2 %3 %4 %5 %6 %7 %8 %9 --stdout|flac - -5 -b 512 -o"%~na.lossy.flac" --tag="LOSSYWAV"="lossyWAV 1.0.1f"

for creating lossyFLAC files quickly....

This post has been edited by Nick.C: May 20 2008, 19:20

[GeSomeone]

Change log 1.0.1d: 18/05/08
Console output has been redirected to 'con', rather than STDOUT.

Apart from loosing the --keep-foreign-metadata, this has the side effect that logging made like
lossyWAV %1 -q 5 >>mylogfile.txt
is no longer there. (My script is (still) based on one from the original dev thread).

Is there an easy way to redirect from CON to file again?

I am having difficulty piping foobar2000 converter output into lossywav - I cannot find any documentation which details the transfer format for foobar2000.

The same here. It should be something like a wav file, although with an incorrect length in the header. Number of bits as specified in foobar2000 encoder setting (for lossless files usually the same as input file).

You could look into code from another codec, that reads stdin and works with foobar, or (after a search ) ask in the foobar2000 development forum next door.

[Nick.C]

Change log 1.0.1d: 18/05/08Console output has been redirected to 'con', rather than STDOUT.

Apart from loosing the --keep-foreign-metadata, this has the side effect that logging made like
lossyWAV %1 -q 5 >>mylogfile.txt
is no longer there. (My script is (still) based on one from the original dev thread).

Is there an easy way to redirect from CON to file again?

I am having difficulty piping foobar2000 converter output into lossywav - I cannot find any documentation which details the transfer format for foobar2000.

The same here. It should be something like a wav file, although with an incorrect length in the header. Number of bits as specified in foobar2000 encoder setting (for lossless files usually the same as input file).

You could look into code from another codec, that reads stdin and works with foobar, or (after a search ) ask in the foobar2000 development forum next door.

I'm going to implement a --writetolog parameter which will append output to lossywav.log in the same directory as the output file (where for stdout, I don't yet know) - this will write pertinent values from the process to this file.

Thanks for the pointer to the foobar2000 forums. I'll have a look there....

[Mardel]

I would use:

CODE

@for %%a in (*.wav) do lossywav "%a" %1 %2 %3 %4 %5 %6 %7 %8 %9 --stdout|flac - -5 -b 512 -o"%~na.lossy.flac" --tag="LOSSYWAV"="lossyWAV 1.0.1f"

for creating lossyFLAC files quickly....

I like tak (-e -fsl 512), cause smaller than flac. I tested.

[Nick.C]

I would use:

CODE

@for %%a in (*.wav) do lossywav "%a" %1 %2 %3 %4 %5 %6 %7 %8 %9 --stdout|flac - -5 -b 512 -o"%~na.lossy.flac" --tag="LOSSYWAV"="lossyWAV 1.0.1f"

for creating lossyFLAC files quickly....

I like tak (-e -fsl 512), cause smaller than flac. I tested.

Whatever you're happier with - I'm just glad it works with more than one codec....

[Josef Pohm]

After a short session, it looks that TAK, FLAC, LPAC and ALS are all compatible with the new channel independent bit depth reduction feature, while WV it's not. When someone else could confirm, and due to the (of course, very well deserved) high popularity of WV, we probably should take care of that. I didn't bother to test WMA so far.

CODE

      1.00,q8 1.01c,q8 Diff.
      6,7r.b. 7,0r.b.

FLAC  40,20   38,55   -1,65
TAK   39,02   37,36   -1,66
WV    40,53   40,57   +0,04
ALS   38,86   37,23   -1,63
LPAC  39,14   37,57   -1,57

[Nick.C]

After a short session, it looks that TAK, FLAC, LPAC and ALS are all compatible with the new channel independent bit depth reduction feature, while WV it's not. When someone else could confirm, and due to the (of course, very well deserved) high popularity of WV, we probably should take care of that. I didn't bother to test WMA so far.

Thanks Josef, that's encouraging (again!). What it does indicate to me is that maybe the --linkchannels parameter is not really required, although it doesn't really add that much to the overall processing time.

I've been thinking that the conversion from reference_threshold array (as calculated by the [unreleased] calculate-white-noise-level-from-rounding) to threshold_index array is too coarse (something I think David alluded to a short time ago). I have "widened" it by a factor of 48 (seems to be a good compromise between memory requirements and additional bits removed) and it has allowed a few kbps to be shaved off the FLAC'ed processed output.

I will post beta 1.0.1g today.

[SebastianG]

It just occured to me that in case of varying "wasted_bits" counts over the channels FLAC's the channel mode "M/S" is one of the two modes out of 4 possible that fail to exploit this. Consider L has 5 zeroed LSBs and R as 4 zeroed LSBs. Then computing S:=L-R and M:=R+S/2 (or L-S/2) results in channels which both have only 4 zeroed LSBs. Actually M has only 3 zeroed LSBs due to the division by two. In this case only "L/R" and "L/S" exploit the 5th zero bit in L.

Since there're no other option besides "-M" to turn adaptive M/S coding on I'm presuming that it also considers "L/S" and "R/S" which is a good idea when used on current lossyWAVs results.

IIRC, WavPack doesn't do M/S but rather interchannel prediction. So, it probably can't exploit different "wasted_bits" counts over the channels in general.

Cheers,
SG

This post has been edited by SebastianG: May 21 2008, 13:03