lossyWAV Development, WAV bit reduction by 2BDecided Options

[halb27]

Thank you very much for the new version, especially for improving on the bit-to-remove routine and for implementing the correction file mechanism.
Hopefully some more people might find lossyWav attractive because of this feature.

This post has been edited by halb27: Jan 3 2008, 19:12

[Nick.C]

Thank you very much for the new version.

It's just occurred to me, but the error was in the detection of -ve clipping samples - so conceivably it has adversely influenced recent fine-tuning of the -3 quality preset settings......

[halb27]

... so conceivably it has adversely influenced recent fine-tuning of the -3 quality preset settings......

What is your opinion on that? Do you think we can try again to arrive at a lower average bitrate? Or should we be more cautious?

[Nick.C]

... so conceivably it has adversely influenced recent fine-tuning of the -3 quality preset settings......

What is your opinion on that? Do you think we can try again to arrive at a lower average bitrate? Or should we be more cautious?

I think that with the bug allowing undetected -ve clipping "through the net" and not activating the clipping-prevention mechanism when it should have, some samples will have sounded worse than they should have - therefore we could probably be a bit more aggressive.

[halb27]

I see, clipping was not detected on -ve occasion, and thus the clipping prevention strategy was not triggered.

With my personal kind of thinking this is not a sufficient reason for going less defensive, but I see you'ld like to have average bitrate a bit lower, and maybe that's true for other potential users as well.

Well, there is no strict reason to have the -nts defaults where they are right now. I personally am happy with them, but I'm happy too with other defaults especially as we've always wanted to have -nts as a major option.

I think we should encourage users to use other -nts value than the defaults, and the defaults themselves can be like that though better not at the extreme edges.
With -3 for instance I think an -nts usage from -nts 0 to -nts 10 is okay, with -nts 0 playing it very safe, and -nts 10 allowing for potential but very minor audible deviations from the original.
With -1 any -nts value >=0 makes sense, and everybody can choose the security margin he likes.
For -2 any -nts value <= 10 is useful (though a large positive value may be more adequate together with -3).

Maybe we should try to write something like this in the wiki (shall I do it in case I'm allowed to?).

[Nick.C]

I see, clipping was not detected on -ve occasion, and thus the clipping prevention strategy was not triggered.

With my personal kind of thinking this is not a sufficient reason for going less defensive, but I see you'ld like to have average bitrate a bit lower, and maybe that's true for other potential users as well.

Well, there is no strict reason to have the -nts defaults where they are right now. I personally am happy with them, but I'm happy too with other defaults especially as we've always wanted to have -nts as a major option.

I think we should encourage users to use other -nts value than the defaults, and the defaults themselves can be like that though better not at the extreme edges.
With -3 for instance I think an -nts usage from -nts 0 to -nts 10 is okay, with -nts 0 playing it very safe, and -nts 10 allowing for potential but very minor audible deviations from the original.
With -1 any -nts value >=0 makes sense, and everybody can choose the security margin he likes.
For -2 any -nts value <= 10 is useful (though a large positive value may be more adequate together with -3).

Maybe we should try to write something like this in the wiki (shall I do it in case I'm allowed to?).

I agree with your reasoned response - I think that I was just a bit eager to reduce the bitrate (as usual!).

Feel free to edit the wiki article - you are a major contributor to the project after all....

[halb27]

...Feel free to edit the wiki article - you are a major contributor to the project after all....

Done, but can you please look it up and correct errors as I'm not a native English speaker.

BTW, I've moved the player comparison link to the codecs section, and added a remark on Rockbox supporting FLAC and wavPack. Guess the codecs section is the more appropriate place especially as the preset section got a bit fatter by my contribution.

[Nick.C]

Done, but can you please look it up and correct errors as I'm not a native English speaker.

BTW, I've moved the player comparison link to the codecs section, and added a remark on Rockbox supporting FLAC and wavPack. Guess the codecs section is the more appropriate place especially as the preset section got a bit fatter by my contribution.

Looking good - thanks again for your input and restraining influence....

[Nick.C]

lossyWAV beta v0.6.1 now appended to post #1 of this thread.

"Fixed" the bug but introduced another (although fairly benign....) expect beta v0.6.2 shortly.

lossyWAV beta v0.6.2 now appended to post #1 of this thread. beta v0.6.1 removed due to bug.

This post has been edited by Nick.C: Jan 4 2008, 12:33

[Nick.C]

I was playing about with FLAC settings on my processed 53 sample set and found the following (all at -b 512):

-0; 45415797B; 3.89s
-1; 46004820B; 4.89s
-2; 44305745B; 4.75s
-3; 41716318B; 4.89s
-4; 42151483B; 5.45s
-5; 40646358B; 8.20s
-6; 40637026B; 9.03s
-7; 40497737B; 29.49s
-8; 40305661B; 38.44s

-3 -m -e -r 2; 40894438B; 13.52s

Is it just me, my sample set or my PC - but does -5 seem to be the sweet spot for encoding?

[2Bdecided]

Is that not the FLAC default?

Cheers,
David.

[Nick.C]

Is that not the FLAC default?

Cheers,
David.

Erm, yes..... Should possibly have stopped to consider why the best time (-5) corresponded to the best time / compression point.

[halb27]

I was playing about with FLAC settings on my processed 53 sample set and found the following (all at -b 512):

-0; 45415797B; 3.89s
-1; 46004820B; 4.89s
-2; 44305745B; 4.75s
-3; 41716318B; 4.89s
-4; 42151483B; 5.45s
-5; 40646358B; 8.20s
-6; 40637026B; 9.03s
-7; 40497737B; 29.49s
-8; 40305661B; 38.44s

-3 -m -e -r 2; 40894438B; 13.52s

Is it just me, my sample set or my PC - but does -5 seem to be the sweet spot for encoding?

File sizes are plausible, but encoding time is strange, as the internal effort of -5 should be higher compared to -3 -m -e -r 2, at least that's what I learnt from the documentation in case I've read that correctly.
That was the motivation behind my struggling for a fast and space saving setting, and it is like that with my system.
I'm a bit unhappy with your 53 sample set for such tests - in this case because they consist of small snippets to a pretty large extent. Do you mind encoding a small set of full length tracks with -5 and -3 -m -e -r 2?
Other than that -5 is a good FLAC setting of course.

[2Bdecided]

Nick,

This is looking really good. Do you feel you're close to a release candidate yet? It seems that the code works well enough, and it's just a case of settling the command line options and documentation.

Would you be able to find time to explain the algorithm as it currently stands? Or should I just read the code Obviously I'm most interested in the changes, but other people might want a full overview without reading 29 pages!

Cheers,
David.

[Nick.C]

Nick,

This is looking really good. Do you feel you're close to a release candidate yet? It seems that the code works well enough, and it's just a case of settling the command line options and documentation.

Would you be able to find time to explain the algorithm as it currently stands? Or should I just read the code Obviously I'm most interested in the changes, but other people might want a full overview without reading 29 pages!

Cheers,
David.

Thanks very much David, I appreciate it.

I'm happy with the code - it's fairly robust (now that I've found that bug in the bits-to-remove routine that only caused a crash if I was trying to write a correction file as well...) and the only outstanding item is the reconstitution of the lossy and lwcdf files to recreate the lossless original.

I will try to put the algorithm into engineering speak (my working language) and pass on to you for formalising into audio-processing language.

I should also try to adequately comment the code to allow sense to be made of it by someone other than myself - as it will form part of the release package.

This post has been edited by Nick.C: Jan 4 2008, 22:42

[jesseg]

I had an idea, it might already be this way. I didn't check. But if the correction file was encoded with the bits reversed (16=1/15=2/etc) for whatever bit-depth needed, wouldn't lossless codecs encode that more efficiently? Since it's not really an audio file someone would use directly, that makes sense for starters, and it shouldn't complicate things if lossyWAV-compliant decoding were ever to be built into any lossless decoders. And there should be practically no speed hit at all.


Also, here's a massively improved version of lFLCDrop. Hopefully the only thing I'll have to add later is the support for automatic re-assembling of a lossless WAV when the correction file is present. And if anyone has any suggestions about things to add or change with the custom settings area in the batch file - now is the perfect time to suggest something.

lFLCDrop Change Log:
v1.2.0.4
- added correction file settings
- added custom encoding setting

lFLC.bat Change Log:
v1.0.0.4
- added flac decoding support
- added correction file encoding support
- added custom settings section
- improved temp file handling

a newer version is out, check for a later post

This post has been edited by jesseg: Jan 7 2008, 01:54

[carpman]

Hi all,

very nice to see LossyWav development coming on so quickly!

@jesseg

just ran flacdrop v1.2.0.4 with lossywav v.0.6.0.2

settings were

Left=33
Top=550
Encoder=D:\active_encoders_all\lossywav\iFLC.bat
Switches=-2
Image=
OutputDir=
StayOnTop=1
Minimize=1
DeleteSource=0

but unlike when I was using
flacdrop v1.2.0.2 with lossywav v.0.5.4.4 (the last versions I ran) no FLAC files were created. The DOS window came up and it was clearly processing -- I checked the temp directory and could see the lossy.wav files being created but no FLAC files in the Output directory (set as the same as the input directory).

Any ideas?

It could be me but I'm not doing anything differently than with previous version.

ps. I did a search on my hard drive for [inputfilenames].flac but nothing came up.

C.

[Nick.C]

I had an idea, it might already be this way. I didn't check. But if the correction file was encoded with the bits reversed (16=1/15=2/etc) for whatever bit-depth needed, wouldn't lossless codecs encode that more efficiently? Since it's not really an audio file someone would use directly, that makes sense for starters, and it shouldn't complicate things if lossyWAV-compliant decoding were ever to be built into any lossless decoders. And there should be practically no speed hit at all.

Unfortunately, as not all of the differences are positive, this wouldn't help as the new bit 1 of every negative difference would be 1.

Something David said *ages* ago popped back into my head and I want to get a second opinion with respect to my understanding:

Looking for zero values in the sample data was mentioned.

I took this to mean "look for FFT's where all the input values are zero" and have implemented a checking mechanism as follows:

When filling up the FFT input array, OR each sample value with a "running total" which is initialised as zero before the filling starts.

If the resultant "running total" is zero then the FFT is full of zeros and does not require to be calculated.

For every FFT not calculated, do not take a 0db value when calculating the bits_to_remove for the codec_block in question (as rounded zero's are still zero = no added noise).

If *every* FFT was full of zeros, set bits_to_remove to zero and simply store the codec_block.

[edit] This approach reduces my FLAC'd processed 53 sample set by a whole 95 bytes. However, it may slightly increase processing throughput..... [/edit]

This post has been edited by Nick.C: Jan 6 2008, 23:03

[jesseg]

That makes sense. I guess I understand little about how FLAC, and other lossless compression works.


carpman, check your system-drive for the directory that was created of the same path as the files you were trying to encode. You should find the flac files and can delete them. I made a silly booboo and forgot to parse and use the drive letter from the input & output locations.

lFLC.bat Change Log:
v1.0.0.5
- fixed drive letter bug

see attached

This post has been edited by jesseg: Jan 6 2008, 23:11

[halb27]

... look for FFT's where all the input values are zero ...

The input values to the FFT: isn't that the wave samples of a block?

[carpman]

carpman, check your system-drive for the directory that was created of the same path as the files you were trying to encode. You should find the flac files and can delete them.

There weren't any FLAC files created on my system-drive [I searched all drives for the FLAC files and none came up]

Anyway, no matter -- I'll use the new version now -- thanks for the quick reply.

C.

[Nick.C]

... look for FFT's where all the input values are zero ...

The input values to the FFT: isn't that the wave samples of a block?

Yes, but at 64 samples and 50% end_overlap and 50% fft_overlap, there are 17 FFT analyses performed per 512 sample codec_block. So, using the existing method, if one whole fft is zero then this brings the whole block's bits_to_remove to zero for 64 samples of silence (which round to silence anyway).....

This post has been edited by Nick.C: Jan 7 2008, 00:20

[halb27]

I see. You look at the codec blocks' subblocks formed by the particular FFTs and ignore the FFT result (not doing the FFT) if the subblock contains silence.
Sounds reasonable though I think for a silent passage only the first and last block (containing partial silence) will take profit of this mechanism as the main silence part affects entire blocks.
I'm not quite sure about the window function. Can the windowing bring FFT input to zero though it's not zero in the wave input? I guess it doesn't but I'm not sure.

This post has been edited by halb27: Jan 7 2008, 10:29

[Nick.C]

I see. You look at the codec blocks' subblocks formed by the particular FFTs and ignore the FFT result (not doing the FFT) if the subblock contains silence.
Sounds reasonable though I think for a silent passage only the first and last block (containing partial silence) will take profit of this mechanism as the main silence part affects entire blocks.
I'm not quite sure about the window function. Can the windowing bring FFT input to zero though it's not zero in the wave input? I guess it doesn't but I'm not sure.

The silence check is carried out on raw audio samples.

As the checking mechanism is now in place, I am wondering about "what constitutes digital (near) silence"?

I have put in place a few lines of code which take the absolute value of the sample rather than the actual value when performing the silence check.

In determining whether to process a single FFT the threshold for "silence" can be set to zero (total silence) or some value above zero (near silence).

Determining what constitutes "near silence" is the tricky bit......

[Nick.C]

lossyWAV beta v0.6.3 appended to post #1 of this thread.