Thank you very much for the new version, especially for improving on the bit-to-remove routine and for implementing the correction file mechanism.
Hopefully some more people might find lossyWav attractive because of this feature.

It's just occurred to me, but the error was in the detection of -ve clipping samples - so conceivably it has adversely influenced recent fine-tuning of the -3 quality preset settings......

What is your opinion on that? Do you think we can try again to arrive at a lower average bitrate? Or should we be more cautious?

I think that with the bug allowing undetected -ve clipping "through the net" and not activating the clipping-prevention mechanism when it should have, some samples will have sounded worse than they should have - therefore we could probably be a bit more aggressive.

I see, clipping was not detected on -ve occasion, and thus the clipping prevention strategy was not triggered.

With my personal kind of thinking this is not a sufficient reason for going less defensive, but I see you'ld like to have average bitrate a bit lower, and maybe that's true for other potential users as well.

Well, there is no strict reason to have the -nts defaults where they are right now. I personally am happy with them, but I'm happy too with other defaults especially as we've always wanted to have -nts as a major option.

I think we should encourage users to use other -nts value than the defaults, and the defaults themselves can be like that though better not at the extreme edges.
With -3 for instance I think an -nts usage from -nts 0 to -nts 10 is okay, with -nts 0 playing it very safe, and -nts 10 allowing for potential but very minor audible deviations from the original.
With -1 any -nts value >=0 makes sense, and everybody can choose the security margin he likes.
For -2 any -nts value <= 10 is useful (though a large positive value may be more adequate together with -3).

Maybe we should try to write something like this in the wiki (shall I do it in case I'm allowed to?).

I agree with your reasoned response - I think that I was just a bit eager to reduce the bitrate (as usual!).

Feel free to edit the wiki article - you are a major contributor to the project after all....

Done, but can you please look it up and correct errors as I'm not a native English speaker.

BTW, I've moved the player comparison link to the codecs section, and added a remark on Rockbox supporting FLAC and wavPack. Guess the codecs section is the more appropriate place especially as the preset section got a bit fatter by my contribution.

Looking good - thanks again for your input and restraining influence....

"Fixed" the bug but introduced another (although fairly benign....) expect beta v0.6.2 shortly.

lossyWAV beta v0.6.2 now appended to post #1 of this thread. beta v0.6.1 removed due to bug.

I was playing about with FLAC settings on my processed 53 sample set and found the following (all at -b 512):

-0; 45415797B; 3.89s
-1; 46004820B; 4.89s
-2; 44305745B; 4.75s
-3; 41716318B; 4.89s
-4; 42151483B; 5.45s
-5; 40646358B; 8.20s
-6; 40637026B; 9.03s
-7; 40497737B; 29.49s
-8; 40305661B; 38.44s

-3 -m -e -r 2; 40894438B; 13.52s

Is it just me, my sample set or my PC - but does -5 seem to be the sweet spot for encoding?

Is that not the FLAC default?

Cheers,
David.

Erm, yes..... Should possibly have stopped to consider why the best time (-5) corresponded to the best time / compression point.

File sizes are plausible, but encoding time is strange, as the internal effort of -5 should be higher compared to -3 -m -e -r 2, at least that's what I learnt from the documentation in case I've read that correctly.
That was the motivation behind my struggling for a fast and space saving setting, and it is like that with my system.
I'm a bit unhappy with your 53 sample set for such tests - in this case because they consist of small snippets to a pretty large extent. Do you mind encoding a small set of full length tracks with -5 and -3 -m -e -r 2?
Other than that -5 is a good FLAC setting of course.

Nick,

This is looking really good. Do you feel you're close to a release candidate yet? It seems that the code works well enough, and it's just a case of settling the command line options and documentation.

Would you be able to find time to explain the algorithm as it currently stands? Or should I just read the code Obviously I'm most interested in the changes, but other people might want a full overview without reading 29 pages!

Cheers,
David.

Thanks very much David, I appreciate it.

I'm happy with the code - it's fairly robust (now that I've found that bug in the bits-to-remove routine that only caused a crash if I was trying to write a correction file as well...) and the only outstanding item is the reconstitution of the lossy and lwcdf files to recreate the lossless original.

I will try to put the algorithm into engineering speak (my working language) and pass on to you for formalising into audio-processing language.

I should also try to adequately comment the code to allow sense to be made of it by someone other than myself - as it will form part of the release package.

I had an idea, it might already be this way. I didn't check. But if the correction file was encoded with the bits reversed (16=1/15=2/etc) for whatever bit-depth needed, wouldn't lossless codecs encode that more efficiently? Since it's not really an audio file someone would use directly, that makes sense for starters, and it shouldn't complicate things if lossyWAV-compliant decoding were ever to be built into any lossless decoders. And there should be practically no speed hit at all.


Also, here's a massively improved version of lFLCDrop. Hopefully the only thing I'll have to add later is the support for automatic re-assembling of a lossless WAV when the correction file is present. And if anyone has any suggestions about things to add or change with the custom settings area in the batch file - now is the perfect time to suggest something.



a newer version is out, check for a later post

Hi all,

very nice to see LossyWav development coming on so quickly!

@jesseg

just ran flacdrop v1.2.0.4 with lossywav v.0.6.0.2

settings were

Left=33
Top=550
Encoder=D:\active_encoders_all\lossywav\iFLC.bat
Switches=-2
Image=
OutputDir=
StayOnTop=1
Minimize=1
DeleteSource=0

but unlike when I was using
flacdrop v1.2.0.2 with lossywav v.0.5.4.4 (the last versions I ran) no FLAC files were created. The DOS window came up and it was clearly processing -- I checked the temp directory and could see the lossy.wav files being created but no FLAC files in the Output directory (set as the same as the input directory).

Any ideas?

It could be me but I'm not doing anything differently than with previous version.

ps. I did a search on my hard drive for [inputfilenames].flac but nothing came up.

C.

Unfortunately, as not all of the differences are positive, this wouldn't help as the new bit 1 of every negative difference would be 1.

Something David said *ages* ago popped back into my head and I want to get a second opinion with respect to my understanding:

Looking for zero values in the sample data was mentioned.

I took this to mean "look for FFT's where all the input values are zero" and have implemented a checking mechanism as follows:

When filling up the FFT input array, OR each sample value with a "running total" which is initialised as zero before the filling starts.

If the resultant "running total" is zero then the FFT is full of zeros and does not require to be calculated.

For every FFT not calculated, do not take a 0db value when calculating the bits_to_remove for the codec_block in question (as rounded zero's are still zero = no added noise).

If *every* FFT was full of zeros, set bits_to_remove to zero and simply store the codec_block.

[edit] This approach reduces my FLAC'd processed 53 sample set by a whole 95 bytes. However, it may slightly increase processing throughput..... [/edit]

That makes sense. I guess I understand little about how FLAC, and other lossless compression works.


carpman, check your system-drive for the directory that was created of the same path as the files you were trying to encode. You should find the flac files and can delete them. I made a silly booboo and forgot to parse and use the drive letter from the input & output locations.



see attached

The input values to the FFT: isn't that the wave samples of a block?

There weren't any FLAC files created on my system-drive [I searched all drives for the FLAC files and none came up]

Anyway, no matter -- I'll use the new version now -- thanks for the quick reply.

C.

Yes, but at 64 samples and 50% end_overlap and 50% fft_overlap, there are 17 FFT analyses performed per 512 sample codec_block. So, using the existing method, if one whole fft is zero then this brings the whole block's bits_to_remove to zero for 64 samples of silence (which round to silence anyway).....

I see. You look at the codec blocks' subblocks formed by the particular FFTs and ignore the FFT result (not doing the FFT) if the subblock contains silence.
Sounds reasonable though I think for a silent passage only the first and last block (containing partial silence) will take profit of this mechanism as the main silence part affects entire blocks.
I'm not quite sure about the window function. Can the windowing bring FFT input to zero though it's not zero in the wave input? I guess it doesn't but I'm not sure.

The silence check is carried out on raw audio samples.

As the checking mechanism is now in place, I am wondering about "what constitutes digital (near) silence"?

I have put in place a few lines of code which take the absolute value of the sample rather than the actual value when performing the silence check.

In determining whether to process a single FFT the threshold for "silence" can be set to zero (total silence) or some value above zero (near silence).

Determining what constitutes "near silence" is the tricky bit......

lossyWAV beta v0.6.3 appended to post #1 of this thread.

Why would it change the output at all. However you round (or not) zeros, they're still zeros. (Where's the "confused" smiley!).

It should be quicker though.

Be very careful if you intend to convert near-silence into digital silence. If you do it, please only for the "lower quality" mode -3. Near-silence is quite easy to encode losslessly anyway, and you could hit all kinds of problems for little benefit.

Cheers,
David.

I am not changing the samples themselves, merely disregarding FFT results which *are* going to be zero by not even calculating them - therefore not including a known 0db result in the minimum_of_all_fft_results calculation when determining the final bits_to_remove.

At the moment the -detection parameter is optional and I would intend to keep it that way.

I've thought this over again, and I think the -detection mechanism as is does already affect a near-silence situation: in a temporal sense, not a sense of amplitude. If we look at a codec block with partial silence some short-term FFT results can remain unconsidered. This does lower the accuracy compared to not using -detection as proved with the 53 sample set. And it's not clear whether this is a welcome thing in every situation (think of a strong transient starting or stopping within a block with silence just before or after the transient).

In the end I also see a bad ratio of benefits against risks, even when considering the current just temporal-near-silence detection.

ADDED: I think our short blocksize of 512 samples (~12 msec) is enough to take care of temporal silence.

Fair enough - I'll park the thought and continue to clean up the code with a view to going RC1 later this week.

Right, to achieve concensus on which parameters should be included in lossyWAV RC1, can I have your thoughts on the following:

Keep:
-1, -2, -3;
-o <folder>
-nts <n>
-snr <n>
-force
-check
-correction
-wmalsl
-quiet
-nowarn
-below
-low

Remove:
-skew <n>
-spf <5x5hex>
-fft <5xbin>
-cbs <n>
-detail

A good selection IMO. I'd just prefer to remove -snr as well and keep only -nts as a quality affecting parameter apart from -1/-2/-3.

About -wmalsl: We wanted to have options to optimize the lossyWAV precedure for specific lossless codecs. So far we have just this switch. Do we really need it? IIRC the switch only addresses codec block size, but isn't the codec blocksize a multiple of 512? I don't see a problem with respect to quality and efficiency with having lossyWAV blocksize = 512 and codec blocksize of the lossless codec a multiple of 512.

lossyWAV v0.6.4 RC1 appended to post #1 in this thread.

At your suggestion I ditched -wmalsl - 2048 is a multiple of 512 after all as you point out. I kept -snr as it has been determined to be an intrinsic element in quality retention during the testing phase.

I found a bug when there was certain characters in directory names or file names, that weren't handled write even inside of quotes, which are now handled properly. Heck, it should even work with unicode now, but I wouldn't bet on lFLCDrop front-end handling it properly.

The batch file has been updated to lossyWAV v0.6.4 RC1 standards. It should still be compatible with older versions of lossyWAV, but the custom settings area doesn't specify the block-size anymore. FLAC blocksize can still be set in the "enc_cust_flacoptions_string" variable at the top, and it's 512 samples by default.

Regarding the file extension handling - If a WAV file has the lossyWAV chuck and already has a ".lossy.wav" extension, the FLAC file's extension will not have another added ".lossy" tacked onto it. However, if a WAV file has the lossyWAV chuck and does not already have a ".lossy.wav" extension, then the FLAC file's extension will have the ".lossy" tacked onto it. If anyone thinks this is an annoying option, I could certainly provide an option in the batch file to turn off the renaming from WAV files that already have lossyWAV chunks, and I suppose I could also provide the option turn off the FLAC encoding of those files all together. Let me know if this would be useful to you.

And finally... the batch file is now over 6 KB

[edit] link removed, newer version later in the thread [/edit]

If you would mention in the documentation (or help file if there is one) that for WMA lossless -cbs 2048 is recommended, we can forget about WMA lossless

But, as 2048 is exactly 4 codec_blocks, is there really a need to retain the -cbs parameter (removed at RC1, along with -wmalsl)?

I don't really understand what you mean. Do you like to have the possibility of using a lossyWAV blocksize of 2048 for the sake of WMA lossless? Or do you want to force a block size of 2048 on the WMA lossless side?

jesseg,

i ran "lFLCDrop.v1.2.0.4. , lFLC.bat.v1.0.0.6 with "lossyWAV_v0.6.4_RC1"

i got the same problem as last time (see: http://www.hydrogenaudio.org/forums/index....129&st=716)

DOS window comes up and does the lossy.wav processing and creates the lossy.wav temp file but then the DOS window closes and no FLAC processing is initiated and the lossy.wav file is deleted.

I'm running WinXP SP2
The FLAC Drop and Lossy Wav programs are running from my 2nd HD (D:) and the Input / Output directory is on my Primary (System Drive) Drive (C:).

Again I've searched for *.FLAC (which would obviously include *Lossy.FLAC) and nothing has been created on either drive and this mirrors what is shown in the DOS process window.

Didn't have this problem with flacdrop v1.2.0.2 & lossywav v.0.5.4.4.

Any ideas?

C.

Are you using the latest version of FLAC? (v1.2.1B) You have to use that or else it will close instantly because older versions of FLAC don't support the foreign metadata feature.

Thanks!
Problem solved. I was using 1.2.0 (i think).

C.

Reading the FLAC wikipedia article, I notice that FLAC will encode any integer WAV between 4 and 32 bits.

When I try to output 32bit from Foobar, I get 32bit Float rather than integer, so I can't test lossyWAV properly, but the internals allow 16, 24 & 32bit integer WAV files to be processed.

I will amend the WAV reading / writing routines to properly allow for 4<=bits<=32 integer values to be scaled properly (as internally, lossyWAV only uses 32bit integers for sample storage, 64bit floats for most calculations).

Thanks for your hard work. I have not had particular interest in this quality/bitrate range, but I decided to try the release candidate.

I think I have stumbled on a problem sample on my very first try.

I browsed my recent albums and selected a track that is a bit difficult for usual lossy encoders. It is Livin' In The Future from Bruce Springsteen's latest album. This track would score well in the "highest lossless bitrate thread". FLAC 1.21 -8 produces 1133 kbps for the complete track and my 30 s. sample is 1162 kbps.

I used the "-3" lossyWAV compression option and my settings were exactly as instructed on the wiki page.

At first I noticed that something may be different, but didn't know what to look for. However, after some trials I understood what I heard and was able to ABX it:



I wonder if anyone else is able to hear the difference. (I can give hints later if needed.)

I uploaded a 30 s. sample here:
http://rs274.rapidshare.com/files/82598575...The_Future.flac (4.15 MB)
(I didn't have enough attachment space at HA.)

Thanks for testing and providing a problematic sample. I'll try it tonight.
Do you mind trying if -2 solves the issue?

Sorry, I didn't pay enough attention to the fact that -cbs would also be ditched. In that case my remark makes no sense. Nevermind

Thanks for the sample Alex B, it's people like you who allow us to refine and improve the quality presets. I've downloaded it and as you say, it's high bitrate - for both FLAC and lossyFLAC -3/-5 (1162.0kbps vs 544.1kbps).

My ears / listening environment have not allowed me to find the problem area(s?) of the sample.

Out of interest, what was your listening environment when you identified the issue?

[edit2] Additionally, maybe instead of trying -2, could you try -3 -nts 2? This may be enough to address the as yet broadly unidentified problem..... [/edit2]

[edit3] This seems to be a sample which activates the anti-clipping mechanism regularly: I tried
lossywav -3 and got 8.7384 bits removed, 0.2577 not removed;
lossywav -3 -nts 0 and got 8.6029 bits removed, 0.2492 not removed;
lossywav -3 -nts -3 and got 8.3549 bits removed, 0.2337 not removed;

lossywav -3 -snr 24 and got 8.3618 bits removed, 0.2295 not removed;
lossywav -3 -snr 27 and got 7.8982 bits removed, 0.1974 not removed;

lossywav -3 -nts -3 -snr 27 and got 7.8069 bits removed, 0.1950 not removed.[/edit3]

[edit1]Don't worry about it - it was a fairly brutal reduction in settings ![/edit1]

I can't hear the difference. Can you give a hint please?
My lossyFLAC -3 bitrate is 417 kbps (filesize = 1528 KB) BTW.

I made a mistake, my bitrate is 418.4kbps for the lossy.flac version. I still can't hear it - however, it prompted me to re-examine the process_codec_block routine and I've managed to speed the processing up by about 50%.

One thing just occured to me - at present only 2 [edit2] 1024 sample FFT[/edit2] analyses are carried out on a 512 sample codec_block -512:511 and 0:1023. This gives 50% overlap over the length of the file. What it doesn't do is carry out an fft analysis with the middle of the codec_block as the middle of the fft. I can easily add in the extra analysis - with no speed penalty [edit2] compared to v0.6.4 [/edit2] due to the vast speedup I tripped over.

Also, maybe the -spf for the 1024 sample FFT should be 22469 rather than 2246C (assuming we have a potential high frequency problem.....)?

Nick, I think we should learn about the problem before trying to fix it.

Hmm... it may be difficult to hear the problem without any hints.

The first occurance is immediately after the first snare hit when the drum is still sounding. It is like the tuning was adjusted slightly. The drums have slightly different pitch after the sharp hit of the drum stick. I ABXed the 0-1 s range. I think I could hear the same later, but it is more difficult because the other instruments and singer's voice are partially masking the effect.

There is also at least one short passage where I could hear a similar effect in the singer's voice, but I need to recheck the exact position. I'll try to find it again and report back.

I used Terrratec DMX 6fire 24/96 & Koss PortaPro in the ABX test, but before that I compared the complete tracks using small powered Genelec studio monitors and became suspicous.

It may well be that -2 makes the problem vanish totally. It wasn't easy to ABX it even though I had a feeling that something is different.

I am not sure if can do further ABXing just now. I've had an exhausting day...

Alex B, thanks for the clarification of the problem.

So relax.
Maybe tomorrow (or whenever) it would be nice if you could test -2.
I can't contribute cause even with your hint I can't abx it. With versions very much earlier however I also made the experience with specific samples that pitch was changed somehow.
So with your excellent hearing you can give a valuable contribution to lossyWAV improvement.

What's not totally correct with the current setting of -3 is that we have decreased the noise sensitivity threshold a bit. We've thought we can allow for that because we have other precautions which however are less effective in the high frequency range.
So this is the first thing to consider.
With -2 we aren't this little bit aggressive, so your ABX result using -2 is very much welcome.
In case -2 is alright it would be nice if you could try -3 -nts 0 as well, as this keeps this little aggressive mode away from -3 as well.
If however even -2 isn't totally satisfying it would be very much appreciated if you could try -2 -nts 3 as this make the noise sensitivity more defensive.