halb27
Sep 24 2007, 13:41
QUOTE(2Bdecided @ Sep 24 2007, 11:34)
.. For my personal use, I would disable the lossyFLAC gain adjustment entirely. Instead, I'd run a ReplayGain album analysis, and apply only the negative ones, before using lossyFLAC. ...
How do you do that exactly:
- ReplayGain using foobar as a 1 step procedure with encoding?
- 16 or 24 bit? dither or not dither?
- How do you make sure only negative replaygaining is applied? Manual control?
Especially the answer to the 16/24 bit dither/not dither question is relevant to me as the answer applies to resampling as well.
Nick.C
Sep 24 2007, 13:49
Attached v0.2.2. Superseded.
halb27
Sep 24 2007, 15:43
Hi Nick,
Thanks for your hard work.
I think it's good to have paramters of the mechanism adjustable in this pre-release state.
However it's hard as at least I have no idea about what some of the parameters are really doing.
For getting the behavior of 0.2.0 -s: is a setting of -skew 9.0 sufficient in order to get exactly the same behavior?
How exactly is the spreading function varying when using -vsfl? From history I guess the variation is with fft length. But how exactly, and what should we expect from it?
bryant
Sep 24 2007, 20:48
QUOTE(Nick.C @ Sep 24 2007, 12:11)
I found a bug in the -v parameter - it was picking the wrong spreading_function_length. I will post v0.2.2 tonight. For the moment, and by special request : v0.2.0 for Bryant.
Thanks Nick, I got it.
And it looks like I wasn't the only one who wanted it!
David
BGonz808
Sep 24 2007, 20:56
Just out of curiosity, will bit-reduction cause ALAC compress any better. I just bought an iPod
and the idea of lossy flac is great but now I'm stuck without my trusty FLAC format if I want lossless music (and rockbox doesn't really appeal to me that much).
Thanks,
Bobby
Nick.C
Sep 24 2007, 23:49
QUOTE(halb27 @ Sep 24 2007, 22:43)
For getting the behavior of 0.2.0 -s: is a setting of -skew 9.0 sufficient in order to get exactly the same behavior?
How exactly is the spreading function varying when using -vsfl? From history I guess the variation is with fft length. But how exactly, and what should we expect from it?
A skew of 9.0 is the same as v0.2.0;
In previous versions, spreading function length was the same for each fft_length, regardless of the length of the fft, i.e. 4 for 64 and 4 for 1024 - taking into account the bin frequency widths, see bins.txt, this seemed to be a bit unbalanced, so if -vsfl is selected the spreading function length vs fft_length is as follows: 2/64; 3/128; 4/256; 5/512; 6/1024. If it is felt that more than 4-bin averaging is excessive, this can be changed at the next revision.
halb27
Sep 25 2007, 00:14
For a promising way to continue testing (and keeping in mind that v0.2.0 -s yielded excellent results) it is important to know what we're testing.
Can you please confirm or correct the following statements:
a) just to make sure the basis:
v0.2.2 -skew 9.0 (no other options) yields exactly the same results as v0.2.0 -s (no other options).
Especially the v0.2.2 noise threshold default is exactly equal to that of v0.2.0?
b) to try out the fft_length dependent bin averaging the following options are useful to test
v0.2.2 -vsfl -skew x -nts y
with x<=9.0 (for instance x=6.0) and y>=-3.0 (for instance y=-1.0).
And as 2Bdecided said -nfc should be used for abxing to make sure no loudness difference is abxed.
What exactly does the weighted spreading function option -wsf do?
Nick.C
Sep 25 2007, 00:55
QUOTE(halb27 @ Sep 25 2007, 07:14)
For a promising way to continue testing (and keeping in mind that v0.2.0 -s yielded excellent results) it is important to know what we're testing.
Can you please confirm or correct the following statements:
a) just to make sure the basis:
v0.2.2 -skew 9.0 (no other options) yields exactly the same results as v0.2.0 -s (no other options).
Especially the v0.2.2 noise threshold default is exactly equal to that of v0.2.0?
b) to try out the fft_length dependent bin averaging the following options are useful to test
v0.2.2 -vsfl -skew x -nts y
with x<=9.0 (for instance x=6.0) and y>=-3.0 (for instance y=-1.0).
And as 2Bdecided said -nfc should be used for abxing to make sure no loudness difference is abxed.
What exactly does the weighted spreading function option -wsf do?
a) Should be almost exactly the same (although the skew in v0.2.0 was 9.0309, i.e. 1.5 x 20 x log(2)).
b) Sounds good.
-nfc is alright - unless the sample clips under bit-reduction. There is no clipping prevention at all when -nfc is used.
-wsf creates spreading functions as follows: [1]; [2/3,1/3]; [3/6,2/6,1/6]; [4/10,3/10,2/10,1/10]; [5/15,4/15,3/15,2/15,1/15]; [6/21,5/21,4/21,3/21,2/21,1/21];
rather than [1]; [1/2,1/2]; [1/3,1/3,1/3]; [1/4,1/4,1/4,1/4] etc. The weighted spreading function tends to 75% below midlength, 25% above midlength as length tends to infinity. I am developing with a variant of this which tends to 7/12 below midlength, 5/12 above midlength.
@2Bdecided: I noticed on running the Matlab script with the same parameters several times in a row on the same input file that the average bits_to_remove value changes....? I can't pin down the cause, does it do that for you?
2Bdecided
Sep 25 2007, 03:10
QUOTE(halb27 @ Sep 24 2007, 20:41)
QUOTE(2Bdecided @ Sep 24 2007, 11:34)
.. For my personal use, I would disable the lossyFLAC gain adjustment entirely. Instead, I'd run a ReplayGain album analysis, and apply only the negative ones, before using lossyFLAC. ...
How do you do that exactly
Hypothetically! Though I've tested it manually.
QUOTE
- ReplayGain using foobar as a 1 step procedure with encoding?
- 16 or 24 bit? dither or not dither?
- How do you make sure only negative replaygaining is applied? Manual control?
Especially the answer to the 16/24 bit dither/not dither question is relevant to me as the answer applies to resampling as well.
It depends on your use. If you're going to gain the files before lossyFLAC, the output should ideally be 24-bit no dither, but adding dither make little difference (less efficient on digital silence!). Nick's portable won't play 24-bit, so 16-bit no dither. I know you _should_ dither, but it reduces efficiency and adds hiss. Without it, you can in theory introduce distortion. Pick your poison.
Whether you should enabled dither
within lossyFLAC is a different question. I have an artificial sample where it's required to avoid quite nasty noise pumping artefacts, but for efficiency I'm normally testing with no dither. The only place I think I've heard a difference is annoyinglyloudsong, but there it's not artefacting - it sounds louder to me without dither, that's all. I should ABX it because I'm probably talking rubbish.
Cheers,
David.
halb27
Sep 25 2007, 04:26
QUOTE(2Bdecided @ Sep 25 2007, 11:10)
... Whether you should enabled dither within lossyFLAC is a different question. I have an artificial sample where it's required to avoid quite nasty noise pumping artefacts, but for efficiency I'm normally testing with no dither. ...
Can you give this artificial sample please?
So you say dithering within lossyWav isn't necessarily the way to go.
Nick, can you provide a switch please to disable dithering? Or in your opinion is there a strong reason for dithering?
Nick.C
Sep 25 2007, 04:38
QUOTE(halb27 @ Sep 25 2007, 11:26)
Nick, can you provide a switch please to disable dithering? Or in your opinion is there a strong reason for dithering?
Don't ask me difficult questions! I'm just the programmer!!!
I had already decided to re-implement the dither_choice option as a -dither parameter (0=none, 1=rectangular, 2=triangular).
Also, I feel that opinion regarding clipping_reduction indicates that the default option (0) should be none, with 1 = fixed reduction taking into account dither amplitude (if any) and 2 = my 2-pass conditional (but consistent across the file) clipping reduction.
2Bdecided
Sep 25 2007, 07:51
QUOTE(halb27 @ Sep 25 2007, 11:26)
QUOTE(2Bdecided @ Sep 25 2007, 11:10)
... Whether you should enabled dither within lossyFLAC is a different question. I have an artificial sample where it's required to avoid quite nasty noise pumping artefacts, but for efficiency I'm normally testing with no dither. ...
Can you give this artificial sample please?
Attached
QUOTE
So you say dithering within lossyWav isn't necessarily the way to go.
Probably not. You might doubt this when you hear this sample though! Remember I know exactly how lossyFLAC works, and therefore I know exactly how to break it. This sample sounds like it's just white noise, but it isn't, as you'll see if you look at the waveform (and more precisely, the sliding paired sample values) in a wave editor.
There's still an issue about rounding/truncating/clipping/dithering. They're all tied together. What's in lossyFLAC6 works well enough, but I think it could be tweaked slightly. It's not a priority.
Cheers,
David.
2Bdecided
Sep 25 2007, 08:01
QUOTE(Nick.C @ Sep 25 2007, 07:55)
@2Bdecided: I noticed on running the Matlab script with the same parameters several times in a row on the same input file that the average bits_to_remove value changes....? I can't pin down the cause, does it do that for you?
Are you re-generating the noise threshold reference table each time? If so, yes. If not, no.
With dither off and a fixed noise threshold table, the output should be identical every time. It's a deterministic process: a computer program where you aren't changing anything!
With dither on, the output will be different every time, but the bits removed should be identical. The FLAC bitrate may vary due to the dither.
I'm still running lossyFLAC6. It's not changed much since I uploaded it on 4th July, but I'll upload it again anyway. (Attached).
Please don't be disappointed it doesn't have any of your improvements. It doesn't have any of my improvements either!
Cheers,
David.
halb27
Sep 25 2007, 14:56
Using v0.2.2 I tried Atem-lied:
a) -skew 9.0 -nfc : I abxed it 9/10, so I wonder if this really is the same procedure as with v0.2.0 -s.
b) -skew 6.0 -nfc -vsfl : sounds okay to me.
I missed a bit the debug mode I got used to.
Nick.C
Sep 25 2007, 15:08
QUOTE(halb27 @ Sep 25 2007, 21:56)
Using v0.2.2 I tried Atem-lied:
a) -skew 9.0 -nfc : I abxed it 9/10, so I wonder if this really is the same procedure as with v0.2.0 -s.
b) -skew 6.0 -nfc -vsfl : sounds okay to me.
I missed a bit the debug mode I got used to.
-debug is now -detail.
I'm re-writing the part of the code which actually does the analyses. I think that there was a difference in the way that David and I determined the bounds of each individual fft analysis - I'm working to correct that now.
The only thing to make sure of at a) would be to add -nts -3.0103 to the command line to see if that makes any difference.
Sorry about the inconsistency.
[edit] I've rewritten the analysis code and the output very closely matches that of the Matlab script (thanks David for the latest version!) I've got a bit more checking to do, but I think that v0.3.0 will be released tomorrow night. [/edit]
Nick.C
Sep 26 2007, 06:48
Attached summary spreadsheet of my 50 sample set, processed using soon to be released lossyWAV alpha v0.3.0 (-3 -dither 0 -clipping 0 -nts 0), against Matlab LossyFLACv6_revised script with same settings, 1024 sample codec_block_size. Matlab script used 5000 iterations to calculate reference_threshold values.
As can be seen, although not identical, 161 extra bits removed over 29799 codec blocks is not too bad a comparison. It can (and hopefully will) be improved upon. Superseded.
halb27
Sep 26 2007, 08:42
QUOTE(Nick.C @ Sep 26 2007, 14:48)
.. As can be seen, although not identical, 161 extra bits removed over 29799 codec blocks is not too bad a comparison. It can (and hopefully will) be improved upon. ...
Very promising indeed.
In theory however chance is the sum of the removed bits is more or less the same but bit removal is different at different spots.
Or is it a block by block comparison adding the deviations per block?
Nick.C
Sep 26 2007, 09:33
QUOTE(halb27 @ Sep 26 2007, 15:42)
QUOTE(Nick.C @ Sep 26 2007, 14:48)
.. As can be seen, although not identical, 161 extra bits removed over 29799 codec blocks is not too bad a comparison. It can (and hopefully will) be improved upon. ...
Very promising indeed.
In theory however chance is the sum of the removed bits is more or less the same but bit removal is different at different spots.
Or is it a block by block comparison adding the deviations per block?
I knew someone would ask that question - the number quoted is for overall change, I will get going with block by block differences (+ve and -ve) and post:
644 bits extra removed, 483 less bits removed, 161 extra bits removed overall - amended results attached.
Will use the same reference_threshold creation technique in Matlab and re-process the Matlab results - to see if there is something more fundamental wrong rather than just noise_analysis results. Superseded.
halb27
Sep 26 2007, 11:15
Thanks for your work.
So this 1127 BTR difference gives quite a different picture than the just 161 BTR summed difference.
From your remarks it looks like the BTR mechanism of the current Delphi version is different from that of the MATLAB version. I think it would be good before starting tweaking to have exactly the same mechanism and result of the Delphi version as is offered by the original version. If the versions do differ it makes things worse when starting to do changes with the MATLAB version.
Nick.C
Sep 26 2007, 11:28
QUOTE(halb27 @ Sep 26 2007, 18:15)
Thanks for your work.
So this 1127 BTR difference gives quite a different picture than the just 161 BTR summed difference.
From your remarks it looks like the BTR mechanism of the current Delphi version is different from that of the MATLAB version. I think it would be good before starting tweaking to have exactly the same mechanism and result of the Delphi version as is offered by the original version. If the versions do differ it makes things worse when starting to do changes with the MATLAB version.
The starting point for the investigation has to be to remove the random element, namely the calculation of the reference_threshold values used to determine the threshold_index arrays (one for each fft analysis). The Matlab version initially used 1000 iterations x (64, 1024, 256) sample fft lengths, i.e. a max of 1024000 sample x iteration. The delphi version uses constants based on a constant 2^25 count, i.e.32MB sample x iteration, i.e. 1048576 iterations at 32 sample fft; 524288 at 64; ....; 32768 at 1024 sample fft.
halb27
Sep 26 2007, 13:53
What about the predefined values?
Or values stored in a file (IIRC the MATLAB script can make use of that)?
There's no need for perfection for these values in the first step - all that counts now is an identical basis for the MATLAB and DELPHI version as you said.
Nick.C
Sep 26 2007, 14:05
QUOTE(halb27 @ Sep 26 2007, 20:53)
What about the predefined values?
Or values stored in a file (IIRC the MATLAB script can make use of that)?
There's no need for perfection for these values in the first step - all that counts now is an identical basis for the MATLAB and DELPHI version as you said.
I have plugged the pre-calculated Delphi values into Matlab and the results from the Matlab script do not seem to have changed - although I can confirm that the reference_threshold values are identical.
I am in the process of a side-by-side block-by-block, sub-block-by-sub-block comparison of the fft analysis results - thankfully keys_1644ds.wav is quite short!
I've identified the error, if not immediately the solution - from the second codec block onwards, the result of the fft analysis of the first sub-block (and only the first sub-block), for each fft_length, for each channel, differs between Matlab and lossyWAV. All the rest are giving identical results to the Matlab script.
Oh well, back to debugging.......
And, I think that I've found the problem..... The audio data is not consistent between Matlab and lossyWAV - I looked at the fft outputs, then at the window_function'ed inputs, then at the bare audio data - there appear to be some discrepencies between the raw audio data, +/- 1 that I have found so far.
@David: I changed over from wavread/write to my previous wavreadraw/writeraw, removed the multiplication / division of inaudio > inaudio_int > outaudio and removed the inaudible addition, in favour of a 20*log10(max(1,min(conv......))).
This has improved the results somewhat, but it's too late to do the comparison and post it.
Nick.C
Sep 27 2007, 00:54
Okay, so having narrowed down the difference to the second codec block onwards, first sub-block analysis only, and bearing in mind that the end-overlap is fft_length/2 and fft_overlap is fft_length/2 - the answer struck me (early) this morning.....
The Matlab script is removing the bits block by block just after the codec-block is analysed, and before the next codec-block is analysed - thus contaminating the audio data in the pre-block-start overlap of the next analysis block.
This hasn't yet been tested, but it seems *too* likely.
Now that I am happy with the Delphi code, please find attached lossyWAV alpha v0.3.0 Superseded.
@David:
In the script (replicated in the Delphi code) when the minimum min_bin value is calculated for an fft analysis the result is *rounded*, i.e. can be increased by up to 0.5 when looking up the threshold_index table to determine the bits_to_remove. Would it not be better to *floor* this value as it would reduce the likelihood of increasing noise above the minimum determined value?
Modified version of your LossyFLAC6_revised attached including the pre-calculated constants to re-create the reference_threshold values. Superseded.
halb27
Sep 27 2007, 01:44
QUOTE(Nick.C @ Sep 27 2007, 08:54)
... The Matlab script is removing the bits block by block just after the codec-block is analysed, and before the next codec-block is analysed - thus contaminating the audio data in the pre-block-start overlap of the next analysis block.
This hasn't yet been tested, but it seems *too* likely. ...
Great work, Nick. So with respect to this your Delphi code is supposed to be better than the MATLAB script.
So it's worth doing intensive listening tests now.
Unfortunately (in this respect) I'm leaving for holidays tomorrow (will be back on Oct 7) and have to prepare a lot for it this evening (most of all find a B&B for the first nights which turned out to be a problem - Lake District, Cumbria, seems to be very popular these days [guess not only these days]).
Anyway I will try at least to do a short test this evening.
But it is most welcome if more members could contribute in testing.
2Bdecided
Sep 27 2007, 04:11
QUOTE(Nick.C @ Sep 27 2007, 07:54)
The Matlab script is removing the bits block by block just after the codec-block is analysed, and before the next codec-block is analysed - thus contaminating the audio data in the pre-block-start overlap of the next analysis block.
Nick, you're a genius - that's exactly what's happening.
The two parts used to be separate (analysis loop then rounding loop) - when I put them together I didn't spot this. Interesting that the effect was so little ("644 bits extra removed, 483 less bits removed, 161 extra bits removed overall over 29799 codec blocks") – it shows how benign the added noise is. It bodes well for this being multi-generation proof and transcode-proof with a noise threshold shift of -6 or -12 dB.
QUOTE
In the script (replicated in the Delphi code) when the minimum min_bin value is calculated for an fft analysis the result is *rounded*, i.e. can be increased by up to 0.5 when looking up the threshold_index table to determine the bits_to_remove. Would it not be better to *floor* this value as it would reduce the likelihood of increasing noise above the minimum determined value?
Yes, probably. If you're going to do that, you should also move the threshold shift down from where it is, into that calculation, otherwise the additional accuracy is pretty meaningless.
QUOTE
Modified version of your LossyFLAC6_revised attached including the pre-calculated constants to re-create the reference_threshold values.
Thank you. Just for clarity: this includes those thresholds, but hasn't fixed the "contamination" bug?
Cheers,
David.
Nick.C
Sep 27 2007, 04:22
QUOTE(2Bdecided @ Sep 27 2007, 11:11)
If you're going to do that, you should also move the threshold shift down from where it is, into that calculation, otherwise the additional accuracy is pretty meaningless.
I will do that for the next rev.
QUOTE(2Bdecided @ Sep 27 2007, 11:11)
Thank you. Just for clarity: this includes those thresholds, but hasn't fixed the "contamination" bug?
Yes - I didn't change the bit-removal procedure.
Nick.C
Sep 27 2007, 05:58
Comparison of "round" method vs "floor" method used when determining bits_to_remove from min_bin values. Noise_threshold_shift moved into this calculation rather than in threshold_index calculation.
Attached comparison for the 50 sample set. Removed.
Rounding or flooring is selected at present using a command line parameter -floor, however, it may be prudent to remove this option but always use floor.
[edit]I feel that due to the small change in bits_to_remove, and for less likelihood of added noise over the min_bin value, that the code should be changed to always floor this result. v0.3.1 will include this.[/edit]
@David, as an aside, I mentioned earlier about randomness in the bits_to_remove in the Matlab script - this would happen if the samples were dithered when bits were removed.
2Bdecided
Sep 27 2007, 06:24
QUOTE(Nick.C @ Sep 27 2007, 12:58)
@David, as an aside, I mentioned earlier about randomness in the bits_to_remove in the Matlab script - this would happen if the samples were dithered when bits were removed.
Yes, it would. Where's the smiley for "hangs head in shame"!
Cheers,
David.
Nick.C
Sep 27 2007, 06:28
QUOTE(2Bdecided @ Sep 27 2007, 13:24)
Where's the smiley for "hangs head in shame"!
Nope, no need - yours was a minor oversight. Floor implemented with noise_threshold_shift moved. v0.3.1 tonight, I think.
Nick.C
Sep 27 2007, 13:12
Revised LossyFLAC6_x.m - no pollution of audio data when removing bits, floor rather than round used in determining min_bin.
lossyWAV alpha v0.3.1 attached.
CODE
lossyWAV alpha v0.3.1 : WAV file bit depth reduction by 2Bdecided.
Transcoded by Nick.C & Halb27 from a script, www.hydrogenaudio.org
lossyWav usage: <input wav file> <options>
Options:
-1, -2 or -3 quality level (1:overkill, 2:default, 3:compact)
-cbs <n> analysis codec_block_size (512<=n<=4608, default=576 bytes)
(should match codec block size used in target compression codec)
-o <folder> destination folder for the output file
-dither <n> dither selection, 0<=n<=2, default=0. 0=no dither; 1=rectangular
dither; 2=triangular dither.
-clipping <n> clipping prevention selection, 0<=n<=1, default=0. 0=none;
1=fixed clipping prevention amplitude reduction, taking into
account dither amplitude (if any).
-vsfl select variable spreading function lengths
(sfl = number of fft bins averaged during convolution of fft
results. short fft = short sfl, long fft = long sfl)
-wsf select weighted spreading functions.
(weighted average of fft bins during convolution of fft results
weighted towards lower frequency fft bins, 5/8:3/8)
-nts <n> noise_threshold_shift=n (-15.0<=n<=0.0, default -1.5dB)
(reduces overall bits to remove)
-overlap <n> fft_overlap = fft_length/n (2<=n<=8, default=2)
(increases number of fft analyses per codec block)
-skew <n> skew results of fft analyses by n dB (0.0<=n<=12.0, default=0.0)
with a (sin-1) shaping over the frequency range 20Hz to 3.7kHz.
(artificially increase low frequency bins to take into account
higher SNR requirements at low frequencies)
-quiet significantly reduce screen output
-nowarn suppress lossyWAV warnings
-detail enable detailled output mode
Options not yet implemented:
-bitdepth <n> forced output bitdepth (16 or 24)
-flac optimizations for use with FLAC
-wv optimizations for use with wavPack
-tak optimizations for use with TAK
Default noise_threshold_shift magnitude reduced (-3.0db to -1.5db) at the same time as the change from round to floor.
Output now *identical*(!) in terms of bits removed per block.
[edit]Default clipping prevention bug report acted upon - v0.3.1 removed, v0.3.1b below.[/edit]
halb27
Sep 27 2007, 15:10
Nick, you're the king, incredibly fast in bringing out the new versions with all the current items included, and wonderful quality judging so far:
I did a quick test with v0.3.1 using -cbs 1024 -skew 9.0 -cliipping 0 (an explicit setting of -clipping 0 was necessary - it seems like 0 isn't the default):
Atem-lied: Could not abx, which is very remarkable as more bits are removed than with say v0.2.0 - quality improvement due to not dithering?
bibilolo was fine to me as well.
Even with 2Bdecided's dither_noise_test.wav there was no audible issue to me, and this is a very artificial sample, and many bits are removed.
Nick.C
Sep 27 2007, 15:20
QUOTE(halb27 @ Sep 27 2007, 22:10)
I did a quick test with v0.3.1 using -cbs 1024 -skew 9.0 -cliipping 0 (an explicit setting of -clipping 0 was necessary - it seems like 0 isn't the default):
Atem-lied: Could not abx, which is very remarkable as more bits are removed than with say v0.2.0 - quality improvement due to not dithering?
bibilolo was fine to me as well.
Even with 2Bdecided's dither_noise_test.wav there was no audible issue to me, and this is a very artificial sample, and many bits are removed.
-clipping 0 *should * be a default setting, I'll investigate.
[edit] Ahem...... mea culpa! Corrected in alpha v0.3.1b[/edit] Superseded by v0.3.2
My gut feeling is that the move from round to floor has made quite a difference where previously bits_to_remove would have been 1 more than the new version (all else being equal) for certain codec_blocks in the sample.
I've been listening to my 51 sample set (added a Black Sabbath extract from Iron Man
) at -3 -nts 0 -dither 2 -clipping 1 -skew 9 -wsf -vsfl and I'm *really* pleased at the result - so much so that this may become my DAP conversion setting.
Nick.C
Sep 28 2007, 01:22
QUOTE(BGonz808 @ Sep 25 2007, 03:56)
Just out of curiosity, will bit-reduction cause ALAC compress any better. I just bought an iPod
and the idea of lossy flac is great but now I'm stuck without my trusty FLAC format if I want lossless music (and rockbox doesn't really appeal to me that much).
Thanks,
Bobby
Sorry for the late reply Bobby - I think in the original thread there's some discussion about the applicability of the method to various codecs and from memory only really FLAC, Wavpack & Tak will benefit. The other way of looking at is is to just give it a try and see if the compressed file is indeed smaller!
Regards,
Nick.
(or, you could Rockbox your iPod for FLAC support!)
Nick.C
Sep 28 2007, 05:25
Performance results for v0.3.1b:
Guruboolez's 150 sample set:
WAV: 252.36MB; 1411.2kbps;
FLAC: 122.17MB; 683kbps;
lossy -1: 95.7MB; 535kbps;
lossy -2: 89.6MB; 501kbps;
lossy -3: 88.1MB; 492kbps.
My "working" 52 sample set:
WAV: 121.53MB; 1411.2kbps;
FLAC: 68.2MB; 792kbps;
lossy -1: 42.6MB; 495kbps;
lossy -2: 39.5MB; 458kbps;
lossy -3: 38.6MB; 449kbps.
I am currently processing a number of albums and will revert with "real-world" results:
AC/DC - Who Made Who :
WAV: 385.3MB; 1411.2kbps;
FLAC: 238.9MB; 875kbps;
lossy -3: 112.2MB; 411kbps;
Gerry Rafferty - City To City:
WAV: 541.8MB; 1411.2kbps;
FLAC: 307.9MB; 802kbps;
lossy -3: 161.2MB; 420kbps;
Jean Michel Jarre - Oxygene:
WAV: 400.7MB; 1411.2kbps;
FLAC: 219.5MB; 773kbps;
lossy -3: 134.8MB; 475kbps;
The Shamen - Boss Drum:
WAV: 663.3MB; 1411.2kbps;
FLAC: 433.4MB; 922kbps;
lossy -3: 170.0MB; 362kbps;
Van Morrison - Astral Weeks:
WAV: 477.0MB; 1411.2kbps;
FLAC: 255.9MB; 757kbps;
lossy -3: 142.6MB; 422kbps.
I carried out a multi-generational experiment and was pleasantly surprised to see that the output matches the input after surprisingly few generations. This was with no dither or clipping prevention. On clipping prevention, up to now there has been no check made to see if the new sample value lies in the range of permitted sample values - v0.3.2 will incorporate this and also issue warnings at the end of processing if samples have required to be clipped to the maximum permissible or minimum permissible value respectively.
You yield in the end to ca. 50% of Lossless (eg. flac) bitrates, in numbers around 450 kbit/s (+-50).
The technical approach of removing bits reminds me to wavpack lossy, which has also great more or less tranparent results at these high - very high lossy bitrates.
Isn't the technical approach between wavpack lossy and this algorithmn very very similar, ie. (nearly) no psycho-acoustic model which might cause artifacts, but instead a higher noise floor, be it perceptual or not ?
(
At a side note, both, the results of wavpack lossy and of this algorithmn could be also similar to DAT at 32 kHz, 12 bit, ie. the way a DAT recorder in longplay position downsamples from 44.1 kHz 16 bit to 32 kHz, 12 bit, to achieve the half of the data.
While for many people without possibility of X/Y or A/B, even not to speak of ABX, DAT Longplay 32 kHz 12 bit is also "transparent").Can somebody of the inventors (of this algorithmn) explain in a few simple words, what is then the difference between wavpack lossy and this new algorithmn ?
(or link to that post, where that has been explained already )
bryant
Sep 28 2007, 09:23
QUOTE(user @ Sep 28 2007, 07:28)
Isn't the technical approach between wavpack lossy and this algorithmn very very similar, ie. (nearly) no psycho-acoustic model which might cause artifacts, but instead a higher noise floor, be it perceptual or not ?
Can somebody of the inventors (of this algorithmn) explain in a few simple words, what is then the difference between wavpack lossy and this new algorithmn ?
There are several differences, but I'll mentioned what I think are the most important.
The first is that this is designed to work with FLAC, or any other lossless compressor that can detect and optimize away redundant zeros in the audio data words. The advantage of this is that it can be used with the many existing FLAC hardware (and software) players without modification or update.
The other major difference is that lossyFLAC is intelligent. WavPack lossy is CBR and simply gets as close as it can to the source given the allowed bitrate. LossyFLAC is VBR because it analyzes the source and tries to determine the most quantization noise it can add and still be completely transparent. In this way it's more like the WavPack lossy “quality” mode I worked on (and abandoned) several years ago.
Both methods rely on
some psychoacoustic principles to achieve their results. In the WavPack case it chooses noise shaping and joint stereo encoding to make the noise less audible. LossyFLAC uses them to help determine how much white noise it can add (i.e., the fact that we are less sensitive to added noise at higher frequencies). However, neither have a psychoacoustic model in the strict sense and, of course, neither use any sort of frequency domain encoding or digital filtering in the signal path.
QUOTE(BGonz808 @ Sep 25 2007, 04:56)
Just out of curiosity, will bit-reduction cause ALAC compress any better. I just bought an iPod
and the idea of lossy flac is great but now I'm stuck without my trusty FLAC format if I want lossless music (and rockbox doesn't really appeal to me that much).
Thanks,
Bobby
I ran a test with .3.1b -3:
(I'm too lazy to type it out, I attached image incase host goes down)
So it has no effect on filesize, slightly negative in fact. LossyWav causes some bad clipping in the case of Ali's Here and Unbreakable btw.
Too bad actually, cause it seemed a good way to save some space on the iPod.
Nick.C
Sep 28 2007, 15:42
QUOTE(Brent @ Sep 28 2007, 20:57)
I ran a test with .3.1b -3:
So it has no effect on filesize, slightly negative in fact. LossyWav causes some bad clipping in the case of Ali's Here and Unbreakable btw.
Too bad actually, cause it seemed a good way to save some space on the iPod.
Thanks for the testing. You can switch on fixed amplitude reduction clipping prevention by using the -clipping 1 parameter.
Attached lossyWAV alpha v0.3.2. Superseded by v0.3.3
Code tidied up further and will now require -force parameter to over-write an existing file. If clipping occurs, warning(s) are issued after processing. I say warning(
s) because it counts clipping over maximum and under minimum separately.
Mitch 1 2
Sep 29 2007, 06:33
Could you please add a command-line parameter to set the process priority to "low"?
Nick.C
Sep 29 2007, 06:47
QUOTE(Mitch 1 2 @ Sep 29 2007, 13:33)
Could you please add a command-line parameter to set the process priority to "low"?
Mitch,
I'll certainly look into it and see if I can - I've never tried it before, but there must be code available to implement it (somewhere....).
Thanks for your input.
Nick.
guruboolez
Sep 29 2007, 07:00
@Nick.C
Did the settings change since version ~0.20? I'm unable to make the last versions (>0.30) work with foobar2000.
What I'm using is:
encoder: C:\windows\system32\cmd.exe
extension: lossy.flac
Parameters: /d /c C:\CODEC\lossywav\LossyWav.bat %s %d
Format is: lossy
bps: 16 bits
my
batch files is the following one:
CODE
@echo off
set lossyWAV_path="C:\CODEC\lossywav\lossyWav.exe"
set flac_path="C:\CODEC\lossywav\flac.exe"
rem echo %lossyWAV_path% %1 -c 576 >>flac_lossy.txt
rem echo %flac_path% -8 -f -b 576 -o"%~DPN2.flac" "%~DPN1.lossy.wav" >>flac_lossy.txt
rem echo del "%~DPN1.lossy.wav" >>flac_lossy.txt
%lossyWAV_path% %1 -c 576
%flac_path% -8 -f -b 576 "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"
set output_file=
set lossyWAV_path=
set flac_path=
Nick.C
Sep 29 2007, 07:50
QUOTE(guruboolez @ Sep 29 2007, 14:00)
@Nick.C
Did the settings change since version ~0.20? I'm unable to make the last versions (>0.30) work with foobar2000.
What I'm using is:
encoder: C:\windows\system32\cmd.exe
extension: lossy.flac
Parameters: /d /c C:\CODEC\lossywav\LossyWav.bat %s %d
Format is: lossy
bps: 16 bits
my
batch files is the following one:
CODE
@echo off
set lossyWAV_path="C:\CODEC\lossywav\lossyWav.exe"
set flac_path="C:\CODEC\lossywav\flac.exe"
rem echo %lossyWAV_path% %1 -c 576 >>flac_lossy.txt
rem echo %flac_path% -8 -f -b 576 -o"%~DPN2.flac" "%~DPN1.lossy.wav" >>flac_lossy.txt
rem echo del "%~DPN1.lossy.wav" >>flac_lossy.txt
%lossyWAV_path% %1 -c 576
%flac_path% -8 -f -b 576 "%~N1.lossy.wav" -o"%~N2.flac"
del "%~N1.lossy.wav"
set output_file=
set lossyWAV_path=
set flac_path=
Settings did change a bit - "-c" is now "-cbs", but a codec_block_size of 576 is now the default setting, so you can remove "-c 576" altogether.
guruboolez
Sep 29 2007, 08:13
It works fine again. Thank you very much!
guruboolez
Sep 29 2007, 08:53
Lossywav is impressive on harpsichord recordings.
It brings a 1062 kbps flac encoding to a pretty nice 436 kbps: 60% reduction!!!
Sound quality was fine, but strangely (because I didn't heard something wrong at first sight and I didn't expect to hear something different from original) my ABX score isn't bad at all:
CODE
foo_abx 1.3.1 report
foobar2000 v0.9.4.5 beta 1
2007/09/29 16:36:04
File A: F:\Fiocco - [Demeyere] Pièces de Clavecin, Oeuvre Premier\CD 1\12-Vivace.flac
File B: N:\12-Vivace.lossy.flac
16:36:04 : Test started.
16:36:29 : 01/01 50.0%
16:36:35 : 02/02 25.0%
16:36:58 : 02/03 50.0%
16:37:04 : 03/04 31.3%
16:37:14 : 04/05 18.8%
16:37:31 : 04/06 34.4%
16:37:36 : 05/07 22.7%
16:37:41 : 06/08 14.5%
16:37:54 : 06/09 25.4%
16:37:59 : 07/10 17.2%
16:38:11 : 08/11 11.3%
16:38:18 : 08/12 19.4%
16:38:21 : 08/13 29.1%
16:38:24 : 09/14 21.2%
16:38:29 : 10/15 15.1%
16:38:40 : 11/16 10.5%
16:38:45 : 12/17 7.2%
16:38:51 : 13/18 4.8%
16:39:00 : 14/19 3.2%
16:39:08 : 15/20 2.1%
16:39:15 : 16/21 1.3%
16:39:23 : 16/22 2.6%
16:39:31 : 16/23 4.7%
16:39:36 : 17/24 3.2%
16:40:04 : 18/25 2.2%
16:40:18 : 19/26 1.4%
16:40:27 : 19/27 2.6%
16:40:40 : 20/28 1.8%
16:40:51 : 20/29 3.1%
16:41:02 : 21/30 2.1%
16:41:07 : 21/31 3.5%
16:41:11 : 22/32 2.5%
16:41:32 : Test finished.
----------
Total: 22/32 (2.5%)
Second half was better than first one (it's usually the opposite). There were no noise, no artefact, but something hard to define (audiophile would call it "lack of soundstage" or something similar).
I'm not completely sure that this ABX score is valid (since I didn't really fix a target: first 16 trials, then I decided to extend it to 24 and when I reached 24 I was so confident that I extend it again to 32 trials). Pio2001 explained once why such test need a better score to be valid.
I uploaded a short sample from this sample (the part I ABXed here):
http://rapidshare.com/files/59085975/fiocco.zip.html
Nick.C
Sep 29 2007, 10:43
From the look of your command line, you're not changing anything that isn't default in v0.3.2. If you're looking to maintain the same processing rate, use "-nts n" where n is less than -1.5 this will reduce the bits removed in the processing - at the rate of 1 bit per -6 of nts (noise_threshold_shift).
The other way to do it would be to use the "-1" quality option to see if that makes a difference to your sample.
Nick.C
Sep 29 2007, 13:29
@Mitch 1 2: I have found a method of lowering process priority, although the permissible priority settings vary with Windows version. Are you running Windows later than NT, ME, 98 & 95? If so then I can provide 3 settings: Normal, Below_Normal and Low priority. If not, Below_Normal vanishes as it is not supported by the O/S mentioned.
I will implement "-below" and "-low" ("-normal" unavailable, but default) parameters to lower the process priority - but below won't work on all versions of Windows.
Nick.C
Sep 29 2007, 15:12
lossyWAV alpha v0.3.3 attached. Immediately superseded by v0.3.3b - output file access bug.
-below and -low parameters incorporated, setting below_normal process priority and low process priority respectively.
Now checks for read-only output files and will attempt to set read/write access, will also report if read/write access setting fails.
CODE
lossyWAV alpha v0.3.3 : WAV file bit depth reduction method by 2Bdecided.
Transcoded to Delphi by Nick.C & Halb27 from a script, www.hydrogenaudio.org
Usage: lossyWAV <input wav file> <options>
Options:
-1, -2 or -3 quality level (1:overkill, 2:default, 3:compact)
-o <folder> destination folder for the output file
-force forcibly over-write output file if it exists.
-cbs <n> analysis codec_block_size (512<=n<=4608, default=576 samples)
(should match codec block size used in target compression codec)
-nts <n> noise_threshold_shift=n (-15.0<=n<=0.0, default -1.5dB)
(reduces overall bits to remove)
-skew <n> skew results of fft analyses by n dB (0.0<=n<=12.0, default=0.0)
with a (sin-1) shaping over the frequency range 20Hz to 3.7kHz.
(artificially decrease low frequency bins to take into account
higher SNR requirements at low frequencies)
-dither <n> dither selection, 0<=n<=2, default=0
(0=no dither; 1=rectangular dither; 2=triangular dither)
-clipping <n> clipping prevention selection, 0<=n<=1, default=0. 0=none;
1=fixed clipping prevention amplitude reduction, taking into
account dither amplitude (if any).
-vsfl select variable spreading function lengths (sfl)
(sfl = number of fft bins averaged during convolution of fft
results. short fft = short sfl, long fft = long sfl)
-wsf select weighted spreading functions.
(weighted average of fft bins during convolution of fft results
weighted towards lower frequency fft bins, 5/8:3/8)
-overlap <n> fft_overlap = fft_length/n (2<=n<=8, default=2)
(increases number of fft analyses per codec block)
-quiet significantly reduce screen output
-nowarn suppress lossyWAV warnings
-detail enable detailled output mode
-below set process priority to below normal.
-low set process priority to low.
Options not yet implemented:
-bitdepth <n> forced output bitdepth (16 or 24)
-flac optimizations for use with FLAC
-wv optimizations for use with wavPack
-tak optimizations for use with TAK
Mitch 1 2
Sep 29 2007, 20:08
I've tested lossyWAV 0.3.3 on Windows XP SP2, and there's a serious problem with output file permissions which is not present in the 0.3.2 version. lossyWAV fails to gain write access even when there is no existing file.
Nick.C
Sep 30 2007, 01:41
QUOTE(Mitch 1 2 @ Sep 30 2007, 03:08)
I've tested lossyWAV 0.3.3 on Windows XP SP2, and there's a serious problem with output file permissions which is not present in the 0.3.2 version. lossyWAV fails to gain write access even when there is no existing file.
That sounds like inadequate testing on my part prior to release. It *should* work if the output file already exists, however this is embarassing - I'll get v0.3.3b out as soon as possible.
This version should work - apologies for the error. - File removed, superseded.
Nick.
Mitch 1 2
Sep 30 2007, 07:02
It's alpha software, so I forgive you.
Now lossyWAV works as expected, and it even handled my conflicting parameter tests!
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please
click here.