lossyWAV 1.1.0 Development Thread., Added noise WAV bit reduction method. Options

[carpman]

Sample 3 - Sine Wave - A440 (generated in Cool Edit)
RG = -11.43 dB

Original FLAC -5 (318 kbps)

-q 10 -b 512 (363)
-q 7.5 -b 512 (363)
-q 5.0 -b 512 (363)
-q 2.5 -b 512 (363)
-q 0 -b 512 (352)

-q 7.5 -b 4096 (296)
-q 5.0 -b 4096 (296)
-q 2.5 -b 4096 (296)

Should I be getting identical bitrates to the original, or is the point that regardless of -q setting lossyWAV is not compromising quality (except at -q0, but hey)?

C.

This post has been edited by carpman: Jul 1 2008, 19:43

[halb27]

Sample 3 - Sine Wave - A440 (generated in Cool Edit)
RG = -11.43 dB

Original FLAC -5 (318 kbps)

-q 10 -b 512 (363)
-q 7.5 -b 512 (363)
-q 5.0 -b 512 (363)
-q 2.5 -b 512 (363)
-q 0 -b 512 (352)

-q 7.5 -b 4096 (296)
-q 5.0 -b 4096 (296)
-q 2.5 -b 4096 (296)

Should I be getting identical bitrates to the original, or is the point that regardless of -q setting lossyWAV is not compromising quality (except at -q0, but hey)?

C.

It would be interesting to see the bitrate of original FLAC -b 4096 and the identical compression option you used with lossyWAV (I only guess you use -5 too). As the lossyWAV bitrate is the same with -q 2.5 to -q 7.5 I expect original FLAC will get close to this bitrate when using -b 4096 as with lossyWAV.
Other than that this is a good sample to see that a short blocksize can make the lossless encoder increase bitrate pretty much with 'simple' music.

This post has been edited by halb27: Jul 1 2008, 20:02

[2Bdecided]

If you have Cool Edit, you can simply load the original .wav and the .lossy.wav and subtract one from the other (mix paste, 100%, invert) to see if there is any difference at all. Or use the foobar2k wave comparator. Or ask lossyWAV how many bits it's removing.

Cheers,
David.

[Nick.C]

If you have Cool Edit, you can simply load the original .wav and the .lossy.wav and subtract one from the other (mix paste, 100%, invert) to see if there is any difference at all. Or use the foobar2k wave comparator. Or ask lossyWAV how many bits it's removing.

Cheers,
David.

Or use the --correction parameter and look at the .lwcdf.wav file.

[carpman]

It would be interesting to see the bitrate of original FLAC -b 4096 and the identical compression option you used with lossyWAV (I only guess you use -5 too). As the lossyWAV bitrate is the same with -q 2.5 to -q 7.5 I expect original FLAC will get close to this bitrate when using -b 4096 as with lossyWAV.

Original FLAC -5 -b 512 = 363
Very close!
Cool.

Didn't think of using mix paste, I'll try all the suggestions -- correction file looks appealing.
Thanks Nick, David, Halb27, much appreciated.

C.

[halb27]

Thanks for your sample.

Makes me reconsider using FLAC for the cases of 'simple' music (solo instruments or very quiet music) where I use wavPack right now.
wavPack is great - especially in these cases - but I'd prefer not to use various codecs (thinking of another DAP in the far future). Your sample encourages me to try FLAC with an explicit large blocksize.

This post has been edited by halb27: Jul 2 2008, 09:11

[Nick.C]

lossyWAV 1.0.1w RC3 attached to post #1 in this thread.

(hopefully I've nailed the WINE crashing issue this time....)

[Dynamic]

Should I be getting identical bitrates to the original, or is the point that regardless of -q setting lossyWAV is not compromising quality (except at -q0, but hey)?

A sine wave concert A (440 Hz) is essentially a pure tone at 440 Hz (the windowed FFT will spread it somewhat) with nothing more than spectrally flat dither noise at other frequencies, which should come to about -120 dBFS per bin with a 1024-point FFT except for the bins around 440 Hz which are presumably full-scale (close to 0dBFS save for spreading caused by windowing)

LossyWAV will look for the noise floor over the frequencies below 16 kHz (the lowest frequency bin). As this is close to -120 dBFS and probably a little lower thanks to the random nature of dither noise, lossyWAV is very likely indeed to retain all 16-bits. The exception is where there's a large negative safety margin to reduce bitrate at the expense of added hiss, as is the case at -q 0.

That explains why you get essentially consistent bitrate with 512 blocksize. I'm not sure what the difference is between flac -5 (lossless, which normally give -b 4096 for 44.1 kHz audio) and lossyFLAC -q 5.0 and -b 4096, which gave 318 kbps and 296 kbps respectively. I guess you should try using just flac -5 or just flac -5 -b 4096 for both lossless and lossyFLAC to compare like-with-like and then see if the lwcdf.wav file has non-zero content (Cool Edit analysis can tell you).

[carpman]

Did a mix-paste in Cool Edit, and just ran a bit compare in foobar:
No differences in decoded data found.

Yet in foobar, bitrates are different:
FLAC Original -b 4096 = 318
LossyFLAC -b 4096 -q 5 = 296

Anyway, the point is made and to be honest I'm just happy that lossyWAV is very careful indeed with the data, and thus it's fine to use for archiving and using as transcode source.

Thanks Dynamic for the explanation.

C.

[Nick.C]

Did a mix-paste in Cool Edit, and just ran a bit compare in foobar:
No differences in decoded data found.

Yet in foobar, bitrates are different:
FLAC Original -b 4096 = 318
LossyFLAC -b 4096 -q 5 = 296

Anyway, the point is made and to be honest I'm just happy that lossyWAV is very careful indeed with the data, and thus it's fine to use for archiving and using as transcode source.

Thanks Dynamic for the explanation.

C.

Remember, when piping in foobar2000, the [edit] FLAC [/edit] seektable will be large and the padding will be 65536 bytes per file. This may be affecting your bitrate (assuming you are carrying out the transcoding using pipes in foobar2000, of course).

This post has been edited by Nick.C: Jul 4 2008, 22:19

[carpman]

Both FLACs were created using foobar2000 (but perhaps only one is using piping? - not even really sure what piping is). One FLAC is created from the other -- piping?

Original FLAC created from WAV.
LossyFLAC created from Original FLAC.

The difference in filesize = 79,214 bytes.
The files are exactly 30 secs in length.

C.

This post has been edited by carpman: Jul 5 2008, 00:25

[Nick.C]

I'll be releasing 1.1.0 final tomorrow night - but I wonder which of the more advanced options included for developmental purposes post 1.0.0b should be removed?

My guess is that --limit should stay but noone seems to be using --highskew. --blocksize is also not necessarily a good idea to keep as it may confuse users as it may be better to keep it at 512 samples throughout.

I've been sorting out the logfile mechanism - it will no longer crash with two or more foobar2000 transcoding threads. Also, I've expanded on --bitdist - now renamed as --blockdist and supplemented by --sampledist which both give information about the block / sample distribution of lsb's and msb's. This will also be written to a logfile.

For the logfile when used with STDIN, I have implemented a --stdinname parameter which is followed by the name which the user wishes to be displayed on the console and also written to the log.

Any suggestions, likes, dislikes, etc will be very much appreciated.

This post has been edited by Nick.C: Jul 11 2008, 19:44

[botface]

I'll be releasing 1.1.0 final tomorrow night - but I wonder which of the more advanced options included for developmental purposes post 1.0.0b should be removed?

My guess is that --limit should stay but noone seems to be using --highskew. --blocksize is also not necessarily a good idea to keep as it may confuse users as it may be better to keep it at 512 samples throughout.

I've been sorting out the logfile mechanism - it will no longer crash with two or more foobar2000 transcoding threads. Also, I've expanded on --bitdist - now renamed as --blockdist and supplemented by --sampledist which both give information about the block / sample distribution of lsb's and msb's. This will also be written to a logfile.

For the logfile when used with STDIN, I have implemented a --stdinname parameter which is followed by the name which the user wishes to be displayed on the console and also written to the log.

Any suggestions, likes, dislikes, etc will be very much appreciated.

I don't use any of the advanced options as I don't really understand what some of them do and under what circumstances it's appropriate to use them. However, I guess that some people do use them so they're probably all worth keeping just in case someone finds them useful. The thing I like about LossyWav is that you can use very basic settings, like the quality level, or even just let it default everything and get good results while on the other hand you have very fine control over how the program behaves if you want to use the advanced settings. Having said that it would be best to remove anything that was put in for monitoring/debugging especially if it's likely to confuse somebody. On that subject is using the same letter but different case for different parameters likely to confuse? EG -s/-S, -d/-D.

[Brent]

Just a small confirmation that iTunes 7.7 still can't make use of lossywav to save bits. Too bad, as my iPod is /the/ place I'd use this ... Ah well, thanks for the good work, it's a very interesting project

[halb27]

I'll be releasing 1.1.0 final tomorrow night ...

Wonderful. Much appreciated.

... but I wonder which of the more advanced options included for developmental purposes post 1.0.0b should be removed? ...

IMO most of the advanced options should be removed in a final release.
--highskew, --blocksize as you think, but if it is up to me I would always make the last beta version available, and keep the advanced options there, but remove the advanced options from the final verison with very few exceptions.
Valuable exceptions IMO are -q, --scale, -s.
The problems with many of the advanced options is also that an average user can't understand them, and a really useful precise description will probably offend many users.
The other problem is the mere number of advanced options which at the end of the day has more of a negative effect than a positive one.

So IMO there should be a high threshold, and an advanced option should have a very promising effect at least to some people in order to justify a specific advanced option within the final version.

The situation is different with beta versions where there are users which want to be a bit experimental.

So for the final version I'd even prefer not to have the --noclips option any more.

This post has been edited by halb27: Jul 12 2008, 11:56

[Nick.C]

IMO most of the advanced options should be removed in a final release.
--highskew, --blocksize as you think, but if it is up to me I would always make the last beta version available, and keep the advanced options there, but remove the advanced options from the final verison with very few exceptions.
Valuable exceptions IMO are -q, --scale, -s.
The problems with many of the advanced options is also that an average user can't understand them, and a really useful precise description will probably offend many users.
The other problem is the mere number of advanced options which at the end of the day has more of a negative effect than a positive one.

So IMO there should be a high threshold, and an advanced option should have a very promising effect at least to some people in order to justify a specific advanced option within the final version.

The situation is different with beta versions where there are users which want to be a bit experimental.

So for the final version I'd even prefer not to have the --noclips option any more.

I'll strip out most of the advanced options from 1.1.0 and also leave 1.0.1x RC4 available with the advanced options left in.

To confirm: --fft32, --analyses, --highskew, --blocksize, --minbits, --noclips & --wine will all be removed for the release of 1.1.0.

[halb27]

...To confirm: --fft32, --analyses, --highskew, --blocksize, --minbits, --noclips & --wine will all be removed for the release of 1.1.0.

That's fine.

[Nick.C]

lossyWAV 1.1.0 attached (with source) to post #1 in this thread.

[halb27]

Thanks a lot, Nick.