I've greatly enjoyed following the progress of this project, with almost continuous updates and all sorts of interesting feedback from all quarters. But I am somewhat ashamed to admit that only this evening have I actually begun playing with lossyWAV in foobar2000 (I had previously only done a few tests here and there, from the command line).

Based on what I've tried this evening, I have a couple of questions. They have probably been answered elsewhere, but I have not yet spotted the answers this evening... and if I've read them in the multitude of lossyWAV pages I've read over the past few months, I've forgotten.

Briefly:

• It was very easy to get lossyWAV working using the suggested structure of C:\bin\lossywav, but it is odd to have to place my codecs in a root-level directory. Every other audio codec resides in a "codecs" sub-folder, within my foobar2000 directory. Is the number of accessible directory levels a limitation of calling the process via cmd, as the encoder? Or will this eventually be altered, to allow a complex structure along the lines of "C:\Program Files\assorted audio stuff\foobar2000\codecs\"? (The program seems to work fine if run from a complex directory structure, but only when called directly from the command line.)

• Since lossyWAV-encoded Redbook audio is still two channels, with 16 bit wordlength, 1411 kbps, and 44100 Hz, is there any reason lossyWAV could not be used to archive material that will later be burned back to a CD-R? (Effectively, isn't the VBR nature of lossyWAV closer to a "padded" CBR?)

Thank you!

- M.

I'm running it from foobar using the following batch file, and it works fine (doesn't need to be at the root):


Note: No spaces in my address, whereas:
C:\Program Files\assorted audio stuff\foobar2000\codecs\
does have spaces, and that may be the issue. See here.

C.

I am becoming more convinced that the "spaces-in-the-path-to-the-executable" problem is in some way due to the way that cmd.exe is being called - I'll spend a bit of time investigating*....
PCM in a WAV file is always most significant bit (msb) justified. So, a 9 bit sample occupies the top 9 bits in a 16-bit word. All lossyWAV does is vary the number of msb's in use per codec-block while maintaining the original sample size (in bytes). The audio is still fully WAV format compliant and there is no reason not to burn back to CD-R (I assume as CDDA rather than data?).

I'm glad that you're enjoying lossyWAV - I've enjoyed it this far as well - although, without David's publication of his method just over a year ago, it would never have happened.

[edit] * I may have cracked it - try:[/edit]

That seems to have gotten it... thank you! (Also, you are correct in the assumption of CDDA versus data.)

- M.

That's another variation, mine was the "short name", just one page ago

I just tested lossyWAV 1.0.1u RC1 using --standard with my usual problem set as well as some regular tracks.
Everything's fine (as was expected - but there's always the possibility with changes in code that something goes wrong).

I was thinking about that too; since cmd is a dos-(related) command short filenames could work too.
Or one adds the dir with lossywav, flac, lame and other audio utils to the path:

In Windows XP this can be done in: (iirc)

Control Panel > System > Advanced > Environment Variables > Path > edit..

Hmm, if it's because of how cmd is launching, and using 8.3 path in... well... path, fixes it...

Then try setting the encoder to be "C:\WINDOWS\system32\cmd.exe" with those quotes around it. That *should* force the method that Foobar is using to launch cmd to use long filenames. There's still a possibility that Foobar is explicitly forcing 8.3 names, and why that would be is beyond me.

Hi all,

Firstly lossyWAV is excellent, really impressed!

Just ran a test on a sample of classical FLACs encoded at -5.
The sample was 110 files that were all between 540 and 560 kbps.
The reason for this selection is that originally I thought that below 550 the savings rarely warrant using lossyWAV (however, this may not be so - but it's tricky to identify which files are suitable and which aren't).

Results:

Files: AVG kbps / Total (MB) (Saving (MB)) / % of orig size
Original FLAC -5: 551 / 2,267 / - / 100%
lossyWAV -7.0 - 535 / 2,203 (64 MB) / 97%
lossyWAV -6.5 - 526 / 2,167 (100 MB) / 96%
lossyWAV -6.0 - 516 / 2,124 (143 MB) / 94%

The saving was less than I expected.
I had a look at a few individual cases and found that a good number of lossyWAV tracks were larger than the originals (from -6 to -7).
So where lossyWAV worked it worked really well, where it couldn't make reductions it increased the file size.

I don't mean this as a criticism of lossyWAV in any way whatsoever, in fact I've been enormously impressed !!!

What is clear is that one cannot tell by the original bitrate whether or not lossyWAV can make a difference. For example I've had good reductions on files < 500 kbps.

So what I'd really like to see is a program that runs the lossyWAV analysis without doing the encoding (like WAVGain).
- User sets a minimum reduction requirement in %.
- Programs identifies all files where that reduction cannot be achieved.
- Then creates an m3u playlist of all the files that can make the user specified reduction.
- The user then loads playlist into encoder UI (i.e. foobar2k) and off you go.

Just a thought / wish.
In the mean time I'm going to do this manually.

By the way, great idea to allow non-integer values for lossyWAV encoding
Very nice.

C.

Better to use lossywav -5 as reference . It is 'standard'.

Yes, it's a basic problem.
Guess your tracks consists of rather 'simple' music (from a lossless codec's view), that is music which is rather quiet at least at considerable amount of spots and/or is made up from one or few instruments only.

First thought: for this kind of music it's really disputable whether it's worth while not to use a lossless codec (and I guess you'll often find that when replacing FLAC by wavPack -hx3 or -hhx3 or even better TAK -p4 or even better Monkey -c4000 the situation is even more shining for the lossless codec).

Second thought:
The first thing to consider is the lossyWAV quality demand. In case you don't use lossyWAV for archiving there's not much use in going beyond the --standard quality, and in this case you do get some savings with most of your samples as compared to lossless. And probably --standard is already some overkill which you can find out for your needs by listening tests.

If you use lossyWAV as a replacement for lossless archiving like me the situation is worse of course. But if the percentage of tracks where lossyWAV brings a remarkable relief in file size is large (my situation) I don't care about the few tracks which I encode losslessly. And I don't care too much about some tracks where I may have chosen a suboptimal decision.
If on the other hand your collection most of the time consists of such tracks which are easy to encode for a lossless codec I wouldn't care about lossyWAV (though I might save some file size on some tracks) and would use lossless straight ahead.

Good point, -5.0 added below:

Files: AVG kbps / Total (MB) (Saving (MB)) / % of orig size
Original FLAC -5: 551 / 2,267 / - / 100%
lossyWAV -7.0 - 535 / 2,203 (64 MB) / 97%
lossyWAV -6.5 - 526 / 2,167 (100 MB) / 96%
lossyWAV -6.0 - 516 / 2,124 (143 MB) / 94%
lossyWAV -5.0 - 491 / 2,022 (245 MB) / 89%

This mini snapshot illustrates the issue very well. In red are the lossyWAV at -5 versions (bitrate at the end):

053. Bach J.S. - [Gould #22] Goldberg Variations - Variation No.21 (1980) - 559
054. Bach J.S. - [Gould #22] Goldberg Variations - Variation No.21 (1980) - 564
055. Bach J.S. - [Gould #27] Goldberg Variations - Variation No.26 (1955) - 554
056. Bach J.S. - [Gould #27] Goldberg Variations - Variation No.26 (1955) - 359
057. Bach J.S. - [Gould #31] Goldberg Variations - Variation No.30 (1980) - 555
058. Bach J.S. - [Gould #31] Goldberg Variations - Variation No.30 (1980) - 559

On 55 lossyWAV achieves a huge reduction (i.e. 56)
On 53 lossyWAV makes it larger (see 54)
Yet the initial bitrate is very similar.

If there was a tool to identify this prior to encoding, then clearly I'd process 55 but leave 53 alone.

@halb27

Yes, I am using lossyWAV as a replacement for lossless archiving. The reason for the test was to identify the cut-off where it was not useful to employ lossyWAV. There are plenty of files in my classical collection > 550 kbps and lossyWAV has a huge effect and is very worthwhile anywhere between -6 to -7.5 (I've not bothered testing > -7.5).

But, as I said, what was interesting were the many samples that were < 500 kbps where lossyWAV also achieved a significant reduction. So the problem, as I mentioned was not simply a case of saying "ah, below x kbps lossyWAV has no value, thus I shall only use it on files over x kbps". The issue is one of prior identification.

Where a reduction of less than, say, 5% is possible, I'm going to leave them as pure lossless FLACs - that's fine. So far - 6.5 looks good/very safe to me. LossyWAV is still young and I'm sure problem samples will arrive as its take-up increases -- though I may be wrong.

C.

lossywav 1.0.1u

flac -x -b 512 --keep-foreign-metadata %1
(-q 10) - 89,96%
(-q 7.5) - 77,15%
(-q 5) - 65,38%
(-q 2.5) - 51,35%
(-q 0) - 34,96%

for example

flac -x %1
(-q 10) - 96,52%
(-q 7.5) - 83,55%
(-q 5) - 70,93%
(-q 2.5) - 56,45%
(-q 0) - 39,69%

In case this is in some way related to my last post:
I'm using lossywav 1.0.1u with the wiki settings for foobar (i.e. -b 512)

C.

Ran into a strange quirk while using piped input and output:

I first noticed a filesize discrepancy when I was testing stuff from the command line and then from foobar2000.

I used these command lines when testing from the console:


and this for encoding through foobar2000 and saved as test3.lossy.flac:



I noticed that the output file from foobar was 7.11MB while everything else was 7.02MB, so I stripped tags from all the files using Mp3Tag and saw that nothing changed.

test1.lossy.flac: 7,363,354 bytes (MD5: bd2ceb8a15bb390cd2d2e4e74efca13b)
test2.lossy.flac: 7,363,354 bytes (MD5: bd2ceb8a15bb390cd2d2e4e74efca13b)
test3.lossy.flac: 7,464,204 bytes (MD5: a472581649d91ec91f35ef17ef4806e5)

Can anyone reproduce this or (even better) tell me why this happens? I verified that neither ReplayGain nor DSP was being applied, and that foobar2000 was set to keep lossless sources at original bit depth.

And differences in number of seekpoints and/or padding ?

This was discussed here and it is to do with piped input from foobar2000, basically the seektable is larger than it ought to be.

If those numbers in brackets are dates, then I'd guess the 1955 recording is quite hissy, while the 1980 recording is not. Lossless encoders have to preserve this hiss perfectly intact - lossyWAV doesn't.


It could be the default lossyFLAC blocksize isn't appropriate for these recordings. Take the lossyFLAC versions and re-encode them to FLAC, but with default FLAC settings. That might help.

Otherwise, it can simply be that lossyWAV can't find anything to remove.

I think we've had this discussion before - it would be useful to have a tool to simply keep the original if it was smaller (or similar).

Cheers,
David.

lossyWAV 1.0.1v RC2 attached to post #1 in this thread.

They are, and it is.


Tried that - the difference is minimal.


We have (see below) and it would.

With lossy processes there is normally one frontier, but with lossyWAV there are two:
1) TRANSPARENCY (the traditional problem): what's the lowest we can go and still achieve transparency
2) FILESIZE (a problem unique, I think, to lossyWAV): what kind of original file can defeat lossyWAV, not in terms of transparency but in terms of filesize.

Although the latter is not as important as the former, it is still an issue, as no one expects a lossy process to increase bit rate.

C.

It's not lossyWAV per se which is increasing the bitrate, rather the -b 512 used in the FLAC encoding command line. For a track which gets bigger, try, as David suggested, re-encoding with FLAC without the -b 512 part of the command line. This will default to the original block length (probably 4096 samples) and the processed file should be the same size or maybe slightly smaller.

Yes, apologies Nick.
Increasing bitrate was inaccurate of me.
Results with your amendment to FLAC's default blocksize:

Bach J.S. - [Gould #22] Goldberg Variations - Variation No.21 (1980) - 559
Bach J.S. - [Gould #22] Goldberg Variations - Variation No.21 (1980) - 564
Bach J.S. - [Gould #22] Goldberg Variations - Variation No.21 (1980) - 554 (default FLAC blocksize)

The point is with such a reduction (559 to 554 with lossyWAV -5) I wouldn't have bothered.
A complimentary tool to pre-scan and identify such tracks, would certainly be welcome. That's really all my point is - and I shall leave it be from now on.

When it comes to practical use of lossyWAV I'm not going to be editing my batchfile each time I get an increase in bitrate ; instead I'm going leave that file alone.

C.

EDIT: It's just hit me that I may have a wrong assumption about lossyWAV:
If, hypothetically, lossyWAV can't do anything to reduce the original, does that mean that its process is lossless?
e.g.
ORIGINAL = 500
lossyWAV -2.5 = 500
lossyWAV -7.5 = 500

1) Will the 2 lossyWAV versions be identical?
2) Will the 2 lossyWAV versions be identical to the original?

Possibly.... - or at least very close to identical.

Ah! So lossyWAV is even safer than I thought. This makes sense of halb27's response a while back as to not being too bothered about the lossyWAV processed files that had almost zero reduction. He knew they were practically lossless.

Thanks Nick. That's cleared up a strange misconception on my part.

C.

They can be absolutely 100% lossless if lossyWAV judges that this is required.

Try a pure digitally generated sine wave, for example.

Cheers,
David.

Sample 3 - Sine Wave - A440 (generated in Cool Edit)
RG = -11.43 dB

Original FLAC -5 (318 kbps)

-q 10 -b 512 (363)
-q 7.5 -b 512 (363)
-q 5.0 -b 512 (363)
-q 2.5 -b 512 (363)
-q 0 -b 512 (352)

-q 7.5 -b 4096 (296)
-q 5.0 -b 4096 (296)
-q 2.5 -b 4096 (296)

Should I be getting identical bitrates to the original, or is the point that regardless of -q setting lossyWAV is not compromising quality (except at -q0, but hey)?

C.

It would be interesting to see the bitrate of original FLAC -b 4096 and the identical compression option you used with lossyWAV (I only guess you use -5 too). As the lossyWAV bitrate is the same with -q 2.5 to -q 7.5 I expect original FLAC will get close to this bitrate when using -b 4096 as with lossyWAV.
Other than that this is a good sample to see that a short blocksize can make the lossless encoder increase bitrate pretty much with 'simple' music.

If you have Cool Edit, you can simply load the original .wav and the .lossy.wav and subtract one from the other (mix paste, 100%, invert) to see if there is any difference at all. Or use the foobar2k wave comparator. Or ask lossyWAV how many bits it's removing.

Cheers,
David.

Or use the --correction parameter and look at the .lwcdf.wav file.

Original FLAC -5 -b 512 = 363
Very close!
Cool.

Didn't think of using mix paste, I'll try all the suggestions -- correction file looks appealing.
Thanks Nick, David, Halb27, much appreciated.

C.

Thanks for your sample.

Makes me reconsider using FLAC for the cases of 'simple' music (solo instruments or very quiet music) where I use wavPack right now.
wavPack is great - especially in these cases - but I'd prefer not to use various codecs (thinking of another DAP in the far future). Your sample encourages me to try FLAC with an explicit large blocksize.

lossyWAV 1.0.1w RC3 attached to post #1 in this thread.

(hopefully I've nailed the WINE crashing issue this time....)

A sine wave concert A (440 Hz) is essentially a pure tone at 440 Hz (the windowed FFT will spread it somewhat) with nothing more than spectrally flat dither noise at other frequencies, which should come to about -120 dBFS per bin with a 1024-point FFT except for the bins around 440 Hz which are presumably full-scale (close to 0dBFS save for spreading caused by windowing)

LossyWAV will look for the noise floor over the frequencies below 16 kHz (the lowest frequency bin). As this is close to -120 dBFS and probably a little lower thanks to the random nature of dither noise, lossyWAV is very likely indeed to retain all 16-bits. The exception is where there's a large negative safety margin to reduce bitrate at the expense of added hiss, as is the case at -q 0.

That explains why you get essentially consistent bitrate with 512 blocksize. I'm not sure what the difference is between flac -5 (lossless, which normally give -b 4096 for 44.1 kHz audio) and lossyFLAC -q 5.0 and -b 4096, which gave 318 kbps and 296 kbps respectively. I guess you should try using just flac -5 or just flac -5 -b 4096 for both lossless and lossyFLAC to compare like-with-like and then see if the lwcdf.wav file has non-zero content (Cool Edit analysis can tell you).

Did a mix-paste in Cool Edit, and just ran a bit compare in foobar:
No differences in decoded data found.

Yet in foobar, bitrates are different:
FLAC Original -b 4096 = 318
LossyFLAC -b 4096 -q 5 = 296

Anyway, the point is made and to be honest I'm just happy that lossyWAV is very careful indeed with the data, and thus it's fine to use for archiving and using as transcode source.

Thanks Dynamic for the explanation.

C.

Remember, when piping in foobar2000, the [edit] FLAC [/edit] seektable will be large and the padding will be 65536 bytes per file. This may be affecting your bitrate (assuming you are carrying out the transcoding using pipes in foobar2000, of course).

Both FLACs were created using foobar2000 (but perhaps only one is using piping? - not even really sure what piping is). One FLAC is created from the other -- piping?

Original FLAC created from WAV.
LossyFLAC created from Original FLAC.

The difference in filesize = 79,214 bytes.
The files are exactly 30 secs in length.

C.

I'll be releasing 1.1.0 final tomorrow night - but I wonder which of the more advanced options included for developmental purposes post 1.0.0b should be removed?

My guess is that --limit should stay but noone seems to be using --highskew. --blocksize is also not necessarily a good idea to keep as it may confuse users as it may be better to keep it at 512 samples throughout.

I've been sorting out the logfile mechanism - it will no longer crash with two or more foobar2000 transcoding threads. Also, I've expanded on --bitdist - now renamed as --blockdist and supplemented by --sampledist which both give information about the block / sample distribution of lsb's and msb's. This will also be written to a logfile.

For the logfile when used with STDIN, I have implemented a --stdinname parameter which is followed by the name which the user wishes to be displayed on the console and also written to the log.

Any suggestions, likes, dislikes, etc will be very much appreciated.

I don't use any of the advanced options as I don't really understand what some of them do and under what circumstances it's appropriate to use them. However, I guess that some people do use them so they're probably all worth keeping just in case someone finds them useful. The thing I like about LossyWav is that you can use very basic settings, like the quality level, or even just let it default everything and get good results while on the other hand you have very fine control over how the program behaves if you want to use the advanced settings. Having said that it would be best to remove anything that was put in for monitoring/debugging especially if it's likely to confuse somebody. On that subject is using the same letter but different case for different parameters likely to confuse? EG -s/-S, -d/-D.

Just a small confirmation that iTunes 7.7 still can't make use of lossywav to save bits. Too bad, as my iPod is /the/ place I'd use this ... Ah well, thanks for the good work, it's a very interesting project

Wonderful. Much appreciated.

IMO most of the advanced options should be removed in a final release.
--highskew, --blocksize as you think, but if it is up to me I would always make the last beta version available, and keep the advanced options there, but remove the advanced options from the final verison with very few exceptions.
Valuable exceptions IMO are -q, --scale, -s.
The problems with many of the advanced options is also that an average user can't understand them, and a really useful precise description will probably offend many users.
The other problem is the mere number of advanced options which at the end of the day has more of a negative effect than a positive one.

So IMO there should be a high threshold, and an advanced option should have a very promising effect at least to some people in order to justify a specific advanced option within the final version.

The situation is different with beta versions where there are users which want to be a bit experimental.

So for the final version I'd even prefer not to have the --noclips option any more.

I'll strip out most of the advanced options from 1.1.0 and also leave 1.0.1x RC4 available with the advanced options left in.

To confirm: --fft32, --analyses, --highskew, --blocksize, --minbits, --noclips & --wine will all be removed for the release of 1.1.0.

That's fine.

lossyWAV 1.1.0 attached (with source) to post #1 in this thread.

Thanks a lot, Nick.