lossyWAV 1.1.0 released., Added noise WAV bit reduction method. Options

[shadowking]

i think -standard quality losswav is aiming for transparency for every sample and every listener . Lower setting like Q3 or 2.5 or lower might very well be transparent for the vast majority of cases. If keeping the bitrate down is a concern, one could use a lower quality like 2.5 and still get very high quality and enough for that transcode to layer 3 if needed.

The other advantage of quality 6 / 7 is that you could make your lossywav archive smaller if needed like transcode it to a lower quality setting with great safety. Example: Say I have a lossywav Q6 archive at home and I need high quality transcodable on-demand files for my internet radio station which is in another location. I could transcode Q6 archive to quality 3 and transcode that to a streaming format like layer 3 or he-aac. Its likely to work well.

Another scenario is sound processing and editing. I could edit Q6 archive dozens of times of my music projects. Another is a high quality backup - transcode Q6 to Q3 as my everyday archive keeping Q6 as the master on some other offline harddrive. the space used will be nearly the same as normal lossless in such a situation but there is the advantage of splitting archives across drives. These overkill setting are lossless replacements. In theory an online music store should be able to use Q6 / 7 to transcode to lossy with great confidence.

This post has been edited by shadowking: Jul 17 2008, 10:51

[2Bdecided]

Another scenario is sound processing and editing. I could edit Q6 archive dozens of times of my music projects. Another is a high quality backup - transcode Q6 to Q3 as my everyday archive keeping Q6 as the master on some other offline harddrive. the space used will be nearly the same as normal lossless in such a situation but there is the advantage of splitting archives across drives. These overkill setting are lossless replacements. In theory an online music store should be able to use Q6 / 7 to transcode to lossy with great confidence.

"Should" is the important word in all those statements. This needs some testing and ABXing before we can claim it comfortably. I did make a start, but got waylaid.

Cheers,
David.

[botface]

To say nothing about vinyl. Gawd! My needledrop FLACs are shrinking by 45% at --standard!

If there ever was a demonstration of the reduced dynamic range of vinyl, lossyWAV has to be it.

I'm not sure that's a valid conclusion. 45% is about right for FLAC:LossyFLAC irrespective of the source. I did some tests a few months ago when I first came accross LossyWav. I didn't keep most of the results but I do still have one set of test data left. This was the first side of Dire Straits 1st LP (or first 5 tracks if you're looking at the CD).

For CD the figures were:
The uncompressed WAVs were 200 Mb
After FLAC processing (can't remember the level I used) it became 111Mb
Processing Via LossyWav --Standard/FLAC 5 it became 63.2Mb.
That's a reduction of around 43% compared with the FLAC'd data and 68.4%(ish) against the WAVs

For Vinyl the figures were:
The uncompressed WAVs (@16/44.1) were 194.4 Mb (presumably slightly less than CD due to speed inaccuracy of the turntable)
After FLAC processing (again I can't remember the level I used) it became 109Mb
Processing Via LossyWav --Standard/FLAC 5 it became 62.6Mb.
That's a reduction of around 43% compared with the FLAC'd data and 67.8%(ish) against the WAVs

I'd say they're pretty similar.

I also checked the same tracks recorded from vinyl at 24/48 to see what happened. The resultant LossyWav --Standard/FLAC 5 files came out at 64.7Mb. That's about 79% smaller than the original WAVs (which were 315.5Mb) but still bigger than the same files at 16/44.1 albeit by only a couple of Mb. However, I interpreted it to mean that recording from vinyl at 24/48 was worthwhile as LossWav was obviously finding some of the extra data to be worth keeping, although it may only be the potential extra data in the 22 - 24 kHz range, while the resulting files were only marginally bigger than the 16/44.1 ones.

Obviously one can't draw any hard conclusions from a few tracks but at the time I did get similar figures with several other albums in different genres.

[2Bdecided]

If you take a 44.1kHz file and simply resample it to 48kHz, I think you'll see the bitrate go up - losslessFLAC or lossyFLAC. There's no mechanism in either to directly recover the redundancy.

That has some relationship to your experiment, though not an exact one.


As you say though, it's interesting the difference was so small. It's not just the vinyl noise floor that lossyWAV is quantising during the music itself.

Cheers,
David.

This post has been edited by 2Bdecided: Jul 17 2008, 15:00

[botface]

If you take a 44.1kHz file and simply resample it to 48kHz, I think you'll see the bitrate go up - losslessFLAC or lossyFLAC. There's no mechanism in either to directly recover the redundancy.

That has some relationship to your experiment, though not an exact one.


As you say though, it's interesting the difference was so small. It's not just the vinyl noise floor that lossyWAV is quantising during the music itself.

Cheers,
David.

Sorry, I don't understand your final paragraph - presumably my lack of understanding on the subject.

Using 24/48 will result in over 60% more data being produced than 16/44.1. I ended up with about 3% extra data after Lossywav and FLAC had processed it. What surprised me was that there was any extra data left at all and that LossyWav didn't "throw away" all of the extra as I was assuming that it would all be below the noise floor. As I say, I took it to mean that there was some useful data in that extra 60%

[2Bdecided]

Sorry - I didn't write that very well.

What I meant to say is that 24 or 16 probably makes little difference in this case, whereas 44.1 vs 48 will.

You could convert the 24/48 to 16/48 and then throw it at lossyWAV+FLAC and see if there's any difference at all. You might prove my guess to be completely wrong.

Cheers,
David.

This post has been edited by 2Bdecided: Jul 17 2008, 17:15

[raygrote]

Lossy Wav really sounds very interesting. I've got a few questions about it.
1. How do I use Lossy Wav with Foober, or if there isn't a way, is their some kind of frontend for it?
2. If I compress a wav to flac, then run lossy flac, will the resulting file be much smaller than just running lossy wav on the wav file? I assume it wouldn't make too much difference, since to my understanding all lossy codecs use lossless compression in some way. Any info please?
Thanks.

[botface]

What I meant to say is that 24 or 16 probably makes little difference in this case, whereas 44.1 vs 48 will.

You could convert the 24/48 to 16/48 and then throw it at lossyWAV+FLAC and see if there's any difference at all. You might prove my guess to be completely wrong.

Cheers,
David.

Good idea. Interesting result.

I converted to 16/48 using Cooledit. I did it twice. 1st time I used triangular dither and no noise shaping (I didn't want to affect the high frequencies too much). The 2nd time I used no dither or noise shaping. In both cases I ran the resulting file through Lossywav --standard/FLAC -5. In both cases the file ended up at 64.36 Mb.

So, it would seem that most, but not all, of the increase in file size is due to hf content above 22050kHz - assuming my choice of dither/noise shaping was appropriate.

I guess that a much wider choice of material and larger numbers of files would be needed before any real conclusions could be drawn but as I said; interesting

[carpman]

1. How do I use Lossy Wav with Foober, or if there isn't a way, is their some kind of frontend for it?
2. If I compress a wav to flac, then run lossy flac, will the resulting file be much smaller than just running lossy wav on the wav file?

For 1 and 2 see: http://wiki.hydrogenaudio.org/index.php?title=LossyWAV
LossyWAV won't make the WAV file smaller, it's a pre-processor for lossless encoders, it processes the WAV file to make it easier for FLAC et al to compress it. The WAV pre-processing is lossy; the subsequent lossless encoding (i.e. to FLAC or TAK) is lossless.

C.

[EDIT: Typos ; EDIT2: Clarity ; EDIT3: Clarification -- OMG!!!]

This post has been edited by carpman: Jul 17 2008, 20:34

[Nick.C]

^^ What he said....

[carpman]

Don't listen to me, just read the wiki; it's clearer and better written.

C.

[raygrote]

Thanks for info.
It still doesn't work with Foober, though. I'm using Foober 0.9.5 and Lossy Wav 1.1.0. I just copied and pasted the commandlines posted on the wiki for flac files, and adjusted everything the way they said, all I did was change the paths to the lossywav and flac files. I set encoder to c:\windows\system32\cmd.exe, extention to lossy.flac, format is lossless )or hybrid), and max bitrate supported 24.
The commandline I enter is:
/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d
And the error I get is
"Source: "E:\data files\audio\abx1.wav"
An error occured while writing to file (The encoder has terminated prematurely with code 1; please re-check parameters) : "E:\data files\audio\abx1.lossy.flac"
Additional information:
Encoder stream format: 44100Hz / 2ch / 16bps
Command line: "C:\WINDOWS\SYSTEM32\cmd.exe" /d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o"abx1.lossy.flac"
Working folder: E:\data files\audio\

Conversion failed: The encoder has terminated prematurely with code 1; please re-check parameters"


I just know I did something wrong.

[carpman]

I set encoder to c:\windows\system32\cmd.exe, extention to lossy.flac, format is lossless )or hybrid), and max bitrate supported 24.

Well that's all correct.

The commandline I enter is:
/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d

Above I've highlighted in red what may be a problem. Try it with a space between -o%d, so it's:
-o %d

Or, paste this into your parameters:

CODE

/d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o %d

Try that first.

C.

[EDIT1] By the way Nick, if that is the problem the wiki needs changing as that's also in the wiki setting for FLAC.
[EDIT2] Just ran mine with -o%d (i.e. without the space) on foobar 9.4.3 and also 9.5.4 and it worked okay. I don't have any spaces (thus e:\"data files"\etc\etc ) in my encoders' addresses. So I'm out of ideas.
[EDIT3] Just ran it with identical parameters including no space with -o%d and with an address with spaces enclosed in " " and it worked fine.

This post has been edited by carpman: Jul 17 2008, 22:19

[halb27]

... /d /c e:\"data files"\codecs\lossywav - --standard --silent --stdout| e:\"data files"\codecs\flac - -b 512 -5 -f -o%d ...

Guess it's the "data files" which causes the problems, or the space between the | and the flac path.
To find out I'd (temporarily?) use a lossywav/flac path like: e:\codecs\lossywav.

This post has been edited by halb27: Jul 17 2008, 23:04

[sauvage78]

-o%d is ok, --stdout| e: maybe is not ... try --stdout|e: without space

This post has been edited by sauvage78: Jul 18 2008, 00:04

[Dynamic]

So, it would seem that most, but not all, of the increase in file size is due to hf content above 22050kHz - assuming my choice of dither/noise shaping was appropriate.

Thanks to the way lossless encoders work, it's almost certainly down to the increased number of samples, because lossyWAV should ignore extra content above 16 kHz (and in any case, good sample rate conversion shouldn't add significant HF content not in the original). The fact there's no difference in bitrate between the two dither types when processed by lossyWAV is confirmation that your dither was below the original noise floor.

There are almost 9% more samples per second at 48kHz than at 44.1kHz, but with limited frequency response, it probably results in lower prediction error per sample (the error is what has to be stored by a lossless encoder), though this is not compensating fully for the 9% increase in number of samples per second.

[raygrote]

I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?

[Nick.C]

I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?

I have noticed that if I switch my server on and try to convert files with foobar2000 immediately I hear it finish its boot then the foobar2000 conversion will fail. However, if I use windows explorer to "look at" the root directory of the external drive where the audio files are first then the foobar2000 conversion will work perfectly.

[botface]

I tried all suggestions, nothing is working. I'm starting to think this may have something to do with cmd.exe? Later I'll try moving the codecs to the directory used in the wiki for the heck of it and see what happens.
Perhaps CMD.exe has problems with external hard drives?

I had a few problems when I had spaces in the directory, quotes or no quotes. So, I moved LossyWav and FLAC to a directory with no saces in the name (C:\auydio\lossywav) and it's worked perfectly ever since. In case it's of any interest here's my command line :
Parameters: /d /c c:\audio\lossywav\lossywav - -q 5 --silent --low --stdout|c:\audio\lossywav\flac - -b 512 -5 -o%d

(Yes, I'm still using the old "q" paramter instead of --standard).

I guess this shows how useful it would be if someone could knock up a GUI front end

[2Bdecided]

There are almost 9% more samples per second at 48kHz than at 44.1kHz, but with limited frequency response, it probably results in lower prediction error per sample (the error is what has to be stored by a lossless encoder), though this is not compensating fully for the 9% increase in number of samples per second.

That's right. Even if you resample 44.1kHz to 48kHz (so there's no increase in the actual information - the extra frequency range is completely empty) the lossless bitrate will still go up. Most lossless codecs simply aren't designed to make use of this kind of redundancy (I think TAK might though).

Cheers,
David.

[Synthetic Soul]

Using Nick's figures above, it seems it is much of a muchness, although WavPack seems to benefit a little less (as can be seen by FLAC actually producing smaller bitrates with most settings):

CODE

             +------------------+------------------+------------------+
             |    TAK           |    FLAC          |  Wavpack         |
+------------+------------------+------------------+------------------+
| lossless   | 749 kbit/s       | 787 kbit/s        | 772 kbit/s      |
+------------+------------------+------------------+------------------+
| --insane   | 634 kbit/s (85%) | 654 kbit/s (83%) | 661 kbit/s (86%) |
| --extreme  | 564 kbit/s (75%) | 583 kbit/s (74%) | 599 kbit/s (78%) |
| --standard | 488 kbit/s (65%) | 508 kbit/s (65%) | 529 kbit/s (69%) |
| --portable | 405 kbit/s (54%) | 425 kbit/s (54%) | 444 kbit/s (58%) |
| -q 0       | 299 kbit/s (40%) | 321 kbit/s (41%) | 342 kbit/s (44%) |
+------------+------------------+------------------+------------------+

In addition, here's some figures for the corpus that I use for my lossless comparison:

CODE

              TAK               FLAC              Wavpack
================================================================
lossless      917 kbps          941 kbps          938 kbps
================================================================
--insane      629 kbps (69%)    639 kbps (68%)    661 kbps (70%)
--extreme     538 kbps (59%)    548 kbps (58%)    574 kbps (61%)
--standard    448 kbps (49%)    459 kbps (49%)    485 kbps (52%)
--portable    362 kbps (39%)    375 kbps (40%)    394 kbps (42%)

As per Nick's results, the difference between codecs is minimal; however my corpus benefits a lot more from LossyWAV preprocessing.

Once again, thank you all for your hard work.

[halb27]

... however my corpus benefits a lot more from LossyWAV preprocessing. ...

The results for your corpus are pretty similar to the results of my small 12 full length regular track test set, and these are very similar to those of Nick's many-albums test set.
If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people.

This post has been edited by halb27: Jul 18 2008, 12:37

[Nick.C]

If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people.

As we're past the second full release, I'll produce a similar table to that above using my 10 album test set. The reason I stick to the 55 problem sample set is that it is (relatively, as I have added a few samples to it over the last six months) constant and relative changes in bitrates at quality presets can be compared. No need to stick to that now - I'll publish some bitrates tonight.

[Nick.C]

If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people.

As requested - results for my 10 album test set:

CODE

             +------------------+------------------+------------------+
             |       TAK        |       FLAC       |     Wavpack      |
+------------+------------------+------------------+------------------+
| lossless   | 820 kbit/s       | 854 kbit/s       | 852 kbit/s       |
+------------+------------------+------------------+------------------+
| --insane   | 615 kbit/s (75%) | 632 kbit/s (74%) | 641 kbit/s (75%) |
| --extreme  | 532 kbit/s (65%) | 548 kbit/s (64%) | 563 kbit/s (66%) |
| --standard | 447 kbit/s (55%) | 463 kbit/s (54%) | 481 kbit/s (56%) |
| --portable | 359 kbit/s (44%) | 376 kbit/s (44%) | 390 kbit/s (46%) |
| -q 0       | 266 kbit/s (32%) | 285 kbit/s (33%) | 296 kbit/s (35%) |
+------------+------------------+------------------+------------------+

Wiki article indicative bitrate table amended to reflect these results.

This post has been edited by Nick.C: Jul 19 2008, 20:38

[halb27]

If there's one thing I don't like with Nick's great work it's his preferred usage of his test set consisting of problem snippets for the biggest part of it which I'm afraid leads to a wrong impression to most people.

As requested - results for my 10 album test set:

CODE

             +------------------+------------------+------------------+
             |       TAK        |       FLAC       |     Wavpack      |
+------------+------------------+------------------+------------------+
| lossless   | 820 kbit/s       | 854 kbit/s       | 852 kbit/s       |
+------------+------------------+------------------+------------------+
| --insane   | 615 kbit/s (75%) | 632 kbit/s (74%) | 641 kbit/s (75%) |
| --extreme  | 532 kbit/s (65%) | 548 kbit/s (64%) | 563 kbit/s (66%) |
| --standard | 447 kbit/s (55%) | 463 kbit/s (54%) | 481 kbit/s (56%) |
| --portable | 359 kbit/s (44%) | 376 kbit/s (44%) | 390 kbit/s (46%) |
| -q 0       | 266 kbit/s (32%) | 285 kbit/s (33%) | 296 kbit/s (35%) |
+------------+------------------+------------------+------------------+

Wiki article indicative bitrate table amended to reflect these results.

Thank you, Nick. A near-perfect match with Synthetic Soul's results as well as mine (371/462/542 kbps for --portable/--standard/--extreme when using FLAC).