Is a process really lossless? Options

[evereux]

Since there seems to be a steady stream of people claiming that a lossless process isn't lossless I've decided to write a mini guide showing how the user can themselves check to see if a process was indeed lossless. Hopefully this will help some gain a better understanding of what lossless is.

For the purposes of this mini tutorial I'll be compressing a file using Wavpack. You can use this methodology for any lossless process, you just need to know how to decompress that file back to wav.

The guide below will be specific to Windows OS's but the principles are true for any other OS. The text within the code boxes are copied from my DOS prompt. To access your command prompt go to Start > Run > Cmd.


Create a new folder in root called lossless and put in this folder your wav file.

CODE

C:\lossless>dir
Volume in drive C has no label.
Volume Serial Number is AC06-24BF

Directory of C:\lossless

04/08/2005  09:20    <DIR>          .
04/08/2005  09:20    <DIR>          ..
04/08/2005  09:19        50,862,044 01._Neighborhood_#1_(Tunnels).wav
              1 File(s)     50,862,044 bytes
              2 Dir(s)   1,017,434,112 bytes free

C:\lossless>

Compress your wav file with your lossless compressor.

CODE

C:\lossless>wavpack -h 01._Neighborhood_#1_(Tunnels).wav

WAVPACK  Hybrid Lossless Wavefile Compressor  Win32 Version 4.2  2005-04-02
Copyright (c) 1998 - 2005 Conifer Software.  All Rights Reserved.

created 01._Neighborhood_#1_(Tunnels).wv in 12.38 secs (lossless, 33.34%)

C:\lossless>

Now we have our compressed file you'll need to rename this file so that when we come to decompress the file it will not over write the original file.

CODE

C:\lossless>dir
Volume in drive C has no label.
Volume Serial Number is AC06-24BF

Directory of C:\lossless

04/08/2005  09:26    <DIR>          .
04/08/2005  09:26    <DIR>          ..
04/08/2005  09:19        50,862,044 01._Neighborhood_#1_(Tunnels).wav
04/08/2005  09:22        33,905,312 01._Neighborhood_#1_(Tunnels)[new].wv
              2 File(s)     84,767,356 bytes
              2 Dir(s)     983,523,328 bytes free

C:\lossless>

Next we will decode the .wv file.

CODE

C:\lossless>wvunpack "C:\lossless\01. Neighborhood #1 (Tunnels)[new].wv"

WVUNPACK  Hybrid Lossless Wavefile Decompressor  Win32 Version 4.2  2005-04-02
Copyright (c) 1998 - 2005 Conifer Software.  All Rights Reserved.

file C:\lossless\01. Neighborhood #1 (Tunnels)[new].wv not found!

C:\lossless>wvunpack "01._Neighborhood_#1_(Tunnels)[new].wv"

WVUNPACK  Hybrid Lossless Wavefile Decompressor  Win32 Version 4.2  2005-04-02
Copyright (c) 1998 - 2005 Conifer Software.  All Rights Reserved.

restored 01._Neighborhood_#1_(Tunnels)[new].wav in 8.46 secs (lossless, 33.34%)

C:\lossless>

You should now have the new decompressed file in your folder.


Now comes the essential part in being able to check if a process is lossless. We do this by running a binary file compare using Windows' FC function. What this does is compare two files and if there are any differences they will be printed to the screen.

CODE

C:\lossless>FC /B C:\lossless\01._Neighborhood_#1_(Tunnels).wav C:\lossless\01._Neighborhood_#1_(Tunnels)[new].wav
Comparing files C:\LOSSLESS\01._Neighborhood_#1_(Tunnels).wav and C:\LOSSLESS\01
._NEIGHBORHOOD_#1_(TUNNELS)[NEW].WAV
FC: no differences encountered


C:\lossless>

As you can see, the above process was perfectly free from any kind of loss and we can be confident this is the case since FC reports no differences encountered.

edit: command changed to FC /B, thanks zZzZzZz

This post has been edited by evereux: Aug 4 2005, 10:03

[evereux]

There are times when using FC /B file1.ext file2.ext from the command prompt will yield different results after compressing a file with a lossless compressor.

There is a very good chance that the reason for this is that the original wav file contained some additional header informations (not related to the audio stream). Once the file is losslessly compressed and then decompressed these header informations are changed or lost.

We can use a tool like Foobar2000's bitcompare to just check the audio stream. To do this you will need to download and install Foobar2000.

Load your 2 wav files into Foobar2000 by simply dragging and dropping the files into Foobar2000's main window.

Highlight your two files by doing CTRL+A.

Right click whilst your files are highlighted and select Bit Compare. See image > foo_bitcompare

Once the comparison is finished the console window will open and you should be presented with the following information. See image > foo_bitcompare results
As you can see, the console window reports:
INFO (foo_bitcompare) : No differences in decoded data found.
INFO (foo_bitcompare) : Finished successfully.

This post has been edited by evereux: Aug 4 2005, 11:17

[Peter]

I believe the commandline should be FC /B rather than just FC, otherwise it compares the files in text mode.

[archagon]

You LIE! It's all an Illuminati conspiracy!

[kl33per]

What if the WAV headers change. Won't this lead to file's not comparing acurately, even though the audio is exactly the same.

[Defsac]

What if the WAV headers change.  Won't this lead to file's not comparing acurately, even though the audio is exactly the same.

Yes. foobar2000's bitcompare would probably be a better option because I believe it only compares actual audio data.

[pest]

What if the WAV headers change.  Won't this lead to file's not comparing acurately, even though the audio is exactly the same.

wavpack stores the complete waveheader and additional chunks...flac doesn't
so this might not work with partially broken wave-files...

'fc /b' is the right command

[PoisonDan]

What if the WAV headers change.  Won't this lead to file's not comparing acurately, even though the audio is exactly the same.

True. For a more reliable comparison, something like foobar2000's bitcompare function should be used. The problem is, when you mention this in the guide you're expecting the user to install foobar2000, while FC is readily available from the OS.

Related to this, do you know of a command-line tool that works similar to foobar2000's bitcompare (i.e. only compare audio data)?

[Defsac]

Related to this, do you know of a command-line tool that works similar to foobar2000's bitcompare (i.e. only compare audio data)?

WavPack will compute and verify MD5 sums during compression and decompression if you use the '-m' argument, you don't need to use an external tool for verification.

This post has been edited by Defsac: Aug 4 2005, 10:09

[bug80]

Hm, interesting. First, let me state that I believe that lossless = lossless, of course.

However, this kind of evidence is a bit tricky. You use one file, which is encoded lossless, to prove your statement. But, how do you prove it works for all files?

I would say, lossless compression is really lossless, untill someone comes up with a sample on which it doesn't work (innocent untill proven otherwise).

This post has been edited by bug80: Aug 4 2005, 10:34

[PoisonDan]

However, this kind of evidence is a bit tricky. You use one file, which is encoded lossless, to prove your statement. But, how do you prove it works for all files?

You can't. That also why evereux specifically wrote:

a mini guide showing how the user can themselves check to see if a process was indeed lossless.

You can prove it for one particular process, but not for everything.

[Sunhillow]

However, this kind of evidence is a bit tricky. You use one file, which is encoded lossless, to prove your statement. But, how do you prove it works for all files?

This would be a bug in the compression algorithm - just like if a .exe had differences after packing and unpacking process.

[evereux]

What if the WAV headers change.  Won't this lead to file's not comparing acurately, even though the audio is exactly the same.

True. For a more reliable comparison, something like foobar2000's bitcompare function should be used. The problem is, when you mention this in the guide you're expecting the user to install foobar2000, while FC is readily available from the OS.

This is why I've used FC for the guide above. If one wav file contains header information what will be outputted to the terminal screen using FC? I'm sure it would be quite obvious that it's only the header data that's different?

Related to this, do you know of a command-line tool that works similar to foobar2000's bitcompare (i.e. only compare audio data)?

WavPack will compute and verify MD5 sums during compression and decompression if you use the '-m' argument, you don't need to use an external tool for verification.

My example uses wavpack but it wasn't my intention for this process to be specific to wavpack. The question PoisonDan asks should be answered in a manner applicable to all lossless processes.

Hm, interesting. First, let me state that I believe that lossless = lossless, of course.

However, this kind of evidence is a bit tricky. You use one file, which is encoded lossless, to prove your statement. But, how do you prove it works for all files?

It's up to someone to show us otherwise instead of blindly stating so. My first post here helps the user to do that.


I'll edit my second post to include Foobar2000s bit compare feature sometime soon.

[kjoonlee]

Lossless compression is a mathematical process.

A mathematical proof would be possible, I'm sure.

[A_Man_Eating_Duc...]

Could we get this stickied?

This post has been edited by A_Man_Eating_Duck: Aug 4 2005, 10:46

[pest]

Could we get this stickied?

yeah for the unbelievers

[Acid8000]

http://www.hydrogenaudio.org/forums/index....showtopic=36043

This somehow gave me an idea. How about, making a .png of a moir pattern, converting to wav, compressing losslessly, then doing the reverse and seeing if it looks the same.

[TCM]

How about, making a .png of a moir pattern, converting to wav, compressing losslessly, then doing the reverse and seeing if it looks the same.

hmm, what's the point? most people don't normally convert their music to png.

in a process like png -> wav -> flac/wv/... -> wav -> png where the two wavs are identical, what additional proof would the png bring? of course, this is ignoring the fact that you assume correct operation of the png compression.