Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lossless Compression Performance Test (Read 6088 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Lossless Compression Performance Test

In view of the recent flurry of Public Listening Tests, and also the new versions of lossless compressor versions, I think it is high time to arrange for a Lossless Compression Performance test.

Now my song collection is kinda limited, genre-wise, so I'd like to enlist the help of all hydrogenaudio members to arrange for a Lossless Compression Performance test. Here's how you can participate:

For Developers/Maintainers:
Please post below, indicating your latest version and a link on where to download the compressor.

For Users:
Download the following Excel spreadsheet (compressed with WinRAR 3.51), enter your data inside, and post it back to me (my email address is inside the spreadsheet)
You may send one Excel file per song, or (preferred) put the results for your tests all in one Excel file.

Excel Spreadsheet for submitting results

Results:
- Performance results will be posted in the HA Wiki. Expected first result: April 21, 2006.
- Statistics will be posted regularly in this thread and also in the HA Wiki.

This is not a listening test; ABX is not necessary!

This test is perpetually open.

From time to time, I will update this, and post the latest result. Especially when newer Lossless Compressors are released.

Lossless Compression Performance Test

Reply #2 –
I have a couple of questions too.

I have used Monkey's Audio for years, but only occasionally the other lossless codecs.

What are the most popular, useful and sensible compression settings for FLAC,  WavPack and OptimFROG and why?

Should the hybrid / dual stream modes be tested too?

WMA lossless and iTunes & dBpowerAMP ALAC are easy since they have only one option.

Would it be useful to test Shorten too? Does anyone use it outside the archive.org?

Lossless Compression Performance Test

Reply #3 –
Quote
Can I also submit results for complete CD images? I guess so, but just want to be sure. [{POST_SNAPBACK}][/a]

It is an audio track like any other tracks. Only the duration is longer than separate tracks usually have.

Edit:

I have noticed that Monkey's Audio does not effectively skip digital silence in some circumstances. I have tested the digital silence handling behavior of FLAC and Monkey's Audio and posted the results here: [a href="http://www.hydrogenaudio.org/forums/index.php?showtopic=41622&view=findpost&p=365879]http://www.hydrogenaudio.org/forums/index....ndpost&p=365879[/url]
(I also reported the problem at Monkey's Audio forum: http://www.monkeysaudio.com/smf/index.php?...g8430#msg8430.)

However, this Monkey's Audio problem is significant only with 24-bit files. It doesn't exist with 16-bit files

The total amount of digital silence included can make a minor difference when a disc image is ripped and encoded instead of the separate tracks. I think that may happen if a silent pregap before track #1 is included in the disc image.

I don't know how the other lossless encoders besides FLAC and MA handle digital silence.

Lossless Compression Performance Test

Reply #4 –
Sorry for not answering earlier. My eyes got infected by some kind of virus and I was bed-ridden for several days.

Sebastian: Yes submit your results on CD images by all means.

Alex B: I don't think Hybrid and/or DualStream is necessary as the aim is for archival.

With that note, let me stress that the aim is to generate files for archival purposes. Thus encoding/decoding time may/may not be important.

I myself use the following tests, just for the sake of satisfying my curiosity:
(sorry I don't recall exactly some switches... will fix later)

flac -5
flac -8
flac --super-secret-totally... (whatever)
flac ... (sorry don't remember, but it has all parameters maxed out without needing --lax)
ofr
ofr --mode best
ofr --mode extranew
ofr --mode bestnew --seek slow
la
la -high
wavpack -h
wavpack -hx
wavpack -hx6

Oh and a question to Guido: In the readme it mentions a switch for evaluation purposes (I forget). I assume that switch generates the maximum possible compression. How do you use that switch?

After measuring the time for encoding and decoding (I do them at least twice and take the smallest time), I purposefully introduce a running bit error (i.e. replacing a portion with FF's or 00's) and test the algorithm's capability of error detection and/or correction.