Hello everyone,

I have the idea of doing a very comprehensive lossless codec comparison, because according to the Hydrogenaudio Wiki the last was about 2,5 years ago. There are some problems which have to be solved, so I ask your help

First, how do I measure the time an encoder/decoder needs to encode/decode? I run Linux, so I was thinking about GNU's time, but would that be unfair as TAK runs on Wine? Wine is not an emulator, so is the overhead probably fixed (in seconds of the total) or negligible? Is there a program similar to time on Windows? There are also some encoders which won't run Wine at all.

Second, what am I going to measure? User CPU time, User + System CPU time, Realtime? I was thinking about both User+ System and Realtime. Probably it is an idea to run these encodings/decoding fully in memory? (via ramfs or something similar) Also I thought measuring the memory usage via time is interesting.

Finally, how about the tracks to encode? I have much metal (quite many different subgenres), much classical, some jazz, rock, some pop, but probably it would be interesting to see how these codecs encode 24-bit or 96kHz material (as I heard FLAC does quite bad comparing to Wavpack)

You may be interested in the following thread:

http://www.hydrogenaudio.org/forums/index....showtopic=50954

I'm not sure that there is a whole lot for you there, as I test on Windows; however if you are interested in my limited experiences in the field, and like to see a pretty graph now and again, it may be worth skimming over.

How are you planning to display the results? I've been meaning to upgrade my comparison for a long time. One thing I'd like to do is to enable users to filter results by various criteria, including genre. If you test a range of genres, and have a decent amount of tracks for each, you could provide stats for any genre, plus stats for a mix of some, or all. Similarly, you could allow users to filter by bit depth. This of course depends on a display format which allows for filtering. If this cannot be done then I would say that you really need to cover as many genres as possible. I'm not sure about merging 16 and 24 bit results: most users are only going to be interested in results for 16 bit (CD) audio. It would be great to see a comparison of 24 bit tracks, but I think results for 24 bits should be kept separate, and I'd say they'd be of less interest to most people.

Finally, after a lot of scripting, here are my first test results. I used a selection of Nightwish' album Oceanborn, the tracks Stargazers, Devil & the Deep Dark Ocean, Sacrement of Wilderness, Passion and the Opera, Moondance and The Pharaoh Sails to Orion. I made a selection because I run all tests from ramdisk (to remove the harddisk as a bottleneck) and I have only 1GB of ram.

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

Now my question is: have I got all the encoders with representable settings? I had some trouble choosing some settings for OptimFROG and MPEG-4 ALS. For codecs such as OptimFROG, I won't test every possible setting: it is way too slow! I'll add Real Lossless, ALAC and WMAL soon, as they have only one setting. As you can see, MPEG-4 ALS gives a SEGFAULT, but only at the very beginning of the track Sacrement of Wilderness... Anyone knows why? There is nothing special with this tracks and its headers AFAIK.

I'm afraid you forgot wv normal mode.

nice work

http://www.icer.nl/test%20-%20Nightwish%20-%20Oceanborn.pdf

Added wavpack normal and ALAC. I tested ALAC using dBpoweramp on Wine, but I can't get Real and WMAL to work, so I'll test these on Windows. When these tests are complete, I'll write some scripts to generate graphs

Thanks for including wv normal.
According to my experience differences in compression ratio between lossless codecs can be more severe with music of a more quiet kind (many kind of classical music especially when done with very few instruments, singer/songwriter resp. folk music).
It would be great if you could add also results for music of this kind.

Okay, I'll write down a list of music I want to test. I'll make selections of about 300MB uncompressed, otherwise it won't fit on the ramdisk while not having to swap the real processing out

- Nightwish - Oceanborn (Symphonic metal)
- Mercenary - 11 Dreams (Melodic Death Metal, 6 out of the 12 songs with the highest bitrate in my FLAC collection come from this album)
- Apocalyptica - Inquisition Symphony (Metal, but just cello's playing)
- Nine Inch Nails - The Slip (Industrial rock)
- Nickelback - Dark Horse
- Tiësto - In Search of Sunrise: Asia
- Coldplay - Viva la Vida or Death and all his Friends
- Britney Spears - Circus (Pop)
- Avril Lavigne - Let Go
- Katie Melua - The Katie Melua Collection
- Various - Aangenaam Jazz 2007 (a selection of jazz music, with vouchers to get discount on the featured CD's... I think this should be quite all-round )
- BLØF - Oktober (quite serene, acoustic dutch pop)
- Howard Shore - The Lord of The Rings: The Return of The King
- Piet Jeegers Clarinet Choir - Volume 4 (A Clarinet Choir playing classical music)
- Gustav Holst - The Planets (Boston Symphony Orchestra feat. conductor: Willian Steinberg)

- Nine Inch Nails - The Slip (24-bit/96kHz)

I think I'll add some more today Please comment on missing genre's... I'll search for some hardcore probably, some more classical and some more solo-instrumental. (mainly Classical)

Edit, Added:
Sting - Nothing Like The Sun (Vinyl Rip, no restoration or filters used)
Vivaldi's Four Seasons by I Musici (Vinyl Rip, no restoration or filters used)

Edit2
I just reread the thread.



Yes, that was my idea too

Very welcome.
With a lot of pop/rock tracks you improve the quality of your statistics for pop/rock music.
When adding classical music, you widen the scope of music under investigation.
When adding quiet tracks of few instruments only, you probably will give some insight that codecs' compression ratio can vary a lot more than from what we expect when considering pop music.

Wouldn't testing ALAC with ffmpeg be more interesting (and easier to automate, and crossplatform)?

It would be way easier to automate indeed, but will it reflect the capabilities of ALAC? I mean, iTunes ALAC codec will be used more? This program doesn't use its own implementation of ALAC, it works via DLL plugins AFAIK

Please also consider testing the EBU Sound Quality Assessment Material (SQAM) CD. This is a "standard" set of test items often used for audio codec evaluation in scientific publications. It contains a lot of single instruments and some classical and 80's pop music.

http://www.ebu.ch/en/technical/publication...h3253/index.php

Thanks,

Chris

Edit: Unfortunately, the music items are not available for download, but the rest should suffice anyway.

I will

Okay, after a second test round I strike off OptimFROG's --maximumcompression and --maximumcompression --experimental as they are so extremely slow, they take 3x realtime. That's not fair and absolutely not useful. Also I'll no longer test MPEG-4 ALS -7 -z3 -p as it keeps throwing SEGFAULT errors. Now I can make some graphs.

dpoweramp has its own ALAC implementation as well, or are you referring to the old iTunes plugin?

Oh, okay, I didn't know that. Then I'll test the latest iTunes and FFmpeg as well.

FLAC -8 -A tukey(0.5) -A flattop can bring +0.1% compression comparing to -8

For Flac there are some hints that there might be non-official settings that could be interesting for end users.

I am specially thinking of:
-l 8 -b 4096 -r 6
-l 8 -b 4096 -M -r 6

These are tuned version of -6 playing with the -M & -m switch, nobody ever tested those settings but if you analyse the switchs behing the flac settings, & compare those switchs to the various compression/speed comparisons out there, there is some hints that "maybe" one of these switchs would achieve compression on par with -5/-6 & yet maybe be faster (it's a trial to swallow some of -4 encoding speedup & some of -3 decoding speedup, increasing decoding speed of -5/-6 is specially interesting for use with cuetools/accuraterip, increasing encoding speed is less interesting IMHO because you cannot encode faster than you rip or decode anyway).

Here is a small table/screenshot of my analyses of the flac switches that leaded me to this conclusion:


I tried to test it for myself but my methodology was obviously not good as sometime it was faster to decode & sometime it wasn't, I think that this is due to my HDD, specially where the files were written on the HDD.

I may be completely wrong & this may be completely useless in the end, but maybe it is worth to try it ...

Anyway good luck with your test.

For some of my test tracks, the files grow even bigger then just -8! I'm not interested in such tweaks for now.



Well, probably I'll test them when these comparison is complete.

np I can live with it, just don't put -l 8 -b 4096 -r 6 & -l 8 -b 4096 -M -r 6 in the same bag as -8 -A tukey(0.5) -A flattop ... I favor speed over compression , specially decompression speed, my switches are expected to compress very close to -5/-6 & maybe be faster. Anything using the -e switch is for people who don't understand lossless IMHO, because lossless is made to be transcoded sooner or later. The problem is that I suspect -m/-M to be a kind of smaller -e. (Lots of wasted time for near zero compression gain).

I won't, I can see the difference They're just both not interesting for me right now.

Also It's usefull to see that encoding from FLAC -8 to LAME is only 1.12% slower than from FLAC -0. There are a lot of bottlenecks like HDD, RAM etc in real encoding scenarios.
http://www.hydrogenaudio.org/forums/index....st&p=608813

Despite decoding FLAC -0 is 22% faster than -8 but encoder LAME is 15x slower than decoding FLAC that's why all speed benefit of -0 is masked only to speed boost +1.12%.

Because of that, I worked from a ramdisk, to eliminate the harddisk as a bottleneck. Indeed, in real scenario's they'll influence encoding, but as eveyone has a different harddisk and a different processor, I'll focus on CPU power, not on harddisk usage.

My first graphs are ready They're based on the results of the first 2 sets of testtracks. Now the question is: what should I do to enhance readability (or is it clear enough?) The graphs are created with JpGraph, many thanks to the authors of this marvelous piece of software ^^

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

Out of curiosity, why use this version of the track? Wasn't it shown previously that the 24/96 files were the same as 16/48 with padding?

It took me a small amount of time to realize what the axes mean. So I'd prefer to explicitly refer to compression ratio and speed.
I'd prefer to give the compression ratio in percent.
I prefer the linear scale of the speed axis. Reader's emotions and speed facts are more in line this way. I'd accept the disadvantage that the very slow codecs are shown with restricted precision this way.

Agreed. Please label the axes. I assume the horizontal axis stands for speed and the vertical for compression ratio?

Also, speaking of ratios in percent, for better comparison, I suggest normalizing the speed axis to the results of the Shorten codec (since that's a classic reference, so to speak) and then to format it in percent as well, i.e. Shorten itself gets 100%, the fastest Flac mode gets nearly 200%, etc.

By the way, thanks for the nice work so far!

Chris

Thks a lot for your test, I think it will affect my collection because I think I will switch from -6 to -4, it confirm what I was suspecting about-m & -M, -M is still interesting & removing it (like -3 or -0) will decrease the compression, but -m is just like -e ... a waste of encoding time for no gain (both -4 & -1 show it). This test made me realize that I was underestimating -M, in the switch I proposed to you I wanted to remove it which was not clever.

To be honest, it is not your test alone which will make me switch, it is the right time for me, I will soon have a new HDD & I need to fix offset on my collection with cuetools, but your test is a nice confirmation of my doubt about the usefullness of the -m switch which is used in -5/-6.

Here is how I analyse your flac result personnaly:


As for the way of displaying result, I think you can just use the same display as this test:
Hans Heijden's
It is pretty much perfect IMHO.

Just add encoders version on top, abbreviations of settings inside, also don't use the same color when result overlaps (tak & wv are green & mixed in your grap) & change the axis.

I'll try to do something similar

I was already busy finding out how to add labels, and I found out Why should I scale to a reference codec instead of just plain wave? Now the axis inform you about how much a codec rougly compress, then the axis will inform you about how the codec outperforms Shorten... That's not really interesting, is it?

Because it is the only 24-bit track I have... I didn't know it was fake. If it just padded, I won't use it I have a 24-bit/96kHz capturing soundcard and a 24-bit/96kHz outputting guitar effect pedal, would that qualify for such a test?

edit: Moved

I added new music too. An about my harddrive: it is about 150 times realtime when copying from harddisk to ramdisk...

Because the information "times realtime" that you have now is a relative one, since it depends on your CPU. I was thinking of a way to make this an absolute measure. I agree that's not optimal either, so I guess we can stick with what you have. But then please provide your CPU brand and model, preferably in the axis label. That way I can estimate the speed for my system.

Chris


Edit: Now I get it, it seems you misunderstood me. I was only talking about the horizontal axis being put in relation to Shorten. The compression ratio (in relation to Wave) is fine the way it is

Yes, I did I was already thinking of making the 'times realtime' some more practical... My CPU is a AMD Turion ML-34, 1800MHz, 1MB cache, and runs on 64-bit Linux, but I don't think such information would be interesting to put in the graph, as MHz don't say anything this age. My idea was to run this test (partly) on more than one computer (I have three computers with quite different CPU's here) so I can look which universal benchmark would match.

Thanks, Turion ML-34 is already enough information



Good idea. Do you have a slow(er) system available? Something in the range of an Atom, or a classic Athlon/Pentium III 1.x Ghz? Would be interesting, since Netbooks are gaining in popularity.

By the way, which albums/songs are included in your last posted analysis? I'm surprised that even the best-compressing codec doesn't reach a ration of 2/3! Has modern music really become that noisy?

Christian

I thought the album was rereleased a short time later with the correct files which were correct?

I have:
- This Turion ML-34
- A Server Pentium 3 of which I currently don't know the exact specs
- An Intel Celeron D, 64-bit

I (well, it is not mine, it is ours here) have a Celeron 400MHz, but it is being used actively, so I have to run this tests if they allow me

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

As you can see, it is not balanced yet. Next I'll add some Jazz, and after that Classical. And yes, most metal (Mercenary in particular, you can listen some tracks here, see the albums 11 Dreams and The Hours That Remain) is very hard to compress. BTW, I added ALAC too. Don't forget to use F5, as I used the same filenames.

If someone can confirm that The Slip is indeed true 24-bit/96KHz (I downloaded it just a few weeks ago) I'll test these seperate

About NIN/slip I recall I re-downloaded it because there was a problem which was fixed, Trent even said thks you. I think it's OK now if I recall well.

See Here

Updated the test again, visit http://www.icer.nl/losslesstest for the results. Please take a look at "What I'm going to do next..." and comment

I just finished the tests with instrumental material. I used the SQAM tracks, some tracks from Best Of Chesky Classics & Jazz and Audiophile Test Disc Volume 3 and some tracks from the Good bye Lenin Soundtrack and some tracks from Le Fabuleux Destin d'Amélie Poulain. In these tracks, there is just one instrument playing at a time (or someone speaking, or silence)

The results suprised me. LA doesn't outperform OptimFROG here, and some codecs do remarkable better (and some other remarkable worse) For example, see ALAC and the first 4 settings of FLAC... what a difference compared to the graphs at http://www.icer.nl/losslesstest! Also decoding shorten is much faster. Wavpack does quite bad in this test, probably because it needs filters (-x settings) to efficiently code this material? Something went wrong decoding the ape files. (that has to do with timing, I don't know what currently)

Maybe I didn't read the results closer enough but I noticed that there was only 7 points for flac when there are 9 flac settings, even considering that 5 & 6 are tied, there is one missing setting ( or a second tied ?). Plz put the settings & encoder versions abbreviation within the table, because it becomes unreadable without the numbers at hand: with a missing setting I don't know which point is which setting anymore. (Either flac -0 or -1 or -2 seems missing or tied from my understanding, results from -8 to -3 seems logic)

Indeed, I see I forgot flac -0 for some strange reason... that's why these graphs were so different I'll add them soon. (in about one hour I think)

Thanks a lot, ktf! Interesting results, and somewhat unexpected for me. So regarding your To-Do list: yes, I think you should keep plotting the instrumental results separate from the previous ones. The compression performance is very different.

Chris

Updated. I also found the bug in my script which was causing the wrong values for APE, it's fixed now

Well gentlemen, indeed three years have passed since I last updated my lossless comparison page. As you read I'm still alive and well! Only my interests shifted a little.

Since I probably won't update the lossless page in the near future, thought some of you might be interested in the excel sheet behind the graphs. And also in an example of a batch file used to run a compress/decompress session. Here you can temporarily find a zip file with both of them.