Time for a new lossless codec comparision?, With some more points of testing perhaps :) Options

[ktf]

Hello everyone,

I have the idea of doing a very comprehensive lossless codec comparison, because according to the Hydrogenaudio Wiki the last was about 2,5 years ago. There are some problems which have to be solved, so I ask your help

First, how do I measure the time an encoder/decoder needs to encode/decode? I run Linux, so I was thinking about GNU's time, but would that be unfair as TAK runs on Wine? Wine is not an emulator, so is the overhead probably fixed (in seconds of the total) or negligible? Is there a program similar to time on Windows? There are also some encoders which won't run Wine at all.

Second, what am I going to measure? User CPU time, User + System CPU time, Realtime? I was thinking about both User+ System and Realtime. Probably it is an idea to run these encodings/decoding fully in memory? (via ramfs or something similar) Also I thought measuring the memory usage via time is interesting.

Finally, how about the tracks to encode? I have much metal (quite many different subgenres), much classical, some jazz, rock, some pop, but probably it would be interesting to see how these codecs encode 24-bit or 96kHz material (as I heard FLAC does quite bad comparing to Wavpack)

This post has been edited by ktf: May 1 2009, 21:27

[Synthetic Soul]

You may be interested in the following thread:

http://www.hydrogenaudio.org/forums/index....showtopic=50954

I'm not sure that there is a whole lot for you there, as I test on Windows; however if you are interested in my limited experiences in the field, and like to see a pretty graph now and again, it may be worth skimming over.

Finally, how about the tracks to encode? I have much metal (quite many different subgenres), much classical, some jazz, rock, some pop, but probably it would be interesting to see how these codecs encode 24-bit or 96kHz material (as I heard FLAC does quite bad comparing to Wavpack)

How are you planning to display the results? I've been meaning to upgrade my comparison for a long time. One thing I'd like to do is to enable users to filter results by various criteria, including genre. If you test a range of genres, and have a decent amount of tracks for each, you could provide stats for any genre, plus stats for a mix of some, or all. Similarly, you could allow users to filter by bit depth. This of course depends on a display format which allows for filtering. If this cannot be done then I would say that you really need to cover as many genres as possible. I'm not sure about merging 16 and 24 bit results: most users are only going to be interested in results for 16 bit (CD) audio. It would be great to see a comparison of 24 bit tracks, but I think results for 24 bits should be kept separate, and I'd say they'd be of less interest to most people.

This post has been edited by Synthetic Soul: May 3 2009, 11:22

[ktf]

Finally, after a lot of scripting, here are my first test results. I used a selection of Nightwish' album Oceanborn, the tracks Stargazers, Devil & the Deep Dark Ocean, Sacrement of Wilderness, Passion and the Opera, Moondance and The Pharaoh Sails to Orion. I made a selection because I run all tests from ramdisk (to remove the harddisk as a bottleneck) and I have only 1GB of ram.

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

Now my question is: have I got all the encoders with representable settings? I had some trouble choosing some settings for OptimFROG and MPEG-4 ALS. For codecs such as OptimFROG, I won't test every possible setting: it is way too slow! I'll add Real Lossless, ALAC and WMAL soon, as they have only one setting. As you can see, MPEG-4 ALS gives a SEGFAULT, but only at the very beginning of the track Sacrement of Wilderness... Anyone knows why? There is nothing special with this tracks and its headers AFAIK.

This post has been edited by ktf: May 15 2009, 10:37

[halb27]

I'm afraid you forgot wv normal mode.

[hero]

nice work

[ktf]

http://www.icer.nl/test%20-%20Nightwish%20-%20Oceanborn.pdf

Added wavpack normal and ALAC. I tested ALAC using dBpoweramp on Wine, but I can't get Real and WMAL to work, so I'll test these on Windows. When these tests are complete, I'll write some scripts to generate graphs

This post has been edited by ktf: May 9 2009, 08:53

[halb27]

Thanks for including wv normal.
According to my experience differences in compression ratio between lossless codecs can be more severe with music of a more quiet kind (many kind of classical music especially when done with very few instruments, singer/songwriter resp. folk music).
It would be great if you could add also results for music of this kind.

[ktf]

Okay, I'll write down a list of music I want to test. I'll make selections of about 300MB uncompressed, otherwise it won't fit on the ramdisk while not having to swap the real processing out

- Nightwish - Oceanborn (Symphonic metal)
- Mercenary - 11 Dreams (Melodic Death Metal, 6 out of the 12 songs with the highest bitrate in my FLAC collection come from this album)
- Apocalyptica - Inquisition Symphony (Metal, but just cello's playing)
- Nine Inch Nails - The Slip (Industrial rock)
- Nickelback - Dark Horse
- Tiësto - In Search of Sunrise: Asia
- Coldplay - Viva la Vida or Death and all his Friends
- Britney Spears - Circus (Pop)
- Avril Lavigne - Let Go
- Katie Melua - The Katie Melua Collection
- Various - Aangenaam Jazz 2007 (a selection of jazz music, with vouchers to get discount on the featured CD's... I think this should be quite all-round )
- BLØF - Oktober (quite serene, acoustic dutch pop)
- Howard Shore - The Lord of The Rings: The Return of The King
- Piet Jeegers Clarinet Choir - Volume 4 (A Clarinet Choir playing classical music)
- Gustav Holst - The Planets (Boston Symphony Orchestra feat. conductor: Willian Steinberg)

- Nine Inch Nails - The Slip (24-bit/96kHz)

I think I'll add some more today Please comment on missing genre's... I'll search for some hardcore probably, some more classical and some more solo-instrumental. (mainly Classical)

Edit, Added:
Sting - Nothing Like The Sun (Vinyl Rip, no restoration or filters used)
Vivaldi's Four Seasons by I Musici (Vinyl Rip, no restoration or filters used)

Edit2
I just reread the thread.

How are you planning to display the results? I've been meaning to upgrade my comparison for a long time. One thing I'd like to do is to enable users to filter results by various criteria, including genre.

Yes, that was my idea too

This post has been edited by ktf: May 15 2009, 16:13

[halb27]

...I'll search for ... some more classical and some more solo-instrumental. (mainly Classical) ...

Very welcome.
With a lot of pop/rock tracks you improve the quality of your statistics for pop/rock music.
When adding classical music, you widen the scope of music under investigation.
When adding quiet tracks of few instruments only, you probably will give some insight that codecs' compression ratio can vary a lot more than from what we expect when considering pop music.

This post has been edited by halb27: May 9 2009, 15:19

[Brent]

http://www.icer.nl/test%20-%20Nightwish%20-%20Oceanborn.pdf

Added wavpack normal and ALAC. I tested ALAC using dBpoweramp on Wine, but I can't get Real and WMAL to work, so I'll test these on Windows. When these tests are complete, I'll write some scripts to generate graphs

Wouldn't testing ALAC with ffmpeg be more interesting (and easier to automate, and crossplatform)?

[ktf]

It would be way easier to automate indeed, but will it reflect the capabilities of ALAC? I mean, iTunes ALAC codec will be used more? This program doesn't use its own implementation of ALAC, it works via DLL plugins AFAIK

This post has been edited by ktf: May 9 2009, 13:50

[C.R.Helmrich]

Please also consider testing the EBU Sound Quality Assessment Material (SQAM) CD. This is a "standard" set of test items often used for audio codec evaluation in scientific publications. It contains a lot of single instruments and some classical and 80's pop music.

http://www.ebu.ch/en/technical/publication...h3253/index.php

Thanks,

Chris

Edit: Unfortunately, the music items are not available for download, but the rest should suffice anyway.

This post has been edited by C.R.Helmrich: May 9 2009, 16:36

[ktf]

Please also consider testing the EBU Sound Quality Assessment Material (SQAM) CD.

I will

Okay, after a second test round I strike off OptimFROG's --maximumcompression and --maximumcompression --experimental as they are so extremely slow, they take 3x realtime. That's not fair and absolutely not useful. Also I'll no longer test MPEG-4 ALS -7 -z3 -p as it keeps throwing SEGFAULT errors. Now I can make some graphs.

[Brent]

It would be way easier to automate indeed, but will it reflect the capabilities of ALAC? I mean, iTunes ALAC codec will be used more? This program doesn't use its own implementation of ALAC, it works via DLL plugins AFAIK

dpoweramp has its own ALAC implementation as well, or are you referring to the old iTunes plugin?

[ktf]

dpoweramp has its own ALAC implementation as well, or are you referring to the old iTunes plugin?

Oh, okay, I didn't know that. Then I'll test the latest iTunes and FFmpeg as well.

This post has been edited by ktf: May 10 2009, 08:19

[IgorC]

FLAC -8 -A tukey(0.5) -A flattop can bring +0.1% compression comparing to -8

[sauvage78]

For Flac there are some hints that there might be non-official settings that could be interesting for end users.

I am specially thinking of:
-l 8 -b 4096 -r 6
-l 8 -b 4096 -M -r 6

These are tuned version of -6 playing with the -M & -m switch, nobody ever tested those settings but if you analyse the switchs behing the flac settings, & compare those switchs to the various compression/speed comparisons out there, there is some hints that "maybe" one of these switchs would achieve compression on par with -5/-6 & yet maybe be faster (it's a trial to swallow some of -4 encoding speedup & some of -3 decoding speedup, increasing decoding speed of -5/-6 is specially interesting for use with cuetools/accuraterip, increasing encoding speed is less interesting IMHO because you cannot encode faster than you rip or decode anyway).

Here is a small table/screenshot of my analyses of the flac switches that leaded me to this conclusion:


I tried to test it for myself but my methodology was obviously not good as sometime it was faster to decode & sometime it wasn't, I think that this is due to my HDD, specially where the files were written on the HDD.

I may be completely wrong & this may be completely useless in the end, but maybe it is worth to try it ...

Anyway good luck with your test.

This post has been edited by sauvage78: May 10 2009, 16:10

[ktf]

FLAC -8 -A tukey(0.5) -A flattop can bring +0.1% compression comparing to -8

For some of my test tracks, the files grow even bigger then just -8! I'm not interested in such tweaks for now.

For Flac there are some hints that there might be non-official settings that could be interesting for end users.

Well, probably I'll test them when these comparison is complete.

[sauvage78]

np I can live with it, just don't put -l 8 -b 4096 -r 6 & -l 8 -b 4096 -M -r 6 in the same bag as -8 -A tukey(0.5) -A flattop ... I favor speed over compression , specially decompression speed, my switches are expected to compress very close to -5/-6 & maybe be faster. Anything using the -e switch is for people who don't understand lossless IMHO, because lossless is made to be transcoded sooner or later. The problem is that I suspect -m/-M to be a kind of smaller -e. (Lots of wasted time for near zero compression gain).

This post has been edited by sauvage78: May 10 2009, 18:03

[ktf]

I won't, I can see the difference They're just both not interesting for me right now.

[IgorC]

Also It's usefull to see that encoding from FLAC -8 to LAME is only 1.12% slower than from FLAC -0. There are a lot of bottlenecks like HDD, RAM etc in real encoding scenarios.
http://www.hydrogenaudio.org/forums/index....st&p=608813

Despite decoding FLAC -0 is 22% faster than -8 but encoder LAME is 15x slower than decoding FLAC that's why all speed benefit of -0 is masked only to speed boost +1.12%.

This post has been edited by IgorC: May 10 2009, 19:37

[ktf]

Also It's usefull to see that encoding from FLAC -8 to LAME is only 1.12% slower than from FLAC -0. There are a lot of bottlenecks like HDD, RAM etc in real encoding scenarios.

Because of that, I worked from a ramdisk, to eliminate the harddisk as a bottleneck. Indeed, in real scenario's they'll influence encoding, but as eveyone has a different harddisk and a different processor, I'll focus on CPU power, not on harddisk usage.

My first graphs are ready They're based on the results of the first 2 sets of testtracks. Now the question is: what should I do to enhance readability (or is it clear enough?) The graphs are created with JpGraph, many thanks to the authors of this marvelous piece of software ^^

Edit: Removed, see http://www.icer.nl/losslesstest for the current results

This post has been edited by ktf: May 15 2009, 10:38

[Zarggg]

- Nine Inch Nails - The Slip (24-bit/96kHz)

Out of curiosity, why use this version of the track? Wasn't it shown previously that the 24/96 files were the same as 16/48 with padding?

[halb27]

... Now the question is: what should I do to enhance readability (or is it clear enough?) ...

It took me a small amount of time to realize what the axes mean. So I'd prefer to explicitly refer to compression ratio and speed.
I'd prefer to give the compression ratio in percent.
I prefer the linear scale of the speed axis. Reader's emotions and speed facts are more in line this way. I'd accept the disadvantage that the very slow codecs are shown with restricted precision this way.

This post has been edited by halb27: May 11 2009, 18:26

[C.R.Helmrich]

Agreed. Please label the axes. I assume the horizontal axis stands for speed and the vertical for compression ratio?

Also, speaking of ratios in percent, for better comparison, I suggest normalizing the speed axis to the results of the Shorten codec (since that's a classic reference, so to speak) and then to format it in percent as well, i.e. Shorten itself gets 100%, the fastest Flac mode gets nearly 200%, etc.

By the way, thanks for the nice work so far!

Chris