Once more: Time for a new lossless codec comparision?, More comprehensive, more detail Options

[ktf]

Hi all,

The recent activity on the FLAC-dev mailing list (it's alive again!) has reminded me of something I announced some time ago here but didn't really finish: make a new comparison of lossless codec performance, because most tests are pretty outdated or not very comprehensive. While there isn't much development at the moment (which I pity) especially TAK has developed and I hope FLAC maintenance development will soon turn to improvement. As I still have all code of last time I made a comparison (I got it cleaned up and running in about 2 hours) the hardest thing to get right is the selection of music. That's why I set up this thread.

Test structure
I will make several comparisons, as last time it became clear that codecs perform very different when given different material. This asks for several tests for different genres. Moreover, the overall test has to be balanced too. While my taste in music is quite diverse, I hope you can help me cover the voids. I've set up some categories, Heavy (mostly metal), Rock, Electronic, Pop, Jazz, World Music, Orchestral, (Classical) Chamber music, Single-instrument/voice and pre-mix material. The starred items will be included in the main test, I'll try to take 5 out of every 'main' genre. Every item (that is, every album) is equally weighted in the final comparison.

Your opinion is important!
So, first of all, which 'main' genres are missing and which categories would you like to be included in the test? If you suggest a genre of category, please recommend me some CD's to buy which more or less 'span' the whole genre. Second, do you think these lists 'span' the whole genre depicted? Please keep in mind it is of course impossible to include every sub-genre, max 5 per genre would be nice.

Heavy
*Mercenary - 11 Dreams
*Dream Theater - Octavarium
Nightwish - Oceanborn
Linkin Park - Meteora
*System of a Down - Mezmerize
*Apocalyptica - Inquisition Symphony
*Metallica - Death Magnetic
Slipknot - All Hope is Gone
Blind Guardian - A Twist in the Myth

Rock (a.k.a less heavy)
*Paramore - Brand new Eyes
*Red Hot Chili Peppers - Stadium Arcadium
*Foo Fighters - Wasting Light
*Nickelback - Black Horse (Onder heavy?)
*Guns and Roses - Chinese Democracy
30 seconds to Mars - This is War
Muse - H.A.A.R.P. (live from Wembley)

Electronic
*Daft Punk - TRON Legacy R3CONF1GUR3D
*Daft Punk - Discovery (which is quite a different subgenre compared to previous entry)
*Tiësto - In Search of Sunrise 7: Asia
*The Prodigy - Invaders must Die (?)
*NIN - The Slip (Is dat Electronic?)

Pop
*5CD's of various popsongs
*Duffy - Endlessly
*Coldplay - Viva la vida or death and all his friends
*Katie Melua - The Katie Melua Collection
*Avril Lavigne - Let Go
BLØF - Umoja

Jazz, Soul
*The Rosenberg Trio - Djangologists (sample: http://www.youtube.com/watch?v=uOC-xZOWFnc)
*John Coltrane - Kind of Coltrane (Live at the half note '63, Vol 1 + 2)
*Joss Stone - Mind, Body and Soul
(I need some more here)

World music/Folk (very hard to define indeed )
*Various - World Music Instruments, Single Reed Instruments (Clarinet)
*Fanfare Ciocarlia - Baro Biao (gypsy brass band)
(please suggest additions)

Orchestral
*Verdi Requiem
*Carneval des Animeaux
Le Sacre du printemps / Petrouchka
*Bach, Magnificat + Mahler, Symfonie nr. 1
Holst - The Planets
*The Lord of the Rings - Two Towers (complete recordings)

Chamber Music
*2 CD's worth of chamber music
*Some unaccompanied chamber choir recording

Single instrument
*EBU Sound Quality Assessment Material (SQAM) CD
Some of my chamber music recordings (hints for myself: kamermuziekexamen + KaMu-blend)
Excerpts from Yann Tiersens Good Bye Lenin and Amelie Poulain
*Ehren Starks - Lines Build Walls

Not categorized (only to balance the overall test)
*BLØF - April (to add some slow, balad-like music)
*Halo: Reach soundtrack

*Some pre-mix, 24-bit material (chamber-music, orchestral music, jazz)

Finally, please don't get upset about the choices I made making this list, just tell what you think is wrong. My musical taste is mainly in classical, rock and metal, so I tried to do the others as well as I could. Music and it's divions into genres is subject of much debate, but that's not the point of this list.

Thanks in advance!

[ktf]

I'll continue

Apocalyptica is orchestral, isn't it?

No. If it should be categorized, it is 'heavy' or 'chamber music'. They're only with four, that doesn't make an orchestra

If you want to cover the more traditional "Electronic" you should consider (any) Kraftwerk, Klaus Schulze or (not any) Vangelis.
As for Jazz, maybe a "classic" like Miles Davis - Kind of Blue.
If you want to add traditional Country, any album from Alison Krauss+Union Station would do.
For chamber music how about Vivaldi's the four seasons (performance of your choice).

Thanks for the additions.

I'm not trying to come up with rare albums, rather the more common ones.

Which makes it easy for me to get hold of it. Thanks!

You have already 2 live albums on the list, those might have as an added characteristic crowd noise/applause. That's OK, I would only suggest, not too many of those.

Actually, most orchestral items are live as well, but there you won't hear much 'live noises'. Besides that, it is very relevant for the other genres. I'll keep it in mind, thanks.

sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)

When comparing between encoders (all encoding the same stuff) there is little interesting to see. This test will update the results, as the last comprehensive test I could find has been some years ago, so this will update these results.

However, last time I found that between genres, encoders can perform different. Wavpack for example, did worse on single-instrument music compared to the other encoders. Probably this will show up with other source material as well. This will point out area's of improvement for certain codecs.

As said earlier, most people won't care much anymore. My motivation was the recent 'awakening' of the FLAC-mailinglist

IMHO, no test should be without the king of instruments --the church organ-- ...

Oh, right, I forgot to add one. I'll use organ spectacular by Jean Guillou, which is IMO very nicely recorded.

To make it more representative, I would add some salsa, mambo, and reggaeton.

Can you suggest some albums?

sorry to sound silly, but what are we hoping to find?
Surely we are only talking about ~5-10% compression differences?
So are we doing compression vs CPU usage? or vs Time taken (as a proxy?)

I don't think this is a silly question at all. I am not even sure if there is any need for this. And if it is, then I think your suggestion is probably the best comparison, along with ranking by size (speed no objective).

Speed no objective? I really won't compress my files to the heaviest Optimfrog compression. While such a list would state it as top-ranked (3 percentpoint better than FLAC -8) decoding ate 50% CPU on my previous computer. Even in seeking you would notice a lag. Speed still is an objective.

There is no need. I just want to make a fair comparison with all styles equally balanced. Current test seem to lack that.

For example, in addition to comparing size, what about the following: make a regression line time vs size, and for each codec report the compression gain relative to the regression line, or the time gain. (The latter is probably a better suggestion, as we have two time variables (encoding and decoding) -- then one could report that codec X encodes 1% slower but decodes 5% faster than average?)

That's an option, but the problem is choosing that line. Tastes differ (there seem to be people actually using the OFR-setting I mentioned above) and I guess the usual graph will supply more information.

This post has been edited by ktf: May 1 2012, 18:53