New lossless codec comparison (Jan '13), Lots of test results, lots of graphs Options

[ktf]

Hi all,

For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here and here.

The full test report can be found here (PDF), the raw data and graphs can be found here (zip).

While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.



The CDs used

• 30 Seconds to Mars - This Is War (rock)
  
• Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  
• Daft Punk - Alive 2007 (electronic, live)
  
• Dan Brown - Angels and Demons (audiobook)
  
• Enya - Amarantine (new age)
  
• Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  
• Gilberto Santa Rosa - Esencia (salsa)
  
• Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
  
• Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
  
• Iron Maiden - Brave New World (metal)
  
• John Coltrane - Live at the half note, vol. 1 (jazz)
  
• Joss Stone - Mind, Body & Soul (soul)
  
• J.S. Bach - Magnificat (orchestral, baroque)
  
• Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
  
• Kraftwerk - Autobahn (electronic)
  
• Lana del Rey - Born to Die (pop)
  
• Lil Wayne - Tha Carter III (hip hop)
  
• Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
  
• Michael Bublé; - meets madison square garden (pop, live)
  
• Mike Oldfield - Tubular Bells (not classifiable)
  
• Rosenberg Trio - Djangologists (jazz)
  
• Rush - Grace under Pressure (1997 remaster) (rock)
  
• System of a Down - Mezmerize (metal)
  
• Szakcsi - Virágom, virágom (world music)
  
• Tiësto - In Search of Sunrise 7: Asia (dance)
  
• Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  
• Various - Latin Village, CD 1: Salsa (compilation of salsa)
  
• Xzibit - At the Speed of Life (hip hop)
  
• Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)

For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report.

Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.

[ktf]

Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

I haven't had any problems lately?

and bitmap instead vector graphs, makes some graphs hardly readable.

To me the charts are very confused (poor color readability, squares too small).

I know of a few PDF-readers which do not scaling bitmap-images properly indeed, however, the toolkit I used was bitmap only. The problem with bigger squares is that they tend to overlap (which they already do with this size)

It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

I tried, but couldn't get it running on my notebook. I should try harder I guess. The problem is that there are a few FLAC-encoders out there, but including them might be confusing. Not everyone understands the difference between an codec and a format, just see MP3-vs-LAME: People complain about MP3 while they are actually complaining about the rubbish encoder some used. I might include it in the next revision of this document.

Can you briefly explain what changed in the outcome compared to your own 2009 and Synthetic Soul's 2008 test?

Nothing stunning really. TAK is even faster now on encoding and ALS -7 didn't work as well as it did last time. So nothing surprising on the part of CD-audio. However, in the PDF there are a few graphs on 96kHz/24bit audio ("High-res") and multichannel audio. Nothing very exciting there either, but some codecs have some pecularities, for example, FLAC being slower than usual on both encoding and decoding when compared to other codecs. There's a note on the performance of MLP (that's the codec used for Audio DVD's) too. The only unusual results are on stereo-encoded mono, which was already known to be a problem with some encoders.

So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.