New lossless codec comparison (Jan '13), Lots of test results, lots of graphs Options

[ktf]

Hi all,

For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here and here.

The full test report can be found here (PDF), the raw data and graphs can be found here (zip).

While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.



The CDs used

• 30 Seconds to Mars - This Is War (rock)
  
• Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  
• Daft Punk - Alive 2007 (electronic, live)
  
• Dan Brown - Angels and Demons (audiobook)
  
• Enya - Amarantine (new age)
  
• Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  
• Gilberto Santa Rosa - Esencia (salsa)
  
• Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
  
• Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
  
• Iron Maiden - Brave New World (metal)
  
• John Coltrane - Live at the half note, vol. 1 (jazz)
  
• Joss Stone - Mind, Body & Soul (soul)
  
• J.S. Bach - Magnificat (orchestral, baroque)
  
• Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
  
• Kraftwerk - Autobahn (electronic)
  
• Lana del Rey - Born to Die (pop)
  
• Lil Wayne - Tha Carter III (hip hop)
  
• Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
  
• Michael Bublé; - meets madison square garden (pop, live)
  
• Mike Oldfield - Tubular Bells (not classifiable)
  
• Rosenberg Trio - Djangologists (jazz)
  
• Rush - Grace under Pressure (1997 remaster) (rock)
  
• System of a Down - Mezmerize (metal)
  
• Szakcsi - Virágom, virágom (world music)
  
• Tiësto - In Search of Sunrise 7: Asia (dance)
  
• Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  
• Various - Latin Village, CD 1: Salsa (compilation of salsa)
  
• Xzibit - At the Speed of Life (hip hop)
  
• Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)

For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report.

Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.

[IgorC]

ktf,

Thank You for sharing interesting comparison.

What decoder and/or application have You used for TAK? Last time TAK native app decodes as fast as FLAC. TAK foobar plugin was somewhat slower.

I was curious to try Monkey's Audio c3000 ('High') on Galaxy II phone with Rockbox app for Android . Got 17 hours of a battery life. And it uses the lowest frequency step so I suspect APE 'Extra High' would last comparable time. Given that an energy efficiency of proccesors grows fast it's not crazy to think about Monkey's Audio on mobile devices. Anyway I don't use lossless on portable and use FLAC lossless storage for a fast transcoding to lossy formats.

[ktf]

What decoder and/or application have You used for TAK? Last time TAK native app decodes as fast as FLAC.

I've used the Takc.exe command line encoder which I downloaded straight from TAK's homepage. This was run over Wine, but that shouldn't make much of a difference, as Wine only emulates system calls.

The reason that TAK decodes as fast as FLAC with you is probably because not your CPU but your harddisk is the bottleneck. In this test, CPU-time was measured and the test were done with a ramdisk to bypass the harddisk, so this might be the difference. After all, those codecs decode so fast on desktop CPU's that harddisks can't keep up.

Regarding Monkey's Audio, I've seen some devices advertising supporting it recently, so your observation is correct.

This post has been edited by ktf: Jan 6 2013, 19:17

[ktf]

I know lossless audio codec comparisons have lost interest with most of you, but only one reply? I hope that PDF wasn't too overwhelming.

Anyway, a short version of this comparison (and the PDF) is on the FLAC website now: http://flac.sourceforge.net/comparison.html

[knutinh]

what kind of pc hardware did you use? did you use special compiles for that hardware?

regards
k

[Fabiolander]

What a job !

Thank you but could you translate the results for newbees ?

[ktf]

what kind of pc hardware did you use? did you use special compiles for that hardware?

The PC was a HP Elitebook 8530w, unmodified. It features an Intel Core2Duo T9600 with 4GB of ram. The CPU has been clocked down to 2.13GHz to avoid thermal throttle. The OS was Kubuntu Linux 12.10 64-bit. The FLAC and WavPack binaries were packed with Kubuntu (64-bit ones, but not optimized for any kind of hardware), all other codecs (except ALAC, WMA Lossless and Real Audio) were run with native Linux binaries or over Wine. The three exceptions were tested on a Windows box, but please refer to the PDF if you want to know more.

Thank you but could you translate the results for newbees ?

I thought the comparison on the FLAC-website is quite accessible to newbies? Can you clarify what it is that needs translation?

This post has been edited by ktf: Jan 19 2013, 22:42

[romor]

Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

Thanks for sharing your results. I believe there is no much interest, because lossless tests were discussed recently

If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs

[urak]

but only one reply? I hope that PDF wasn't too overwhelming.

To me the charts are very confused (poor color readability, squares too small). Thanks

[Gregory S. Chudo...]

Great job!
Shameless self-promotion on my part follows:
It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

[C.R.Helmrich]

It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

Indeed, I would have liked to see that comparison as well to find out how the claim "capable of so much more" translates into numbers.

ktf, other than that you comparison (e.g. using a ramdisk) is very well done! But the amount of data is indeed a bit overwhelming. Can you briefly explain what changed in the outcome compared to your own 2009 and Synthetic Soul's 2008 test? I see the inclusion of a RealAudio lossless codec, but other than that the ranking seems to be the same? Meaning TAK and FLAC are most efficient, ending up to the lower right of the scale most of the time?

Chris

[ktf]

Is sourceforge slowly dying? This is not the first time lately I can't access SF resource.

I haven't had any problems lately?

and bitmap instead vector graphs, makes some graphs hardly readable.

To me the charts are very confused (poor color readability, squares too small).

I know of a few PDF-readers which do not scaling bitmap-images properly indeed, however, the toolkit I used was bitmap only. The problem with bigger squares is that they tend to overlap (which they already do with this size)

It's a pity you didn't include CUETools Flake/FlacCL encoders, FLAC format is capable of so much more, and the limitations of libFLAC hide that.

I tried, but couldn't get it running on my notebook. I should try harder I guess. The problem is that there are a few FLAC-encoders out there, but including them might be confusing. Not everyone understands the difference between an codec and a format, just see MP3-vs-LAME: People complain about MP3 while they are actually complaining about the rubbish encoder some used. I might include it in the next revision of this document.

Can you briefly explain what changed in the outcome compared to your own 2009 and Synthetic Soul's 2008 test?

Nothing stunning really. TAK is even faster now on encoding and ALS -7 didn't work as well as it did last time. So nothing surprising on the part of CD-audio. However, in the PDF there are a few graphs on 96kHz/24bit audio ("High-res") and multichannel audio. Nothing very exciting there either, but some codecs have some pecularities, for example, FLAC being slower than usual on both encoding and decoding when compared to other codecs. There's a note on the performance of MLP (that's the codec used for Audio DVD's) too. The only unusual results are on stereo-encoded mono, which was already known to be a problem with some encoders.

So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.

[Dynamic]

So, in short, nothing really new or stunning, but I think it's just a fair comparison encompassing a wide range of musical genre's. The only conclusion that I draw from this which wasn't drawn before AFAIK, is that there's no codec that behaves particularly good or bad at certain kinds of music: choral music gives you about the same pattern as heavy metal, only the average compression achieved is different. If you want to see that for yourself, see the raw data, which includes graphs for all CDs I tested.

I suspect a lot of that is less to do the genre's musical nature and more to do with the loudness war (continuously filling all 16 bits for metal, usually using the lowest 12 to 14 bits for choral, leaving about 2 to 4 bits of extra random/unpredictable noise to encode much of the time). I suspect that heavy metal and choral music both adjusted to, say, 83 dB SPL Album Gain level, would be a much closer match. (This tends to be supported by lossyWAV's results, and pre-lossyWAV tests of applying Album Gain (technically a loss of data, albeit mostly noise) before lossless encoding)

[ktf]

I suspect a lot of that is less to [..]

What do you mean by 'that'? The average compression achieved or the observation that no codec performs much better on certain music? In the first case, I don't agree (see below), for the latter, I don't understand what you mean.

For example, take a look at the graphs (see the raw material ZIP) for Rush - Grace under Pressure and Howard Shore - The Hobbit: An Unexpected Journey. Both are DR 10 (see here for more information on DR), the average compression of the first ('80 rock) is about 70% and for the latter (2012 soundtrack material) it is 51%. As the DR measure is the same for both, a difference this large (19 percentage points!) can't be assumed to be side-effects of the loudness war only. However, still, the graphs look very similar when comparing codecs amongst each other.

edit: I just checked the replaygain values of both albums, for the first it is -7.0dB, for the second it is -4.9dB. That's a difference of 2.1dB. If we assume every 6dB (or 1 bit) of noise added halves the achieved compression, then "adding 2.1dB of noise" to the soundtrack would result in a compression ratio of 1 - ( 1 - 0.51 ) / 10 ^ (2.1 / 20) = 62%. This is still 8 percentage point away from the observed 70%, so I can't be all loudness war. But then again, ReplayGain is probably not a good measure either...

edit2: I just measured the RMS, for Rush it's -14.6dB and for Howard Shore its -13.6dB, so it's even the other way round: the one that is compressed more has a higher RMS in this case...

This post has been edited by ktf: Jan 21 2013, 15:19

[Dynamic]

I was meaning the average compression for a particular genre and that the lossless compressed bitrate had a lot to do with the average loudness (in my experience, Replay Gain was a good benchmark) as you surmised.

I've got the impression that lossless bitrates above about 900 kbps were pretty rare in the late 90s and early 2000s CDs, but are now commonplace in much of the 2010s popular releases, with over 1000kbps being pretty frequent.

I'm not familiar with how the DR value or even its version of RMS is calculated in detail (e.g. gated or not), which is sometimes necessary, because it mentions things like ignoring long-term dynamics and presumably combining measurments reflecting medium term variations and transient attacks to derive the DR number. People get in trouble making assumptions about the ReplayGain method too.

I'm not sure what type of thing is on that soundtrack. One thing that tends to be the case on albums with multiple CD releases over the decades is that a 1983 release, a 1993 release, a 2003 release, and a 2013 release say, would tend to get louder decade by decade (so they sound loud enough on a CD changer, I guess) and the lossless bitrate would tend to increase too.

My expectation (based on only my anecdotal evidence and selective memory, and potentially a fautly mental model of why!) is that there's a tendency that any popular music format (pop, rock, dance) follows that trend to increased loudness at first, then finally reduced dynamic range, and I wouldn't be surprised that picking out those genres only it might be possible to plot a graph versus year.

One neat idea for someone with the right CD collection is picking all versions of a long-running pop compilation such as the UK & Eire's "Now, That What I Call Music!" pressed from the mid 80s volume 10 double CD to the present day volume 83) and plotting bitrates of the lossless files versus date, we'd see a steady increase in loudness and bitrate over the years (as a trend - there's bound to be variation), and almost certainly a decline in DR value too (albeit that it seems fairly constrained).

x-axis = Date of Release, Left y-axis = FLAC bitrate, Right y-axis = ReplayGain value
Dual plot - bitrate vs date, RG Album Gain vs date (or versus the NOW! number, as in NOW! 36)

[ktf]

If I may comment on latex article: it has unnecessary huge margins if intended for screen reading, which coupled with many formats tested and bitmap instead vector graphs, makes some graphs hardly readable. I wish more tables then graphs

To me the charts are very confused (poor color readability, squares too small). Thanks

I have updated the PDF, this time it is optimized for screen reading and readability of the graphs has been improved. No changes have been made to the contents (so no new test results). New version download

This post has been edited by ktf: Mar 19 2013, 09:45

[ktf]

Once again hi all,

Today I've been busy trying to add refalac. Does anyone have an opinion about which switches to add? I only saw --fast, are there any other? Can't find them in the usage info.

[lvqcl]

BTW, there are command-line WMAL encoder and decoder (both require Windows Media Format runtime)


...and if you want to use Windows for WMAL tests: you can use timer.exe from 7-Benchmark (7bench1200.7z)

This post has been edited by lvqcl: Mar 22 2013, 19:27

[spicymeatball77]

I've got a bit of a noob question. I just RockBox'd a Sansa Clip+ and I'm trying to find the best lossless encoding that balances file size and battery life. Right now I'm using FLAC-8. I'm assuming that on the decoding graph a higher value on the X-axis equates to better decompression ie. better battery life?

[Dynamic]

Yes, and the flac -8 point is the leftmost value on FLAC, though all are pretty close. While it doesn't translate 1-for-1 to battery life, decoding speed is a strong indicator (well correlated).

[Propheticus]

If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.

[marc2003]

I've got a bit of a noob question. I just RockBox'd a Sansa Clip+

admittedly old now (2010) but there have been benchmarks done already.

http://www.rockbox.org/wiki/CodecPerforman...ARM926EJ_45S_41

CODE

flac_5.flac     2936.56% realtime     Decode time - 5.99s     8.17MHz
flac_8.flac     2748.43% realtime     Decode time - 6.40s     8.73MHz

you can always run tests yourself to see if there have been any improvements to the rockbox code in the intervening years.

[db1989]

If I look at those graphs I see no point in using anything higher than FLAC -5. The extra compression you get is minimal and en-/decoding takes longer. Quite extreme case of diminishing returns I'd say.

I think you’re actually talking about -4. In the upper graph, that’s the one with about 135 degrees between it and the next setting on the right, and as Dynamic said, the graph goes from -8 on the left to -0 on the right. The lower graph seems to have only 8 settings represented (rather than 9), although I presume that’s just a result of visual obstruction as some of them are very close together.

This post has been edited by db1989: Apr 5 2013, 18:37

[Propheticus]

Yes -4 is the most clear point as you said. But -5 seems to be the point after which the graph levels out (asymptotic almost)*. So anything higher is virtually wasted effort.

*also see the document and individual cases.

This post has been edited by Propheticus: Apr 5 2013, 19:17

[spicymeatball77]

I transcoded my FLACs from -8 to -5 and there was *very* little size increase. This will likely help out my battery life at a tiny size cost. Thanks for the input guys.