New lossless codec comparison (Jan '13), Lots of test results, lots of graphs Options

[ktf]

Hi all,

For the last few weeks I've been busy writing a new lossless audio codec comparison, after lots of talk about it here and here.

The full test report can be found here (PDF), the raw data and graphs can be found here (zip).

While the full report has lots of graphs it also has lots of text, so the 'main' graphs of the report are posted below.



The CDs used

• 30 Seconds to Mars - This Is War (rock)
  
• Confido Domino Minsk - Sacred choral music from White Russia (choral, religious)
  
• Daft Punk - Alive 2007 (electronic, live)
  
• Dan Brown - Angels and Demons (audiobook)
  
• Enya - Amarantine (new age)
  
• Fanfare Ciocarlia - Baro Biao: World Wide Wedding (world music, brass)
  
• Gilberto Santa Rosa - Esencia (salsa)
  
• Giuseppe Verdi - Messa da Requiem (Berliner Philharmoniker feat. conductor Claudio Abbado) (classical, romantic)
  
• Howard Shore - The Hobbit: An Unexpected Journey (soundtrack, orchestral)
  
• Iron Maiden - Brave New World (metal)
  
• John Coltrane - Live at the half note, vol. 1 (jazz)
  
• Joss Stone - Mind, Body & Soul (soul)
  
• J.S. Bach - Magnificat (orchestral, baroque)
  
• Koninklijke Militaire Kapel - [no name] (military brass, vinyl rip)
  
• Kraftwerk - Autobahn (electronic)
  
• Lana del Rey - Born to Die (pop)
  
• Lil Wayne - Tha Carter III (hip hop)
  
• Martin O'Donnell & Michael Salvatori - Halo: Reach O.S.T. (ambient-ish, soundtrack)
  
• Michael Bublé; - meets madison square garden (pop, live)
  
• Mike Oldfield - Tubular Bells (not classifiable)
  
• Rosenberg Trio - Djangologists (jazz)
  
• Rush - Grace under Pressure (1997 remaster) (rock)
  
• System of a Down - Mezmerize (metal)
  
• Szakcsi - Virágom, virágom (world music)
  
• Tiësto - In Search of Sunrise 7: Asia (dance)
  
• Various - Jeff Waynes Musical Version of The War of the Worlds (radio drama-ish, rock, orchestral)
  
• Various - Latin Village, CD 1: Salsa (compilation of salsa)
  
• Xzibit - At the Speed of Life (hip hop)
  
• Yann Tiersen - Le Fabuleux Destin d'Amélie Poulain (minimalism, soundtrack)

For more information on why I chose these CDs, what strange things happened with the pure speech data (Dan Browns audiobook) and tests on 96kHz/24-bit or surround sound audio, check the full report.

Its labeled revision 1 because I intend to keep it updated. Any comment (on my grammar too, as I'm not a native speaker) is welcome.

[Dynamic]

I was meaning the average compression for a particular genre and that the lossless compressed bitrate had a lot to do with the average loudness (in my experience, Replay Gain was a good benchmark) as you surmised.

I've got the impression that lossless bitrates above about 900 kbps were pretty rare in the late 90s and early 2000s CDs, but are now commonplace in much of the 2010s popular releases, with over 1000kbps being pretty frequent.

I'm not familiar with how the DR value or even its version of RMS is calculated in detail (e.g. gated or not), which is sometimes necessary, because it mentions things like ignoring long-term dynamics and presumably combining measurments reflecting medium term variations and transient attacks to derive the DR number. People get in trouble making assumptions about the ReplayGain method too.

I'm not sure what type of thing is on that soundtrack. One thing that tends to be the case on albums with multiple CD releases over the decades is that a 1983 release, a 1993 release, a 2003 release, and a 2013 release say, would tend to get louder decade by decade (so they sound loud enough on a CD changer, I guess) and the lossless bitrate would tend to increase too.

My expectation (based on only my anecdotal evidence and selective memory, and potentially a fautly mental model of why!) is that there's a tendency that any popular music format (pop, rock, dance) follows that trend to increased loudness at first, then finally reduced dynamic range, and I wouldn't be surprised that picking out those genres only it might be possible to plot a graph versus year.

One neat idea for someone with the right CD collection is picking all versions of a long-running pop compilation such as the UK & Eire's "Now, That What I Call Music!" pressed from the mid 80s volume 10 double CD to the present day volume 83) and plotting bitrates of the lossless files versus date, we'd see a steady increase in loudness and bitrate over the years (as a trend - there's bound to be variation), and almost certainly a decline in DR value too (albeit that it seems fairly constrained).

x-axis = Date of Release, Left y-axis = FLAC bitrate, Right y-axis = ReplayGain value
Dual plot - bitrate vs date, RG Album Gain vs date (or versus the NOW! number, as in NOW! 36)