Public Listening Test [2010], Discussion Options

[rpp3po]

While you have written this, I have updated my last post. I think all tools except MediaInfo show consistent relative rates compared to undiluted raw AAC (including constant frame overhead).

This post has been edited by rpp3po: Feb 5 2010, 13:06

[.alexander.]

Alexander, I also supported the idea to use hand picked files, to get as close to 128 kbit/s average, at first. But I don't do anymore. Let us just use a setting for each encoder that will result in a collection average of ~128kbit/s. Some contenders like Quicktime TVBR, which is only alterable in increments of 8, don't allow fine tuning or even want to downsample at some settings. This would be a mess after all. Using one preset for each encoder really seems the most practicable approach.

I don't usualy quote myself, buy why not

use looped samples. Then pick fragments with lowest bit-length for each encoder. This way you could check whether CBR is that bad.

This can be useless in case of uncontrained VBR, but very easy to implement. CT encoder is also restictive in some way, why don't adapt to its traits? e.g. adjust its bitrate to match average bitrate of others?

[rpp3po]

Blind testing is impossible if each non-adjustable encoder gets different fragments. It's also only a cosmetic change.

This post has been edited by rpp3po: Feb 5 2010, 13:15

[.alexander.]

The input would be same for each encoder (1000 times looped 30sec sample). Then output stream is split into 1000 fragments of same duration (30 sec) and then the fragment with lowest amount of bits is chosen for listening. The only difference is in bit reservoir state at the begining of fragment.

This post has been edited by .alexander.: Feb 5 2010, 13:36

[rpp3po]

Ok, I understand. But IgocC has already decided that this won't happen. We should tolerate that. He's doing most of the work. Without that discussions could go on well into 2011...

[nao]

Quicktime on the Mac doesn't have any problem with odd bitrates, it's rather a bug of the Windows port. Both nao's XLD and Apple's afconvert front-end don't have this problem.

QuickTime on OSX has the same limitation. Afconvert and XLD are not affected by the limitation because they use CoreAudio (AudioCodec) API instead of QuickTime to access the AAC codec.

[rpp3po]

Thanks for the info, nao!

It is the same codec, nevertheless, only different APIs. You basically said that, just wanted to clarify.

CVBR 120 results in 128 kbit/s (instead of 129 as Q60) with the above collection. A 1 kbit/s difference shouldn't be a problem.

This post has been edited by rpp3po: Feb 5 2010, 15:31

[frozenspeed]

So when does this listening test start? How can I participate?

[C.R.Helmrich]

rpp3po,

I have 87 lossless cross-genre albums on my notebook right now. I'll check in a couple of minutes how close its average is going to get to 128kbit/s at Q 59.

PS I have finished the 87 albums. I get a collection average of 129 kbit/s (median at 134 kbit/s) for Q 59.

For completeness sake, would you mind listing the individual albums? And when you encoded them, was each album one single FLAC file or did you encode track by track?

Thanks,

Chris

[C.R.Helmrich]

Could someone give a summary to what has been decided so far? Is the discussion about codecs and their settings over? I am asking because there was already a samples thread somewhere and usually you choose samples after choosing codecs (at least it was so in the past).

Let's see.

http://www.hydrogenaudio.org/forums/index....showtopic=77932 We will test Apple true VBR vs. Apple constrained VBR vs. Nero 1.5.3 vs. winner of internal pre-test (DivX vs. Coding Technologies). The poll is also in-line with the newest discussions.

http://www.hydrogenaudio.org/forums/index....showtopic=77809 128 kbps. Also supported by the discussion here which has been exclusively about 128-ish VBR bitrates. Note that the CT encoder (which is CBR only) will be tested at 130 kbps since it supposedly uses ADTS.

Important short-term decisions missing:

Nero quality value: rpp3po, could you please run the nero encoder over your 87-album test set with -q 0.41 and report the bitrate here? Thanks!

Now, help me out. What settings are we using for iTunes CVBR and DivX?

Once that's fixed, I think we can finish the item selection.

Chris

(Edit: changed the quality value slightly)

This post has been edited by C.R.Helmrich: Feb 6 2010, 12:45

[rpp3po]

For completeness sake, would you mind listing the individual albums? And when you encoded them, was each album one single FLAC file or did you encode track by track?

I don't feel comfortable doing that. A set 87 individual albums (including several rarities) is sufficient for a one in billions fingerprint and I don't want that linked to my internet alias. I know millions share info like that on last.fm, spotify, and whatnot, let alone Facebook & Co., but I do not. 87 albums might not be a problem (only a possibility) today, but what's in five, ten years? The internet doesn't forget very fast. If anyone can't live with these terms feel free to upload your own results.

The encoding was done track by track. The reported average is that reported by Foobar's properties dialog for all selected tracks.

Nero quality value: rpp3po, could you please run the nero encoder over your 87-album test set with -q 0.41 and report the bitrate here? Thanks!

I can do that later today.

PS It is running now. It is going to take 6:30 hours in a virtual machine. I have sorted the queue by track numbers so that l can do a half-way check in 3 hours and evaluate if going for 0.4 would be better.

PPS Watching Nero groan in VMware was a pain. So I gave Dibrom's new Foobar package for OS X a try and intregrated NeroAACenc and the ALAC component. It rock's! Restarted the whole thing, new estimated time 2:50h, speed 32x. It looks like 0.41 is correct btw.

This post has been edited by rpp3po: Feb 6 2010, 17:17

[IgorC]

rpp3po and Chris,

After nao's acclaration I think --CVBR 128 --normal is fine as it produces the same output as itunes on Windows OS. --CVBR 128 --normal has very close bitrate to --TVBR 60 --highest. (See previous posts)

Divx -v 4

Alex B's method for calculation bitrate should be applied. http://www.hydrogenaudio.org/forums/index....st&p=685755

This post has been edited by IgorC: Feb 6 2010, 17:46

[rpp3po]

What's the benefit of attributing up to 5% more bitrate to CVBR over TVBR than needed, if we don't have to? qtaacenc is perfectly able to accept the settings 120, 122, 124, 126, and 128 on the Windows platform and uses the same encoder as iTunes. The contestant could still be truthfully called "iTunes VBR encoder with 128 kbit/s average target setting".

This post has been edited by rpp3po: Feb 6 2010, 17:53

[IgorC]

What's the benefit of attributing up to 5% more bitrate to CVBR over TVBR than needed, if we don't have to?

The difference is much less from my previous findings.
http://www.hydrogenaudio.org/forums/index....st&p=683265

qtaacenc is perfectly able to accept the settings 120, 122, 124, 126, and 128 ...

Actually it doesn't.

[rpp3po]

Actually it doesn't.

Ok, we didn't have a true clarification of the meaning of "odd" bitrates, yet, wether it means odd numbers or odd in the sense of not (96, 122, 128). There was just a clarification why it worked on the mac but didn't on Windows. But since you insist, I guess you double-checked, and believe it.

When there really was no way on Windows to reproduce the files in the test, that's really a drawback and I'll accept it. There's still the possibility to ABX my own set of files, if CVBR and TVBR come out too close in the test.

This post has been edited by rpp3po: Feb 6 2010, 18:28

[rpp3po]

Nero has finished. 0.41 was a good call by C. R. Helmrich!

QT CVBR 121 --normal: average 129 kbit/s, median 130 kbit/s, min 83 kbit/s, max 136 kbit/s
Nero AAC 1.5.1.0 q 0.41: average 129 kbit/s, median 133 kbit/s, min 81 kbit/s, max 143 kbit/s
QT TVBR Q(59-68) --highest: average 129 kbit/s, median 134 kbit/s, min 72 kbit/s, max 168 kbit/s

Lowest bitrate is the same as QT CVBR: Radiohead: "Motion Picture Soundtrack". Highest is Björk, "Cocoon" from "Verspertine Live". The top 30 bitrates are quite different from QT, I'm looking forward to the comparison.

I had only done a ~35% run of QT CVBR 128 and then switched to lower rates, because the result was too high. Tonight I'll let it run to 100% for completeness.

This post has been edited by rpp3po: Feb 6 2010, 22:44

[Alex B]

Divx -v 4

Before anyone encodes a huge amount of files and goes through the needed muxing step here are my first results of DivX VBR AAC encoding.

I encoded my usual reference sets (25 various and 25 classical tracks) @ -v 4 and -v 5.

I used Speek's Batchenc program as a GUI for DivXAACEnc.exe. http://members.home.nl/w.speek/batchenc.htm
For -v 4 a command line that works is:
[path to the encoder]\DivXAACEnc.exe -i <infile> -o <outfile.aac> -v 4

After encoding I muxed the raw AAC tracks with Mp4Box v.0.4.6 (dev.20091013). http://kurtnoise.free.fr/mp4tools/
Once again I used Batchenc as a front end. A command line that works is:
[path to the muxer]\MP4Box.exe -add <infile> <outfile.m4a>
According to the documentation the m4a extension makes MP4box to automatically create an "iTunes compatible" file (whatever that is supposed to mean).

Then I measured the individual track bitrates with foobar and calculated the results with a spreadsheet (I need to do that because my reference sets have tracks that were carefully selected only because of their individual audio qualities. The varied track durations would add incorrect weighting in this case.
EDIT: For instance, foobar calculates the overall bitrate of the complete selection and thus longer tracks have more weight than shorter tracks in the displayed average value.

The results:

-v 4

CODE

DivX -v 4 Various

AC_DC - Highway to Hell 127
Adiemus - Boogie Woogie Llanoogie 122
Barry White - Sho' You Right 128
Björk - Possibly Maybe - LFO (Lucy Mix) 120
Davis Bowie - Starman 127
Dido - Life For Rent 123
Duran Duran - Astronaut 129
ELO - Livin' Thing 123
Erich Kunzel & The Cincinnati Pops - Theme from The Pink Panther 109
Evanescence - Going Under 126
Faithless - Mass Destruction 121
Garbage - Bleed Like Me 124
Jamiroquai - World That He Wants 114
Kraftwerk - Tour de France Etape 1 131
Morrissey - Irish Blood, English Heart 129
Paco De Lucia - Rumba Improvisada 118
Santana - Oye Como Va 131
Simply Red - You Make Me Believe 125
Sting - Desert Rose 127
The Beatles - Let It Be 119
Tina Arena - Symphony of Life 118
U2 - Vertigo 129
Whitney Houston - Queen Of The Night 124
Yello - Planet Dada 135
Yo-Yo Ma - Libertango 114

Average 123.72


DivX -v 4 Classical

Aldo Ciccolini - Satie, Sports Et Divertissements, Le Flirt 95
Alfred Brendel - Beethoven, Piano Sonata No 15, Op 28 Scherzo 102
Baroque Festival Orchestra - Vivaldi, Sinfonia in C major, 3. Presto 114
Berlin - Mahler, Symphony No 8, 2 Chailly, Ewiger Wonnebrand 110
Berlin Philharmonic Orchestra - Mozart, Requem in D moll K 626, Sanctus 119
Berlin Philharmonic Orchestra - Strauss, Also Sprach Zarathustra (Thus Spoke Zarathustra) 108
Christophe Rousset - Farinelli, Il Castrato (OST), J.A. Hasse, Generoso risuegliati 108
Concentus Musicus Wien - Bach, Matthäus Passion BWV 244, Da ging hin der Zwölfen Einer 99
Daniel Barenboim - Mozart, Piano Concerto No 3 in D major, K40-3 103
Gérard Lesne - Vivaldi, Sonate Op 2 No 3 pour violon & bc, III. Adagio 107
Giuseppe Sinopoli - Elgar, Cello Concerto, Serenade for Strings, Enigma-Andante 98
Itzhak Perlman - Paganini, 24 Caprises, No 1 In E 112
Jascha Heifetz - Sarasate, Zigeunerweisen, Op 20 No 1 116
Jessye Norman - Angels we have heard on high (Trad.) 113
Kirov Orchestra & Chorus - Khatchaturian, Gayaneh, Säbeltanz 117
Leslie Howard - Liszt, Douze Grandes Études, S 137 No 1 in C 91
London Sinfonietta - Saint-Saëns, Le carnaval des animaux, Hémiones 95
London Symphony Orchestra - Ravel, Daphnis et Chloé, 10. Tres modere 95
Marie-Claire Alain - Bach, Wo Soll Ich Fliehen Hin BWV 646 107
Michael Nyman - The Piano OST, A Bed of Ferns 100
Orchestre Symphonique De Montreal - Elgar, Enigma Variations No 1 104
Philharmonia Slavonica - Bach, BWV 1067 Rondeau 113
The Cleveland Orcestra - Ravel Valses, nobles et sentimentales, 1. Modere 115
The Philadelphia Orchestra - Tchaikovsky, The Nutcracker, Op 71a (Ballet Suite) No 2 109
Zbigniew Preisner - Trois Couleurs Bleu (OST), First flute 88

Average 105.52


DivX -v 4 Overall average 114.62

-v 5

CODE

DivX -v 5 Various

AC_DC - Highway to Hell 152
Adiemus - Boogie Woogie Llanoogie 145
Barry White - Sho' You Right 154
Björk - Possibly Maybe - LFO (Lucy Mix) 142
Davis Bowie - Starman 152
Dido - Life For Rent 148
Duran Duran - Astronaut 154
ELO - Livin' Thing 147
Erich Kunzel & The Cincinnati Pops - Theme from The Pink Panther 130
Evanescence - Going Under 151
Faithless - Mass Destruction 143
Garbage - Bleed Like Me 148
Jamiroquai - World That He Wants 135
Kraftwerk - Tour de France Etape 1 156
Morrissey - Irish Blood, English Heart 155
Paco De Lucia - Rumba Improvisada 140
Santana - Oye Como Va 156
Simply Red - You Make Me Believe 148
Sting - Desert Rose 153
The Beatles - Let It Be 141
Tina Arena - Symphony of Life 140
U2 - Vertigo 155
Whitney Houston - Queen Of The Night 148
Yello - Planet Dada 161
Yo-Yo Ma - Libertango 135

Average 147.56


DivX -v 5 Classical

Aldo Ciccolini - Satie, Sports Et Divertissements, Le Flirt 113
Alfred Brendel - Beethoven, Piano Sonata No 15, Op 28 Scherzo 120
Baroque Festival Orchestra - Vivaldi, Sinfonia in C major, 3. Presto 136
Berlin - Mahler, Symphony No 8, 2 Chailly, Ewiger Wonnebrand 129
Berlin Philharmonic Orchestra - Mozart, Requem in D moll K 626, Sanctus 143
Berlin Philharmonic Orchestra - Strauss, Also Sprach Zarathustra (Thus Spoke Zarathustra) 129
Christophe Rousset - Farinelli, Il Castrato (OST), J.A. Hasse, Generoso risuegliati 129
Concentus Musicus Wien - Bach, Matthäus Passion BWV 244, Da ging hin der Zwölfen Einer 116
Daniel Barenboim - Mozart, Piano Concerto No 3 in D major, K40-3 121
Gérard Lesne - Vivaldi, Sonate Op 2 No 3 pour violon & bc, III. Adagio 128
Giuseppe Sinopoli - Elgar, Cello Concerto, Serenade for Strings, Enigma-Andante 116
Itzhak Perlman - Paganini, 24 Caprises, No 1 In E 132
Jascha Heifetz - Sarasate, Zigeunerweisen, Op 20 No 1 137
Jessye Norman - Angels we have heard on high (Trad.) 134
Kirov Orchestra & Chorus - Khatchaturian, Gayaneh, Säbeltanz 140
Leslie Howard - Liszt, Douze Grandes Études, S 137 No 1 in C 107
London Sinfonietta - Saint-Saëns, Le carnaval des animaux, Hémiones 112
London Symphony Orchestra - Ravel, Daphnis et Chloé, 10. Tres modere 113
Marie-Claire Alain - Bach, Wo Soll Ich Fliehen Hin BWV 646 126
Michael Nyman - The Piano OST, A Bed of Ferns 119
Orchestre Symphonique De Montreal - Elgar, Enigma Variations No 1 123
Philharmonia Slavonica - Bach, BWV 1067 Rondeau 134
The Cleveland Orcestra - Ravel Valses, nobles et sentimentales, 1. Modere 136
The Philadelphia Orchestra - Tchaikovsky, The Nutcracker, Op 71a (Ballet Suite) No 2 129
Zbigniew Preisner - Trois Couleurs Bleu (OST), First flute 103

Average 125


DivX -v 5 Overall average 136.28

Summary:

DivX -v 4
Various 123.72
Classical 105.52
Average 114.62

DivX -v 5
Various 147.56
Classical 125.00
Average 136.28


Apparently -v 4 produces quite low bitrates. -v 5 is closer to the test target, though a bit on the high side.

This post has been edited by Alex B: Feb 6 2010, 23:53

[C.R.Helmrich]

Thanks a lot for the results, rpp3po! I estimated the -q value from the discussion here.

Thanks also to Alex! Sigh, now how on earth are we going to integrate the DivX encoder into the test in a fair way? Increase the nero and QT TVBR -q values so that they match the iTunes 128 CVBR and DivX -v 5 average bitrates? Or just kick out DivX?

Chris

This post has been edited by C.R.Helmrich: Feb 7 2010, 00:31

[rpp3po]

CVBR 128 --normal is through.

All results so far:

QT CVBR 128 --normal: average 134 kbit/s, median 134 kbit/s, min 88 kbit/s, max 144 kbit/s
Nero AAC 1.5.1.0 q 0.41: average 129 kbit/s, median 133 kbit/s, min 81 kbit/s, max 143 kbit/s
QT TVBR Q(59-68) --highest: average 129 kbit/s, median 134 kbit/s, min 72 kbit/s, max 168 kbit/s

Sigh, now how on earth are we going to integrate the DivX encoder into the test in a fair way? Increase the nero and QT TVBR -q values so that they match the iTunes 128 CVBR and DivX -v 5 average bitrates? Or just kick out DivX?

The next higher Q value for QT TVBR would land us in the 140-150 kbit/s range. Initially it was said that we take CT or DIVX. Maybe we can get closer results for the CT encoder?

PS I thought a little about why my relative average for QT CVBR is much higher than Igor's. I think it is because my collection includes more old recordings. The two less constrained encoders don't allocate more than about 80-90 kbit/s to those while the CVBR mode does barely produce anything below 100 kbit/s. Without those old Jazz albums Nero's and QT's TVBR average would be higher, closer to CVBR.

This post has been edited by rpp3po: Feb 7 2010, 01:43

[IgorC]

rpp3po

Thank you.

Ok, I see. CVBR 121 is more closer to TVBR 60. As Windows users haven't access for odd CVBR bitrates I will ask you to encode files for the test.


Alex B,
I sent email to Divx delevoper to see if it's possible to get ~128-130 kbps.

[rpp3po]

No problem, Igor. I can do that.

While you have written this, I added some thoughts to my last post about why our averages were different. I think without the old part of my Jazz collection (and Jazz was quite overrepresented), the three averages could have been closer together (>130kbit/s). So I could live with both choices: providing custom, Apple encoded CVBR samples or accepting iTunes as encoder for CVBR. Whatever finds the most consent...

This post has been edited by rpp3po: Feb 7 2010, 01:48

[IgorC]

Your old Jazz music is part of real scenario.
On my collection CVBR 128 has also slightly higher bitrate 131 kbps while TVBR 60 is 129 kbps for my and your collections.

CVBR 128 is a little bit higher (2-4%) than TVBR 60.
CVBR 121 is on par but with some old recorded music.

Then CVBR 124-125 would be the best balance.

This post has been edited by IgorC: Feb 7 2010, 01:59

[.alexander.]

Above is a graph of average bitrate for looped emese sample. iTunes / 128kbps

[C.R.Helmrich]

Thanks, Alexander. Is that CBR? If so, could you do the same for iTunes CVBR 128?

Thanks,

Chris

[.alexander.]

Thank you for the interest Chris,
The sample was encoded with iTunes VBR so probably it is CVBR.