Public Listening Test [2010], Discussion Options

[C.R.Helmrich]

Before I purchased Audition, I used GoldWave to cut and edit samples: www.goldwave.com/

I think now that the looping issue seems to have disappeared, we should revert to the intermediate decision to run CVBR at 124 kb/s or so. Otherwise, we'd have to raise the bit rate for TVBR as well, right? IIRC, halb27 or rpp3po offered to encode using iTunes on a Mac?

Edit: Found the original discussion here. Since I can't find anything previous of that sort: rpp3po, do you volunteer to CVBR-encode the concatenated test sample when it becomes available?

Chris

This post has been edited by C.R.Helmrich: Mar 27 2010, 20:42

[IgorC]

Yes, CVBR 124 kbps is better to be compared to --TVBR 60 and Nero q041.
Who is MAC user?

rpp3po was going to encode to CVBR 124 but he isn't with us anymore. http://www.hydrogenaudio.org/forums/index....showtopic=79075
He was great guy here.

This post has been edited by IgorC: Mar 27 2010, 20:44

[C.R.Helmrich]

Oh my, I didn't see that thread! That's so sad Thanks for pointing me to it!

Bit off-topic, but I still have some messages from him in my inbox. It was about this delay removal tool Synchrotron. Maybe we should give that tool a try as a tribute to/in memory of him (or however you say that in English...)?

Chris

[.alexander.]

I think now that the looping issue seems to have disappeared.

Well then, it's a good time to make bets about (bitrate overhead) vs (quality scores) correlation coefficient.

[C.R.Helmrich]

Fyi #1, Apple seems to have updated its encoder. See nao's version number report. Let's test this new version even if it turns out to deliver bit-identical encodings for our target bit rate.

Fyi#2, we are still open for suggestions on which item to use as test sample #16. If you know a sample which is a piece-of-cake ABX at 128 kb and which has not been mentioned in that thread, please don't hesitate to let us know about it. The deadline is today... well, maybe the end of the week

Thanks,

Chris

[IgorC]

The choice of samples is completed now.

The problem is bitrate.

1. Is anybody MAC user here? We need shifted bitrate for CVBR --124 and it's possible only under MAC.

2. As already all people know we choose CBR (there is no VBR that would feet into average 130 kbps) for Divx but it doesn't produce 128 kbps neither. The real bitrate is 125-125.5 kbps. It's at least 2.7% less compared to other encoders. Is it ok for you or should we do not include Divx encoder into test?

Suggestions are welcomed.

This post has been edited by IgorC: Apr 14 2010, 04:04

[IgorC]

At this moment we have encoded samples with:
1. QuickTime 7.6.6 . TVBR 60 Highest. Here it produces the same output as older 7.6.5
2. Nero 1.5.4.0 -q 0.41 (VBR)
3. CT encoder 8.2.0 (Winamp 5.5.572 (January 13, 2010) dll & MediaCoder GUI) 130 CBR

This post has been edited by IgorC: Apr 14 2010, 04:39

[halb27]

... Divx but it doesn't produce 128 kbps neither. The real bitrate is 125-125.5 kbps. It's at least 2.7% less compared to other encoders. Is it ok for you or should we do not include Divx encoder into test?

Suggestions are welcomed.

I wouldn't care about a 2.7% difference.

[C.R.Helmrich]

Me neither. But for fairness sake I will evaluate the DivX encoder against a 128-kb bitstream of the CT 8.2 encoder (true bitrate about 126 kb).

Igor, please prepare/send the DivX and CT 128-kb bitstreams so I can do the pre-test. Thanks!

Edit: Found someone at the office with a full version of Winamp, and we noticed that it's possible to create both MP4- and ADTS-formatted LC bitstreams (format conversion in the playlist library). See this tutorial. So actually, it won't be necessary to make a 130-kb encoding to compensate for ADTS overhead.

Chris

This post has been edited by C.R.Helmrich: Apr 14 2010, 14:11

[Alex B]

DivX -v 5 = 136.3 kbps
iTunes 128 vbr = 133.6 kbps
QT tvbr 65 highest = 130.2 kbps
Nero -q 0.42 = 132.9 kbps
CT (Winamp 5.572) = 128.0 kbps

average = 132.2 kbps (=> test "target" = 132 kbps)

The bitrates are approximately within +- 3%.

My reasoning - the bitrate tests in Excel format (I tested my usual 25+25 complete tracks):
http://www.hydrogenaudio.org/forums/index....st&p=700607


Some notes:

-- Nero -q 0.42 would be closer to the average than -q 0.41 with my test set. (I tested 0.41, 0.42 and 0.43)

-- For QT CVBR, please don't consider using anything else than a setting that can be found in iTunes.

-- Winamp 5.572 "MP4 LC-AAC 128 kbps" produces 128 kbps files according to foobar. (not 130 kbps)

-- Use the real Winamp program for encoding "CT/Winamp MP4 LC-AAC", not MediaCoder.

-- DivX -b 128 produced 124.8 kbps with my test set - propably caused by this:

Known issues for DivX Plus HD AAC Encoder Beta 1:

The CBR mode does not pad frames if the input audio complexity does not require using all of the available bitrate. ...

(a quote from: http://labs.divx.com/node/11682)


Sorry, I don't have time to write more now. Testing and creating the Excel sheet took too much time already and I am late from something else...


EDIT

A quick edit though,

-- It would be interesting to see how DivX VBR works even though its bitrates are on the "high side" with some complex tracks. IMO, its behavior is more like true VBR than Nero's or QT's "true" VBR behavior. It is more in the line with LAME, Vorbis or Musepack VBR behavior. For comparison I added LAME -V5 to my Excel sheet.

-- I used the latest versions of all SW (checked the versions today).

-- Fixed the "quote" link.

-- I have posted some useful instructions for the DivX encoder here: http://www.hydrogenaudio.org/forums/index....st&p=686087. Additional info can be found from the above "quote" link.

This post has been edited by Alex B: Apr 14 2010, 15:49

[IgorC]

Alex B
A big thank you. I'm totally agree with all your statements and your setting will be used if everybody agree.
Only one thing. I've sent a PM to Benski (Winamp's developer) asking if bitrate shifting 132 for CT (MediaCoder) is ok. While real bitrate for 132 is actually 133. 133 is average for other 4 encoders.

Chris, I will send you all samples as soon as we reach agreement about CT encoder.
Sorry.

This post has been edited by IgorC: Apr 14 2010, 17:51

[C.R.Helmrich]

Thanks, Alex!

Igor, I already got the following bitstreams:

• Apple QT 7.6.6 TVBR. Got it from you.
• Apple iTunes CVBR 128kb. I installed iTunes on Win32 on Monday, encoded it by hand from the GUI.
• nero 1.5.4.0 -q 0.42, I also checked 0.41, but 0.42 is closer to the Apple encoders on our test set.
• Dolby/CT 8.2.0 128kb MP4. Encoded by hand via Winamp 5.572. See my previous post. I think we should use that one.

So the only thing missing is the DivX VBR bitstream (I agree with Alex, VBR is more interesting in our case).

If you want I can forward you my bitstreams for cross-checking tomorrow.

Chris

[benski]

Alex B
A big thank you. I'm totally agree with all your statements and your setting will be used if everybody agree.
Only one thing. I've sent a PM to Benski (Winamp's developer) asking if bitrate shifting 132 for CT (MediaCoder) is ok. While real bitrate for 132 is actually 133. 133 is average for other 4 encoders.

Chris, I will send you all samples as soon as we reach agreement about CT encoder.
Sorry.

Sure, setting the bitrate manually shouldn't result in any problems. 132kbps won't be as "well tuned" as 128kbps but the quality should always be nominally better than 128kbps

[C.R.Helmrich]

Good, then we take Winamp 132kb MP4 and DivX -v5. Once I receive the bitstreams from Igor, we will do the pre-test. If anyone else is interested in joining, please send a personal message to me or Igor. A few extra listeners won't hurt.

Chris

[IgorC]

ok, I will send you the files in a few minutes. CT 132 and Divx -v5

This post has been edited by IgorC: Apr 14 2010, 20:18

[Sylph]

Question: is anyone interested in adding the — if I have the right info — Liquid Audio AAC now in AVS Audio Converter 6 to this listening test?

And Fraunhofer's from MAGIX is also out of the picture?

[IgorC]

Impossible.
Too late. The sample are already defined and no any other encoder will be accepted.

Before I will start to argue I would like to ask you. Where were you during 3 months of discussions? You knew perfectly about it.

But even if it wasn't late those encoders still had very tiny possibility.
Reasons:
1. Did you ABX Liquid Audio AAC encoder yourself? It's awful as there is no any simple lowpass. No talk about something more advanced tools.
2. Sizes of packages of MAGIX. 100-200 MB just for AAC encoder? Apple is exclusion as it's well know here and personally I think it's reasonable price for high quality encoder.
3. Those combos encoders OGG-AAC-MP3-AVR-BLA encoders.... etc etc.. have no serious encoders.
4. AAC from RealPlayer was excluded as it has adware (plus not so good quality CBR encoder) and today it's shame for company who put adware to their products. Maybe MAGIX has no direct adware but it's somehow get some extra control in OS which isn't funny.

This post has been edited by IgorC: Apr 15 2010, 18:09

[C.R.Helmrich]

Relax, Igor My 2 cents:

1. We had defined a limit on the number of encoders in the test. This limit is reached. So we like to test only the best encoders available. I trust Igor when he says that the Liquid Audio encoder is not one of the best.

2. Since the encoder version used in the Magix software, the Fraunhofer encoder has undergone some considerable quality tuning. We decided not to include the latest Fraunhofer encoder because IIRC it's currently not availble in a cheap software or as a demo version "for everyone" (and because I'm helping to coordinate the test).

If you like to know how the current Fraunhofer encoder sounds, I'm sure there will come a time when you can listen to it through some software.

Chris

[Sylph]

Wow. One wants to help and gets attacked. Wow. I have MAGIX so it wouldn't cause me any additional effort to encode.

And I actually didn't think you had any knowledge about AAC and any such stuff when this thing started so I thought it was going to flop. But I see C. R. Helmrich joined with many others and it's going great!

[Sylph]

If you like to know how the current Fraunhofer encoder sounds, I'm sure there will come a time when you can listen to it through some software.

Chris

I think that time will not come, sadly. With every day that passes, it seems less likely a Fraunhofer AAC will be implemented anywhere. Especially with QT & Nero AAC popularity.

But it's strange MAGIX hasn't implemented the latest one in MP3 Maker v16.

[Alex B]

Here's a line graph of the bitrates in my test. It shows interesting variation from a track to another. Naturally, it doesn't tell how the bitrate varies inside the files in individual frames, but seems like the encoders don't always agree with each other when they estimate the needed bitrate. For comparison, I included LAME -V5.



The tested tracks are:

CODE

Various

1 AC/DC - Highway to Hell
2 Adiemus - Boogie Woogie Llanoogie
3 Barry White - Sho' You Right
4 Björk - Possibly Maybe (Lucy Mix)
5 Davis Bowie - Starman
6 Dido - Life For Rent
7 Duran Duran - Astronaut
8 ELO - Livin' Thing
9 Erich Kunzel & The Cincinnati Pops - Theme from The Pink Panther
10 Evanescence - Going Under
11 Faithless - Mass Destruction
12 Garbage - Bleed Like Me
13 Jamiroquai - World That He Wants
14 Kraftwerk - Tour de France Etape 1
15 Morrissey - Irish Blood, English Heart
16 Paco De Lucia - Rumba Improvisada
17 Santana - Oye Como Va
18 Simply Red - You Make Me Believe
19 Sting - Desert Rose
20 The Beatles - Let It Be
21 Tina Arena - Symphony of Life
22 U2 - Vertigo
23 Whitney Houston - Queen Of The Night
24 Yello - Planet Dada
25 Yo-Yo Ma - Libertango

Classical

26 Aldo Ciccolini - Satie, Sports Et Divertissements, Le Flirt
27 Alfred Brendel - Beethoven, Piano Sonata No 15, Op 28 Scherzo
28 Baroque Festival Orchestra - Vivaldi, Sinfonia in C major, 3. Presto
29 Berlin Philharmonic Orchestra - Mozart, Requem in D moll K 626, Sanctus
30 Berlin Philharmonic Orchestra - Strauss, Also Sprach Zarathustra (Thus Spoke Zarathustra)
31 Berlin Radio Symphony Orchestra - Mahler, Symphony No 8, 2 Chailly, Ewiger Wonnebrand
32 Christophe Rousset - Farinelli, Il Castrato (OST), J.A. Hasse, Generoso risuegliati
33 Concentus Musicus Wien - Bach, Matthäus Passion BWV 244, Da ging hin der Zwölfen Einer
34 Daniel Barenboim - Mozart, Piano Concerto No 3 in D major, K40-3
35 Gérard Lesne - Vivaldi, Sonate Op 2 No 3 pour violon & bc, III. Adagio
36 Giuseppe Sinopoli - Elgar, Cello Concerto, Serenade for Strings, Enigma-Andante
37 Itzhak Perlman - Paganini, 24 Caprises, No 1 In E
38 Jascha Heifetz - Sarasate, Zigeunerweisen, Op 20 No 1
39 Jessye Norman - Angels we have heard on high (Trad.)
40 Kirov Orchestra & Chorus - Khatchaturian, Gayaneh, Säbeltanz
41 Leslie Howard - Liszt, Douze Grandes Études, S 137 No 1 in C
42 London Sinfonietta - Saint-Saëns, Le carnaval des animaux, Hémiones
43 London Symphony Orchestra - Ravel, Daphnis et Chloé, 10. Tres modere
44 Marie-Claire Alain - Bach, Wo Soll Ich Fliehen Hin BWV 646
45 Michael Nyman - The Piano OST, A Bed of Ferns
46 Orchestre Symphonique De Montreal - Elgar, Enigma Variations No 1
47 Philharmonia Slavonica - Bach, BWV 1067 Rondeau
48 The Cleveland Orcestra - Ravel Valses, nobles et sentimentales, 1. Modere
49 The Philadelphia Orchestra - Tchaikovsky, The Nutcracker, Op 71a (Ballet Suite) No 2
50 Zbigniew Preisner - Trois Couleurs Bleu (OST), First flute

This post has been edited by Alex B: Apr 15 2010, 22:00

[Polar]

And I actually didn't think you had any knowledge about AAC and any such stuff when this thing started so I thought it was going to flop.

It's hard not to label such an utterance as blunt and ignorant. If you had done your homework, you'd know about Igor's numerous and elaborate double-blind AAC listening tests which he took the trouble about to document them and share them with everyone right here, in this very forum. The search button is right there at the top.

[Sylph]

I'm not talking about ABX, it's other stuff.

[IgorC]

There is a simulation of ABC-HR and fake Sample01 packages.
Please report if there will be any error.

I have done the next steps:
1. Java abchr - java from here http://www.hydrogenaudio.org/forums/index....st&p=683924
2. Put FLAC 1.2.1 and FAAD (http://www.rarewares.org/aac-decoders.php) decoders in the same folder.

Links:
ABC-HR h*tp://www.mediafire.com/?gmdrnmmuwyj
Sample01 h*tp://www.mediafire.com/?inkymjmtdjj


P.S.
Low anchor is Itunes LC-AAC 64 kbps VBR

This post has been edited by IgorC: Apr 19 2010, 01:54

[Alex B]

I tried the simulaton. Technically it works fine. Some comments:

- In the last MP3 test the bat files were in the BIN folder. The isolated location made easier to find them.
For reference, I uploaded Sebastian's MP3@128 "ABC-HR_bin" here: http://www.hydrogenaudio.org/forums/index....st&p=701994 (In case you don't have it archived.)

- You didn't include actual instructions. Here is Sebastian's readme.txt from the last test:

CODE

Public MP3 Listening Test @ 128 kbps
------------------------------------

Welcome and thank you very much for taking part in this listening test.
Your results will be very valuable.

Here are the instructions:

1. Decompress the "ABC-HR_bin.zip" package to its own folder.

2. Download one ore more sample packages (location of the packages is
below).

3. Place the sample package ("SampleXX.zip") in the same folder as
ABC/HR and uncompress it. Make sure you keep the directory structure
intact or the configuration files will not work.
For an idea how the folders should look like (in case you
downloaded all sample packages), check "folder-setup.png".

4. WINDOWS USERS:
Navigate to the "bin" directory and run "DecodeXX.bat" ("XX"
being the number of the sample package you want to prepare for testing).
If you would like to prepare all packages, run "DecodeAll.bat".
Wait until the command prompt screen disappears.
*NIX (Linux, OS X, BSD, Solaris, etc.) USERS:
Linux users are asked to use Wine with "wine wcmd /c DecodeXX.bat" from
the "bin" directory.
OS X users are asked to handle decoding of samples themselves (sorry).

5. You need the Sun Java Runtime Environment version 1.5 or newer to run ABC/HR.
If you don't have Java, download it from here:
http://www.java.com.

You need to use the ABC/HR for Java version provided in the
"ABC-HR_bin.zip" package together with this readme. Older versions are not
compatible with the configuration file format.
WINDOWS USERS:
Double-click "abchr.jar".
*NIX USERS:
Run "java -jar abchr.jar" from the shell.

Once ABC/HR is open, click "Open ABC/HR Config..." and load the
file "SampleXX.ecf" (again, "XX" being the number of the test you want
to take - make sure you ran the batch file respective to that package
before).

6. Take the test. If you need information on how to properly do that,
check ff123's page: http://ff123.net/64test/practice.html.

7. After you finish the test, save the results, (7-)ZIP, RAR or ACE them
together and mail the file to mail@listening-tests.info.
The test is scheduled to end on November 3rd, 2008. No results
will be accepted after that date, unless the test is extended. Possible
extensions will be announced at the test page (http://www.listening-tests.info).

You don't need to test all samples to participate. Even one single
result is already very helpful. Of course, the more you test, the better
for the final results' significance.

All results and comments I receive will be published. If you want
to be associated with your results, please enter your (nick)name in the
"Show name in results file" field in ABC/HR (check the checkbox next to
it to enable the field). Otherwise, your results will be anonymous.

---------------------------------------------

These are the sample packages:

http://.../~sebastian/Sample01.zip
http://.../~sebastian/Sample02.zip
http://.../~sebastian/Sample03.zip
... etc

- OR (alternate download links) -

http://.../Sample01.zip
http://.../Sample02.zip
http://.../Sample03.zip
... etc

The average package size is 4 MB.

Thank you very much!

Best regards,
Sebastian Mares

- Regarding the low anchor, perhaps 64 kbps is too low for emese (the demo sample). In the last test the low anchor choice was a bit unfortunate. Sebastian wanted to evaluate the progress from the first popular FhG MP3 encoder and picked an old FhG MP3 encoder from "really rarewares". However, that happened to be a buggy prerelease version (Roberto's description wasn't exactly accurate) and the encodings were really too bad to be anyhow useful.

I wonder if e.g. 80 kbps would be suitable for the test samples. The difference should be audible, but not like a night and day difference. Ideally the low anchor should give some hints about what kind of artifacts the contenders might produce. I.e. it should produce similar type of artifacts in a more pronounced form.

EDIT

Remember to include the licenses. You can copy them from Sebastian's test package.

This post has been edited by Alex B: Apr 22 2010, 00:13