The public listening test of the german c't magazine is over.
They also did a private listening test with some audio experts (including Andree Buschmann).

Overall public rankings (best to worst):

64kbit/s:
Wave - Ogg Vorbis - MP3Pro - WMA - AAC - RealAudio - MP3

128kbit/s:
Wave - Ogg Vorbis - WMA - RealAudio - MP3Pro - MP3 - AAC

Note that the 128kbit/s files rank very close. Ogg Vorbis was judged by the experts to be the best lossy encoder while MP3/MP3Pro come in last (the results vary alot from one person to another).

I scanned the page with the online results:

Results of the c't Online-Tests


Overall number of online participants was 7240. There were 6080 valid ratings, 3295 of those for the 64 kbit/s test set.

Explanation of the picture: Top to bottom: Best to worst. Prozent = Percent, Platz = Place. Example: In the 64 kbit-test, Ogg Vorbis was rated as the best codec (better than the hidden reference) by 25% of the participants. MP3 was rated as worst by 90% of the participants.

Whoa, this is great news...

Are the raw results available?

You mean, all 6000+ submissions, in order to make your own statistical analysis with them? Hmm, i don't think they are.

However, you can try your luck and e-mail c't about it...

Quite strange that AAC 128 is the worst, even worst that MP3, since, if I'm not wrong, it was designed to be an improvement over MP3, which itself (MP3) was designed to be "near-cd" quality at 128 Kbps.

Which makes me wonder: any info about the used encoders?

Certainly.

MP3: MusicMatch 7.2

MP3Pro: MusicMatch 7.2

AAC: FhG IIS-A Eval Build, August 23

Ogg Vorbis: 1.0

RealAudio: HelixProducer 9 Plus

WMA: Windows Media 9 beta 2798

I just can't understand this tests, why they use only lower bitrates? i never use less than 192 Kbps in a MP3 or --q6 in a OGG. Can you explain me that? i'm really curious

Did they post the overall rankings of the private listeners?

If I understood correctly, the listeners couldn't distinguish the 128kbps Ogg from the original. There is no point in testing at higher bitrates in this test setup.

More details:

http://forum.doom9.org/showthread.php?s=33...&threadid=33110

I have some issues with this:

MP3, MP3Pro and AAC are marked as 'kostenlos' (arguably)

MP3 is marked 'all operating systems', but Vorbis isn't

The encoding times of Vorbis look really really strange (compare the 64kbps to the 128kbps numbers)

The test must have been hacked by the ogg-vorbis mafia!




They don't say anything about VBR at 64 kbit. So maybe they used ABR (slower?)

[quote=Garf,Sep 8 2002 - 01:17 PM][/QUOTE]
If I understood correctly, the listeners couldn't distinguish the 128kbps Ogg from the original. There is no point in testing at higher bitrates in this test setup.[/quote]
Now that's a good thing

They did, but only for the "freestyle event", the 160 kbit/s test with own equipment and test samples.

The 8 invited "expert listeners" took the normal tests (64 and 128) on equipment provided by c't, and an additional run at 160 kbit/s (VBR) on that equipment (in addition to the freestyle test). The individual results are published.

These are the overall "freestyle" test results:

160 kbit/s results of 8 invited expert listeners

In case someone doesn't understand german , "arguably" is not the translation of "kostenlos"... (it means "free").

Sort of. [edit: oh yes, these are only the 160 freestyle results]
Wave - OGG Vorbis - AAC - WMA - RealAudio - MP3 - MP3Pro
The individual rankings differ by a great deal. I haven't done any maths, but judging from the graphs, I think the only things you can say with a certain amount of confidence is: out of the lossy bunch, Ogg is best while MP3Pro is worst.

The private contestants also had to do a different test than the public. They were given a 60s clip with various samples in them (including the ones from the public test). Each got 3 CDs with the 64, 128 and 160 tracks. Track 8 was the reference and inside of the other seven they placed another hidden reference. The contestants had to rank each sample set. They were given 20 minutes for the whole task. Hardware setup: 2x Yamaha CDX-596 and an A/B switch. Monitor speakers by Quested.
In a second round with unlimited time, the participants could bring in their own music which was put on a cd in a likewise fashion at 160kbit/s and listen through their favorite headphones or whatever they liked (the Heise guy listened to the samples in his car).

Also, this is not correct: Marked as "kostenlos" (free) are: MP3, MP3pro, Ogg Vorbis and Windows Media 9. It might also be that they mean the applications, not the codecs themselves.

The AAC rating has a footnote (3) that says: "Developer version, not yet available", so they mean the used program.

Looks harder to conclude things from this. It looks like MP3Pro sucked clearly, whereas the rest isn't very different from each other.

Vorbis does seems to score quite well again.

The main problem of the test is the selection of the test samples:

- three fully different samples with huge loudness changes were merged
- first clip has bad quality also as WAV file

For testing I first generated 9 files from each file and compared this pieces.
It is very likely that >99.9% didn't do that.

To my mind the bad result of the AAC was related to the 0.27 dB level reduction.
0.27 dB level difference is nearly inaudible, but combined with a 18 kHz this becomes audible.

Is there any info about the settings and the exact codec versions they used? On their website they mentioned a closer description of the different encoders "from page 102 on", I think... I would like to know if they used CBR or ABR in order to compare them all on a fair ground or if they used VBR with the ones that offer it. Also I'm missing the info if they took v1.1.0 of mp3Pro or the elder v1.04. If they didn't provide these infos in the printed issue, I will not bother to buy it tomorrow...

The Heise newsticker has an article about the test. Basically it says Ogg is pretty good and at 128 there's not much difference, buy the magazine. German: http://www.heise.de/newsticker/data/anw-08.09.02-000/
If you want to get totally frustrated by usnubstantiated claims, rude remarks and pure ignorance, then also dive into the newsticker forum.

They don't mention the concrete encoding parameters used. You can criticise the test in many ways. c't has written a small side note responding to some of the criticism. They had several concerns about filesize and the number of test samples driving potential testers away. Keep it simple. However they advocated a blind comparison which I think is a big step forward and hopefully many users became aware that they should be judging by ear and not by graphs. Unfortunately in the software comparison a few pages later in the mag, they once again draw frequency plots and look at the cutoff. Nevertheless c't does mention that a lower cutoff is preferred instead of warbled sound. And after all, the results are similar to the ones we allready have from ff123's 64k goup listening test.

I uploaded all results from the test I made on august 28 and 29.

http://www.uni-jena.de/~pfk/mpp/ct.txt

It looks like I should do another test and write all down on a nice webpage.

People for which Ogg Vorbis, Real Audio and MP3Pro sounds good should NOT use Musepack,
because their ears seems to be fully different from my ears and Musepack is tuned for my ears.

[kidding] And if using WMA? [/kidding]

Indeed. I liked Real and Vorbis in the test. Now I made a comparision with MPC 1.1b quality 4.55, and rated them higher again.
BUT: my decision was mainly based on the Otello excerpt, MPC sounds extremely noise-reduced (wheras vorbis introduces echo and Real Audio is a little brighter).
In the other excerpts (especially Jazz), MPC beats Real Audio.

I don't like Vorbis at low bitrates either, and Musepack is very appealing to me

Imo this is a bit generalizing comment. You are talking about 128kbps average here right? But somebody might get an impression that if one likes Vorbis, he cant like MPC even at higher bitrates..

I mean 128kbps average is pretty damn low for a subband coder although with PNS. Imo it should be emphasized that MPC although being a subband coder can have a comparable quality even at low (128kbps average) bitrates.

For the range from 100...130 kbps Musepack has

- no Intensity stereo
- PNS is bad implemented (simple SMR based switching criteria)
- LPC is not implemented
- It should be possible to squeeze some bits from better lossless encoding

I have serious problem with Vorbis and the c't test sample. These problems
seems a lot of people not to hear. So MPC may be tuned in the wrong direction???

Problems are:
- Stereo imaging problems (I would call it jumping voices) [64 kbps, 128 kbps]
- Too much treble for percussions [64 kbps, 128 kbps]
- too less treble for reverbation and low levels [64 kbps, 128 kbps]
- unstable quantization noise [64 kbps, 128 kbps]
Not in this test:
- encoding of male voice or alto female voice

I think it should be possible to solve these problems at 128 kbps,
but it is very likely a long way to go.

OK, after reading thoroughly through both articles (yes, I couldn't help buying it ) and trying to squeeze out just a little bit more about the codecs and their settings, I can give at least these additional infos:

My 64 kbps darling AAC is an evaluation version from FhG, as we already know, built especially for this test, as it seems (date: 23.08.2002). It is capable of using "Long Term Prediction", part of the MPEG-4 tools, as opposed to QuickTime 6.0 AAC (can only use the "Low Complexity" profile) that ff123 had used in his former comparison. The cutoff frequency for the tested FhG AAC is stated at 13.3 kHz. No Spectral Band Replication was involved here, only mentioned as a new technique in Coding Technologies' AAC+. Does anyone know if PsyTEL's current AAC version uses LTP for low bitrates, too?

The MP3 and mp3PRO codec was the one in MusicMatch 7.2, so that probably means FhG FastEnc and Coding Technologies. Can anyone who actually uses MusicMatch shed some light on this and maybe take a look at the exact codec version of mp3PRO in this software, please?

RealAudio actually was RealAudio Surround from Helix Producer 9 Plus, meaning that they ignored Real Network's warning that there might be related artifacts when used with pure stereo signals (I heard a warbling underwater sound with the bass in "Horny"). Nevertheless c't mentioned this warning, which is a good thing, and added that in pre-tests this Surround version already sounded better than RealAudio 8. But they couldn't help drawing that dreaded conclusion from the graphs again that "a 13 kHz lowpass wasn't enough for most of the listeners' ears and so RealAudio was rated second worst" (reminds me of some similarly "eloquent" articles in the Heise forum).

WMA already was WMA9, so the test might be used as a comparison for anyone who wants to know more about this new Microsoft codec. But as for all encoder settings, it is not mentioned if they used CBR, ABR or VBR (and which option of WMA9's different VBRs, if in use). So I can only guess from one remark right at the beginning that they probably used CBR all the time, because they only mentioned a variable bitrate for the 160 kbps "freestyle" part of the test, where all participants could bring in their own prefered music. If this is true, then it still would not be clear what they did with mp3PRO and 128 kbps and in which files they used it, because MusicMatch apparently can only use 96 kbps and VBR for a maximum bitrate of approximately 130 kbps (mentioned on page 104).

So maybe it is necessary to mail the guy that was responsible for this test and ask him about the missing informations. For example using CBR or ABR with Ogg Vorbis could perhaps explain the apparent background noise behind Kylie's voice that mainly contributed to my OGG ranking (second worst with 64 kbps).

It is not completely fair to put it that way. MPC was not compared by the other testers. BTW you didn't add remarks for the mpc encodings, were they "perfect" to you?

About tuning in a direction, what direction are you tuning? The lower bitrates?
To be honest I don't know of mpc-users that use anything lower than --standard (then again, how many do I know? )

The things you mentioned are important for transparency. If you hear anything wrong and can improve it that would of course be OK. The other way round, ignoring problems others would hear because you would not hear them, would be wrong . Did you still haven't found any test listeners? I read about a kind of MPC mailing list for MPC developers/testers.

I'm sorry I won't be of any help as test listener. I tested myself and I didn't come up with reliable results, when listening for the umptiest time I didn't hear the differences I (thought? I) heard the first time.
-
Ge Someone

English translation of the heiesticker article is here

Thanks /.

I heart no special characteristic problems with the current MPC.
No comments do not mean that there are no problems.
See also loudspeaker evaluation of loudspeaker tests of "Stiftung Warentest".
There's a quality evaluation and a evaluation of mentionable coloring.

One thing I started to wonder.. Which AAC decoder c't used? I just hope that they didn't use the buggy FAAD. I have no idea how long the FAAD-decoder actually was buggy though, but if they used the buggy FAAD, then the results are no surprise..

To my mind the AAC encoded files sounded great.
An it was audible more silent than the Ogg Vorbis file.
An a more silent file is always a l00zer.

CODE

--- 128 kbps ---
Vote   Offset    Level    File       Coding          ODG    Remarks
 2a  -1507      0.9888   6147.shn   MP3            -0.44   HF-noise
 1    -196      1.0000   6148.shn   uncompressed    0.00
 7    -197      0.9915   6149.shn   RealAudio      -0.68   some warpling, horrible rasping (krkrkrrr)
 4     +47      1.0052   6150.shn   WMA            -1.25   MF/HF-noise
 5    -196      1.0081   6151.shn   Ogg Vorbis     -0.52   much too sharp, aggressive, Rauschatmen, artifical sounding noise (zzzzzzz)
 6    +146      0.9941   6152.shn   MP3Pro         -0.84   warpling, pre-echos, modulation (schlschlk)
 2b  +1799.38   0.9699   6153.shn   AAC            -0.55   too much treble
 2c      0      0.9995              MPC            -0.17

Link

Sorry, I don't know which decoder they used or how they would have produced the referenz.wav at all (should have asked that in my mail to Sven Hansen, too... ). I think this reference was not sounding as good as one might have expected, not only with Kylie, but also with Mousse T's "Horny" (almost no stereo image there except for the reverb on the voice and by no way a typical "Jazz" recording because of all the synthesized instruments) and even worse with the opera excerpt that showed a constant hissing noise on the left channel and was sounding much too compressed altogether. It's hard to believe that the record company Deutsche Grammophon would have released a relatively new CD (from 1997) like this. But I don't own the CD, so I can only guess...

And which results are no surprise - mine or the overall results from all online testers?

The background noise was audible in all three parts.
It sounded in the direction of HiCOM compander noise.

The encoding were made from the hidden reference, not for the reference.shn.

Yes, but it really disturbed me when it changed poor little Kylie's voice so that she almost sounded like Tina Turner... I think this effect is contributing to the "Vorbis sound" giving an impression of brightness or harshness, at least at 64 kbps (I did not participate in the 128 kbps test, so I can't say anything about that part). This might also be the reason that so many people (25%) prefered Ogg Vorbis even above the original WAV.

Another part of this problem could be what you called "jumping voices": In the opera sample I heard that the cymbal crash in the orchestra tutti did not stay on the right side where it belonged in the original, but it showed up in the centre of the stereo image where it can be perceived more easily of course, although it probably wasn't louder than in the original. But this did not disturb me as much as this obvious "roughening" in the first sample.

I've noticed that Dibrom stays out of most of these debates about sound goodness of ogg vs mp3. I think the results in this test would have been much different with LAME 3.92 as the encoder. If you can wait for a LOT of processing, LAME VBR can produce superb sound. I really think Dibrom and associates deserves kudos for continuing to make audible improvements in a format based on such old technology and that even Dibrom seems to want to see fade away...

"kostenlos" = "free of charge".

You don't have to pay for usage. It does not mean "free".

Frank Klemm giving me english lessons. That's just about the limit.

From where I quote.....

"Parallel to InterNet-heard met c't eight experts in the Peppermint pavilion, the studio of the music producer Mousse T. except Mousse T. sharpened the Sopranistin Carmen Fuggiss of the state State of Hanover, a blind clay/tone engineer, a clay/tone master, a 12-jaehriger pupil and the developer of the audio codec MUSE luggage (in former times MP+) Andree shrub man the ears. "

....which to me seems more like Double-Dutch than English. Having four young grandkids I tought I was quite good at Double-Dutch but this passage defeats me.



ß

What a delicious Babelfish salad again... It tastes like Triple-Dutch mixed with a pinch of Pidgin English to me.

What they were trying to translate was that c't invited eight experts to Hanover that met in the studio of producer Mousse T who also took part in the test. The other participants were a female soprano singer from the opera house in Hanover, a blind sound engineer (probably mentioning this because he might have better hearing abilities than people with eyesight), a supervising sound engineer ("Tonmeister" in German), a 12 year old school kid (with young ears and singing in a choir, too) and Andree Buschmann (former developer of MP+, now Musepack and showing up with "castanets", "spahm" and "fat_boy", by the way ). They missed out the c't chief editor (prefering to do the "freestyle" part of the test in his car) and a professional headphone tester from Sennheiser who came up with his state-of-the-art working gear (Sennheiser Orpheus, tube headphone amp and so on).

This last person seemed to be the only one who was able to handle the very difficult test situation, because his results (at 64 kbps) were almost equal to mine. In the first part they had only 20 minutes for three different rows of test files at 64, 128 and 160 kbps. In my opinion this was almost impossible for anyone who wouldn't do this every day (like the headphone tester). I had to stretch my 64 kbps test on two half days, most of the first day trying to figure out a valid and reliable test method which had to be done because of the test setup (3 very short samples in one file for each encoder and bitrate).

If you guys din't like low bitrate lossy music ... Use lossless ones g*d d*amn it! How hard can i be ....

Besides, If compresion algoritmens kepp getting better and better you may very well soon have lossless music that is takaing up less space than high bitrate lossy.