Hello.

There have been some delays, but here are the so awaited results of the latest 64kbps group blind listening test.

http://audio.ciara.us/test/64test/results.html

And the final plot:
Each vertical line segment represents the 95% confidence interval (using ANOVA analysis) for each codec.


Note: Lame MP3 is 128kbps "high anchor" in this test, FhG MP3 is the "low anchor"

Zoomed version without the anchors:


Codec specifications:

• Ahead/Nero 6.0.0.15 HE AAC VBR profile Streaming :: Medium, high quality
• Ogg Vorbis post-1.0 CVS -q 0
• MP3pro (from Adobe Audition 1.0) VBR quality 40, Current Codec, allow M/S and IS, allow narrowing, no CRC
• Real Audio Gecko (from Real Producer 9.0.1 64kbps
• Windows Media Audio v9 VBR quality 50
• QuickTime 6.3 AAC LC 64kbps, Best Quality
• Lame MP3 encoder 3.90.3 --alt-preset 128 --scale 1. (high anchor)
• FhG MP3 encoder (from Adobe Audition 1.0) 64kbps CBR, Current codec, allow M/S, no I/S, allow narrowing, no CRC. (low anchor)

Thanks to everyone that participated and helped!

Best regards;

Roberto.

yummie

hm, first time that i voted most of the codecs worse than the average (especially wma was 1.77 in my case )

seems that my hearing evolves (dunno if this is a good thing )

edit: perhaps an even more zoomed in graph (leaving out lame too which is also at 128kbps) between 4 and 2.5 would be nice

Interesting...

I expect Vorbis to be a little higher, and wow, Lame is really up there.

Bloody 5:20 AM here.

I'll read criticisms, comments & al. in 7 hours :B

One last thing: if anyone is interested, the results are all available as a single zip here:
http://rarewares.hydrogenaudio.org/rja/comments.zip

Enjoy!

Best regards;

Roberto Amorim.

BTW: If the server starts with that redirection fag0try: Blame Verloren

One last thing: People are invited to announce it on Slashdot, Kuro5hin, RAO or anywhere you think it fits. This is the best test I performed so far and, IMO, with the most interesting results. So it deserves (i think) some advertizing

Roberto, can you please move the anchors (low-high) to the right - and/or, indicate that LAME is 128 kbps, otherwise the test graph results might be very misleading

The group results seem to be:
Lame > He AAC, MP3Pro, Vorbis (with He AAC > Vorbis) > Real, WMA, QT AAC > FhG.

Listed in descending order of mean, with >'s only where they're significant.

I tested 11 of the 12 samples, so I thought it'd be quite interesting to apply a similar analysis to my results. I get:
Lame > Vorbis, MP3 Pro, He AAC > WMA, Real Audio, QT AAC > FhG.

So my results agree with the group, except that instead of He AAC > Vorbis, I found Vorbis > He AAC (although it wasn't quite at the 5% significance level). This may be influenced by the fact that I didn't test Polonaise, which seems to be a poor sample for Vorbis -- it's quite possible that, with that sample in, I'd be even closer to the group average.

Well, it's good to know that I agree with the majority .

PS. neoufo51:
Lame is 'really up there' because it's at twice the bitrate of everything else! It was in the test to see whether any of the codecs lived up to the hype of some of them (wave at WMA ) -- the marketing is often claiming 128kbps performance at 64kbps. The results of this test indicate that none of this generation of codecs are there yet.

Perhaps grayed

Sure. But tomorrow, since I would nearly have to rebuild the spreadsheets.

Or maybe someone wants to do that for me?

http://pessoal.onda.com.br/rjamorim/plots.zip

(nevermind the comments, they are from the 128 test)

It might be also interesting to replace "Lame MP3" with "Lame 128"

G'night to you all :B

Here are the averaged results of my own tests on just five samples, in case anyone's curious as to what an untrained newbie can and cannot hear...

LAME MP3 (128kbps anchor) - 4.76
LC AAC - 4.66
MP3Pro - 4.64
HE AAC - 4.64
WMA Std - 4.32
Real - 4.06
FhG MP3 - 3.72
Vorbis - 3.64

At least the upper anchor sounded the best to me...otherwise I'd run out tomorrow and get my hearing checked.

The ironic part? Vorbis is my lossy codec of choice.

I obtained different results, with bad notation for vorbis (unfortunately, I forgot the matrix on another computer). I'm not at ease with vorbis at this bitrate during a blind test : it sounds too particular (hiss, desquilibrated tonal range : more treble, poor low-medium, and limited stereo), and it's easy for me to detect the encoder. I'm rating vorbis, and not an unknow encoder. So it isn't blind anymore.

HE-AAC was my favorite : often the best - never the worst. Sometimes betrayed by a grainy texture, the same as mp3pro one. No noise packets, as heard with the first releases of the encoder.

Lower anchor was rarely the worse file I rated : on 8 files, I rated other encodings as worst one. I prefer an excessive lowpassed sound without artifacts than a richer sound, but destroyed by flanging. Personnal taste.

WMA9 (I hated this encoder) was as often the best file than HE-AAC. But it was three time the worse for me. So it isn't a reliable encoder, but on some situation, it works very well.

LC-AAC was first on two sample (02 and 09), last on one (11).. Vorbis best on one (04), and last on two (06 and 09).

That's not all that odd, really -- if you've listened to Vorbis encoded files a lot, you've probably grown more sensitive to its distinctive features, and so you can pick Vorbis out of a crowd more easily than some other codec which has other problems. When MP3 first appeared, it took quite a while for even people with 'good' hearing to detect the problems inherent in that format, even though the problems would stand out a mile if you could listen today to your 7/8 year old MP3 encodes.

With just five samples the scope for meaningful statistics is reduced, but we can say that you were probably able to detect Real, FhG MP3 and Vorbis from the original, and not able to detect AAC or MP3 Pro.

It takes time to be sensitised to codec problems, which is why most people can happily listen to encoded tracks which would lead many of us to commit suicide with a pointy stick within 30 seconds...

edit: my zoomed in pic is already on the official presentation page

Hey, I think it might be worth emphasizing (or at the very least, mentioning!) on the original post that Lame is at 128 kbps and the others at 64 kbps. That fact took me a while to realise, and I suspect the post will be leaving a lot of people with the impression that Lame at 64 kbps is better than the other codecs at 64 kbps, when this is obviously not the intent!

Ditto. Well, I wasn't sure it was vorbis (because I've never used it), but I hated it and soon recognised it in all subsequent samples. It just killed the stereo. This would have been less obvious for non-critical listening over speakers (i.e. not in the sweet spot), but over headphones it was useless!



I'm sure I wrote in one of them "I can hear the low pass, but it's less annoying that what some of the others are doing!". There was nasty temporal smearing that destroyed the "fun" of the music in a way that a low pass doesn't.



Roberto, what do the results look like if you transform each person's scores into rankings, before doing the analysis?

Cheers,
David.

btw(!) Thanks for conducting such an excellent, and interesting test!

EDIT: Can hydrogen audio set up a directory of listening test results (with or without samples)? It would be useful to have them all in one place, and backed up for if/when other peoples servers go offline.

What do other people think of this idea?

Cheers,
David.

EDIT2: Does this mean that any codec producer who says "sounds as good as 128kbps mp3 at only 64kbps!" can now be taken to court for false advertising?

Note than Vorbis sufffered by an artifact (reverberation or hollow sound), introduce by latest CVS encoder, used in this test. See :
http://www.hydrogenaudio.org/forums/index....pic=7197&st=25&

(I'm not enterely sure that previous encoders didn't have this flaw).

I can't say that for me:
lame > HeAAC > Real > Vorbis, mp3pro, QTaac > WMA > FhG

Like in the c't test I'm a "Real Audio" fan!

I tested only 4 samples though (exactly those with the least participation B)).

Very likely. You'd be amazed (and I'm embarrassed to say) how many 5.0's I noted...about half of the test groups across the samples I tested. Oh well. That just means I need more practice listening for artifacts. (But in the meantime, I save a *lot* of hard disk space with medium-low bitrate music that sounds just fine to my ears.)


Well if they say it here, anyway, they'll be flogged with TOS rule #8.

Slashdot is going to be hell if this is posted and the lame 128kbit sample isn't properly labeled in the graph :-O

Also, how many participated? Ah.. between 26-43. Not a very large sample then.

Might the Wiki be a suitable place for this repository?

My results looked like this: Lame, HE AAC > MP3Pro, Vorbis > Real > WMA, QT, FhG. So I pretty much agree with the overall results.
That listening test really was a very interesting expereince. I found it especially surprising just how well you can train yourself to hear artifacts. In the beginning I was rarely able to tell more than 3 codecs from the original. But after some hours of continuous testing (I tested all of the 12 samples), I mostly had no problems identifying the encoded samples instantly. I even repeated some of the tests I did early on, because I tended to generally rate everything lower after some training.
As for the ratings, I did rate WMA and QT almost exactly the same overall as FhG. Still, these codecs have quite different artifacts. Especially WMA really annoyed me because, on most of the samples, I could hear a constant very high frequency ringing to an otherwise more or less well reproduced sample. Sometimes this made the sample just plain unlistenable (it's most noticeable in Waiting and NewYorkCity). I never rated WMA better than 2.0 because of this.
Generally, my ratings are based on high frequencies. I have pretty good high frequency hearing, so if a codec lowpasses excessively, I find the sound detail-less and dull. On the other hand, things like stereo image are not really important for me (I didn't use headphones).

WOW, very interesting! The results are so far different from my own that I think I should have my hearing checked. Here's the original post I had in the other thread:


Obviously, everyone else is hearing a problem with WMA that I'm not. I'm going to have to look over the comments to see. Real Audio, FhG and MP3 Pro had very damaging lowpasses to my ears. I have no idea how either got rated as high as they did. They sounded horrible to me!

I guess everybody assumes someone else will do it for them

The results are statistically valid regardless of the small sample size.

Will the next test be a 96kbps test, with the same player?
Would be interessting if LC can beat HE in that range or if Lame 128 will be beaten by any codec.

he-aac isnt available for 96kbps afaik

I agree 100%. I often knew right away which one was vorbis, and I struggled to not let that influence my ratings. Am I rating it too high because I'm an OSS true-believer? Am I rating it too low because I'm overcompensating?

I also agree that its performance was spotty. A few samples had very serious problems (for me, those included Illinois, Polonaise, gone). I hope 1.0.1 fixes these issues and some of the problems exposed in the 128k test.

bitrate = 95
channels = 2
samplerate = 44100
codec = AAC+SBR
tool = Nero AAC Codec 2.5.5.3


Nero 6 can do it. Whether it's smart I don't know.

The actual quality problems won't be fixed until 1.1 I guess. Most of the major issues I heard have been there since 1.0RC2 or so...

which settings did you use to create such a file?

Umm, just select cbr 96kbps and High Efficiency profile..

hm, i meant the vbr profiles, as i dont think that ahead wanted to make it possible for example to create he-aac files with 448kbps by using the cbr option

also the presets use lc aac for ~96kbps (vbr and cbr)

Well you can't do that, it changes to LC if you try to create a cbr 112kbps HE file. 96kbps cbr is the highest allowed HE profile setting.

Gosh! I only listened to one sample (New York City)... but I gave Vorbis a 1.3 Even the muffled lo-fi sounds of QT and WMA sounded nicer to me than the lavish sweeps of distortion and generous, unrestrained servings of noise that were dished out by Vorbis. I really wasn't expecting that at all. Still, I suppose I shouldn't judge it on one sample alone...

And Lame really is very good, (even if twice the bit-rate)... it's nice to know that.

Anyone else think it interesting that Ogg Vorbis was beaten by QT in the 128kbps test, but crushed QT in this test?

Of course, HE-AAC beat Ogg Vorbis, as did MP3Pro. Thus, SBR technology beats out Ogg Vorbis.

Not really. It was pretty much expected, since QT doesn't have a High Efficiency profile.

Great test! However, I wonder who is doing 64kbps encoding. I believe that most people use at least 128kbps or 96kbps at the very minimum...flash card prices are getting lower and lower...anyway, that is why I'm very glad that a 128kbps test already took place...
I hope that other test will follow...how about a 160kbps test

By the way: What is HE-ACC?

Thanks

HE AAC is one of the profiles of MPEG-4 AAC. It uses SBR (Spectral Band Replication) in order to achieve a high efficiency at low bitrates. Currently there's only one publicly available HE AAC encoder implementation - Nero AAC/AAC-HE encoder.
Check here for more info about HE AAC.

Actually, I'd like to see it go even lower... a 32kbps test (with samples containing some music, but mainly speech) would be a fairly good reflection on low-bitrate streaming -- it's around the rate used by the BBC's RealAudio streams, anyhow. Many people found it hard to detect artifacts even at 64kbps, so as we increase the bitrate the likelihood of getting decent statistically valid results crashes through the floor.

Thinking about this test has revived an old idea of mine, which would be to test the samples without the original present -- the users would then mark which sample sounds better, rather than which sounds closest to the original. I've not yet been able to figure out a decent way to analyse the results, though. Not having a scale opens the possibility of a non-transitive chain: i.e. a set of samples X_1,..., X_k where X_1 is preferred to X_2, X_2 to X_3, ..., *and X_k to X_1*. I'd love to see something like that happen in practise.

This is the table of my results:



According to the averege the best my preferred encoder seems to be HE AAC even if i should say that i’m quite surprised for the result of mp3PRO (or maybe i’ve overestimated HE AAC).

I’ve done a sort of ranking for the best, real antagonists at 64 Kbps. The first position is colored in green, the second in yellow and the third in red. With this direct comparison Mp3PRO is (according to my preferences) often better than HE AAC.

Mp3Pro has shown a detectable lowpass (16Khz) but this is not the real problem for a 64 Kbps, artifacts are more annoying. There is an interesting thing i’ve perceived: with HE AAC the high frequencies seem unnatural, attenuated, as if it was lowpassed. While doing the blind test i imputed this to the lowpass, but later i’ve discovered that the HE AAC files are lowpassed at about 20 kHz (surely inaudible for me). Does FAAD use dithering when decoding ? If not, i think that the fact could be explained with the SBR “problem” of which guruboolez give us an excellent description.

While the two codecs above scored a very close quality level, i can’t say the same for Vorbis that is often behind the others two. I surely agree with guruboolez: Vorbis in this bitrate range is easy detectable because of noise and exaggerated highs (with sharp attacks the result is quite annoying).
We all are waiting for the 1.1 version that should give better result with this type of artifacts.

At the end, i sincerely want to thank Roberto for his effort organizing this useful test. The number of participants is increased and this is a clear indication of good organization.

hmm, surprisingly wma was very close to the low anchor FhG on some samples.

wma was probably the standard edition?
lame was probably abr 128?

* Ahead/Nero 6.0.0.15 HE AAC VBR profile Streaming :: Medium, high quality
* Ogg Vorbis post-1.0 CVS -q 0
* MP3pro (from Adobe Audition 1.0) VBR quality 40, Current Codec, allow M/S and IS, allow narrowing, no CRC
* Real Audio Gecko (from Real Producer 9.0.1 64kbps
* Windows Media Audio v9 VBR quality 50
* QuickTime 6.3 AAC LC 64kbps, Best Quality
* Lame MP3 encoder 3.90.3 --alt-preset 128 --scale 1. high anchor
* FhG MP3 encoder (from Adobe Audition 1.0) 64kbps CBR, Current codec, allow M/S, no I/S, allow narrowing, no CRC. bottom anchor

Well, definitely not from me. I'm not interested in testing 96kbps.

Anyway, I replaced all the plots. Now Lame MP3 is Lame 128. This should help avoid confusion.

@bond: Thanks, I used your zoomed in version

great that it was usefull

At first, let me thank Roberto for his efforts: thank you! It was nice to see that the participation level was high.

For me HE AAC and MP3 Pro came out on top, while Ogg was just mediocre. Forget the rest. The below is only true for my personal ratings and usually the anchors are disregarded, unless mentioned otherwise.

The most surprising result was with sample 06, Illinios. Here HE AAC totally sucked, while Ogg shone (and the others did well too). This is contrary to the average public ranking.

On sample 9, Polonaise, MP3 Pro was worst. MP3 Pro also has the highest standard deviation of all encoders (including anchors).

Sample 07 was also interesting. While all other encoders dipped real low in quality, HE AAC was doing well.

Also worth mentioning is that I didn't rate Lame 128k as best on 6 samples. In 5 of these cases, HE AAC was rated higher than Lame.

WMA std was rated worst 9 out of 12 times. WMA std was also most consistent in quality. It consistently sucked.

On the 4 occasions that FHG was not rated lowest, WMA std was worst.

Me! I use 64-80kbps Vorbis to put albums on an USB RAM stick and carry home to use in my mom's portable while coding.

Time to switch to HE-AAC for that...

You got that right!



Doesn't work. Why? Well, see you own comment above...

Hello.

Menno came with an idea: Replacing "Lame 128" and "FhG MP3" with "High/Low Anchor" on the plots.

As to completely avoid confusion, since several people are thinking Lame won, and it wasn't there to win or lose to start with.

Any comments? Suggestions on alternatives?

Yes, it's a good thing. A different color maybe for each anchor, in order to avoid subconscious confusion.
Is it possible to re-assignate a different place for the different plot ? I mean : from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.