I've recently finished my latest multiformat test @128kbps, featuring iTunes AAC, LAME 3.93.1, WMA9 VBR and Vorbis 1.01.
Results were interesting enough...

On 7 of 12 "classical" samples I used (thanks to Roberto for sharing) results were more, than just interesting...
AAC: nearly perfect, but sometimes have little troubles with hi-freq (percussion) signals and noices.
Vorbis q4: very slightly better, than AAC, but its attempt to encode hi-freqs made it fail on one sample (too annoying difference). Generally the same, as AAC.
LAME -b128: Poor. Difference is mostly obvious. It either artifacts badly or has degraded sound.
WMA Q75: Complete crap. Mostly even worse, than LAME (sic!). Incredible amount of artifacts.

If anyone interested in details I could provide them...

What do you think about WMA? Should I retest it in CBR (no pro, 'cause test was oriented on codecs with HW support) or just leave this crap? Any suggestions?

Codecs chosen:
iTunes for AAC for high quality in previous tests and price
LAME 3.93.1 - just it seems to be one of the most popular ones
WMA9 VBR - kinda the best (even though WMA8 CBR seems to be the most popular one)
Vorbis 1.01 for possible uality improvements over 1.0

So overall it's just a demonstration with no tweaking (this test was for newbies, not audiomaniacs).

Why did you not use alt-preset 128 for LAME?

Also, were these all ABX tests?

More details on the samples used would be nice.

I think he means the samples I used in my 64kbps test. (and will use in the MP3 test)

In case you want to check them out:
http://rarewares.hydrogenaudio.org/samples/samples.rar

@Yaztromo: I expected this question
Very simple. It was a test. oriented on newbies, so I decided to use default or popular settings, without tweaking anything. And -b128 is a way more popular, than ap128
And, surely, it was a blind test, using ABCHR. In many cases I used direct ABXing. Those cases, where I got reliable results with some efforts are given 4.5 (see below)

@rjamorim: Roberto, could I make a link to your sample set in my article?
btw, were there any "killer" samples? Myblood seem to be a good sample for getting a clear picture of how format's noise shaping works

This article was made for one Russian magazine, so I can't publish it right now for at least two reasons (Russian and © stuff), but test results would be given below. btw, if anyone is interested in proofreading this text in Russian (don't even ask to translate, it's 70k symbols already ), just e-mail me.

A list of samples (all marks would be given in that order):
Mybloodrusts
Newyorkcity
Bigyellow
Gone
Dafunk
Enolagay

AAC results:
4
3.5
4.5
3.5
3.5
4.5

Vorbis results:
3.5
4
4.5
4.5
2.5
4

WMA results:
2
2
3
2
3.5
3

MP3 results
2.5
3
4
2
3
3

Reading results: they're only given to compare one format against another. Absolute values won't give you reliable results.

Another thing. I already noted, that this test is made for amateurs, who aren't interested much in command line tweaking. They only want to know quality related stuff. Next, each of these formats has support in portables, so 128K test might be interesting from this side.

I'm asking for help once again: had anyone experienced quality related problems with VBR mode on WMA9 files? In my previous test (http://www.computerra.ru/compunity/dos/25596/index.html) it gave me poor results. So, how do you think, would it be more honest to update test with WMA9 CBR?
Anyway, from my experience of work wih WMAs I seriously doubt, that resuts would change much: WMA8 at 96K is annoying even on my iRiver. I even get crazy on my old WMA8 160 kbps recordings...

One more funny thing: after this test I seem to become much less tolerable to artifacts. Now even some TV programs drive me crazy... I hear these damned encoding artifacts :\

Well, yes, as long as it doesn't become much popular. I don't want to abuse HA's bandwidth...



The worst sample there is Waiting. There are no traditional problem samples there.

Actually it's "classic", not "classical".

"Classical" suggests music like Beethoven or Mozart, you know, hmm, "classical music"

I think that was why he put quotation marks around classical (i.e. "classical") meaning orchestrated music in general. I agree the word is overly (and incorrectly) used though. Anyway...

Interesting test although not entirely unsurprising. By averaging out your scores you'll see that AAC actually outperformed Vorbis (although I don't think thats statiscally valid and the difference is so insignificant that you couldn't draw any solid conclusions from it). They are as you say generally the same. WMA9 result is a little surprising, although considering WMA9Pro didn't outperform any codec (except LAME of course) in Roberto's second 128kbps test then you couldn't expect WMA9Std to do to well. The fact that it didn't perform closer to LAME though (considering LAME was using -b128 instead of --alt-preset 128) is a little disturbing.

@Roberto:
Thanks.

@AtaqueEG:
Please, don't blame me; I wrote this late at night And thanks for correcting

@kl33per:
You can see, that AAC score is very slightly higher in average, but on most samples Vorbis sounds slightly better. That was the most surprising thing for me as AAC is becoming an object of worship on HA, while Vorbis is often blamed for quality.
I tested WMApro some time ago and liked it a lot, but there's still hardware compatibility issue, so I didn't use it in my current test.

As I already said, my conclusions are simple: AAC and Vorbis are clear leaders, which often give you "near transparent" quality at that size, being a good replacement for mp3 on HW players. Using LAME MP3 at -b128 (you can't be sure, that ap128 would be better) is not a good idea, even though it sound is better, than WMAs. Sooth to say I expected WMA to be slightly better, than LAME ;\ Shame on MS for making such a crappy codec. It seems, that they don't care about quality at all. Using theoretically better codec plus VBR mode and performing worse, than MP3 is a shame. It has very noticeable artifacts, so even MP3's preecho is nothing compared to it.

No MPC?

it works on my linux box

"HW support" generally means portable hardware support (iPod, iRiver, Rio, PocketPC, etc.) or support on proprietary systems (Phatbox, Kenwood Music Keg).

But you probably already knew this.