Hello

I would like to announce that the 128kbps Extension listening test is now closed, and the results are available at my Listening Tests page:

http://audio.ciara.us/test/

Here are the overall ratings:


Please post comments at this thread.

Best regards;

Roberto.

Nice test. The results clearly show how good the newer generation of codecs are. Looks like Blade served its purpose well. I wonder if there are still some Blade 128's out there on p2p? I wonder how Fraunhofer's FastEnc or the older "Radium" hack would have compared to Lame?

Death2.wav is really interesting. The bitrates for Ogg and MPC don't suggest anything out of the ordinary (about 115 kbit/s), but WMA9Pro really tanks. A lot of people comment on noisy transients, with something also wrong with the stereo during those parts.

Several people have very good high frequency hearing, and can hear the lowpassing of MPC and AAC (both around 16 kHz). One person (gecko) seems to find this to be a significant defect.

If new samples are chosen for the next test, I think Waiting.wav should be retained. It seems to be one of those killer samples which have a big effect on every codec.

ff123

SWEET!

this will help me decide on what portable (player) to get.

looks like aac and ogg are very good indeed at this bitrate, really good news for portables, since those 2 are most likely to be the ones to get implemented first, (i'de love to get the iriver perl, ogg support)

and the ipod already had aac support iirc ...

VERY interesting that Musepack came out on top, albeit marginally. Thanks to Roberto, all the test takers and everyone who helped. These comparisons are very useful. Keep up the good work!

I did two different tests : one on a portable PC, with a (I suppose) crap audio chip (Conexant AC) but good headphone ; another with my main soundcard, Terratec DMX6Fire.

Results are here :

http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm
http://membres.lycos.fr/guruboolez/AUDIO/t...MF/TERRATEC.htm

Comments (log file), (in anglische) are available too :
http://membres.lycos.fr/guruboolez/AUDIO/test_MF/


I've two different winners :
WMA9PRO on portable, and Quicktime AAC on Terratec (never mpc).
The three best are close each others ; Vorbis was fourth, on two test, not as good as the three others, but more constant than WMA9PRo, and maybe MPC.
LAME is now old, and Blade looks more as a joke than as an encoder.

I'm a bit surprised to see mpc "wining" here. I hope that Frank Klemm is just in holydays : I didn't see him for some month on HA. The development of mpc can't stop !

EDIT : Roberto, thank you another time for publishing these results immediately after then end of the test

I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

Well done, Roberto. B)

I too was surprised at how well the codecs performed generally, some tests were *extremely* hard. (However I have to say that in some cases where added background noise was the only problem, I had absolutely no chance of detecting it because my sh*tty equipment hisses like mad. Gotta get a better soundcard soon. )

I'm beginning to lose hope for Vorbis. It gets beaten by MP4 in most cases, and generally performs worse than one would expect, given the fact that this type of bitrate range is considered Vorbis' speciality. Add to that the sluggish development (also with regards to hardware support), and its future doesn't look too bright...


On a side note, @ff123: Have you considered perhaps adding XML support for the results file to ABC/HR? That would make parsing the results into a database a breeze, and they could be published on one HTML page much more easily.

I wouldn't loose hope it just shows how competitive things are in that bitrate area, really there is not much between AAC, Ogg, WMA and MPC. Big bucks have gone into developing some of those codecs and it is good that ogg achieves the same without patent infringing (if WMA wanted to use some technology as AAC, they would just cross license patents, that was not an option for Ogg).

I think the 'winners' in years to come will be AAC and WMA, both of these will have large online commercial music shops and wide support on portable players.

heh

It seems that you are sensitive to MPC artifacts, Guru. rjamorim wants your ears, I am very happy not to have them

Lev > Is this what appeared when you tried to reach the two results matrix I uploaded ? It works with me... Can someone confirm ?

P.S. I created the webpage with MS Word XP (I haven't anything else installed, and I'm bored by notepad ) : It's big, and maybe problematic.

Works for me, the filtering is through his company proxy server (unilever).

For the record, if you have six players and five tie for first place, the sixth player is in sixth (or last) place, not second place. If you have six players, and two tie for second place, there is no third place; the next place is fourth place. Win. Place. Or Show. The rest don't really matter, you know.

Taking into account the deviations possible in the results, I see it fair to tie AAC, MPC, OGG & WMA in joint 1st.. (MPC could be the lowest of the four and OGG the highest if the errors fell in a parculiar way)

At first I thought it is surprising that 4 very different codecs can achieve 4 levels of quality that converge so closely.. but then I thought, does this just mean they are approaching the soft-limit of achievable compression? Until radical re-thinks in compression schemes (SBR @ 128k, wavelets) I wouldn't imagine a codec *could* progress much further than this ~4.5 limit.

Thankyou for organizing this test Roberto, these results provide the interesting food for thought that can only come about from a professionally conducted group test

Yep - its a Unilever filter... you didnt include 'bondage' or 'granny fisting' in the title bar, did you?

I agree with Mac - its almost like a plateau has been reached, and asking people to judge encoders is almost like asking people if they like celery. Guruboolez seems sensitive to MPC artifacts, whereas many of us aren't, hence it slightly won in the test. Its almost like a genetic thing, rather than a matter of training or practice.. I have a self proclaimed talent for picking out Oggm for example.

But yep, I love test results like this. I love figures as opposed to words. Thanks

I think one certain conclusion we can make is that anyone selling 128k anything and calling it "CD quality" needs to be thrown in jail.

A couple notes (based on my listening):

For me, each codec had at least one sample with a pretty catastrophic falure that I would probably notice in casual listening. MPC came the closest to being "acceptable" to me for all samples, but had a quite noticeable stereo separation issue with TheSource. Nobody else seemed to mention that problem though...

Blade did serve its purpose, however, I had one sample where lame lost to blade. I one other person ranked them in that order for that sample (Waiting), and another had them tied, so I might not be completely crazy. Also on death2 if it weren't for blade for comparision, WMApro would have gotten < 2 and AAC would have been scored quite a bit lower too.

I was able to ABX a lot more of these than I initially expected to. My ears are not all that trained (well I suppose they're getting there).

I would have ranked them (based on my results before I saw everyone elses):
1. MPC
2. WMApro
3. AAC and vorbis tied (very close, throw out one test and they'd flip-flop)
5. lame (fairly far back)
6 . blade (way way back).

For my results, MPC was the most consistant, almost always either getting first, or very close (with the one exception.) WMApro was ususally near the top, with one pretty big failure, and somes falling back to the middle. AAC consistantly came in around third, close to MPC and WMApro, while vorbis tended to jump around more. Lame usually trailed but occasionally came out near the top, and blade was always way back. For me MPC was definately "least likely to have serious problems" which is pretty consistant with the consensus of these forums.

My ears got trained as I went along... Problems that were hardly noticeable initially became really easy (specifically the "brightness" of vorbis on some samples, and the absolute crappiness of blade on all samples). Other things got harder from having listened to them too much. I think I would have fairly different results if I were to do another test (WMApro, AAC and esp. vorbis would probably get lower scores now that I am more familiar with their problems).

I thought it was interesting how much people varied in their sensitivity to different problems. Even though I knew harpsichords were supposed to cause trouble, I really couldn't identify too many with Bachpsichord (and I did try very hard). OTOH, I'm the only one (I think) that mentioned the stereo problems with TheSource and MPC.

I'd have to say I was "heartened" by the the final results. While I had MPC as a winner, I don't ever see it having success in the portable market or widespread support. After looking at my results, my sentimental favorite (vorbis) didn't look like it could stand up against against WMApro, nor could AAC (which I reguard as "less evil" than WMApro). To see that there's pretty much a four way tie for first is a bit surprising. If vorbis gets further tuning, I think it could stay competitive.

Who gets the golden ear award? Guruboolez? Gecko? I suppose handing out awards isn't a good idea. It might encourage people to fake results. Plus from my comments, I think I'd get the "excessive verbosity" award.

I want to say thanks one more time to rajamorim (espescially for the speedy tabulation) and to all the other participants.

Here is the ranking table for this test (based on overall results) :

http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm

Mpc never goes below the third place (only codec to perform that). Must be the most constant encoder (thanks to VBR I suppose) of the five challengers (with lame, always at the two last places).

Vorbis can't seriously claim one of the first three place.

Lame is outdated (Lev, you could call that 'granny fisting' )

Wasn´t the test open to 5th august? I´ve done test 1 - 10 already, with comments etc., just hadn´t enough time to do the last both test.
Now I´ve done the ranking of myself. also the last both samples are not included my result is a bit different:

1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0

I always rated the blade sample to 1, because it´s clearly the worst, also on some samples Blade performs better. The results are not manipulated and didn´t thought that would come out (I thought AAC clearly beats Vorbis and Musepack as higher-bitrate codec is the worst after blade). here are results for sample 1-10:

If you like I can also post all my comments to the samples.

that would be interesting to see.

That's the ranking I've expected.

I hope this test will encourage Xiph.org to do some further Vorbis tuning.

I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it?

Just read Roberto's presentation :

wma9pro VBR 128 two pass (it's more ABR, and very accurate : 128 kbps)

WMA9PRO is available on Microsoft website. Just DL for free Windows Media Encoder 9. If you're running on 98SE OS, take a look at Spoon's dBPowerAMP. It's an easy way to encode with WMA9 (pro, lossless, voice...)

I will add a table with all the bitrates to the results page later today.



Oh, sorry, S_O, but quoting the readme:
"7. After you finish the test, save the results and mail them to
rjamorim@<blanked>. The test ends on August 3rd, 2003. No
results will be accepted after that date."





Some info for you guys:

Next test will be at 64kbps. I'll compare Vorbis, HE AAC, WMAv9, MP3Pro and Real Audio Cook. But that's discusseable, and I don't want to discuss it right now. I gotta take a break...

Hopefully, people won't have to struggle that much to find artifacts in this test.

I plan to start this test by the beginning of September, probably the 3rd.

People are too fed up with the current samples, so I think I'll only keep Waiting (per ff123's suggestion) and do a call for samples next week.

No decision has been taken yet though, and I sense there'll be a very heaten discussion about whether choosing VBR whenever possible or only CBR. I'll start a pre-test thread as the test gets closer.

Thank-you very much.

Best regards;

Roberto.

These results are very supprizing, even on MusePack.net it says that Ogg is probably better for bitrates <140Kbit.

I find it very interesting that aac being cbr and wma pro being two passed reach such a high quality. I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).

Looking at the results, I am also glad mpc was included in the test since it is a serious contender.

I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).

Thanks Roberto, for performing the test and such a quick evaluation!

I'm surprised too. The WM9pro encoder used for this test is the first never released. I never heard a baby singing so good for its birth.

Quality is amazing for half samples of this test, which is simply remarkable (first place, with sometimes great notation !). More tuning is needed - death5 is totally crap - the biggest flaw in my opinion on 12 samples, even when including lame.

Last but not least, wma9pro is gapless... AAC/MP4 isn't for the moment.

Iam Happy that my Favorite Codec WMA show the rest that he is better and not so Crappy like Everybody said.

But i have One little Problem a wma9pro Encoded File dont be played in Winamp 2.95.

Is there a plugin that wma9pro played without the Windows Media Player.

ThX

Unfortunately, that's the "one little problem" with using wma9pro right now. Although it sounds better than the older wma format, it breaks compatibility with current decoders. I'm sure that will change in due time.

ff123

WMA9pro is a good surprise. But standard WMA9 is really worse.
There's no plug for winamp. Foobar2000 can play WMA9 PRO-LOSSLESS... without problem. Just install Special Release offered by Case, on :
www.ca5e.tk
www.musepack.fr.st

Thx i test foobar.

But i Hope a in_put Plugin for winamp will follow.

Call me what you want but foobar is not my Favorite Player,Sorry but i respect the work on it.

Ciao

I've got to go back and listen to that again when I get home. I thought it was really obvious (present to a lesser degree with lame too), and my ears were barely capable of noticing stereo collapse in FloorEssence or any other samples. I ABXed 10/11 (and the miss was while fishing for other defects) so I don't think I imagined it.

Can one of you sharp-eared folks check it also to help me verify that I'm not insane?

I only noticed stereo collapse in one small section of one sample, and it was with Ogg.

BTW, I like your quotes, phong. Here's my favorite:

"It sounds like an leprechaun is making love to the whammy bar."

ff123

Well, after listening to a few of the samples, I noticed I started getting much better results in hearing differences and ABXing by using vivid imagery. I'd listen to the original, and try to picture it in my head, then listen to the encoded version, and create an image for that. Then the "X" would automatically bring up the appropriate image. Some of the images made their way into the comments; on others, I thought better of it.

delete in_wm.dll and add wma to the extension list in in_dshow's configuration

imho vorbis definitely lost the test

and i will use mp4/aac in the future


(btw. cant wait anyomore for the 64kbps test )

Man, LAME really is dated at this bitrate. Hopefully no one misinterprets this test and decides from it that all these codecs perform equally the same at higher bitrates. Please remember that this is only at 128kbps. alt-preset standard, vorbis gt3, musepack standard, etc will be tested later, if that would even be a valid or necessary test. Taking from the difficulty that this test has shown, it would be no wonder even harder to test the higher bitrates.

Even though I know that it is incorrect to make blanket statements, I'm sure that because of this test we will see them. If it wasn't for compatibility, I think you would see people leaving LAME in droves now. While I guess it might be plausible to make judgements from this data that LAME is dated, that other codecs have seen greater development as of late, and then as such decide to use a different codec for all your encoding needs, I don't think it would be a good idea to directly link this test to overall sound quality.

Oh bollocks, /. sucks:
2003-08-04 19:06:04 128kbps audio codec listening test (articles,music) (rejected)

Will they? When? and by who?

Surely not by me. First, because these bitrates are transparent, we would get most of the codecs reaching score 5. Second, because I am only willing to take one other VBR test. The critics on a VBR test are such a PITA that I definitely don't even consider conducing several ones.

Too bad to read this negative impact, consequence of the recent and uniform critiscism posted the last week. Just ignore them : you did a very nice job ! Some myths are now dead, and without your test, they'll probably stay alive.


EDIT, after reading the answer below : my apologies for people concerned by my unfounded message.

Dude, to tell you the truth, I keep wondering if it's not better to keep these results out of /. :-/

I can only imagine the flood of unfounded criticism about bitrates, samples, methodology, and that I should have included Flac in the test (yeah, I read a comment like that in the AAC test article!)

@Guruboolez: No, the problem wasn't that thread. At least, people were (trying to) make constructive criticism. (although I'm kinda sad because they didn't bother to do this welcome criticism when the time was right, I.E, the pre-test thread)

What really made me sick was people saying my test resoults would be worthless in my e-mail. I got to a point that I didn't even read them properly, just hit the delete button.

Hm, at least how I remember the test, the lowpass was usually one of the most significant things to notice on the aac/mpc samples. I consider this a good thing actually, since lowpassing can be easily fixed without further issues (save higher bitrate) in mpc (and aac?) and you have almost no other artifacts to begin with.

Looking back at my results, I didn't notice the lowpassing very often (only on 4 samples) and I just assume right now that the low rating wasn't a results of the lowpassing but other defects. I was in a hurry and didn't bother to do extensive comments so I just noted down some things that were particularly interesting imo. You could look at it this way that I didn't notice significant lowpassing on all the other samples (other than I would have expected).

Do you think? I don't see a clear winner/loser between MPC/WMA/AAC/Vorbis. The results are very closed.
Roberto, you've done a good job and that's what really matters. Congratulations.

I don't want to be considered as an anti-vorbis zealot, and I'm not specialist in statistic, but :
http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm

Vorbis took the two first places one time (on 12), and the two lasts 7 times. It's far the worst performance, compared to mpc, aac and wmapro (can't include seriously any mp3 encoder as pretender of the throne).

According to my own results, vorbis isn't exactly at the same level as mpc, mp4 and wmapro. Not too far, of course, but not as transparent as its challenger.

For me personally Ogg Vorbis came out on top. I voted it best on 7 samples (once as worst on "waiting"). Second best on the remaining samples.

Here are my personal codec averages:

vorbis: 3.85
wma pro: 3.32
mpc: 2.87
aac: 2.77
lame: 2.23

blade: 1.23

With the amount of overlap between confidence intervals for most of the samples (considering our top 4 contenders of course), many (or most) are a statistical tie. Yes, vorbis might not ACTUALLY have first or second place in many samples, but statistically speaking, lots of those third and fourth place finishes are a statistical tie with first. The confidence intervals could be narrowed by having more listeners, but the average scores would change too. If the additional listeners preferred vorbis, it could move to the front. Alternatively, if you were willing to accept, say 80% confidence instead of 95%, you might be able to say more, but be less sure about it.

Now, there could be some interesting things that could be done to analyse these numbers further. I think the standard deviation of each codec would be an interesting number (though I think I'd know who's gonna win that contest). I've got a perl script in the works right now to convert the raw data to .csv so that it can be played with in a spreadsheet. I'll post that later tonight.

I'm sure that if you stuck to either a full ABR or full VBR philosophy there would be fewer critics. Either find the settings that produce an average of 128kbps (or whatever target) across a broad spectrum of music, or find the settings that produce that same average for each individual sample. It would be much harder to find flaw with such a test.

The most important thing isn't really to speculate with statistical tools, but to test by yourself, and finally make your choice according to your own subjectivity, training, hearing abilities, etc...

Generally speaking, and according to the posted results, mp3 - created by Fraunhofer for 1/11 encoding - is now completely outdated. Wma standard family is probably out-of-date too.

Now, we can hope some hardware support for some of these new formats. 120-130 kbps encodings are now close to transparency. Therefore, they aren't sufficient at this bitrate range for archiving/HQ listening, but perfectly suitable for comfortable listening on any portable player (impossible with 64 kbps encoding, crap, even with best algorythms). This is very important for flash memory players. Most of people would be happy with these codecs : they will fit more music without suffering from big artifacts (flanging, pre-echo...).

A new era is beginning...

I wonder if Sony said the same thing about Betamax...

No hardware support without widespread consumer support, and even if a codec surfaced that was perfectly transparent at 32kbps, not a dent would appear in MP3's dominance without... (?) (I'd say "mindshare").

Most important, thanks for the test Roberto! Great work.

Suprising news regarding WMAPro. Any news on when it will be portable compliant? Since I purchased dBPoweramp, I am ready to encode my collection using WMAPro for my portable use. But my flash and HD players cannot read it.

(MP3 APS with LAME still rules for the combination of transparency and portability! - when the 128 restriction is relaxed)

/me needs MPC portable! NOW! ;-)

I've found the test very amusing except thear1 sample - deafening experience.
BTW, I normally listen to similar music too, but all other samples were MUCH quieter and I've set my volume accordingly...

WMA9Pro has some very serious failure modes, but it was to be expected, it's 1.0 release.
Vorbis has constant quality, good, but is easily detectable.
AAC is very good, for me is the only one which could compete with MusePack.
Can't wait for hardware support and freely available better encoders.

I guess I'm fed up with 'MP3 artifacts' - they sound damn nasty to me.

fewtch: I'd say 'without advertising and hardware support'.

Thanks for organising the test Roberto. Excellent stuff!

I was one who flagged a concern about the bitrates, and I am genuinely sorry for any concern or constenation that I may have caused to others at HA. What caught me by surprise was that we had the discussion thread where people posted a shed load of bitrate test results etc, to get the right q numbers, but then when I started getting into the test proper, seeing a mpc sample hit 197 just caught me by surprise.

I don't think that in the end it really matters, as there are now at least three codecs that got close to the money in the 128 kbits range, despite some using more bits to do so with particular samples. We now have some new successors to LAME, and at least two of them have current or hopefully pending hardware support. B)

Thanks again for all your efforts Roberto.

As guru said, and I've also mentioned previously, it still comes down to personal preference, and testing/deciding for yourself. Hopefully this test has opened a few people to alternatives to their own favourite format. B)