I'm very sorry I couldn't participate in this test.

However, many thanks to you Roberto - and isn't it nice to have such interesting results? I don't think many people expected this.

To me, these bitrates are very interesting. It's amazing both how good, and how bad, some things can sound at 128kbps.

Also, I bet almost every sample in this test was light years better than most people's typical experience of 128kbps lossy coding!

Cheers,
David.

I already asked him about this.
http://www.hydrogenaudio.org/forums/index....ndpost&p=210584

EDIT:It's certainly Bitrate VBR 128kbps, 44kHz, stereo VBR 1pass.

BTW, a thread is made at Slashdot about the test results.
Vorbis And Musepack Win 128kbps Multiformat Test

harashin> I need to try again. I probably did a mistake. Thanks


Other question: did someone post these results on minidisc ddicated boards? It would be interesting, because a lot of MD users often said in these boards that atrac3@132 = mp3@192...

Thanks rjamorim for this test, and all testers for their time and results!

And imagine, the first Hungarian internet music store uses WMA instead of MP3., because it has "better wuality" (And DRM, but they could have used AAC for DRM'd music)

Yeah, and thanks for the Hungarian references in the samples! (Bartók, Dances Hongroises - or whatever, they're "Magyar táncok" in Hungarian )

Actually, I used to hear things to the effect of "LP2 (132kbps) sounds better than a 320kbps MP3!!!" As much as I would love to send an email to the webmaster of the MD Community page or create a thread on one of their forums, I don't think posting the results will have much of an effect. If my past experiences are any indication of the MD community's openess to such data, it will be ignored or disputed via the usual subjective, empirically-dubious arguments one might expect to be prevelant on such boards. For example, when I ABX'ed ATRAC-R with ease a few years ago and posted the results in the most unbiased manner possible, I was either flamed for being "anti-MD" or just outright ignored. The truth hurts, eh?

Roberto, kudos on another great test!

Cygnus X1> I've also read that MD > CD, because MD is 292 kbps and CD 176

This current test is nevertheless different: there's not only one “biased” tester who posted false results, but a whole community, with 18 samples, on ABX conditions. Some opinions might change

Cool, these results clearly show that

WMA 1ST DEATH :[

/me wonders what the Extremetech editors will think when - if - they see these results...

This just in:

http://slashdot.org/comments.pl?sid=108647&cid=9238730

and

http://slashdot.org/comments.pl?sid=108647&cid=9238686

"actual ranking is Vorbis, iTunes, MPC, Lame, WMA, Atrac3" after "a quick-n-dirty compensation, [using] the average scores times 128 over the average bitrate."

ff123

Not very hard to guess that these "compensators" appear who don't want to understand the concept of vbr and how the test was conducted. Though last time I remember something like this on HA also.

Here it is. I probably won't add this graph to the results page. There are already plenty of graphs there.
http://www.rjamorim.com/test/multiformat128/plot18b.png



God, no. If people are that uninformed, they shouldn't be even reading those results.



Penultimate



That will lead peopel to linking bitrates with scores, just like happened at /. - and that is wrong.



No worries, next test will be at 48kbps. Even people with crappy $5 speakers (like me ) and tone deaf will be able to participate.



It hasn't changed a bit since last test. And I personally believe including WMA Pro in that test was a mistake (my second biggest mistake in test conduction, perhaps). When I included it, I expected microsoft would soon start pushing it with all the might of their marketing department to make it replace WMA Std. Alas, that didn't happen. Microsoft seems to have settled on focusing WMA Pro on DVD players and industry usage, and keeping WMA Std. for consumer usage (portables, online stores, ripping at home...)

Moving on to next post...

Those samples had too few listeners and/or results were too close to each other. When that happens, friedman.exe doesn't output the LSD (which is essential to build the confidence intervals) and says that results are "not significant" (what in practice means they are tied)



MWAHAHA!

Time for me to give it a try
Even the "Waiting" sample was un-ABX-able for me, when I tried at higher bitrates, I might be deaf

Some body has posted the results at http://forums.minidisc.org/viewtopic.php?p=22300 Nobody has dered to answer yet. Maybe all the minidisc guys have got heart attack after reading the results.

why not? i would vote for that everyone who thinks about buying songs from a music store, which offers wma@128, should be forced to stare at this graph for two hours

maybe adding lame to this graph would be also good, to proove that probably even the "kazaa music store" will offer better quality at more reasonable prices than wma-based ones

Let's all go over there and flame them. Mwahahaha!

http://microsoftusernetwork.com/forum/viewtopic.php?p=275

I have actually been waiting for the complaining about unfair bitrates to start. I personally find it hard to understand the complaining, after the reasoning behind it has been explained over and over.

Btw: Slashdot seems like a nice place to waste time. Why bother with the facts when you can assume things instead and base the discussion on these assumptions (in this case why bother reading how the test was performed. Lets just assume how it was done and base the discussion on that). I only visit Slashdot when someone post a link on this forum, and I don't plan spending more time there either.

Reading discussions at Slashdot, makes me want to thank the Hydrogenaudio staff for keeping Hydrogenaudio a source of information, instead of a source of speculation, assumption and incorrect information. A big thank you to you all!

Also thanks to Roberto for yet another interesting test. Too bad they are always conducted when I have to prepare for my exams, but I don't really have a suitable listening environment for performing such tests anyway.

You won't find many lame -V5 --athaa-sensitivity 1 at Kazaa, I reckon

IMHO the test is meaningless. If codecs can go over the 128 kbps average then essentially these codecs have cheated.

Don't get me wrong I've encoded most of my tunes into Vorbis as I feel it's a better codec than others, but I would like to see a fair comparison that fits the title of this listening test!!

oh, come on! not again...

Good Test
Just the average bitrate is a little bit to high imo (itunes is ok though)...

*edit* havn't seen ep0ch's post before...

Heh, it was asked for Just my opinion!

I don't see how you can possibly disagree that it would have been a worse test if each sample had been encoded to give an average 128kbps per sample... oh I'll shut up to keep you all happy!!

*yawns* Well the failure of CBR codecs in this test are due to their developers: they did not write a code smart enough to alter bitrate allocation dynamically. VBR is the key issue that requires proper tuning -> the test is fair.

Well ATRAC has a cool name and that's all about it I guess. It looks like this test will be referred many times to whoever claiming ATRAC@132 sounds better than MP3@320 in the upcoming years. Maybe it's a good idea to open a Codec Comparison Forum to post these tests for future easy access.

I really wish people wouldn't post links to slashdot. It's an incomparable juggernaut of stupidity. I can't resist the urge to go over and yell at them. It usually takes me about half an hour to realize that I'm just sticking my finger in a dike.


AHHHHHHH!! You've got slashdotitus! Do they make a vaccine for this yet?

*runs off to wash hands*

I take your points into consideration but tbh i don't think my ears are up to the challenge of telling the two apart. Although im still undecided as to what format i will use.. i can never seem to make up my mind on these things


thanx to rjamorim for a very informative listening test.


-Brian

It would have been worse because it wouldn't simulate a real-life situation. In real life, you'd choose one setting, and encode your music with it.. you wouldn't spend time encoding every single song 3 or 4 times until you reach 128kbps average.

My thanks to Roberot for organising such an enlightening test after the string of setbacks, and to AoTuv for improving Vorbis to such an impressive level

Roll on the 48k test!

Same here. I poked my head into the discussion surrounding this particular news item. Had to get out before my anger welled. Lots of big mouths just waiting for the chance to open and blah blah blah BLAH. How totally useless. The headlines are all that is interesting.

Vorbis winning (I know, MPC is very close) a listening test? That didn't happen for a long time

Heh, as a tradition I would like to thank Roberto and partisipants for the test !
Good work, guys !

But I have one question about MPC average bitrate.
It seems (I may be wrong) that for those two samples (Debussy, CouldBeSweet) bit allocation mechanism or something else fails badly.
So, if we exclude those two samples, average bitrate of MPC will be about 142Kbit for such a setting (close to avg bitrate that was in previous test).
I can not tell it professionally, but may be those two bitrate values should be excluded from average bitrate calculation ?
As I remeber, it can be count, if this bitrate results are statistically significant to include in calculation...

ItCouldBeSweet was purposely inserted into the test to compensate somewhat for the higher average bitrate of the other samples (Debussy was not chosen specifically for bitrate).

I don't think it's a matter of the bit allocation mechanism failing, it's that the samples were chosen such that their average bitrates were generally higher than 128 kbit/s. The rationale was that having such samples would make defects easier to detect. This is true for defects like pre-echo, but as we saw from the test, if having a high bitrate helps the VBR codecs, having a very low bitrate can also hurt it. In other words, problem samples can be found at either end of the bitrate spectrum.

I think the bitrate criticism has some validity, but probably not to the extent that the overall results would have been significantly different if the average bitrates were closer to 128 kbit/s. It's an oversimplification to assume a linear degradation with average bitrate.

ff123

Would I be opening up a can of worms to ask if this indicates that 3.96 should now be the recommended compile?

Oh, I didn't know it...
Thanks for clarification ff123 !
I've got the point, it is a material to think about.

Agree completely
This was not a criticim, really. English is not my native as you see, sometimes I can not explain my thoughts clearly, sorry.
But I will try
My point about MPC(not vorbis) bitrate was:
when you count average on statistical data column, you must exclude values that are outside 3 sigma boundaries. Only in this case result will be statistically valid.
So, in other words, did MPC with used setting produce an average bitrate of 136bit *really* ?
It seems, that not. That two strange samples breaks a statistics a bit, because they were specifically chosen (at least one of them). So, users can be confused (possibly), when the real average bitrate with such a setting will be 142Kbit...
BTW, this do not affect rating calculations, only a bitrate...
This is my IMHO, of course.
Any opinion (and a clarifcation that I'm wrong too ) will be greatly appreciated

Yes, I understand your point.

Ideally, you'd like the bitrate distribution to look somewhat like a bell curve with its mean at 128 kbit/s.

The two samples with extremely low bitrate do not compensate very well for the other 16 samples which are generally skewed above 128 kbit/s.

For the 48 kbit/s test, if there are VBR codecs, I think we should strive to have about an equal number of bitrates above and below the average bitrate (which should work out to be 48 kbit/s on average across the sample set).

The rationale about wanting to use "hard" samples does not apply at low bitrates.

ff123

IMHO, this test is pretty good evidence that Lame 3.96 performs better than 3.90.3 at this bitrates, at least. Personally, this is enough to convince me to use 3.96 for mid-bitrates from now on, whatever the recommended version happens to be.

Ehhh. When writing a previous reply start thinking about it.
Things are not that easy with VBR encodings ...
May be two pass ABR is the best encoding mode ?

I think that this test should have little relevance as far as the recommended LAME version goes. Remember --preset standard is the setting we are really worried about there.

It is the best for test conducers, for sure :B

Too bad only WMA implements it.

Yep, that is what I mean.
Anyway, it is great that such a test are performed !
Thanks again !
EDIT:

He-he

http://forums.minidisc.org/viewtopic.php?p=22321#22321

Hmmm... If the results are too close to each other then it doesn't make sence to find everything equal. For instance, in the leahy sample vorbis gets 4.68 and atrac get 3.76. If the confidence intervals are so tight there is no way these two are statistically equal. And if there are too few listeners then you cannot make any statistical tests on the samples anyway. BTW how many listeners do you considers too few?

Is there a way you can upload the text files with the individual ranks for each sample tested? It is a real pain to build the tables manually from the xml files and I have to exclude ranked references from the start. I'm asking because maybe I can help with providing statistical results for these samples.

Quite the opposite - the confidence intervals are so broad that they all overlap - so there are no winners and no losers in that sample.



To make me happy, I need at least 20 valid results/sample.



First, download the .rar package containing all the XMLs. Decompress it to an empty folder.

Then, install python and Phong's wonderful Chunky:
http://www.phong.org/chunky/

At the folder you decompressed the RAR, run


The codecs.txt should be:


It'll create all result tables (good to be fed to friedman.exe) at the empty folder, and will discard the ranked results that haven't been ABXd to a confidence of 0.05. Chunky is just too wonderful to be true! OMG!

Regards;

Me.

Thanks for the plug Roberto. :-)

If you have windows you don't need Python installed (the standalone windows binary version should work). You should also try out the --help option to get some other options. My personal favorite is the --spreadsheet option to output all the scores in a nice spreadsheet (CSV) format.

I intend to add an option for outputting the listener comments as browseable HTML.

I've tried to make the code fairly accessable, though it's gotten a bit crufty in recent versions (the XML support, for example, isn't as pretty and clean as I would like). The existing code is "incomplete"; there are some features I'd like to add still, but it does all the heavy lifting already (i.e. parsing result files into useful data structures and filtering out bad results). So, if you feel like "doing something" with the data and you know Python, feel free to jump in fix features or add bugs.

Thanks a lot Roberto!

I think this test showed us, one more time, that open source is still better than any paid stuff. I don't want to start any Open Source political fight here, as I am not an free software defender most of the times. But, hey come on!

I think what impressed me most in this test was LAME's climb. LAME is doing the impossible with MP3, to improve it even more. Perhaps Gabriel should make a very simple preset that uses this configuration so maybe we can see more and more nice MP3 around.

Anyway, congrats to all for the nice encoders. By the way, where is the winner? Did he see the results already?

Indeed. Aoyumi should show up to receive a big round of applause.

I get a 404 when I try to donwload the windows binary. I'm not on my linux box right now so I'm downloading python for windows (9MB take a long time to download on 56k )

Ooops, should be fixed now.

This site apparently interviewed Roberto about the test:

http://p2pnet.net/story/1525

They got the contestants wrong, though.

ff123

Yeah, the site author mailed me earlier today asking for comments, and for that sexy picture.

I'll mail him asking him to correct the competitors list.

Another news site mentioning my test:
http://www.afterdawn.com/news/archive/5257.cfm