Teh introduction, aka Let's talk about techmology. Massive.
Seeing the WMA discussion in the upcoming listening test, it made me curious about the Pro vs Standard performance. So I did a little test with 14 samples.


Samples
I used samples from a previous private listening test, so I'm not 100% certain of the samples names, but they should be as following (random order): Dafunk, enolagay, experiencia, flooressence, gone, hongroise, itcouldbesweet, layla, leahy, midnightvoyage, scars, waiting, rosemary, hongroise.


Encoding settings and bitrates
Now I have to say that it's not exactly easy to get a specific bitrate from the WMA encoder. As a result, the bitrates vary a lot. One thing to be considered is that those are just testing samples. I'm sure with real music the results would be more consistent. So even though the bitrates comparison might not look to be fair, I see it more as a plus for the encoder that it allocates more bits when necessary. That's why the WMA Pro encodes have a higher bitrate even though they use lower encoding settings. I used dBpowerAMP Music Converter to create the files, as I still have nightmares from the last time I tried that Microsoft media encoder thing, or what's it called. The settings I tested:

Windows Media Audio 9.1 Professional, VBR Quality 10, 44 kHz, 2 channel 24 bit 1-pass VBR
average bitrate: 70 kbps

Windows Media Audio 9.1 Professional, VBR Quality 25, 44 kHz, 2 channel 24 bit 1-pass VBR
average bitrate: 111 kbps

Windows Media Audio 9.1, VBR Quality 50, 44 kHz, stereo 1-pass VBR
average bitrate: 85 kbps


Table with bitrates and results for each sample




Here be the results. No graph, just text. You have to read. No graph. No, really.

Windows Media Audio 9.1 Professional, VBR Quality 10, 44 kHz, 2 channel 24 bit 1-pass VBR - Score: 2.1

Windows Media Audio 9.1 Professional, VBR Quality 25, 44 kHz, 2 channel 24 bit 1-pass VBR - Score: 4.5

Windows Media Audio 9.1, VBR Quality 50, 44 kHz, stereo 1-pass VBR - Score: 2.9


Conclusion?
I hate Excel. Oh, you mean... You have to make up your own. This was a quick test. I decided to not do any ABX tests, just rate them without examining them too much. I listened to the sample rated it. After I listened to all 3, I listened to them quickly again and adjusted the rating. WMA Pro Q25 was the best, obviously. It sounded pretty good and I would have to listen to them more carefully and use ABX to reliably identify some of the samples. But the bitrate is higher than WMA Standard Q50. So, it's not really a fair comparison. But as I said, for one, WMA doesn't give you many options to adjust the bitrate and second, I think the bitrates would be closer together with real music. I quickly tried the samples with a WMA Standard Q75 setting, and I would rate the quality of those a bit (<0.5 point) above WMA Pro Q25. But those files had an average bitrate of 150kbps.


Last words
I'm no Guru, so keep that in mind. This test was just meant to fulfill my curiosity. Oh, and for the test I used the Sennheiser HD580 headphones connected to my nForce2 onboard audio. OK, that's it, I hope I didn't forget anything...

Results?

Or was it meant as a bitrate test rather than listening test?

Watch out Guru!, a challenger to your expansive, thorough, amazingly well thought out tests has arrived!, all without doing any ABX'ing!, awesome...

In the table, you fools! j/k

OK, I need to make it more obvious, the results are a bit lost in the table. Don't expect any fancy graphs from me, I said I'm no Guru!

Ahh! Now I see! I just skipped that part first since it looked just like a bitrate table...

Interesting! From the looks of it, Pro would certainly give Vorbis and Itunes a run for their money. Thanks for your test.

But was it a blind test or not ?

Damn, I knew I forgot something.

Yes, of course, it was a blind test. I used ABC/HR for Java, Version 0.5b.

sorry, i don't get it, did you compare diferent codecs at diferent BR? what's the point? shouldn't they be in a range?

As I mentioned, it's impossible to get matching bitrates for the encodes, because of the limited available WMA settings. So I compared what I got. And also, those are test samples. General music should produce bitrates which don't differ that much. It's not really a codecs fault if it allocates additional bits for parts that need it. And this can be obvious with short samples, as used in this test.

Getting them in range was the most difficult part of his test.

I just did an unscientific, nonblind comparison with WMA pro at Q10 versus Ogg at q0, and Ogg has better frequency response yet it's also smaller in size.

I might have to eat my words regarding just how well it matches up against the competitors at higher bitrates.

It is possible to get matching bitrates, you just have to use 'Bitrate based VBR (2-pass)' instead of 'Quality-based VBR'. See the Sebastian Mares' posting.

I made something similar for the first multiformat listening test, and compared WMA and WMAPro at 128 kbps. It was really clear that both encoders are not in the same class:

http://www.hydrogenaudio.org/forums/index....topic=10551&hl=

WMAPro is definitely a major improvement. Unfortunately, 30 months after this test, WMAPro is still not supported by the industry.

Thank you for the test

pro25 is better than std50, pro10
std50 is better than pro10

With p < 0.01

...and that's exactly what we would expect from a single format/codec with the same average bitrates. I guess the only conclusion you can draw from this test is that WMA Pro is NOT superior to WMA STD in the 80 kbps-ish area.