How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx

It would be nice if they stated the bitrate of the HE-AAC used in the test.

one would asume that the bitrate was close to that of the wma it was being compared to... but that IS just an asumption

i liked this line: "71% of all listeners indicated that WMA Pro was equal to or better than HE AAC."

there is a 48kbps multiformat test coming up isnt there? I'll wait for those results

@[JAZ] no problems at that kind of a bitrate.. but then again i dont know, i don't really use wma, i just keep the dbpoweramp encoder around for occational tests

Bitrate is in the pdf and is indeed 64 kbps. Even precise steps to how the clips were encoded are given in there.

Of course using the same rethoric you could say that 63% of the participants rated HE-AACv2 equal or better than WMA.

Anyone notive the MS codec used?



I don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

Kristian

** Test codecs:

WMA Pro at 64Kbps? Interesting...It might be the codec they are making for Vista.
Let's see if now they ditch WMA Standard (or start to do so, even though "PlaysForSure")

OTOH, wouldn't HE-AAC V1 achieve better results than HE-AAC v2 at 64kbps?

**Test methodology:
Faults ( IMO ):

From the .pdf, it seems that nor ABX neither ABC-HR was done when electing the samples(1), plus the randomness in which the samples where played is questionable(2).

Overall, most of the results seem random(3), except for a few samples(4) where there could really be a difference.


(1) "The first clip to be played for every song was the reference clip. After listening to the reference clip, the participant would hear two additional clips, Clip-A and Clip-B. The participant was asked to compare both Clip-A and Clip-B to the reference file, and determine which sounded more like the reference file. NSTL would replay any of the three clips for the participant upon request, but once a decision was made on a song, the test would advance to the next song"

(2) BBABBBAAABAA ( B->HE-AAC first, A-> WMA first. ). They did this for 2/3rd of the participants, and inverted the order for the other 1/3rd.


(3) For most of the samples, the listeners elected "Identical". The percentage in which they elected wma or he-aac is similar. No big differences in prefference (only ~5% although generally is in favour of WMA...)

(4) The results, from my POV (point of view):
BobMarley,ColdPlay,Greenday,si02 -> Favour WMA

SimpleMinds,BlankBaby,Floyd -> Seem to Favour WMA, but with a small margin.

LimpBizkid,DireStraits -> There really is a difference, but prefference between one or the other is random (or averaged)

GuanoApes,Maroon5, -> There could be a difference, but the distribution is much averaged.

TheVerve -> Seem to favour HE-AAC, but with a small margin.


**Conclusion:

It could be true that WMA Pro 10 at 64Kbps is better than Nero 7, HE-AAC v2 at 64Kbps from the results shown. It definitely shows some problematic samples (especifically BobMarley,Greenday,si02) for HE-AAC, and that WMA Pro 10 is now competitive at this bitrate.
Let's see if it can be included in the multiformat low-bitrate test that we will do soon.



[Edit -> Forgot to say a few things]

> don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

9.1 is the latest, the confusion comes from WMP 10.

This report, to tell you the truth I wouldn't believe 1 word of it, even if it appears to be from a separate testing agency (guess who pay their bills...). The same goes for any company, Nero, FhG, any self-tests or sponsored tests should not be given a second thought, it is all advertising spiel. That is why listening tests by non-affiliated groups / people are worth their weight in gold.

I found it interesting that at the very bottom of the pdf report, they write "NSTL does not guarantee the accuracy, adequacy or completeness of the services provided to Microsoft or the data included herein." I know that disclaimers are common, but in this case it seems ironic especially for a company that on the same page claims "NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools to hardware developers, software publishers, government agencies and corporations."

Also for such an large testing group I think they should have included more types of music such as classical and jazz. It's nearly all rock/pop oriented.

.... they forgot to add "maybe"
I didn't know they were using some new wma 10+ codec... just have to wait till it's available to see how it really fares

I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.

@Woodinville, is there any way for the public to obtain the codec used in the test - so some kind of public verification could be done in the future, or this codec is to be released with Vista?

There is a new 48 kbps multiformat listening test scheduled for, most likely, March (after 48 kbps AAC listening test) - and testing the latest WMA codec would be a very very nice thing, especially after latest claims from the NSTL.

I'll make 2 factual observations:

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

Edit: Since this behaviour seems to have been fixed/may not have existed in the version used, you can ignore this for the sake of argument. I'll replace it by: 1) It seems impossible to verify the test results and setup.

2) They consulted with MS how to set up WMA and got new tools that are apparently not publicly available. They didn't consult with us, and used an old version of the Nero AAC encoder.

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.

If this is the case, somebody should contact Microsoft because they can't keep that comparison online if they know that it is completely false. Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

And doesn't that mean that Microsoft's hyper super ultra new and amazing WMA 10 Pro + is only tied with lc aac instead of he aac.

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?

No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.

You select that "Nero Digital Audio (HE-AAC)" out of a list of MP3Pro, TwinVQ, WMA, AIFF, WAV etc.

By default, it will encode to 128kbps LC-AAC after selecting that.

Selecting the CBR/Stereo - 64kbps [HE-AAC] preset works correctly.

You have to go inside the encoder settings, force it to 64kbps CBR, and ignore the explicit combobox that says "LC (Low Complexity) AAC", to get it to fail this way. And that is exactly what their description says they did.

PS. I do agree this part of the GUI could be done better, but it can only fail this way by tinkering with some "advanced" settings and ignoring others. I don't think removing the LC-AAC options is good, either.

I agree with you. It would be a lie. But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

EDIT: emphasis in the quote

If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???

You could contact NTSL to obtain the .wav files used in the test - both reference, A, and B samples.
You could then encode the reference using the same software and settings as described by NTSL and make one LC-AAC and one HC-ACC.
This would allow you to find out what settings where in fact used.

What about volume leveling?

With the test procedure described by NTSL the encoder raising the volume the most would normally win such a comparison.

Not that I suspect neither WMA PRO nor AAC of boosting the volume, however software is software and IMHO the testing metholody should have taken this issue into account. At least provide an analysis of the gain of the resulting wav files.

Hmm...

I followed the procedure described in the test report and got this Nero encoding setting:





Isn't this a HE setting?

You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.

Yes. Does it automatically switch to HE-AAC for you? Older versions of Nero 7 don't seem to, and one newer one I have here doesn't either.

I read the testing procedure and it says that person also went on to the detailed encoder settings and changed them. As I already said, using the defaults, or using a preset, all would have produced a good result.

In other words, your frontend is working correctly without bug when it suddenly goes from HE profile to LC one just because the user manually set the bitrate to 64 kbps after selecting the HE profile.
It's something I don't understand, but you're probably right.
BTW, I can't reproduce either the bug you're mentionning.

I don't have a version "2", the only available option is selected in the screenshot.

The only other things I did were these:

7. - open settings
8. - select CBR 64 kbps
9. - save

Edit

I tried to find out what exact version I have. The first version number I found is 7.0.1.2 in the about Nero Burning Rom dialog. I downloaded the package in late November.

Bah, would have to locate exactly the version they used then, or see if we can get the samples.

Since the chance of a recent Nero encoder having been used seem 0 (test was conducted in October, we released new one in November), it doesn't really matter to me.

Fact remains: MS gave them an encoder, told them exactly how to use it, and they didn't apply the same carefullness for the competitor. Is that a fair test?

No. But did Nero pay anything for this test? . This is pure business/marketing after all.
I'm not questioning these results, they might be perfectly valid. It's just that tests like these annoy me quite a bit, because IF the results were in competitors favour the results would never been published.
How many similar tests have been conducted where the results weren't 'good enough' and thus everything was just shelved? it's impossible to know.

Good point indeed. The older (< Nero 7) HE-AAC encoder is clearly worse than the one bundled with Nero 7 (and also worse than Coding Technology implementation - see here).

But there's something which leads me to believe that HE-AAC was used and not LC-AAC: it's the results of si02.wav (= castanets). It's the only sample which revealed a huge difference between AAC and WMAPro (correct me if I didn't understand the graphs). To explain so big difference, either the AAC encoder performed really poorly or the WMAPro encoding was as sharp as a razor blade. The second is unlikely (WMAPro doesn't handle very well pre-echo). But if HE-AAC was really used, then the results would be coherent: SBR encodings have currently really poor performance for pre-echo. Consequently, even with weak performance WMAPro should appear as clearly better than HE-AAC, but not LC-AAC (with a modern implementation like Nero) with a sample like castanets/si02.
From this and according to my own experience, I'd say that HE-AAC was used. But for all other samples, results are surprising me. From my experience HE-AAC is usually better than WMApro. But if the test include the same family of HE-AAC encoder I've tested this summer (HE-AAC at 80 kbps was as poorer as LC-AAC from iTunes and not that far from standard WMA) then the tests results could be in coherence with my experience.

Too bad that the test was performed in october :/

Hey! They used WMA 10 Pro+, not WMA 9.1 Pro. The version 10, let alone +, isn't eyet available publically.

Assuming that WMA10+ correspond to a new encoder and not WMP 10, that's right. But is Microsoft able to transform an encoder which doesn't perform really good in sharpness even at 192 kbps to something excellent at 64 kbps? I don't think so. I've never seen before such metamorphosis.

Well I would personally really appreciate if we could perform a public test with latest Microsoft and latest Nero Digital Audio encoders.

Discussing this internal test without any data (except the test report) sounds pretty pointless to me.

Am I missing something here? How could this test have utilised Nero 7 in October, when in fact it had not even yet been released?

Moreover, what is the point in challenging He-aac v2 at 64kbps, when that implimentation is designed for optimum bitrates of 32 or 48kbps?

The premise makes little sense, but how did they get NERO 7 prior to release. Hummm???

For sure, October Nero7 release contained 2-years old AAC encoder, which was replaced somewhere in November with the improved SBR encoder.

Concerning HE-AAC v2 (HE-AAC + Parametric Stereo) - it was also introduced in November, and for sure there is no way to activate it in the GUI at 64 kbps - as it was not even designed for such bit rate.