NSTL WMA Pro listening test, MS claims test proves WMA Pro > HE-AAC Options

[Woodinville]

i think they are 'supposed' to represent a % quality.. but they are just high-to-low (or should it say low-to-lower... uh oh.. i better watch out for tos8)

Have you heard problems with a ~240Kbps VBR (Q98) wma 9.1 file to say it is low quality?

How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx

[Cyaneyes]

How about this new report (DBT and all) on WMA Pro quality?

http://www.microsoft.com/windows/windowsme...Comparison.aspx

It would be nice if they stated the bitrate of the HE-AAC used in the test.

[gameplaya15143]

It would be nice if they stated the bitrate of the HE-AAC used in the test.

one would asume that the bitrate was close to that of the wma it was being compared to... but that IS just an asumption

i liked this line: "71% of all listeners indicated that WMA Pro was equal to or better than HE AAC."

there is a 48kbps multiformat test coming up isnt there? I'll wait for those results

@[JAZ] no problems at that kind of a bitrate.. but then again i dont know, i don't really use wma, i just keep the dbpoweramp encoder around for occational tests

[stephanV]

It would be nice if they stated the bitrate of the HE-AAC used in the test.

Bitrate is in the pdf and is indeed 64 kbps. Even precise steps to how the clips were encoded are given in there.

Of course using the same rethoric you could say that 63% of the participants rated HE-AACv2 equal or better than WMA.

[kritip]

Anyone notive the MS codec used?

15. Set the “Codec” to Windows Media Audio 10 Professional+

I don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

Kristian

[[JAZ]]

** Test codecs:

WMA Pro at 64Kbps? Interesting...It might be the codec they are making for Vista.
Let's see if now they ditch WMA Standard (or start to do so, even though "PlaysForSure")

OTOH, wouldn't HE-AAC V1 achieve better results than HE-AAC v2 at 64kbps?

**Test methodology:
Faults ( IMO ):

From the .pdf, it seems that nor ABX neither ABC-HR was done when electing the samples(1), plus the randomness in which the samples where played is questionable(2).

Overall, most of the results seem random(3), except for a few samples(4) where there could really be a difference.


(1) "The first clip to be played for every song was the reference clip. After listening to the reference clip, the participant would hear two additional clips, Clip-A and Clip-B. The participant was asked to compare both Clip-A and Clip-B to the reference file, and determine which sounded more like the reference file. NSTL would replay any of the three clips for the participant upon request, but once a decision was made on a song, the test would advance to the next song"

(2) BBABBBAAABAA ( B->HE-AAC first, A-> WMA first. ). They did this for 2/3rd of the participants, and inverted the order for the other 1/3rd.


(3) For most of the samples, the listeners elected "Identical". The percentage in which they elected wma or he-aac is similar. No big differences in prefference (only ~5% although generally is in favour of WMA...)

(4) The results, from my POV (point of view):
BobMarley,ColdPlay,Greenday,si02 -> Favour WMA

SimpleMinds,BlankBaby,Floyd -> Seem to Favour WMA, but with a small margin.

LimpBizkid,DireStraits -> There really is a difference, but prefference between one or the other is random (or averaged)

GuanoApes,Maroon5, -> There could be a difference, but the distribution is much averaged.

TheVerve -> Seem to favour HE-AAC, but with a small margin.


**Conclusion:

It could be true that WMA Pro 10 at 64Kbps is better than Nero 7, HE-AAC v2 at 64Kbps from the results shown. It definitely shows some problematic samples (especifically BobMarley,Greenday,si02) for HE-AAC, and that WMA Pro 10 is now competitive at this bitrate.
Let's see if it can be included in the multiformat low-bitrate test that we will do soon.



[Edit -> Forgot to say a few things]

This post has been edited by [JAZ]: Jan 12 2006, 19:23

[spoon]

> don't keep up on the WMA codecs, is this a new version?? I thought 9.1 Pro wsa the latest avaliable??

9.1 is the latest, the confusion comes from WMP 10.

This report, to tell you the truth I wouldn't believe 1 word of it, even if it appears to be from a separate testing agency (guess who pay their bills...). The same goes for any company, Nero, FhG, any self-tests or sponsored tests should not be given a second thought, it is all advertising spiel. That is why listening tests by non-affiliated groups / people are worth their weight in gold.

[singaiya]

I found it interesting that at the very bottom of the pdf report, they write "NSTL does not guarantee the accuracy, adequacy or completeness of the services provided to Microsoft or the data included herein." I know that disclaimers are common, but in this case it seems ironic especially for a company that on the same page claims "NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools to hardware developers, software publishers, government agencies and corporations."

Also for such an large testing group I think they should have included more types of music such as classical and jazz. It's nearly all rock/pop oriented.

[gameplaya15143]

"NSTL is the leading independent information technology testing organization for the computer industry, dedicated to providing high quality services and test tools...."

.... they forgot to add "maybe"
I didn't know they were using some new wma 10+ codec... just have to wait till it's available to see how it really fares

[Woodinville]

I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.

[Ivan Dimkovic]

@Woodinville, is there any way for the public to obtain the codec used in the test - so some kind of public verification could be done in the future, or this codec is to be released with Vista?

There is a new 48 kbps multiformat listening test scheduled for, most likely, March (after 48 kbps AAC listening test) - and testing the latest WMA codec would be a very very nice thing, especially after latest claims from the NSTL.

This post has been edited by Ivan Dimkovic: Jan 14 2006, 09:42

[Garf]

I must admit that I am both offended and annoyed by the various insinuations in this thread. Hence, I shall say nothing here, and await evidence of the various insinuations and suggestions.

I'll make 2 factual observations:

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

Edit: Since this behaviour seems to have been fixed/may not have existed in the version used, you can ignore this for the sake of argument. I'll replace it by: 1) It seems impossible to verify the test results and setup.

2) They consulted with MS how to set up WMA and got new tools that are apparently not publicly available. They didn't consult with us, and used an old version of the Nero AAC encoder.

This post has been edited by Garf: Jan 14 2006, 16:02

[Gabriel]

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.

[Latexxx]

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

If this is the case, somebody should contact Microsoft because they can't keep that comparison online if they know that it is completely false. Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

And doesn't that mean that Microsoft's hyper super ultra new and amazing WMA 10 Pro + is only tied with lc aac instead of he aac.

This post has been edited by Latexxx: Jan 14 2006, 14:03

[guruboolez]

Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?

[Latexxx]

Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?

No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.

[Garf]

1) If I follow the test procedure, the Nero files will be 64kbps LC-AAC, not HE-AAC. In their very detailed description, they simply didn't enable HE-AAC in the settings dialog. You can easily verify this with Nero 7 (even the free demo version should work).

You mean that when selecting “Nero Digital Audio (HE-AAC v2) (*.mp4)” only AAC-LC is used by default? If this is the case, you probably have a user interface design problem.

You select that "Nero Digital Audio (HE-AAC)" out of a list of MP3Pro, TwinVQ, WMA, AIFF, WAV etc.

By default, it will encode to 128kbps LC-AAC after selecting that.

Selecting the CBR/Stereo - 64kbps [HE-AAC] preset works correctly.

You have to go inside the encoder settings, force it to 64kbps CBR, and ignore the explicit combobox that says "LC (Low Complexity) AAC", to get it to fail this way. And that is exactly what their description says they did.

PS. I do agree this part of the GUI could be done better, but it can only fail this way by tinkering with some "advanced" settings and ignoring others. I don't think removing the LC-AAC options is good, either.

This post has been edited by Garf: Jan 14 2006, 14:35

[guruboolez]

Of course they could still keep it on their page but that would likely be illegel if they keep claiming that the paper proves wma's superiority to he aac.

Illegal? Do you mean that results of a test shouldn't be revealed if one term of the comparison is buggy?

No. But if they really have tested lc aac and know it, they can't keep lying by keeping that document publically available.

I agree with you. It would be a lie. But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

EDIT: emphasis in the quote

This post has been edited by guruboolez: Jan 14 2006, 14:33

[Garf]

But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???

[sehested]

But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???

You could contact NTSL to obtain the .wav files used in the test - both reference, A, and B samples.
You could then encode the reference using the same software and settings as described by NTSL and make one LC-AAC and one HC-ACC.
This would allow you to find out what settings where in fact used.

[sehested]

What about volume leveling?

With the test procedure described by NTSL the encoder raising the volume the most would normally win such a comparison.

Not that I suspect neither WMA PRO nor AAC of boosting the volume, however software is software and IMHO the testing metholody should have taken this issue into account. At least provide an analysis of the gain of the resulting wav files.

[Alex B]

Hmm...

I followed the procedure described in the test report and got this Nero encoding setting:





Isn't this a HE setting?

[guruboolez]

But in this case, the conducer or the laboratory has clearly set Nero to use HE-AAC. There's apparently a bug in the software, and the conducer can't be responsible of this (and can't necessary notice it while performing the test).

If you fiddle around in the settings dialog, change some settings, and it clearly says "LC (Low Complexity) AAC", how on earth can you say that "it is clearly set to HE-AAC"?

Where is the bug???

You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.

[Garf]

Isn't this a HE setting?

Yes. Does it automatically switch to HE-AAC for you? Older versions of Nero 7 don't seem to, and one newer one I have here doesn't either.

This post has been edited by Garf: Jan 14 2006, 15:25

[Garf]

You should read the testing procedure again. Step#6:
Select "Nero Digital Audio (HE-AAC v.2)....
The person has set Nero to use HE-AAC v.2 as described in the frontend. If the result is LC-AAC, then it would rather look to a bug than a feature.

I read the testing procedure and it says that person also went on to the detailed encoder settings and changed them. As I already said, using the defaults, or using a preset, all would have produced a good result.

This post has been edited by Garf: Jan 14 2006, 15:28