Although we had a pre-test discussion already, I wanted a fresh start. This doesn't mean it should take ages before a decision is made like it was the case before.

So - now that the new WMA codec is out and ready for testing, I decided to delay the MP3 listening test and organize this long-awaited multiformat test. The candidates so far are:

Nero HE-AAC
AoTuV Vorbis Beta 5
Windows Media Audio 10

Possible anchors are:

LC-AAC at 48 kbps as low anchor
LC-AAC at 128 kbps or 96 kbps as high anchor

Ivan / Nero Crew and Aoyumi, can you recommend settings for your encoders?

What other contenders would you like to see (preferably other formats)?

In the meanwhile, I am going to re-read the old thread and see if I can find anything useful there.

I would restrict the test to :
• the most serious competitors for low bitrate (it excludes MP3, MPC, AAC-LC... one of them could nonethless be introduced as anchor)
• living formats (no MP3Pro even if this format should still be competitive)
• one implementation per format

The purpose of those restrictions is to drastically limit the number of competitors. As a consequence I only see the following formats:
• AAC (High Efficiency profile)
• Vorbis
• WMA Standard
• WMA Professional
(+ AAC-LC or MP3 as anchor and also to please people interested by these hardware-friendly formats).

The only true problem so far is the choice of HE-AAC implementation: Nero Digital or Coding Technologies?


The choice of settings is not easy. Vorbis should compete as VBR (the CBR mode of Vorbis isn't recommended and is painfully slow; I believe that nobody uses it). HE-AAC VBR is possible, thanks to Nero. And if we go for VBR, then there's no problem with the choice of HE-AAC implementation.
WMA & WMAPro may cause problems. They're both working with VBR but we would be terribly lucky if one VBR preset gives us the expected bitrate. VBR 2-pass can't be an option (it was discussed last year) unless you're (Sebastian) ready to accept full tracks in order to build the WMA samples (it means: encode each full track in 2-pass mode, decode it and then precisely extract the 20...30sec desired sample - but it also imply that people have the complete track of all samples).

To sum-up:

As 48k is almost exclusively used for streaming audio, RealAudio should be there. Also Coding Technology's HE-AAC (the Winamp dll), as it is by far the dominant codec for HE-AAC Shoutcast radio.

mp3pro is largely abandoned so I see little point in testing it. Maybe ATRAC3, but I guess that's quietly being phased out by Sony as well - do their Walkman phones even support it? WMA-standard is not very interesting I think, as WMA Pro is quickly taking over.

Anchors look fine.

CT (Winamp) HE-AAC at 48 kbits gives approx 46-46.5 kbits. While Nero VBR/CBR, Vorbis and WMA CBR have not problem with bitrate precision.

According to the WME, for WMA/WMAPro @48kbps, we have an overhead of 6 kbps. So, it's around to 54 kbps.

Did you verify it?
I also noticed in the past an overhead with WMA, but it seemed to be a fixed one (e.g. +50kb per file). The impact is completely null for full tracks but distort the average bitrate of short encodings.

EDIT: I just looked on some freshly WMA encoded tracks and the final bitrate is correct (expected bitrate + 1 kbps).

No real verification sorry. I just read these infos in extra settings.

I would like to test WMA Professional and WMA Standard and see how they allocate bits (=> build a bitrate table). Should I encode each track of my albums individually, sum up the bitrates and divide by the number of samples (roughly 600) or can I encode full CD images (roughly 50)? I would say encoding each track is near to real-world usage.

BTW guys, do you know any tool other than WMP 11 that can encode using the new WMA codec?

I'd go for separate tracks.
I'm starting on my side a bitrate table for both WMA products. But it will take some time and it only concerns classical music.

I will let my PC encode this night if I find the right tool. My small collection:

Some tips for WMA conversion:
• foobar2000 using WMCmd.vbs script: it works, but I'd like to know if it's normal that each encoding needs so much time to begin... Unusable on my side.

• Winamp 5.31: works great (WMA Std appears as v.9.2 and WMAPro as v.10) but the transcoder needs compatible components (converting from flac files works but not from wavpack).

• dBPowerAmp : should work with v.11 (it worked well with beta versions of WMP) but I don't think that WMA components are available for dBPowerAmp v.12

• VUPlayer : should work, but IIRC the converter doesn't have many encoding options. At least transcoding from WavPack should work with this one (contrary to winamp).

Me again. In Windows Media Encoder I have four options (translated from German):

CBR
Bitrate VBR (Highest value)
Quality VBR
Bitrate VBR

CBR and Bitrate VBR (Highest value) can be ignored. Do you know if "Bitrate VBR" is the 2-pass VBR mode?

Guru, were you successful? I just set up fb2k for encoding and with WMA9PRO (which is 10, but the VBScript file is older than the codec) Q10_44_2_24 which is the smallest quality setting, I get around 68 kbps per song.

I didn't try WMAPro yet.
Before encoding a complete library of full tracks, I'm doing preliminary experimentation on short samples (faster processing). What I can say yet is:
• MrQuestionMan 0.8 doesn't report correctly the bitrate of VBR encoded files (WMA only). I'm currently using foobar2000 but I didn't checked the accuracy yet.
• WMAstd q10 gives me 42 kbps on average [foobar2000] with classical short samples. Bitrate varies from 27 kbps to 93 kbps. But I have several files at 27 kbps, that's why I start to suspect foobar2000 to not give correct values on all files.

Nobody said anything about mp3PRO. Bleh, you edited your post.

I am experimenting with Alphaville's "Forever Young" album. WMA 10 Professional at Q10 produces around 64 kbps. Bitrate based VBR (2-pass) produces around 48 kbps. Didn't try WMA Standard, yet. However, WMA Professional is not looking very good since plain quality settings won't work for the target bitrate of this test. This brings three possibilities:

1. Use 2-pass VBR and try to obtain full tracks. However, this might be problematic because the people who are sharing the files would do something illegal and me purchasing CDs is not an option. As for cutting out the samples precisely - wouldn't ABC/HR compensate for inaccurately cropped samples?
2. Use CBR for WMA (Professional and possibly Standard depending on the outcome).
3. Inrease testing bitrate to 64 kbps.

Edit: Re-checked bitrates with Mp3tag and WMP instead of Windows Explorer and the figures I posted before were wrong. Seems that Explorer is off by ~5 kbps: the files actually have ~5 kbps less. 64 kbps for Q10 is still too far away from 48 kbps.

WMA Standard at Q10 seems to produce ~51 kbps.

WMA standard -q10 SHORT SAMPLES:


The difference between foobar2000 & 'real bitrate' (=computed by 'hand') is expected: it must correspond to the impact of the WMA overhead which is significant when compared to the small size of the files. As proof, the difference is much smaller with non-classical samples, which are on average 3 times longer.

A bitrate table based on full tracks should give more reliable values. It will be performed overnight. But I guess that WMA Std -q10 is fine for a 48 kbps comparison.

Seems that we have similar results for WMA Standard - and I used full songs. Thanks!

WMAPRO -q10 SHORT SAMPLES:


While VBR10 might be usable with classical music the bitrate is too far from 48 kbps with non-classical music (~61 kbps). My values are again close to yours, Sebastian.
I guess that we could forget WMAPro@VBR for this (48 kbps) test. If I'm correct we still have the choice between CBR and the 2-pass VBR mode, right? Each mode lead to different issue:
-CBR: 'unfair' comparison, or "apples and oranges", or "anti-WMA trolling", or... {choose yours}
-2Pass: legal issue (i.e. sharing full tracks over the net) or changes in organisation (i.e. either allowing other people to prepare the WMA samples or providing yourself all future samples).

I'm not 100% sure that it will work, but I can provide a WMA encoder for Winamp that let's you have more fine-grained control over the quality value (e.g. 8 or 9). I'll go ahead and try it, and let me know if it would help you out.

It would be nice!

No luck it doesn't appear that you can set the quality to anything except for 10/25/50/75/90/98

Well, using about the same bitrates WMA Pro's 24 bits files are incomparable to the other contenders' 16 bits anyway, since the latter naturally consume less space, leaving WMA Pro in a clear disadvantage. To improve its chances a little it should be encoded in bitrate-based VBR mode instead, which allows 16 bits stereo at 48 kbps.

Alternatively, since 16 bits files are 2/3 the size of 24 bits ones, you could also allow the WMA Pro codec to participate with 48 kbps*1.5=72 kbps.

Vorbis, AAC, MP3... aren't 16 bits.

You forgot WMA Standard, which is also part of the test. Would be new to me if it encoded to anything besides 16 bits.

Edit: Nope, it doesn't, here's part of its specification: "This codec samples audio at 44.1 or 48 kilohertz (kHz) using 16 bits, similar to the current CD standard, offering CD quality at data rates from 64 to 192 kilobits per second (Kbps)."

It's wrong, because lossy encoders aren't working with fixed bitdepth. And even if you were right, it would be the developer decision to "resample" to 24 bit all 16 bit source material and this behaviour should therefore be tested as it - according to the developers' original decision.

Ah, I see, thanks for the clarification. I didn't know about that, therefore I relied on calculations based on uncompressed materials' behaviour.

Interestingly, you can ask the Windows Media Format SDK decoder to give you 24bit samples for WMA standard, but the least significant 8 bits are always 0. I suspect that Microsoft's decoder implementation is integer based (it's either that or it's stupidly doing a 32bit float->16bit PCM ->24bit PCM conversion)

Anyway, WMAPro at CBR 96 kbps and less is necessary "16 bit" ("24 bit" doesn't seem to be available for those bitrates - not with graphical frontends as Winamp & VUplayer at least).

Hmm, this is really bad news. I know that 2-pass VBR results would not mimic real world usage, but wouldn't this still be better than testing plain CBR? I know it was used in some of Roberto's tests.

foobar2000 doesn't compute correctly the average bitrate for WMA VBR files (tested with WMA STD q10 and with foobar2000 0.9.4.1).

Example with a long file (~20 minutes).

TECHNICAL INFORMATION


The file size and the length are good, but the computed bitrate is necessary false:


4.409.475 bytes for 18 minutes and 50 seconds correspond to 30...31 kbps and not 56 kbps (which is what foobar2000 reports for this file).

Now lets see what other softwares are saying:

• Windows Media Player 11 => 30 kbps CORRECT
• Winamp 5.31 => 30 kbps CORRECT
• MrQuestionMan 0.8 => 44 kbps WRONG

Winamp brings additionnal information in the file property box. I noticed and reported in the past that three fields in metadata are dedicated to the bitrate: "Bitrate", "CurrentBitrate" and "OptimalBitrate". And the values are:


I guess that foobar2000 reports the WMA "Bitrate" field: 56 kbps for "55917". But despite its name it's the wrong one. foobar2000 shouldn't therefore be used as bitrate measurement for WMA VBR files (CBR must be OK). The greatest discrepency I found was 80% (32 kbps for one file while foobar2000 says 58 kbps) but in most cases it's much lower (~10% on average).


To finish, the average bitrate for my set of 150 classical COMPLETE tracks (~16 hours) was 41.43 kbps. It's too low. But I noticed in the past that WMA uses more bitrate for other musical genres; I'm quite certain that 48 kbps would be reached with alternate bitrate table.

This is WMA Standard 9.2, quality based VBR with Q10, correct?

Yes (sorry, it was clear for myself only):
WMA 9.2 ['Standard'] VBR 10 - encoded with Winamp from flac source.

I agree with you, maybe CBR too.

OK, what to do with WMA Professional 10 then? The two possible solutions for me are: encode to CBR or encode samples to 2-pass VBR as it was done in previous listening tests.



CBR is CBR. There isn't much you can do wrong computing CBR bitrate.

Going to install Ubuntu 6.10...

• The past use of 2-pass encoding on short samples was a mistake. Now we're aware that 2pass can't be used like that, we can't reproduce this mistake anymore.

• Well, if VBR isn't possible with WMAPro we should either go for CBR or give up with WMAPro for this test... The latter is not the best solution IMO.


I would:
- determine what exact bitrate WMA STD correspond to (is it 48? 50? 51?)
- adapt HE-AAC and Vorbis to match this bitrate (it should be possible if Nero Digital will be used)
=> three competitors will have highly comparable settings
- then for WMAPro, which would be slightly handicaped, I would introduce in the final representation of results a kind of separation (a border, a different color...) to recall that there's a slight limitation to any direct comparison of results between WMAPro and the three other competitors.

It's worth noting that I can supply a custom CT-based Winamp encoder to hit any bitrate you'd like.

It would be nice but I wonder if we shouldn't follow the results of the previous listening test and use Nero Digital instead of Coding Technology (CBR only) implementation.

small correction. CT is "short window ABR". I presume CT's goal is streamability, so it doesn't stray away from the designated bitrate too far or for too long, but it's definitely not CBR.
example bitrate distribution (on a 51kbps file I just encoded as a test): http://filebox.vt.edu/users/ballison/51.png

ATRAC3+ uses CBR only, right?

Real seems to use 44 kbps CBR which is not an option for this test.

Yeh I can conferm that, I've been useing it for a year or so!

CBR only!

If WMA 10 is going to be used with CBR, I would like to add ATRAC3+ there. Some players, especially in Asia IIRC, support the format so it would fit into the portability category.

I don't like the preoccupation with a nominal and, most likely, inaccurate bit rate measurement -- what matters is the size of the file. My point is that the files being compared should all be as close in size as possible.
Surely we all know that HE-AAC is the codec for these rates, so both Nero and CT should be included. Also, wasn't there supposed to be a new Nero encoder before year end? It would be silly to do the test just before Ivan Dimkovic shows us new tricks...

Most likely, there won't be any significant change in the 48 Kbit/s HE-AAC codec, so I would definitely propose to use the codec used in CT vs. Nero shootout.

Question is, what bit-rate mode to use - as Nero encoder supports several bit rate allocation methods (ISO CBR, more relaxed ABR, VBR 1 and 2 pass)

I would personally go for ABR, as this was the mode used to test against Coding Tech.

Isn't Real just some flavour of AAC in a Real container nowadays?

I think that "aoTuV" should be used with a quality base.
That is because it is tuned up in quality mode.
I have done only the minimum test about managed mode.

Thanks for the feedback Aoyumi and Ivan. I will use the settings you guys recommend since you know the internals of your codecs better than anyone else.

Edit: Do you guys think 6 contenders are too much? If not, my list of contenders is:

WMA Standard 9.2
WMA Professional 10
Nero Digital HE-AAC
CT HE-AAC (Winamp)
Ogg Vorbis AuToV Beta 5
ATRAC3+

If 6 is too much, we could exclude either CT HE-AAC (Winamp) or ATRAC3+.

What to use as anchors? Do we need both low and high anchor? If low anchor is needed, I would go with LC-AAC at 48 kbps. If high anchor is needed, I would go with 80 or 96 kbps LC-AAC.

At this bitrate, I think we need low & high anchors. I'm not sure what, though.

Out of topic, even Zune will not support WMAPro

v1 or v2 (I guess v1 because of previous test)

Unfair, it's still 'Beta'.


and yeah... ABR

Edit: my mistake, pls just delete this post.