I'd say include both. It would be interesting to see how their quality are related to each other and whether pro is able to compete with the best codecs out there, and it would make wma supporters silent, they couldn't complain about the lack of wma pro in the test.

I think including WMA standard is a must: It is widely used by end users, plays on practically all portables and is used by most music download stores.

The problem with WMA Pro, IMO, is that Microsoft never targeted it at the end user market, and probebly never will.

It's simple to notice that: All online music stores are selling WMA Std, and Windows Media Player won't rip your CDs to WMA Pro either. Microsoft isn't pushing it for portable player adoption either.

Therefore, I wonder what would be the real use of testing it. Microsoft itself doesn't want us using it.

If I knew Microsoft would never push it for the consumer market, I probably wouldn't have featured it even in my first multiformat test. I only did it because I believed, at the time, that Pro was set to replace Std.

If it is correct that MS will never target WMA Pro to the end user market, I agree. But with the competition from AAC, I think it is strange that WMA Pro won't replace WMA Std at some point. Personally, I think it will.

What about itunes mp3 as the low anchor? It's got to be the most popular of the old mp3 encoders and lots of people still use it. It scored pretty miserably at the last mp3 test

Well, it lost, but 3 is still pretty high as low anchor.

Actually, recent Windows Vista Betas now have WMA Pro as a ripping option in WMP 11. Perhaps MS is deciding to make it mainstream after all. If it's going to be a standard feature in the next gen mainstream OS, it would make sense to include it in the test for the purpose of comparison.

But by then, this test will be outdated. For at least the next two years or so, WMA Pro is targeted at professionals who use it at high resolutions and higher bitrates than 128kbps. Whenever WMA Pro becomes a consumer codec (for use with portables/music stores etc), I agree it should be tested.

Somebody mentioned also VBR...it would be nice to include CBR vs. VBR comparison for MP3. Too bad it would be too much work.

And I still think that WMA Pro should be included, people will have harder time discrediting this test.

Maybe it would be better to define the scope of the test, and then codecs could be included or excluded at the hand of that.

Now I see many arguments (popularity, completeness, support for portables) for and against inclusion of certain codecs, but any of those arguments are only good or bad if you know exactly what it is exactly you are trying to test.

If you just want to test what (on average) the best codec is, then you should definetly include WMA Pro. But if you want to test what the best codec for portable use is, it shouldn't be included.

Since I couldn't really find exactly what is being tested, the discussion about yes/no certain codecs seems a bit messy to me.

I agree. It all depends on the scope of what formats the generous tester wants to listen to. If it is more of a mainstream listening test then I don't think WMA pro should be included but ATRAC3 should definately be included as well as WMA std, mpeg-4 AAC (iTunes), Nero AAC, and Lame 3.97b1. If it is more of a test to include a broad range of formats then WMA pro, WMA std, AAC (iTunes and Nero), Lame 3.97b1, and other formats should be included. This is a lot of work for the listener and I appreciate all his work.

Personally, I would like to see ATRAC3 just to show people on the PSP boards how bad the format is (or convince other people who dive into the Sony formats). I would also like to see WMA std as this format is everywhere. I would also like to top the rumors that a 64kbps WMA is the same as a 128kbps mp3. I would also like to see QuickTime 7 (iTune 6) and Nero 7's AAC encoders going up against Lame 3.97b1.

Again, thanks for the work.

side note, maybe 11 days is too little? I would rather have it open by 20 days.

What about using the encoder with the oldest sourcecode as a low anchor: blade. It should reliably deliver the worst result (and if it doesn't, that's a very interesting result too), and will show how much mp3 has actually progressed.

With sony dropping minidisk, i'd personally favour wma pro over atrac for inclusion in the test.

I second that.

That criteria is kinda wacko. You mean the oldest open source MP3 encoder? Then it would probably be 8hz-MP3. The oldest MP3 encoder is L3enc (and hey, it's based on source code!). But there are even older encoders like Indeo Audio, IMA ADPCM and Lucent's PAC.

I think he meant using Blade because it's not much different than the original specification source code (and it was well spread) which I have read a couple of times on these boards.

Popular formats, that's why I think ATRAC should be left out.
Since WMA Standard is more popular than Professional, I'd also go with Std. and a low anchor like l3enc.

If you really want to include a low anchor, then I'd vote for HE-AAC. And the other contender should then be WMA Standard. While I'd like to see how it compares to WMA Pro, the lack of compatibility and encoding tools make WMA Pro pretty much useless.

my vote would be L3enc as the low anchor, then nero, itunes, vorbis, lame, and unfortunately, both wma std and pro. these tests are benchmarks, and they really need to be as conclusive as possible. testing 7 encoders will certainly be a tedious task for all of the participants, but we, the ha.org crowd are a tough bunch, we can handle it. maybe we might consider leaving the test open for a few extra days to get more participants...

i just really think we would be remiss to leave out wma std because it is so widely used, or wma pro, because many "audiophiles" will argue that it is a viable competitor to the other encoders tested (and it likely will be more viable than wma std, at least). therefore, its performance will need to be tested against the other new encoders...

just my 2 cents. and btw, many thanks to sebastian mares for volunteering to set this up for all of us.

Use HE-AAC as a low anchor, force Nero to output a 128kbps HE-AAC stream
Then use the test results as a graphic reminder of why HE-AAC should never, ever be used above 64kbps.
It's a better low anchor than random_ancient_MP3codec IMHO.

I would like to see how Atrac3Plus performs. I am an owner of Sony portable device and I use it because the device is able to play Atrac files gaplessly. I use only 256kbps with that format but I accidentally once used 64kbps A3P and was surprised that it didn't sound THAT BAD. So I'm quite curious. I really would like if the word "Plus" actually means any progress and step to quality(I've never used the old Atrac so I don't know how it sounds like).I think HA is the only place where I can find the unbiased results without any hype.

Btw, SonicStage 3.3 (the last one since it's being replaced with the Connect Player) support ATRAC3+ at 128kpbs (vs older ATRAC3 132kpbs)

And since Sony's MP3 players, CD, Hi-MD and the PSP(?) support both MP3/ATRAC3, it could be interesting to have it in
Some peoples may be tempted to convert to ATRAC3+ since playback time is about 25% longer (lower power consumption I guess)

I'd like to see WMA Pro included, because I'm curious to see if Microsoft is actually capable of producing a competitive audio codec.

I have to ask a basic/important question: Who are the results of this test targeted at?

If it's the frequenters of HA then one set of codecs may be appropriate, but if it's general consumers of 128kbps digital audio (either purchased or home-brewed) then the codecs chosen should be relevant to the general consumer market (current, popular, accessible).

I vote for WMA STD, and a low anchor. We can come back to WMA PRO in 2007 once Vista is released (the codec may well have changed/updated by then anyway).

In that case WMA Std, but not Pro, yes. Although, as low anchor maybe it is interesting to use HE-AAC(v2)...

Yes, but how many people use ATRAC with that hardware? I'd say only the minority - the majority using MP3 for compatibility reasons.



As stated, the test should feature popular formats which you would probably meet in "everyday's life". 128 kbps is also an excellent bitrate for portable players, music store downloads, etc.

As you mentioned, WMA Pro should be tested when it's used by the large mass.

I think a lot of peoples download/own MP3s at 320kpbs CBR and transcode/re-encode when they transfer it to thier flash memory device
Some may be also ripping a CD directly to it

Yes, but still, compared to WMA, it's used by far less people and that's why I think it shouldn't be included. The question is whether it's really important to replace the low anchor with another format.
WMA (pro or std) has to be tested and I think most of you agree. It's too important to be left out. So this makes room for really another competitor or a low anchor. I'd go with the second.

Hey, you're running the test, so you decide what to include. You can't wait for everybody to come to consensus, that's just nog going to happen.

I would suggest you wait until January to choose the codecs. If you do not, you may obsolete your test.

I though you were required to have evidence for making such assertions.

Identical version numbers aren't good enough for you?



Why?

WMA's been upgraded to version 9.1 across the board (std, pro, lossless, voice) since the last test, in case you missed the memo (courtesy of the absence of an official announcement from MS, most people seem to have, so don't worry ). There have been improvements. FAAC I don't know much about at all.

In addition, the Pro arm of the WMA codec is set to be included among the ripping options available in WMP 11, so progress is being made.

I wonder, have you explored WMP 10's encoding options?

Well, no. I misread kotrtim's statement.

Few people do, yet they are usually the first to say something derogatory about WMP or WM formats at any chance they get without having even tried either of them. It's practically prejudicial.

Personally I don't comment on formats I have no direct experience with. As such, the formats I'd like to see compared are: WMA 9.1 Standard, WMA 9.1 Pro, LAME 3.97 MP3, OGG (not familiar with the specs) and MPC (same) at the very least. Is there any merit to including an FHG MP3 encoder also for comparative purposes? I'm curious to see how well it does against LAME.

When reading the results I would also want all of those included, but while doing the actual testing I would want as few as possible, and absolutely no more than 5 + low anchor... High anchor is useless in a 128 kbit test. My choice would be: Vorbis, Lame, iTunes, WMA Pro + low anchor. If people insist to have a 5:th competitior, then maybe include MPC or Nero or WMA Std or ATRAC or something else that was discussed before....

Really, though, what is the fascination with WMA? Pro may be a significant improvement but the closed nature of the format offers it no benefits over the competitors, other than hopeful inclusion in future hardware. I'd like to see Pro in the mix if only to show that it has nothing to offer over the competitors, quality-wise. Yes, I'm making a fairly baseless prediction.

WMA Pro is a good contender choice, it's shipped with Vista beta already which makes it already more widespread than ATRAC3 will ever be.
Plus I just want to see how good a WMA product with the label "Pro" could be

^emphasis mine

As far as I know, the fact in itself that a project is either open or closed source does not affect the quality of its output. An open source project with poor developers will be outdone by a closed source project with good developers, and the reverse is true also.

As to the "fascination" with WMA - it's a common format. Get used to it. That's just a fact. WMA Pro will become more common also when WMP 11 comes out, as screenshots show it being included as an encoding option. As such, they should be compared. That doesn't mean it will necessarily triumph or lose, but there can't be a fair comparison if a major competitor is not included.

WMP9 beta included PRO encoding. It included even lossless encoding as an option. (I remember burning audio CDs full of samples just to test PRO, because WMP wouldn't take WAV as input, only CDDA). Both options disappeared in the final version.

I see too much wishful thinking in this thread. People here are pushing PRO much more than Microsoft itself.

I think it's sort of hard to know what Microsoft is planning with Pro, so maybe everyone should just stop making assumptions one way or another.

The question should be: Is WMA Pro worth testing?

I think the answer is probably yes. Whether or not it is actually being marketed one way or another is pretty much irrelevant for two reasons:

1. It has the potential to be competitive in this test.
2. It has the potential to become a widespread format.

Most of the other alternatives proposed for testing in place of WMA Pro (most notably ATRAC) fail to meet one or both of those conditions.

Given that one of the presumed goals of this test is to highlight progress in various encoding technologies, and given the fact that WMA is very widespread, it really doesn't make a lot of sense to me to leave Pro out of the picture -- especially when just about everyone here knows that WMA Std is going to most likely be inferior to Pro across the board. Leaving it out amounts to an oversight that people will likely (and perhaps rightly) point to as a flaw in the test.

There is something to be said for keeping the number of codecs tested to a low number, and even perhaps delaying the testing of WMA Pro until some point in the future. However, I believe that will only lessen the relevance of this test perhaps in the somewhat near future.

So, again, my vote is for WMA Std and Pro either with or without an additional codec in the form of an anchor, and no ATRAC.

Personally, I don't promote it at all. And I don't use it also...
I just think including WMA Pro is a way to gain a bit of "credibility" - this community will find a bit easier to promote its views if it'll be a bit more "marketable" to common folks, by including the tech in which large part of them believe (but that's still only a "bit" of course...many don't care about ABX, or even quality for that manner...)

BTW:

MPC has one tremendous advantage applicable to people who download their music (discussing this practise here only) and want very good quality, but not at the expense of lossless (and if they don't own portable player of course )

When you download something in MPC, it's almost guaranteed it'll be very good (especially since REX guidelines are very often used). With OGG...not always, you can find files encoded in ancient versions (and see this only afer download), besides greater popularity leads to less strict ripping (OTOH that's a plus when it comes to actually finding something). And MP3...like OGG, but much more...

Good point. But I think it's not really a good idea to feature a format because it's good for file trading purposes

Some of us would prefer to see a more futureproof test, that's all, even to silence the critics on Slashdot, as a few have commented. If you exclude Pro, a lot of people might discount your test as being unfair. As a compromise, you could do a limited test of it.



A lot of people don't buy Hondas or Fords and get along quite well too. That doesn't mean Car and Driver should exclude the brands from a roundup of family sedans. That would be an unfair test. Another example: a roundup of pickups that excluded the Toyota Tundra because it's not a traditional American brand. WMA's either 2nd or 3rd behind MP3 in popularity, and Pro is no longer in beta. If you kick either out, you run the risk of people seeing the test as being incomplete (see Dibrom's post).

As for being enthusiastic about the codec, that's because I like it, just like other users like OGG, MPC, etc. Apparently a lot of people seem to have problems with others liking MS products or anything closed source, mostly for prejudicial reasons as I posted earlier. That and it seems to be cool to hate MS. They have the right to do that if they wish. None of what I said suggested WMA would win or lose. I'd just like to see it fairly represented, instead of bashed by people who've never personally touched it, listened to it, or otherwise used it.

I had one post in another thread where I spoke about my personal experiences with different formats (I was new to HA and didn't know about the ABX rule, I apologized and recanted) but I don't bash other formats, especially ones I haven't used. People seem to enjoy doing that to WMA, however, and that's just wrong, fairly speaking. Nothing I've posted in this thread asserts the superiority or inferiority of WMA. I want to see a fair test and the results therof.

I have an idea here, lets try to be democratic!

Can the poll be tweaked so that it can let a person pick exactly 5 or 6 choices
depending on the maximum limited spaces for these formats....

I see some boards are able to do this, I'm not sure "Invision Power Board v2.0.4" can do this.

i completelly agree.


at the risk of being off topic, would you buy a Honda (WMA) if it had a lot of restrictions, you have to take it only to honda to reapair it, or a toyota, without restrictions, and open to anyone to repair (Vorbis)?

Heh, you obviously know far more about this than i do.

As markanini said, i suggested blade because it used the original specification source code, but seeing as that code was meant as a demonstration, i suppose l3enc would be a good option also. The other encoders you mention aren't mp3 encoders are they? I meant mp3 of course.

Blade was spread though, i'm not sure which of these two encoders spit out more files to the web.

Nownow, you are making assumptions and you surely know the old cliche' about those. I've used WMA in the not so distant past and found that horrid metallic high frequency clang noticeable at even 128 with the std codec. It is true that Pro improves on this issue (and very decently from my recollection) but I cannot very easily encode to this format with the software of my choice, as with the more open formats in discussion here today (thanks for your clarification of my post above, kwanbis. )...and, again, the only place it will play is on my PC. My Iaudio I5 flash player and Iriver CD mp3 player (both of which support modern incarnations of the vorbis encoder) both only support the wretchedly outdated std but no pro, so it's off-limits to me because the metallic ringing is something my tin ears can't even stand. This is where I have trouble understanding blatant support for the wma format, as good as Pro may or may not be! Nero HE-AAC v.2 codec is also a fairly closed format with limited (nonexistent at this stage as far as I know) hardware support beyond Windows-based PCs but after hearing the amazing clarity of sound at even 40 kbps, I can't help but give the project respect. In reference to Dibrom's post, I also can only comment on the basis of the WMA presence/support in the past and up to this point, so my comments may very well become obsolete come 2006 or at some unknown date in the future...

I apologize if I've said too much, I don't mean to incite a format war.

Back to the thread topic and in reference to what I mentioned earlier, I would definitely vote for wma pro for inclusion in this test...in regards to std, only if we're wishing to win over a handful of the more inquisitive type from "Joe Public" would I be interested in its inclusion. ATRAC3...what kind of future does it have now that the MD has reached near obsolescence?

There is nothing closed about HE-AAC v2.

You can get the full specification from the standardization organisations, and there's a complete open sourced GPL decoder. (There's another open source one for HE-AAC v1, and another open source one for HE-AAC v2)