Greetings!

I am planning to start a multiformat listening test at 128 kbps on November, 30th which should end on December, 11th. The test start can be postponed if necessary.

Edit: The test has been postponed: Stard date is December 5th and end date is December 25th if everything goes well.

Edit: The discussion about codecs is now over.

The following codecs are going to be included:

• iTunes / QuickTime AAC
• Nero Digital 4.2.1.0
• LAME 3.97
• Ogg Vorbis, AoTuV 4.5
• WMA Professional 9.1
• Shine as low anchor

I'm open to suggestions regarding the settings to use. The only thing that's for sure is iTunes / QT AAC which will be used at 128 kbps VBR.
I'd like to ask for Garf's or Ivan's opinion regarding Nero Digital (CBR or VBR, etc.).
As for the rest of the formats, suggestions from the community are welcome, too.

I have some samples in mind already and we'll start discussing them soon. While at it, would you guys prefer 18 or 12 samples?

Regards,
Sebastian

IMHO
1. MusePack - on the last test it performed very well at that bitrate.
2. WMA - lot of people use it - it would be good to prove WMA is not the best codec as lot of people in the world think.
So my vote for MusePack and WMA.

I'm with de Mon.

wma standard and pro

Hi. Thanks for your efforts in organizing the test.

I would suggest skipping musepack as well for the reasons already mentioned and also because it is rarely used at this bitrate.

WMA standard would be the low anchor here I think.If WMA Pro is finding hardware support these days it might be useful to have it blind tested as well.

It breaks my open source loving heart that FAAC is not more actively developed. If it has not changed significantly then testing is probably not warranted at this time.

$.02

There should be one WMA at least, otherwise the retards populating Slashdot will claim it wasn't tested because we are afraid our beloved codecs would be owned badly by it.

I wouldn't feature MPC. But that's what you should expect from an MPC hater like me.



18. Otherwise a well know person around here will use the few samples as an excuse for his format of choice not winning, in case it loses (and if it wins, he'll stay quiet)...

For the low anchor, I suggest using something that sounds bad for sure, like l3enc.

Atrac?

ATRAC3 has certainly improved since SonicStage 2.0 (especially in 2.2 I think) but would still be the low anchor. Probably more interesting than including another MPEG layer 3 codec like l3enc. So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.3.

New Nero AAC LC/HE Encoder is 4.2, not 3.0...

If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRAC superiority which is quite widespread in other communities.

Maybe. The last time the atrac community just attacked the testing methodology used.

Die Hards........

This sounds like a good idea, and at the bitrate that the test will be done, the results should be very interesting.

Didn't we do it already more than an year ago?

Testing ATRAC3 is beating a dead horse, IMO, even if it improved since SS 2.0. I doubt it got much better (Sony has other things to worry about, like DRMing their format to the limit and spreading rootkits throughout the world), it's forcedly DRMd - DRM is not an option like in WMA - and even though it was bad, it wasn't bad enough to act like an anchor. The anchor is there to avoid people ranking samples too low just because there isn't anything worse, even though said samples are prefectly acceptable otherwise.


IMO, good anchors would be old or weird encoders from RRW (like l3dec, MBaacenc, QDesign, AUPECg2), or a lowpass.

If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.

I agree with that.

I don't - retards from Slashdot (which rjamorin mentioned) will claim that Pro isn't included because of its superiority (and that it will Soon™ be supported by portables anyway...)

Erm... you could use pretty much the same argument for MPC. And that didn't keep it from being tested on my older tests.



Indeed, I agree. F*** SlashDot. We're not testing for the clueless hordes populating that place.


I just thought of another reason not to use ATRAC3: Sebastian is a nice guy. He doesn't deserve to feel the infinite pain involved with installing and using SonicStage

By the way, has WMA Pro ever really been compared in a listening test like this? If so, could I be provided with a link? If not, could it be included in this test? I'm curious to see how good it really is.

http://www.rjamorim.com/test/128extension/results.html

Quite outdated though.

L3enc would be an interesting low anchor, as it would allow to check evolution of "state of the art" encoders. However, it might be a little too good for a low anchor.

Even 1.0?

http://www.rjamorim.com/rrw/l3enc.html

A low anchor would be nice, but personally I would probably prefer WMA Standard and WMA Pro.

I think ATRAC would be a waste of time because:
1. It's already pretty well known by most people who actually do any real research on the topic that it's inferior to just about all modern alternatives.
2. It's fading in popularity, and will continue to do so quite rapidly in the near future.
3. The people that feel that ATRAC are superior seem quite resistant to persuasion by evidence.

So, ATRAC isn't really a worthwhile choice.

I don't think MPC is really a good choice either, because:
1. It isn't really used at this bitrate (not explicitly that is, although VBR may sometimes cause it to average out at this bitrate), even if it can perform well.
2. I don't know if it's fading in popularity, but it's not really growing either. In terms of relevancy for the results, I think MPC would be less useful for a large audience.
3. It hasn't seen any serious development in a long time, whereas other alternatives might have, and would be more interesting from a results perspective -- MPC's performance is pretty well established either way, so it could serve possibly as some sort of anchor, but beyond that it's not going to provide much new data.

I understand that a low anchor is important, especially in order to provide a reference point illustrating just how much some of the codecs in this test have improved either upon recent iterations, or older/inferior codec designs in general.

Maybe the best choice would be to add WMA Standard and WMA Pro, AND a low anchor. That has the danger of making the test too tedious and complex, but maybe it is still doable...

I don't understand why everyone insist to inlude WMA Standard instead of the Pro version... You don't cripple other formats the same way, but wma everybody wants WMA to be bad and just to make sure it is, only test the ancient WMA Std. Ask Woodinville which version to use, and I'm pretty sure he suggest using Pro instead of Std.

I definitely would like to see both WMA Standard and Pro included. I think testing against a lower anchor can be done in an another test. You can say whatever you want about WMA, but it at least had gapless support way before some of the other codecs had it (and some still don't have it, yes, I'm looking at you Apple). And I'm very much interested in the differences between WMA Standard and Pro.

I think we can drop WMA or even FAAC safely now, there's no improvements (version is still the same) since the last test, it is fine to just refer to the previous results

Unlike iTunes, Nero, LAME...they are all improved

WMA Pro vbr 128 kbps 2-pass is not in the previous test, it is interesting and more meaningful to add WMA Pro this time

Actually, if you're referring to the Multiformat 128kbps listening test done by rjamorim in May 2004, WMA has since been upgraded to version 9.1, a fact that seems to escape many people. Microsoft has done a rather poor job of making this widely known, so it's understandable that it's flown well below the radar of most listeners.

Besides that, it's the codec supplied on the majority of new PCs sold today. Omitting it from the test would be like omitting the Honda Accord from a roundup of family sedans.

I was under the impression the WMA Std did not get an upgrade with the WM9.1 release. Has anyone actually compared the output of 9 and 9.1 to see if there is in fact an update?

I just received green light from Aoyumi. He also told me to use a quality preset, so I think I'll go with something around 4 (exact quality level depends on the samples and the resulting average bitrate).



Well, just in case...



Running on a virtual machine with Windows 98.

I assure you that Microsoft didn't just change the version from 9 to 9.1 just for fun. Forums regarding the topic:

http://www.hydrogenaudio.org/forums/index....showtopic=31029

and

http://www.hydrogenaudio.org/forums/index....topic=27037&hl=

According to the second link, there is an ABX-audible difference between the two versions. I use WMA for my personal ripping purposes, and the main improvement I've noticed is a bitrate increase for each VBR Quality setting. At the highest setting it's a nice compromise between the quality of the lossless and the compatibility of the lossy versions of the format.

One is playable on portable players, the other one is not.
Btw, they are NOT the same format, but two different formats.

I vote for Atrac3 and WMA Standard because I have a feeling that Atrac3 could beat WMA nowadays.

And not even Windows Media Player supports ripping to WMA Pro.

The freely available Windows Media Encoder 9 does. WMA Pro is aimed more at content producers than consumers such as most of us.

For what it's worth, here's my take.

My preffered linup would be Nero, iTunes, Vorbis, Lame, WMAStd, WMAPro, and a Low Anchor. Obviously, this introduces one more codec than on previous 128kbps tests conducted by Roberto. In the first 128kbps test their were six competitors and no low-anchor. Theoretically, the low anchor should be easy to ABX from the original, thus not really require any extra time or effort. As such I think having seven codecs is acceptable, as long as one is a low anchor.

As far as bitrates go, you'll probably want to independantly verify, but I've encoded 291 tracks from an assortment of genre's using the new Nero codec (4.2.1.0). The closest preset I found was the VBR/Stereo - Internet 90-100 kbps [LC AAC]. On my test tracks, this provided an average bitrate of 133kbps. Some may consider this too far from the target bitrate, but I do believe it is probably not possible to get closer to the target of 128kbps.

For anyone about to bring up the issue of fairness, I quote Roberto:


For reference, at 133kbps, my tracks total 1024mb. If the average were 128kbps, they would total 985mb. This represents only a 4% increase of bitrate/filesize. I would imagine the margin of error for this test is going to negate a 4% bitrate increase.

WMA Std will probably make a suitable low anchor, since this is a low bitrate test after all. WMA Pro is an interesting contender, and has not been tested adequately.

ATRAC3 has been tested to death, and the ppl who still use it are not interested in listening test results. Keep the rootkit company's software safely away

seeing as how HA is seen as some-what reliable source for audio info (writers from Wired come here) i think WMAPro should be tested just to prove that Pro doesnt always meen better (see http://www.rjamorim.com/test/64test/results.html and http://www.rjamorim.com/test/32kbps/results.html for example).

The problem with several codecs is not only the fatigue, but the need to iterate codecs several times to come to a ranking among them. I realized that at my low bitrate tests. Even though there was very little fatigue involved, as reference was instantly recognizable for most cases, people spent too much time checking samples again and again to see who won over who.



In my tests, the limit I tried to pursue was 10% bitrate difference from the format consuming least bits to the format consuming most. Therefore, 133 would not be a problem as long as there is no format consuming less than 120.



Neither of these tests featured WMA Pro. Pro actually doesn't even go down to 32kbps, and only goes down to 64kbps using a VBR mode that's completely unreliable as far as bit allocation goes.

I'd say ATRAC doesn't really have to be tested since it's used by a very limited user group, unlike MP3, AAC, Vorbis or WMA.
Also, I don't think that the ATRAC lovers will accept the test results anyways.

Out of curiosity, I'm interested how WMA Pro, ATRAC3 or Musepack would compete. But nevertheless, WMA Standard and FhG-fastenc are more often used with commercial audio-software.

I suggest Coding Technologies aacPlus @ 64 kb/s as low anchor. Maybe it's quite close to 128 kb/s with some samples.

Sorry? If this encoder has 64 kbit/s CBR mode, how do you expect it to get close to 128?


As for me, I vote for WMA Std and Pro to be included in this test. And as low anchor... maybe FAAD or Blade (less likely).

Sounds like he means quality, not bitrate. Somehow I believe that there really are some samples to which this would be a reasonable assumption. So including 64 kbps HE AAC would be interesting indeed and so how far low bitrate codecs have came since 1995.

But somehow I believe that including a lower bitrate codec without including FhG @ 128 would be unapproprite.

I don't think MusePack should be tested at 128 kbps because it was optimized for higher bitrates. As for ATRAC3, see my comment above.
Including WMA is another story.

One more question regarding samples - what would you say about using the same sample set as in Roberto's last multiformat 128 kbps test?

Shouldn't this be discussed in it's own thread?

True, I didn't get that. Maybe because I was assumpting that even low anchor encoder should get close to 128 kbps bitrate...

Well, the development of low-bitrate codecs should be tested separetely, I guess. I had a disussion about a 64 kbps listening test a few months ago, but the test was cancelled because Apple didn't release an HE-AAC encoder as I expected and at that time, the new Nero version was also about to come out "soon".

Well, yes, you are right, but I wanted to see what you think about the idea before opening a special thread just for this.

For my own listening test (test is finish - results tomorrow), I used:

• AAC: Nero Digital VBR
• AAC: Apple iTunes VBR
• MP3 LAME 3.98 VBR
• Vorbis: aoTuV VBR

as low anchor, a very old AAC encoder
as high anchor, LAME 3.97 -V2 --vbr-new

I discarded atrac3+, MPC or WMA (both Std and PRO). Two of them are not usable with portable players (I don't cound PDA in this category), and the two remaining (atrac3+ and WMAstd) are really poor and simply uninteresting. Testing them is fine for curiosity, but for practical usage it's a waste of time IMO.

I got 125,7 kbps for Nero Digital -internet high with my 150 reference tracks.
For the samples "non-classical" samples, I got 133 kbps with the same setting.

Note if iTunes will be tested in VBR mode, there will be the same issue. iTunes AAC works as iTunes MP3 in VBR mode: the target bitrate also corresponds to the minimal bitrate. In other words, the encoder don't encode anything below 128 kbps (apart digital silence maybe). Consequently, the bitrate is necessary superior to 128 kbps. I got 133.33 kbps for classical and 137.x for non-classical.
Same issue will occur with LAME -V5.
Only exception is aoTuV, which benefits for a very flexible VBR mode.

These problems might be a good reason to discard all CBR encoder from the test and to use VBR encodings only outputting to the same approximate bitrate.
LAME -V5, iTunes AAC and Nero Digital are fortunately is this case. Same for Vorbis of course. I'm not sure that WMA VBR (Std and Pro) could be close to ~130 kbps. If they're not, it may end the debate about using or not this format in the test.
kl33per, or someone else, could you try different WMA settings to see if a 1-pass VBR mode could stay close to 130-135 kbps?

Sorry for posting this in three replies...

About WMA or ATRAC as low anchor:

- during the first multiformat test, Blade was used as low anchor. Final note = 1.99.
http://www.rjamorim.com/test/128extension/results.html
- during the second listening test, WMA standard get a much higher note (3.65).

In other words, WMA seems to be too good to be considered as low anchor.

Yes, as mentioned already. If we discard ATRAC, we could test WMA and feature a low anchor. WMA is used by a lot more people than ATRAC.
Whether the standard encoder should be tested because it works on portables and it's also available in a lot of music stores or the professional due to its higher quality is uncertain. What do you guys think?