Multiformat 128 kbps Listening Test, Pre-Test Discussion Options

[Sebastian Mares]

Greetings!

I am planning to start a multiformat listening test at 128 kbps on November, 30th which should end on December, 11th. The test start can be postponed if necessary.

Edit: The test has been postponed: Stard date is December 5th and end date is December 25th if everything goes well.

Edit: The discussion about codecs is now over.

The following codecs are going to be included:

• iTunes / QuickTime AAC
• Nero Digital 4.2.1.0
• LAME 3.97
• Ogg Vorbis, AoTuV 4.5
• WMA Professional 9.1
• Shine as low anchor

I'm open to suggestions regarding the settings to use. The only thing that's for sure is iTunes / QT AAC which will be used at 128 kbps VBR.
I'd like to ask for Garf's or Ivan's opinion regarding Nero Digital (CBR or VBR, etc.).
As for the rest of the formats, suggestions from the community are welcome, too.

I have some samples in mind already and we'll start discussing them soon. While at it, would you guys prefer 18 or 12 samples?

Regards,
Sebastian

This post has been edited by Sebastian Mares: Nov 29 2005, 16:47

[de Mon]

IMHO
1. MusePack - on the last test it performed very well at that bitrate.
2. WMA - lot of people use it - it would be good to prove WMA is not the best codec as lot of people in the world think.
So my vote for MusePack and WMA.

This post has been edited by de Mon: Nov 12 2005, 20:20

[Raptus]

I'm with de Mon.

[kwanbis]

wma standard and pro

[kennedyb4]

Hi. Thanks for your efforts in organizing the test.

I would suggest skipping musepack as well for the reasons already mentioned and also because it is rarely used at this bitrate.

WMA standard would be the low anchor here I think.If WMA Pro is finding hardware support these days it might be useful to have it blind tested as well.

It breaks my open source loving heart that FAAC is not more actively developed. If it has not changed significantly then testing is probably not warranted at this time.

$.02

[rjamorim]

There is room for two other competitors or one competitor and a low anchor. Maybe MusePack and some version of WMA (Professional or Standard)?

There should be one WMA at least, otherwise the retards populating Slashdot will claim it wasn't tested because we are afraid our beloved codecs would be owned badly by it.

I wouldn't feature MPC. But that's what you should expect from an MPC hater like me.

While at it, would you guys prefer 18 or 12 samples?

18. Otherwise a well know person around here will use the few samples as an excuse for his format of choice not winning, in case it loses (and if it wins, he'll stay quiet)...

This post has been edited by rjamorim: Nov 12 2005, 21:26

[Sebastian Mares]

WMA standard would be the low anchor here I think.

For the low anchor, I suggest using something that sounds bad for sure, like l3enc.

[kennedyb4]

WMA standard would be the low anchor here I think.

For the low anchor, I suggest using something that sounds bad for sure, like l3enc.

Atrac?

[CiNcH]

ATRAC3 has certainly improved since SonicStage 2.0 (especially in 2.2 I think) but would still be the low anchor. Probably more interesting than including another MPEG layer 3 codec like l3enc. So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.3.

New Nero AAC LC/HE Encoder is 4.2, not 3.0...

This post has been edited by CiNcH: Nov 12 2005, 22:43

[de Mon]

So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.

If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRAC superiority which is quite widespread in other communities.

This post has been edited by de Mon: Nov 12 2005, 23:00

[kennedyb4]

So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.

If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRACK superiority which is quite widespread in other communities.

Maybe. The last time the atrac community just attacked the testing methodology used.

Die Hards........

[skelly831]

So I vote for ATRAC3 @ 132kbps with SonicStage 3.2 or 3.

If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRACK superiority which is quite widespread in other communities.

This sounds like a good idea, and at the bitrate that the test will be done, the results should be very interesting.

[rjamorim]

If most people are against MusePack... we could really include ATRAC. With ATRAC included - the test will break TWO BIG MYTHES - about WMA and ATRAC superiority which is quite widespread in other communities.

Didn't we do it already more than an year ago?

Testing ATRAC3 is beating a dead horse, IMO, even if it improved since SS 2.0. I doubt it got much better (Sony has other things to worry about, like DRMing their format to the limit and spreading rootkits throughout the world), it's forcedly DRMd - DRM is not an option like in WMA - and even though it was bad, it wasn't bad enough to act like an anchor. The anchor is there to avoid people ranking samples too low just because there isn't anything worse, even though said samples are prefectly acceptable otherwise.


IMO, good anchors would be old or weird encoders from RRW (like l3dec, MBaacenc, QDesign, AUPECg2), or a lowpass.

This post has been edited by rjamorim: Nov 12 2005, 23:07

[JeanLuc]

If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.

... (Sony has other things to worry about, like DRMing their format to the limit and spreading rootkits throughout the world) ...

[Sebastian Mares]

If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.

I agree with that.

[zima]

I don't - retards from Slashdot (which rjamorin mentioned) will claim that Pro isn't included because of its superiority (and that it will Soon™ be supported by portables anyway...)

[rjamorim]

since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.

Erm... you could use pretty much the same argument for MPC. And that didn't keep it from being tested on my older tests.

I don't - retards from Slashdot (which rjamorin mentioned) will claim that Pro isn't included because of its superiority (and that it will Soon™ be supported by portables anyway...)

Indeed, I agree. F*** SlashDot. We're not testing for the clueless hordes populating that place.


I just thought of another reason not to use ATRAC3: Sebastian is a nice guy. He doesn't deserve to feel the infinite pain involved with installing and using SonicStage

This post has been edited by rjamorim: Nov 13 2005, 00:48

[MuncherOfSpleens]

By the way, has WMA Pro ever really been compared in a listening test like this? If so, could I be provided with a link? If not, could it be included in this test? I'm curious to see how good it really is.

[rjamorim]

If so, could I be provided with a link?

http://www.rjamorim.com/test/128extension/results.html

Quite outdated though.

[Gabriel]

L3enc would be an interesting low anchor, as it would allow to check evolution of "state of the art" encoders. However, it might be a little too good for a low anchor.

[rjamorim]

However, it might be a little too good for a low anchor.

Even 1.0?

http://www.rjamorim.com/rrw/l3enc.html

[Dibrom]

A low anchor would be nice, but personally I would probably prefer WMA Standard and WMA Pro.

I think ATRAC would be a waste of time because:
1. It's already pretty well known by most people who actually do any real research on the topic that it's inferior to just about all modern alternatives.
2. It's fading in popularity, and will continue to do so quite rapidly in the near future.
3. The people that feel that ATRAC are superior seem quite resistant to persuasion by evidence.

So, ATRAC isn't really a worthwhile choice.

I don't think MPC is really a good choice either, because:
1. It isn't really used at this bitrate (not explicitly that is, although VBR may sometimes cause it to average out at this bitrate), even if it can perform well.
2. I don't know if it's fading in popularity, but it's not really growing either. In terms of relevancy for the results, I think MPC would be less useful for a large audience.
3. It hasn't seen any serious development in a long time, whereas other alternatives might have, and would be more interesting from a results perspective -- MPC's performance is pretty well established either way, so it could serve possibly as some sort of anchor, but beyond that it's not going to provide much new data.

I understand that a low anchor is important, especially in order to provide a reference point illustrating just how much some of the codecs in this test have improved either upon recent iterations, or older/inferior codec designs in general.

Maybe the best choice would be to add WMA Standard and WMA Pro, AND a low anchor. That has the danger of making the test too tedious and complex, but maybe it is still doable...

[ErikS]

I don't understand why everyone insist to inlude WMA Standard instead of the Pro version... You don't cripple other formats the same way, but wma everybody wants WMA to be bad and just to make sure it is, only test the ancient WMA Std. Ask Woodinville which version to use, and I'm pretty sure he suggest using Pro instead of Std.

[Gambit]

I definitely would like to see both WMA Standard and Pro included. I think testing against a lower anchor can be done in an another test. You can say whatever you want about WMA, but it at least had gapless support way before some of the other codecs had it (and some still don't have it, yes, I'm looking at you Apple). And I'm very much interested in the differences between WMA Standard and Pro.

[kotrtim]

QUOTE(JeanLuc @ Nov 13 2005, 12:23 AM)
If you go with WMA, I'd suggest to go for WMA Standard since WMA Pro does not play on any portable device and thus is completely useless in a real-world-scenario.
*



I agree with that.

I think we can drop WMA or even FAAC safely now, there's no improvements (version is still the same) since the last test, it is fine to just refer to the previous results

Unlike iTunes, Nero, LAME...they are all improved

WMA Pro vbr 128 kbps 2-pass is not in the previous test, it is interesting and more meaningful to add WMA Pro this time