64 kbps listening test 2005, Pre-test thread Options

[Sebastian Mares]

Greetings!

I am planning to conduce a multi-format 64 kbps listening test under Roberto's supervision. The exact date for the beginning of the test is not set, since I am waiting for Apple to release their HE-AAC encoder sometime in April (according to the rumours).

So far, here is what I was thinking about. Feedback and suggestions are welcome.

Codecs:

• Nero HE-AAC
• Apple HE-AAC
• Ogg Vorbis
• Windows Media Audio
• LAME 128 kbps as high anchor
• Fraunhofer 64 kbps as low anchor
• mp3PRO maybe

Nero and Apple HE-AAC will be included for sure.
What I don't really know is which Vorbis version to use - either AoTuV3 or 1.1. Same applies to WMA - either Standard due to compatibility or Professional for quality. Last but not least, should mp3PRO be tested again as reference (since there aren't any changes from version used in Roberto's last test)?

As you can see, there will be 6 or 7 (depending on mp3PRO) codecs.

Samples:

• DaFunk
• EnolaGay
• experienca
• gone
• Illinois
• mybloodrusts
• NewYorkCity
• riteofspring
• Scars

I am planning to use 18 samples - 9 from Roberto's old listening test and 9 additional ones. What do you guys suggest?

The results will be calculated in the same way Roberto's results were calculated.

So, what are you thoughts?

Regards,
Sebastian

Edit: Spelling and typo in the number of samples planned to use.

This post has been edited by Sebastian Mares: Mar 22 2005, 16:00

[Sebastian Mares]

As difficult sample, I would suggest Kraftwerk to replace Waiting (which was used in the last test and is not missing).

[sehested]

Good initiative. I will look forward to the test.

I suggest to leave mp3PRO out, since nobody use it.

As for WMA I would recommend using WMA std. It is more widespread than PRO and is what most people refer to as WMA. People already have a hard time distinguishing between std. and PRO so choosing PRO is likely to lead to misinterpretation of the test results. As an alternative one could consider including both WMA std and WMA PRO, anyway my vote on WMA std.

What about using LAME 3.97a8 as low anchor?

[Latexxx]

I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.

[PaleGreen]

I'd like to see both WMA Standard & Pro included.

[Latexxx]

I'd like to see both WMA Standard & Pro included.

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.

[rjamorim]

I'd like to see both WMA Standard & Pro included.

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.

I wouldn't include WMA standard because it has already been tested and, as far as I know, it hasn't changed much, if any, since

[sehested]

I wouldn't include WMA standard because it has already been tested and, as far as I know, it hasn't changed much, if any, since

May I then suggest we use it as reference?

[rjamorim]

May I then suggest we use it as reference?

FhG 64kbps (low anchor) didn't change much either, AFAIK.

[sehested]

May I then suggest we use it as reference?

FhG 64kbps (low anchor) didn't change much either, AFAIK.

In that case we can leave out WMA completely

[Gabriel]

Perhaps Lame could be used as low anchor?

[Latexxx]

May I then suggest we use it as reference?

FhG 64kbps (low anchor) didn't change much either, AFAIK.

In that case we can leave out WMA completely

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?

[guruboolez]

I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.

I second that suggestion. Atrac3plus wasn't tested yet (only atrac3@132 kbps). It could be very interesting to see how will perform this format, which correspond (according to optimistic estimations made by Sony) to atrac@132 and to atrac@292 kbps:

http://www.mdfr.com/faqs/atrac_roadmap_small.jpg

But it's necessary to upload them as lossless files (flac or something else).

[Thoifl]

What I don't really know is which Vorbis version to use - either AoTuV3

I would take AoTuV3, since it should be tweaked for low bitrates and is little testet.

[rjamorim]

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?

Good point. But then again, most slashdotters can go .... themselves, for all that I really care...

I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.

I second that suggestion. Atrac3plus wasn't tested yet (only atrac3@132 kbps). It could be very interesting to see how will perform this format, which correspond (according to optimistic estimations made by Sony) to atrac@132 and to atrac@292 kbps:

http://www.mdfr.com/faqs/atrac_roadmap_small.jpg

But it's necessary to upload them as lossless files (flac or something else).

Well, same would have to happen with MP3pro, that is another alternative for 7th codec.

So, I guess the discussion is about of wether Atrac3+ or MP3pro should be tested.

I already feel sorry for Sebastian if people choose Atrac3+

This post has been edited by rjamorim: Mar 22 2005, 16:58

[DreamTactix291]

What I don't really know is which Vorbis version to use - either AoTuV3 or 1.1. Same applies to WMA - either Standard due to compatibility or Professional for quality.

Since aoTuV b2 (which 1.1 is essentially) has been fairly well tested it might be interesting to see how the relative newcomer aoTuV b3 fares IMO. Even worse possible scenario of regression I can't see it being really that much worse than 1.1 (which I doubt it is worse but only testing can prove either way).

As far as WMA I'd go standard. Sure it's not as high quality as pro but I've never even come across a WMA Pro file I didn't make myself in the wild.

[rjamorim]

Perhaps Lame could be used as low anchor?

Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?

[sTisTi]

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?

WMA std should definitely be included. I'm sure it will fail badly against Vorbis and HE AAC, so this test will be one more evidence against Microsoft's ridiculous claims and demonstrate the low quality of WMA compared to other modern codecs.

[guruboolez]

atrac3plus could be tested over mp3pro, for some reasons:
- mp3pro was tested previously, and hasn't progress since.
- atrac3plus wasn't tested
- there are more device playing atrac3plus than mp3pro (all modern HiMD units, many CD players, Jukebox and Sony flash USB devices).
- atrac3plus have more supporters (I'd use another word to describe them...) than mp3pro, which has progressively lost importance with time and competition.

[Latexxx]

I already feel sorry for Sebastian if people choose Atrac3+ 

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.

[guruboolez]

Perhaps Lame could be used as low anchor?

Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?

MP3 at 64 kbps must be a low anchor. It doesn't matter to use lame or fhg. All we need is a encoder which sound worse than all other competitors.
Testing lame at 128 and at 64 kbps is IMO a good idea.

[guruboolez]

I already feel sorry for Sebastian if people choose Atrac3+ 

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.

I must warn you that basic MD lovers won't consider results as significant unless you've made encoding directly from device, and not from SonicStage. Hardware encoding is often considered as better than software encodings (placebo or not: I don't know — blind listening tests were never done to my knowledge).

[rjamorim]

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.

Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.

I must warn you that basic MD lovers won't consider results as significant unless you've made encoding directly from device, and not from SonicStage. Hardware encoding is often considered as better than software encodings (placebo or not: I don't know — blind listening tests were never done to my knowledge).

That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.

This post has been edited by rjamorim: Mar 22 2005, 17:06

[Gabriel]

Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?

I am not sure if the difference would be that big between FhG and 3.97b.

Anyway, whatever the choosen anchors, I'd like the graphs of results mentionning "low anchor" and "high anchor" instead of the codec name.

[Latexxx]

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.

Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.

That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.

I have a working SS installation if that matters.

There are some hardware atrac3 vs SS atrac3 abx threads somewhere.