Greetings!

I am planning to conduce a multi-format 64 kbps listening test under Roberto's supervision. The exact date for the beginning of the test is not set, since I am waiting for Apple to release their HE-AAC encoder sometime in April (according to the rumours).

So far, here is what I was thinking about. Feedback and suggestions are welcome.

Codecs:

• Nero HE-AAC
• Apple HE-AAC
• Ogg Vorbis
• Windows Media Audio
• LAME 128 kbps as high anchor
• Fraunhofer 64 kbps as low anchor
• mp3PRO maybe

Nero and Apple HE-AAC will be included for sure.
What I don't really know is which Vorbis version to use - either AoTuV3 or 1.1. Same applies to WMA - either Standard due to compatibility or Professional for quality. Last but not least, should mp3PRO be tested again as reference (since there aren't any changes from version used in Roberto's last test)?

As you can see, there will be 6 or 7 (depending on mp3PRO) codecs.

Samples:

• DaFunk
• EnolaGay
• experienca
• gone
• Illinois
• mybloodrusts
• NewYorkCity
• riteofspring
• Scars

I am planning to use 18 samples - 9 from Roberto's old listening test and 9 additional ones. What do you guys suggest?

The results will be calculated in the same way Roberto's results were calculated.

So, what are you thoughts?

Regards,
Sebastian

Edit: Spelling and typo in the number of samples planned to use.

As difficult sample, I would suggest Kraftwerk to replace Waiting (which was used in the last test and is not missing).

Good initiative. I will look forward to the test.

I suggest to leave mp3PRO out, since nobody use it.

As for WMA I would recommend using WMA std. It is more widespread than PRO and is what most people refer to as WMA. People already have a hard time distinguishing between std. and PRO so choosing PRO is likely to lead to misinterpretation of the test results. As an alternative one could consider including both WMA std and WMA PRO, anyway my vote on WMA std.

What about using LAME 3.97a8 as low anchor?

I would take Sony's Atrac3plus @ 64 kbps. It has shown some potential with non-problem samples and it would be great to compare it to other formats.

I'd like to see both WMA Standard & Pro included.

I wouldn't include wma pro because no internet radio uses it and it would only confuse people.

I wouldn't include WMA standard because it has already been tested and, as far as I know, it hasn't changed much, if any, since

May I then suggest we use it as reference?

FhG 64kbps (low anchor) didn't change much either, AFAIK.

In that case we can leave out WMA completely

Perhaps Lame could be used as low anchor?

If wma would be left outside, half of wannabee slashdotters would be asking where is wma, the cd quality at 64 kbps codec. Were the HA Lame and mpc lovers afraid of it?

I second that suggestion. Atrac3plus wasn't tested yet (only atrac3@132 kbps). It could be very interesting to see how will perform this format, which correspond (according to optimistic estimations made by Sony) to atrac@132 and to atrac@292 kbps:

http://www.mdfr.com/faqs/atrac_roadmap_small.jpg

But it's necessary to upload them as lossless files (flac or something else).

I would take AoTuV3, since it should be tweaked for low bitrates and is little testet.

Good point. But then again, most slashdotters can go .... themselves, for all that I really care...



Well, same would have to happen with MP3pro, that is another alternative for 7th codec.

So, I guess the discussion is about of wether Atrac3+ or MP3pro should be tested.

I already feel sorry for Sebastian if people choose Atrac3+

Since aoTuV b2 (which 1.1 is essentially) has been fairly well tested it might be interesting to see how the relative newcomer aoTuV b3 fares IMO. Even worse possible scenario of regression I can't see it being really that much worse than 1.1 (which I doubt it is worse but only testing can prove either way).

As far as WMA I'd go standard. Sure it's not as high quality as pro but I've never even come across a WMA Pro file I didn't make myself in the wild.

Dunno... wouldn't FhG be more recommended, since it is, at least in theory, better at low bitrates thanks to IS?

WMA std should definitely be included. I'm sure it will fail badly against Vorbis and HE AAC, so this test will be one more evidence against Microsoft's ridiculous claims and demonstrate the low quality of WMA compared to other modern codecs.

atrac3plus could be tested over mp3pro, for some reasons:
- mp3pro was tested previously, and hasn't progress since.
- atrac3plus wasn't tested
- there are more device playing atrac3plus than mp3pro (all modern HiMD units, many CD players, Jukebox and Sony flash USB devices).
- atrac3plus have more supporters (I'd use another word to describe them...) than mp3pro, which has progressively lost importance with time and competition.

If you mean preparing samples, it is pretty easy nowadays using a program called himdrenderer.

MP3 at 64 kbps must be a low anchor. It doesn't matter to use lame or fhg. All we need is a encoder which sound worse than all other competitors.
Testing lame at 128 and at 64 kbps is IMO a good idea.

I must warn you that basic MD lovers won't consider results as significant unless you've made encoding directly from device, and not from SonicStage. Hardware encoding is often considered as better than software encodings (placebo or not: I don't know — blind listening tests were never done to my knowledge).

Ah, that's good news, indeed. But he'll still have to install SonicStage. That flaming piece of poo ruined my win2000 installation.



That is certifiable bollocks. Hardware encoding has no reason to be better than software encoding, quite the opposite: hardware encoding faces limitations related to battery consumption, real-time processing on underpowered DSPs, integer-only processing, etc, etc.

I am not sure if the difference would be that big between FhG and 3.97b.

Anyway, whatever the choosen anchors, I'd like the graphs of results mentionning "low anchor" and "high anchor" instead of the codec name.

I have a working SS installation if that matters.

There are some hardware atrac3 vs SS atrac3 abx threads somewhere.

Oh, yes, after I reinstalled my system from scratch it installed without problems. But it is (was?) a terrible buggy program, neverthless.

I think it's somewhat related to the security and encryption crap Sony installs everywhere.

Might be usefull to do the SonicStage installation under VMWare or something similar.

I can prepare the samples if required.

what are the up- and downsides of the differen vorbis version, which could be used?


i would also love to see atrac3+ to be tested

and if there is space for another codec, apple lc-aac would be interesting too

I don't think so. Who uses LC AAC at such bitrates, if one can use HE AAC with much better quality?

it would show the quality difference between he-aac and lc-aac at such a bitrate

well, just realised that we already tested that in your 64kbps test

Exactly

I don't think the gap got much different since.

Nokia users who did not replaced the tinny memory card they got with the phone?

I would be interested in seeing MPC in the test. I know its not tuned for these bit rates but it did suprisingly well at 128 and Im hoping that the new team does some tuning at low bit rates after work on 7.5 or 8 is well underway. While it currently has littly hardware support, its low CPU overhead for decoding could help extend battery life on flash players where decoding is the major drain on batteries.

As you said yourself, it's badly tuned at these bitrates, so I don't see what is the point of such test currently.

Maybe if the dev team really tunes low bitrates after SV8 is released, then that might warrant a new test.

OK... So far, MusePack and LC AAC will definitely not be included in the test. MusePack isn't the encoder to test at 64 kbps, the same applies to LC AAC.

MusePack's emphasis is on quality and transparency and was never designed for such low bitrates. People using MusePack will at least encode at 128 kbps. As Roberto already stated, if things change after SV8, MusePack should be included in a 64 kbps listening test.

As for LC AAC, I doubt that there are any major improvements since the last 64 kbps test (where Apple's implementation was used) and for such a low bitrate, it is logical to use HE-AAC. Also, I think it's a good idea to have a direct comparision between Nero HE-AAC and Apple HE-AAC and that makes AAC appear twice in the test.

Hmm yeah but either way, do those MD fanboys conclude all recording/copying/converting shoudl be done by the device rather than a PC??
Don't get me wrong, I have an MD player (old school model from 1996 or so) and I quite like it. It was very good back then too. And I did most of the copying using the device itself. Two words: *big hassle*.
I don't think most MD users would do that if they were given the alternative.
So, even if we asume that hardware encoding is better, that we really should use software encoding (there are hardware MP3 encoders too, after all) - at least until ABX tests prove there is a noticeable difference.

Vote for atrac3+ and wma std.
WMA pro would be more interesting but less useful

Vote for Atrac3+

I would like to suggest the following sample for the listening test:

Supertramp - School

The 19s sample I have uploaded contains acoustic guitars, piano, cymbals and bass.

I have good experience with this sample in revealing subtle difference between encodings.

Let me hear what you think about the sample.

If desired I can upload a 30s version of the sample.

Sample proposition: the beginning of "Money" by Pink Floyd.

I do not have it available, but I am sure some Pink Floyd fan could upload it.
Basically it is background music with coins and cash machine sounds. I think that the coins coud be interesting.

Yeah, I can... The clipped remastered version or the "old" one?

SHINE ON

I think it would be interesting to use the SSE optimized aotuvb3/archer version of Ogg Vorbis.

A direct comparison to the non SSE version would be best, but since differences are expected to be small anyway, this is an opportunity to test it on multiple samples. If any problems come up, it could be tested against the regular version after the test.

Of course if you want to do the encoding at the user end, you'll have to distribute separate versions of the sample packages for people with non SSE capable CPUs.

The problem in using Archer and aotuv3, IMO, is that the results won't be as valid as we would like them to be. If vorbis performs badly, we'll not know if it is becauseof a format weakness or just because the tweaking broke something. After all, these are beta versions - not for production environments.

Now, IF the Vorbis fellas around here conduce a little test before the real thing, like happened in my multiformat @ 128 test, then the choice of what version to use will be supported by more reasons than just "let's give it a try".

Well, it wasnt really tuned for 128 either was it and it is tied for first

Nero HE-AAC and Apple HE-AAC (has it already been released??) are tested, what about FhG / Coding Technologies HE-AAC? It is the only HE-AAC implementation available for free (in Helix Producer). I think it should be included.

You're right. But I believe the differences regular/SSE would be minimal and this may be a chance to detect flaws (I don't expect any that result from the optimizations). This could be verified after the test by checking the problem cases against the regular version.

But if you need to run a separate test anyway then you might as well do it beforehand. Maybe there are serious flaws which could ruin Vorbis' reputation!

The archer tunings are at RC3 right now and I think we will see a final soon. Ah, but of course the archer tunings will still remain beta, my bad.

edit: added quote

The HE-AAC encoder from Helix Producer hasn't been improve since last 64kbps listening tests. There is only new thing : introduction of new target bitrates (32 & 48 kbps).

I just tried it on one sample at quality 1 and 2. MPC is not a contender at this bitrate. Sounds like you are underwater, bubbles everywhere - great effect. In comparison, Lame encoded at 64k is hifi. No ABX required.

I have two more candiates for listening test samples...

They are both exceptionally well mastered using the Blumlein technique where the entire sound picture is captured at a single point, carefully chosen to achieve the optimum balance between direct sound and reverberation.

This makes unnecessary all forms of electronic correction and manipulation and the music signal travels by the absolutely shortest possible path between the microphone and the final CD.

The first sample is recorded in an old stone church:

Vaquero

The second sample is recorded in a jazz club:

Four Brothers

What's unussual with these recordings is their depth of image, timbre and dynamics.

For more info on the recording:

Opus3 Recordings