Well, I guess it will take at least one more week for the test to start. Apple's HE-AAC encoder isn't even out yet.

Also, again, Vorbis tests are welcome to help deciding which encoder to use.

I reckon HE-AAC is still a bit off, definitely more than 1 week.

I know this is a little late in the thread and I may have missed it, but I think you ought to do both WMA and WMAPro.

Reason being is because WMA is widespread, but WMAPro is superior to Standard. It would keep people from becoming confused and still giving the WMAPro codec a fair chance(to keep the WMA fanatics off your back).

Rumour has it that Tiger will be released mid-april. Of course, Apple rumours are NEVER trustworthy.



I think the level of usefulness of WMA Pro at 64kbps is too small. Microsoft clearly sends you the message that they only want you using it above 128kbps (in CBR mode at least). Forcing it to go lower with VBR results in too random bitrates that would make the test biased for one side or the other.

Also, it's worth remembering such low bitrates are useful for a) limited memory hardware or b) streaming. a) is pointless because no portable player supports WMA Pro. b) makes no sense because VBR doesn't mix well with streaming.

Yea, I don't know of any portable support. If I remember right someone on this board said that MS never intended it to be.

Anyway, was just a suggestion for some folks concerns. I don't use WMAPro, have messed with it a little in the past, so didn't matter much to me.

Suggestion: if you plan to publish a bitrate table, and in order to avoid (at least to limit) the usual discussion about 'fairness', 'apples & orange', etc... why not publishing a bitrate table of the full encoded song instead of the short samples (~20 sec), which are probably higher.

It's not really easy (you have to download full samples, or have to ask to people submitting samples to encode their files for you and then report the exact value), but it could be more representative of the bitrate of very short samples. On visual tests (video, cf. doom9.org), we have the bitrate for the full movie, and not for the selected frames. What do you think?

Regarding the low anchor, I would personnally like Lame @abr.
My own justification is:
*that would probably not change that much the notation of the low anchor, and anyway the purpose of the low anchor is to be low.
*that would allow me to have fine input regarding Lame's true performance @64k compared to modern codecs

This is a great idea.
And it would give you a "real life" scenario, showing the actual behavior of the codec with full songs, and how it would act on such tracks.

I second that suggestion. During the 32 kbps test, I gave most sample only between 1 and 2 because then I decided to follow the comments near the scale while (it seems that) a lot of people use the whole scale without taking into account the comments '“perceptible but not annoying” and so on...).

I mean that the situation should be clarified: should we use the whole scale or follow the comments?

I will modify to the labels to excellent - poor.

Perhaps you could post your choice of the labels so we can discuss and perhaps improve them, in order to make them more descriptive and as unambiguous as possible? I'm not sure "good" and "excellent" labels would help me make my results on scale with others' results...

Tiger has gone GM, the question is whether HE-AAC will be included in QT 7 after all..

Sebastian...I don't know if you are already begun the preliminaries for these tests, but may I suggest to add a new candidate : Enhanced AAC+ v2 ??

Brandon Siegel has updated its CLI encoder for dbPowerAmp. It can now reach up to 64 kbps for stereo files. Check this thread directly on dBAmp forum for more infos.


Another question : have you got enough samples or can we post new ones ?

HE-AAC v.2 = AAC + SBR (v.1) + Parametric Stereo (=> v.2). According to some tests requested by Ivan ~one year ago, Parametric Stereo have a negative impact at "high" bitrate (> 40 kbps IIRC). Of course, it was noticed on Nero AAC encoder, and it doesn't necessary mean than other encoders have similar reactions. Therfore, it could be worth to test.


To be entirely honest, I don't really like the idea of testing two HE-AAC encoders (Nero & Apple) in a same multiformat test. I don't like it, but I understand the reason I think. I would rather make preliminary tests, in order to choose the best challenger for each format (as we do for wma, vorbis...). There are many possible contenders:

- Nero AAC VBR & CBR (does VBR produce stable quality at low bitrate?)
- Apple VBR & CBR (same comment)
- Coding Technologie (in Real products) AAC
- the one pointed by kurtnoise.

But to make this possible, we need people to test them first.

Well, with Apple and Nero, the two most popular HE-AAC encoders, I have enough HE-AAC encoders covered.

Regarding samples, I have enough now, thank you. It's a bit hard to decide which ones to use, since almost all of them are more or less killer samples.

I guess you are right, but who wants to conduce another pre-test? Look at how many people tested Vorbis.

Edit: By the way... Regarding WMA, I guess I will stick to Standard since Pro isn't really "tuned" for such bit-rates and as Roberto said, 64 kbps is used for either streaming or portable usage. None of those areas are covered well by Pro.

I had this in mind when I said "I don't like it, but I understand the reason I think"
Preliminary collective tests are possible, without someone to conduce it from A to Z. Remember last 128 kbps listening test: there were many choices for vorbis (CVS, aoTuV, Quantum Knot, Modest Tuning), and the final decision was based upon personal tests. We know the result, more than positive for Vorbis

Yes, but who has the time to spend on testing? I don't think I can justify a winner based on five or six test results.

Edit: BTW, everyone is free to conduce pre-tests.

I can't find the exact thread, but if I remember correctly, there were no more persons to test various fork of Vorbis in order to select the good one last spring.


Time is not the problem. Motivation is probably a greater one. I don't think that many people will be interested to test HE-AAC implementation that are not used otherwise.

Nevertheless, removing one redundant challenger could make the test easier (each additionnal codec makes evaluation and the establishment of the hierarchy harder). There are currently 7 contenders (including anchor): that's very much, especially for untrained people.

Well, the most I can do is dump ATRAC3+ and if enough tests are available, one of the HE-AAC encoders.
The reason why I wanted both HE-AAC encoders to be featured is that a lot of work was done to improve the Nero encoder and it is also interesting to see how Apple performs against the big competitor (so there is a direct comparison).

This?

Yes, this one. I've found the exact post I had in mind. Roberto had only 5 set of results (+ 2 partial results) to make a decision. Also note that aoTuV beta 1 was pre-tested, and that the non-tested aoTuV beta2 was included in the final collective test... and won (tied with mpc). Taking some risks is not necessary a bad thing

From personal tests the "fast" mode on Nero causes some serious problems/distortions around 96kbps or lower in LC-AAC and somewhere below 64kbps it causes trouble on HE-AAC (32kbps HE-AAC in fast mode sounded horrific! High mode sounded much better)

high mode with the "streaming" profile gives good quality 64kbps HE-AAC, fast mode sounds similar at 64kbps, but the bitrates only average around ~50-55kbps, i haven't done any ABxing to determine any quality differences though

So Quicktime 7 will be widely available in a little over two weeks. Is that enough time to narrow down the contenders? And can anyone verify that Quicktime 7 includes an HE-AAC encoder? AAC isn't even mentioned on the preview pages for Quicktime 7.

Well, I am still waiting for some Vorbis tests. Now that AoTuV PB 4 is out, somebody might want to test that against Xiph 1.1 and Archer.
Regarding HE-AAC tests, I doubt that anyone would like to test on their own. Also, I would rather get the test started pretty soon after Apple releases their HE-AAC encoder, so there won't be much room (time) for pre-tests. Therefore, I guess I will stick to my first decision featuring both codecs in the main test.

I was just visiting Apple booth at NAB Las Vegas and saw what they have and asked this. Answer is there's no HE-AAC in Quicktime 7, maybe in the future.

Too bad. I suppose this test will have to wait then, as QT HE AAC would be the most interesting novelty.

I'm still interested in the results without QT HE AAC. It'd be nice to re-compare where the various codecs are after a year of development.

I believe Sebastian will leave the decision to the forum members. He left me some offline messages on ICQ talking about not being online for a few days, but I couldn't understand why or when he will return.

Well, IPS is updating the forum software to 2.1 Alpha and during this process, the blogs are offline, too. The problem is that I had a surgery near the Coccyx on Friday and am not allowed to sit or lay on my back for two weeks. I also have to go to the doctor and let him clean the open wound (which is a painful process, unfortunately) each morning.

ATRAC VS MP3PRO
=============
If it's not settled yet, I think the arguments for Atrac3+ and against Mp3Pro are pretty solid. I vote for Atrac3+ as well.

ATRAC encoder
------------------
However, I think somebody should really test whether the best (Sony) hardware Atrac encoders are _different_ from the latest SonicStage software release (assuming same version of Atrac now).

It's all very fine to assume there are no differences, but that's not a proof in scientific terms and it leaves way too much room for useless idle speculation, which is going to be rampant anyhow.

Maybe somebody with connections to the Atrac3 forum people could encourage a user there to encode a few tracks both with a hardware encoder and Sonicstage and see if they are bit-accurate?

Then again, even if we'd find out that hardware encoders are different in their output, it still doesn't solve the question of which encoder to use.

As such, for the sake of implementation easiness I recommend going with the encoder that the person doing the encoding is most comfortable using.

If somebody complains, we can ask him/her to conduct her own tests.

ANCHORS
=======
As for anchors, I think they should be of clearly higher/lower quality in many respects.

Remember, this is a subjective analysis and some people find some artifacts annoying, while other people are almost ignorant of them.

That is, making the anchors too difficult to spot will only muddy the results.

Bitrate bloat (esp. WMA Std)
=====================
I know this issue is not a favourite amongst many of us, but how will the bit-rate averaging issue be handled?

While a 5-25% difference in avg bitrate may not always be critical at 128-160kbps, it can have serious skewing at 64 kbps testing, no?

I for one have noticed in my own testing that getting WMA 9.1 Standard 2-pass VBR (ABR) to achieve anywhere near the advertised bitrate is really hard, considering there isn't much flexibility in choosing the target bitrates.

For example, I'm now encoding to 128 ABR for a test of mine and WMA9 constantly gives 140-160 ABR on most tracks, even though the target is set to 128 kbps (2-pass vbr, 9.1 WMA std, encoded from dbPowerAMP rel.11).

Can this issue be handled in any meaningful manner? Is it a problem with the chosen sample set?

BTW, the problem of average bitrate fluctuation with OGG (aotuv b3) and MP3 (lame 3.96.1) is of much smaller magnitude - at least on my encodings.

SOFTWARE
=======

What software will be (can be used) to conduct the test when the testers download the sample pack?

I'm not very fond of ABCHR Java version myself.

Also, this is probably a FAQ, but I couldn't find an answer for this by searching, how will the samples be decoded into the final test form (in regards to clipping, gain/limiting, dither)?


CHECKING
=======
Would it be possible to check the test data submissions for accidental clicks?

For example, if for a certain sample set both samples are rated below 5.0, this is more than likely a mistake. It wouldn't be too difficult to check for this, if it's not already checked for.

I have had this happen to me on various occasions during the previous test (when clicking play/stop buttons and moving sliders): i'd rate a sample that I had ABXed properly at say 3.5. However, I had accidentally also moved the paired sample rating to 4.8 without noticing it.

Other than that, my hat goes to you on staring to pull this test together. It's not easy work, but somebody's gotta do it

regards,
Halcyon

PS I hope you recover soon from you operation. Take it easy though. Health is more important than testing

Hardware and software versions of atrac3plus won't be bit-identical because all hardware encoders (Hi-MD players) always resample the audio at 44,1 kHz even if the original is at that sample rate to compensate jitter.

Ah well, another good reason to forget about the comparison or use HW encoders, imho. BTW, do you have a source for this? Do all Sony Atrac HW gear use asynchronous sample rate conversion by default?

Windows Media Encoder seems to handle bloat reasonably.
Bit rate (expected): 64.02 Kbps
Bit rate (average): 64.19 Kbps

According to my tests, I managed to get bitrates around 64 kbps for most of the test samples using WME, too.

Any news on what the final low anchor will be? Is it still to be Adobe Audition 1.5 FhG Encoder? and if so, in what mode? VBR (ABR)? CBR?

Most probably same as in Roberto's last 64kbps test: 64kbps CBR, allow M/S, no I/S, allow narrowing, no CRC.

I'm way late looking in on this, but I sure would have liked to see the bsiegel AAC+ encoder for dbPoweramp tested. The max rate supported is 64kbps.
I declare it the shiznit for Pocket PC users.

How about using the Coding Technolgies AACPlus encoder as released in the new Magix MP3 Maker 10 Deluxe package for the listening test? See thread about it at:
http://www.hydrogenaudio.org/forums/index....showtopic=34361

The CT encoder is sure to be included in many new products shortly. This is the first consumer implementation of a file based encoder using the CT AACPlus encoder I know of. It seems to sound pretty good to me compared to Nero and Quicktime, and it really should be put to the test I believe. Thanks!

As already mentioned, I don't mind people doing some private listening tests to decide which HE-AAC encoder to be used.

I've already submitted my review of Magix MP3 Maker 19 Deluxe in the thread that guest101 initiated. I really do not think this codec can be excluded from testing because XM Satellite is so much in the news and their claims for high fidelity so loudly shouted (see: http://tinyurl.com/bk4c9 ).
Certainly, inclusion of this codec in the test will result in more widespread attention of the test results than would otherwise be the case.

Why would you use an older version of WMA, when you are using the state of the art with an AAC codec? I think that isn't very equitable.

1. I don't know any hardware players supporting WMA Pro.
2. Most people rip to WMA Standard because that's the default setting in WMP AFAIK.
3. Music stores offer music in the WMA Standard format.

So it's just like AAC-HE, give or take a very little. Don't you think it's a bit improper to be comparing the state of the art in one technology to 3 year old technology elsewhere?

I would suggest that you use at least WMA-Pro, in the interest of simple equity.

Well, I am not sure, but did WMA Pro change since the last test?

Can WMA-Pro be even encoded properly at 64kbps?

AFAIK, even Microsoft themselves don't advertize WMA Pro for such bitrates. The CBR settings only go down to 128kbps, and forcing 64kbps with VBR 10 doesn't work very well.

As far as I can tell even at VBR Q10 you have to have either 5.1ch 16 bit or 2ch 24 bit, you can't have 2ch 16 bit with WMA Pro.

I really doubt the people here would be confused. Most people who read these forums know the difference between WMA Standard and Pro.

I really think both should be included. Standard, because that is the WMA that by far most people use, and most mp3 players don't support Pro.

Pro, because I (probably many others as well) would like to know how it compares with the others. It can be played back on a Pocket PC (or Windows Smartphone), as well as from a desktop PC, and hopefully there will be more mp3 player support for it in the future.

For your information, the test will start once Apple's HE-AAC is available.

hmm.. For all we know it could be years away.