Hello.

I'm creating this thread so that we can discuss how the upcoming AAC @ 128kbps listening test will be handled.

The encoders I plan to feature are:

-Nero AAC encoder VBR profile Streaming :: Medium (@Ivan: Fast or HQ mode?)
-Apple iTunes 4.2 128kbps
-FAAC "whatever VBR setting comes close to 128kbps"
-Compaact! "same thing as above"
-Winamp AAC encoder 128kbps
-FhG l3enc 1.0 as anchor

Some codecs that might be considered to replace another:
-Real Producer
-NCTU AAC
-Emuzed
-Imagine Technology


I am personally very fond of the idea of using l3enc as anchor. It'll be a good insight on how perceptual audio coding developed since the first MP3 implementation. It'll be like "the oldest vs. the newest".

Of course, as usual, EVERYTHING is still open to discussion. Remember I am conducing these tests for you, not for me, so please make your opinions heard.

Actually, only one thing is not open for discussion: the amount of codecs. There won't be more than 6 (including anchor), no matter what.

Some other topics I want to discuss:

-Samples tested. IMO we should definitely replace some samples. Specifically Polonaise (I would suggest Fossiles as replacement) and Illinois (I would replace it with our old friend Fatboy). Do you believe something else should be replaced?

I wouldn't like to replace Waiting. I love that sample for it's wacky behaviour, so I want to force you guys to listen to it over and over again until your ears pop out of your skull :B


-Should only LC be tested? Or maybe test other (better?) profiles when the encoder supports them, like FAAC and Compaact. I think an argument for this would be that it's like VBR vs. CBR - you shouldn't penalize some codecs because others lack that feature. A very good argument against this would be that LC is waaaay more supported now and probably in the future. Opinions?


-Do you think there is any need for encryption (it will definitely be used on the Multiformat tests)? If so, we'll only use Schnofler's comparator, since ff123's doesn't support encryption.


These are the subjects I can think about ATM. I'll post more ideas/questions as I come up with them.

Thanks for your help.

Best regards;

Roberto.

Well, isn't winamp AAC practically the same thing as quicktime AAC except an older version? I would be in favour of NCTU AACenc instead if this is the case. It would be interesting to see how it would fare in comparison to other codecs which rely less on ODG and other objective measures in development.

Yeah, but what will you say when it comes in third?

P.S.: Come on, somebody had to say it.

That is a very good point, indeed.



I will give up all this mess and become a hermit

6 encoders is the absolute maximum imho, I'd rather drop the anchor to be honest.

128kbit AAC will likely be even harder than the mp3 test, and listening gets a little tedious after the 3rd or 4th sample, so adding exotics like NCTU is a big nono

I would like to see you keep the WinAmp encoder due to its widespread use among users. Would be nice to know how well it encodes. Also I would like to know of Imagine Technology's codec as I use that with their Cool Edit Pro/Adobe Audition plug-in. Thanks rjamorim for all the kind advice your post on HA. You're very helpful!

No, there must be a bottom anchor. It doesn't have to be l3enc, but it's a good way to keep things in perspective.

How about VQF as anchor? According to AudioCodingWiki , it is(was?) a part of MPEG-4 Audio version 1.

Re: the profile issue: I'd say just stick with the LC profile, for the simple fact that I'm not personally aware of any hardware player that can handle higher profiles like Main. Plus, I also question whether using the more complex profiles drastically improves sound quality (i.e., a statistically significant improvement over LC). I haven't seen any data comparing the different profiles of popular encoders (FAAC, Nero, Dolby, et al), so while higher profiles should sound better in theory, to what extent (if any) has yet to be studied.

Actually, TwinVQ is part of MPEG4 audio. It's slightly different from VQF, and in MPEG4 audio it's limited to veeeery low bitrates - around 8-16kbps

That said, I was planning to use it as an anchor in the multiformat test that will come after the AAC test.

I trust in your greater wisdom
Also, thank you for going through the trouble once again.

For the anchor, FhG 1.0 would surely be funny, but I persoanlly would prefer something that is used more widely in old files and has a bad name to it. (hint: Blade 128k) Maybe it is not so bad as many ppl like to call it?

For the profile, only plain LC please

For samples: I would surely like to see "mybloodrusts" and "daFunk" from the mp3 test

I already ruined Blade's reputation


(First multiformat @ 128kbps test)



I wouldn't think so...

Oh. Thanks for the clarification.

Ack.

dont get that...
one codec more would make sense to me!

and i would choose real for this extra codec, as i dont like their 192kbps is better than 128kbps marketing blabla...
first of all i want to know how their codec does compared to itunes

I guess you have no idea the utter pain in testing several (6+) encoders, most of them getting pretty close to transparency :B

hehe

BTW, I would expect Real to be similar to Winamp.

well as i attended every test till now, also the first one, i think i have a small idea of what this means

but when it will be finished i think most will say "it was worth it"

anyways you said the codec amount is not to be discussed and i respect this
and i also dont want that real with its .ra sh*t gets popular

RM is just another container format, playable in many players these days, but I have already mentioned that AAC wrapped in RM is just a temporary solution. Our alternative file writers were not ready in time for Beta. RealPlayer 10 Gold and RealProducer (Helix Producer) 10 Gold will write AAC to another format...

The Real (HE-)AAC encoder was licensed from Coding Technologies, is the only HE capable encoder in addition to Nero (CT did invent aacPlus/HE by the way), and their AAC base encoder probably started out from a similar codebase as the FhG AAC encoder. Personally I would be very curious how it compares to the rest of the AAC encoders out there. "Installing" Helix Producer is as simple as unzipping the files, and if included in the test, I will make sure Roberto has no trouble whatsoever with the encoding or installation (don't want to repeat trying to install RealPlayer.. ), as well as losslessly convert the .ra files to whichever format is preferred, .aac, or .m4a.

As a sidenote, I think it is a good idea to be very clear about which is the actual codec provider, for example for WinAmp, who did they license the encoder from? For Real, like I mentioned, this would be Coding Technologies.

e ai Roberto

would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first

Nullsoft (AOL actually) licensed from Dolby.

CodingTechnologies licensed their base AAC encoder from FhG (FhG consumer AAC encoder, actually)

FhG and Dolby share a lot of codebase, according to an AAC developer.

That's why I guess they would be similar

Maybe I should start a poll with AAC codec options? What do you think?

Dude, you pointed out the exact problem in your suggestion. How can I use an anchor that might even surpass some of the actual competitors? :B

well ... you mean it to be an anchor ... but what happened to xing? ... i second the opinion of using LAME ...

Well I think Apple iTunes (Quicktime 6.5) and Nero 6 should be in for sure since they are 2 quite different codecs and 2 of the most popular AAC encoder apps. Apple is based on Dolby Consumer I believe with heavy tweaking and Nero is heavily tweaked and is based on the Coding Technoligies codec I believe. I thought WinAmp would make a good baseline Dolby Consumer (without tweaking) comparison against the iTunes/Quicktime 6.5 encoder.

That would take care of 3 of your 6 choices in my opinion.

RealNetworks would be nice, to see a mostly untweaked Coding Technologies codec compared against Nero (heavily tweaked) one. Hopefully RealPlayer 10 will play .m4a and .mp4 (including HE AAC) natively when it gets out of beta

Blah!!!

Please consider all the bad comments on Xing done here and elsewhere over the years, and then come tell me I did a bad choice!

No more than 6 codecs, as 6 is already enough. I also second the latest LAME version for anchor. Atleast keep iTunes, Nero and Winamp in the test, as these are the most widespread codecs. RA's AAC codec might also be more spread than some of the others (NCTU, etc) once it's out of beta, so i'd probably want to include that one also, even though all these codecs may be based on the same codebase - optimizations might be different.

Nero is 100% in-house development



FAAC must be definitely in, because of implications of open source/yadda yadda.

I actually think Faac is more essential in this test than Winamp



Real codec is absolutely unrelated to Nero, as I explained above. So, no, it won't work to compare them in these grounds.

it sure it was a good choice, but it, turning not into the anchor, didn't harm the test ...

It harm in the aspect that there's no perspective. Codecs dipped too low on scores. I wanted Xing to be like Blade was in the first 128kbps multiformat test. Always at the bottom, to avoid some otherwise reasonable codec being labeled very bad because others were better.

I really like you original choices for the 5 AAC encoders (not including the anchor which appears to be open to much debate):

-Nero AAC encoder VBR profile Streaming :: Medium (@Ivan: Fast or HQ mode?)
-Apple iTunes 4.2 128kbps
-FAAC "whatever VBR setting comes close to 128kbps"
-Compaact! "same thing as above"
-Winamp AAC encoder 128kbps

Perhaps you will stick with them, as that selection should give you a smattering of the various AAC encoders. I trust your judgment on this rjamorim

What about FhG AAC ?

I'm sure we can get permission / find a way to encode samples for test purposes

nero is very important
i would like to see real as compared to itunes
FAAC is a must (NCTU-AAC was based on FAAC i believe)
throw in winamp if you really want to
for the anchor use the itunes MP3 (we know it sucks and will act as a comparison between itunes AAC and MP3)

just my opinion, as long as nero, itunes, and faac are in there i'll be happy.

Well, I agree that would be interesting, but I am not fond of testing codecs noone else has access to.

I mean, so what that it won, I can't use it anyway... :/

I support Roberto's initial codec choices fully, including l3enc as lower anchor.
More than six codecs is absolutely insane and those that ask for it have probably never participated fully in one of Roberto's tests, or have an unhealthy amount of free time and patience.
None of the other choices IMO merits the place of any of the initially chosen contestants, since they are either 1) In beta or not-quite-final state 2) not really popular or relevant enough.

I also think Roberto's choice of codecs is nice as it is. The only other codec I'd be interested in, is the NCTU codec, for the same reason Bonzi stated in the second post of this thread (proving that listening tests are superior to ODG results). But none of the codecs of the original line-up seems debatable to me, with the possible exception of compaact (but I'd prefer testing a serious contender instead of just slapping some encoder which isn't really used anyway).
Lame as an anchor is a bad idea in my opinion. Not only because it might repeat the "Xing disaster", but because it would considerably increase the difficulty of the test and it's difficult enough as it is right now. With Lame there'd be six real codecs instead of five codecs and one ... well ... joke. And it will help general motivation if there's at least one codec everyone should be able to identify.

Now to an interesting question

I haven't really made up my mind about that one. The problem, of course, is that a comparison between an LC codec and one which uses additional tools will be completely useless to people who plan on using AAC on their portables. But it might also provide a very interesting outlook into the future. The latter, though, only holds if we can really expect considerable advantages from using the additional features. I'd really like to hear some comments from the developers on this. If the advantages are only slim, there's no point in using them, in my opinion. But if we can really expect a definite increase in quality, maybe it would be interesting to have FAAC use them, to see how it compares to the supposedly better, but then somewhat handicapped, commercial encoders.
But these are only some thoughts, I haven't really decided for myself yet whether that's a good idea.

About the anchor:

I would very much like to see Lame in this test, as this would allow us (and me) to see how it performs against current AAC codecs.

However, it would make the test harder, and with less defined scale. Using something that we think will really be a low anchor is a better idea. This way the scale of notation will probably be more uniform.
Using l3enc 1.0 is, from an "historical" point of view very interesting.

So even if I would be interested to see Lame compared to AAC codecs, I think that it would be better for the purpose of this test (which is to compare AAC codecs) to use another anchor. I like the idea of l3enc 1.0.

doesnt faac maybe do well as "anchor" (maybe bringing the same or worse quality as lame)?

Obviously both the Apple and Nero codecs should be included, to see how they've each progressed since the last test. Furthermore, I think compaact should definately been included, and as it seems to be undergoing fairly regular development as well as trying to establish an active user base here at HA, I think it's time we got an evaluation of it's quality. I would also like to see how far FAAC has come being a free alternative. Finaly, I'm probably most interested in the NCTU codec for the reasons Bonzi has stated. As Winamp should be similar to iTunes (and apparantly Real as well) I think it should be dropped. I think the anchor is also good choice and is deffinately a requirement.

On the issue of Lame. Clearly in the last Multiformat test, LAME was behind by a fair margin. Furthermore, this is specifically a AAC test and should not feature other formats.

FAAC has improved a lot recently and a lot of people will be suprised to hear how well it performs at 128kbps.
I've already found samples on which it performs better than LAME and I wouldn't be suprised, if their overall performace would turn out to be similiar. This is pure speculation though and not backed up by any tests (yet).
Gabriel's idea of including LAME seems very interesting to me, but LAME's performance is probably too good to serve as an anchor.

dev0

Did you read my post? Perhaps I did not managed to properly expose my opinion?

1. Who uses Compaact? What's the reason behind including it in this test? (Not intending to sound demeaning in any way - just curious since I never seen it being used anywhere...)

2. Please replace the Rite of Spring sample! It drives me mad...

No problem. But please suggest another sample to replace it

Edit: Woot! 4444 posts

Hi,

you all know I am biased. That said, I would really like to see compaact! in this test. Surely it has not so much users as Nero, but it is only 3 months old, and we are a very small company, not being able to make so much marketing. I personally think that compaact! has potential for being a popular encoder.

Regarding different profiles:
The main profile is not useless, it gives higher quality than the LC profile. However, the quality increase is not great (but of course greater than with the silly LTP profile).
I personally think every competing encoder should give its best, regarding settings and profiles in this test. But again, I'm biased...

Regards,
Alexander

Well, I think the following codecs should be included:

• Nero
• iTunes
• FAAD
• Winamp
• Real

I would either use l3enc or LAME for the 6th encoder.

• Nero
• iTunes
• Real
• FAAC
• Compaact
• Anchor

I would vote to use FAAC1.17 as an anchor since it's much worse than latest FAAC and it could be interesting to compare both versions.
If you don't like FAAC1.17 then I would vote for LAME to be an anchor
(even if it turns out not to be the worst it's still very interesting comparison)


Yes, Waiting is definitely not to be removed.


Those other profiles in FAAC are likely to be broken so it shouldn't be used.

since this will be a 128 test HE for nero won't help. im not sure what the main profile for AAC really is (and no idea what the difference is for LC). i think that all codecs should use LC simply because it is the setting that can be used on portables. i don't know about anyone else, but the only time i would encode to 128 is if i had limited space on a portable. otherwise i would aim for a transparancy setting.

i think an early version of FAAC is a good idea. i would like to see how the open source encoder has progressed. besides won't lame be against AAC and the other codecs in the next multiformat test? there is no point in using it as an anchor since is might end up like Xing. pick a truely poor encoder.

Nero
itunes (real and winamp are assumed to be similar)
FAAC
compaact
NCTU-AAC (real or winamp could replace, but since NCTU doesn't use short blocks - or something to that effect - i would like to see how it performs)
FAAC 1.17 - anchor

oh, and will there be an vorbis test before the multiformat test to compare the various tunes?

I'm completely opposed to including NCTU, until they have sorted out their licensing issues.
rjamorim's original selection seems to be the most sensible to me and l3enc would probably do fine as an anchor.

dev0

Well, I don't think you can compare it like that. Xing was chosen because pretty much everybody thought it would fail miserably. You can't say the same thing about Lame in this test. Using Lame here is to much risk messing up the point of the test: Which is the best AAC encoder?

In the MP3 case I think we were lucky to have iTunes serving as a "light" anchor.

And I guess we will see Lame against the AAC winner in the next multi-format test?

According to them, they already sorted it out (I.E, started developing an encoder from scratch and gave up developing from Faac)

But still, due to their quite immoral past behaviour, I'm not very inclined to take them seriously and test their encoder.

About Faac 1.17: It has already been tested in the first AAC@128kbps test, so you can use that test to compare how it fared against others. I don't see much point testing an old version again.

But, in MP3's case, I would really like to know how it sounded when it was premiered



Of course. I hoped to test another encoder since Lame was already tested in the former multiformat test. But since it won the MP3 test... heh

Actually, I think that's another good reason not to test Lame here. It has been tested in the 128kbps extension test, the 64kbps test, the MP3 test, will be tested in the next multiformat test... :B

This may seem a stupid question but does the Quick Time Professional AAC Codec differ from the I-Tunes one or are they one and the same? I thought I read somewhere that there were slight differences but I cannot remember where I read this!

Depending on the answer I would like to see:

• Nero
• Quick Time
• FAAC
• Compaact
• Winamp
• Anchor