AAC @ 128kbps listening test discussion Options

[rjamorim]

This may seem a stupid question but does the Quick Time Professional AAC Codec differ from the I-Tunes one or are they one and the same? I thought I read somewhere that there were slight differences but I cannot remember where I read this!

They are the same encoder. The difference is that in QuickTime you have the option to encode in three quality modes: Good, Better and Best. iTunes is hard-coded at Better. Best supposedly is the same as Better in 16bit streams, and only shows any improvement in 24bit streams.

[mmortal03]

would it make any sense using LAME instead of FhG l3enc? that way we could see how much better (if better at all ) AAC is when compared with the best MP3 encoder
I wouldn't be surprised at all if it came out in first

Dude, you pointed out the exact problem in your suggestion. How can I use an anchor that might even surpass some of the actual competitors? :B

I doubt LAME at 128kbps would surpass AAC, it was the lowest in the previous 128kbps multi-format test (besides Blade). What has changed at 128kbps since then that woukld put it over AAC now? LAME is a good choice as anchor.

[rjamorim]

I doubt LAME at 128kbps would surpass AAC, it was the lowest in the previous 128kbps multi-format test (besides Blade).  What has changed at 128kbps since then that woukld put it over AAC now?

In the multi-format is was worse than QuickTime. But I can't predict how it will behave against Compaact or Faac.

Besides, as I recently posted, Lame has been featured in too many tests. Enough is enough, IMO.

[knik]

So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.

[emtee]

@rjamorim:
Maybe you should start a poll to decide what will the anchor be, reminding users that LAME as already been tested in the previous listening tests and will be tested again in the multiformat listening test, after this one.

I would like to see these in the test:

• Nero
• Apple
• Real
• FAAC
• Compaact

For the anchor, i vote for a MPEG-4 codec. Maybe an early version of FAAC or Psytel.

[mmortal03]

I re-read some of the posts, and I have changed my mind. The next multi-format will pit LAME against AAC, so let's keep this one an all AAC test, including anchor.

So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.

I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME. These tests have definately had interesting results. Yours might win


Edit: BTW, knik, was that entire quote yours, or were you answering someone elses question? It seems as if you are FOR LAME in the first sentence, but then you say there is no need for it in the second sentence...

This post has been edited by mmortal03: Feb 8 2004, 18:33

[knik]

I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME.  These tests have definately had interesting results.  Yours might win

I'm sure the results will be very interesting with or without LAME but including LAME would make it even more interesting.

[mmortal03]

Gotcha, you want to know how LAME does in comparison to the OTHER AAC codecs, not just the winner. I can understand that.

[Continuum]

So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.

I second that! The AAC winner might not be useable for all situations, so it's quite interesting to compare every AAC competitor with the best MP3 solution available.

Besides, what's wrong with using Lame in many tests? It allows (to an admittedly small amount) inter-test comparability (Another meaning of the word anchor?), while FhG l3enc 1.0 will only make the used rating range smaller.

[knik]

So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

IMHO there is no point testing LAME against AAC winner again since it already lost badly in such test and latest AAC encoders can only be better.

Edit: BTW, knik, was that entire quote yours, or were you answering someone elses question? It seems as if you are FOR LAME in the first sentence, but then you say there is no need for it in the second sentence...

The whole point is that including LAME here is at least as useful as including it in the multiformat test.

[bidz]

I would wait till after this test, as once we see the results, it might not even matter any more to you how it compares to LAME.  These tests have definately had interesting results.  Yours might win

I'm sure the results will be very interesting with or without LAME but including LAME would make it even more interesting.

Agreed. I really want to know how the latest version of LAME stacks up against most AAC encoders, not just the winner.

Using L3enc as anchor would be "funny", but would it be of any use? i don't think so, but having LAME as anchor would be of use for many people. This way they can see how all AAC encoders, not just the winner (currently Quicktime/iTunes) performs against the best MP3 solution.

This post has been edited by bidz: Feb 8 2004, 18:57

[MGuti]

IMO, this is a test to determine the best AAC codec, not compare MP3 and AAC. if you really want to know how LAME (a free mp3 encoder) fairs against FAAC (a free AAC encoder) run asimple test. with 2 encoders it is much easier than with 6 and LAME included.

if early FAAC is out of the question than the early MP3 encoder sounds best. it will act as a good anchor.

in truth, if we REALLY wanted to be able to compare every codec in relation to each other we would have to run a test including every codec at once. that is far too many to do in a single test (time and patience being the limiting factors).

stick with AAC encoders and leave LAME out. you never really know how the test will turn out anyway.

[SirGrey]

I also like the idea to use l3enc as an anchor, but may be it will be better to use it's latest version (2.71 or whatever was the last) ?
To compare to the first mp3 codec in it's best, not the buggy first version ?

[rjamorim]

Well, the anchor rating isn't supposed to be useful to users. It's supposed to be bad (Lame doesn't fit that criteria) and put things into perspective. That's why formal tests would rather use lowpass as anchor

Let's be realistic: If Lame is included, it won't be an anchor. It will just be another competitor. An MP3 competitor in an AAC test. You probably notice the mess here...

What people seem to forget is that the bottom anchor is supposed to fail! It's not there to compare against the actual competitors.

Anyway, that's my point of view. If the majority of users want, I'll feature lame. But I'll be sincere: I don't like the idea of adding a MP3 competitor to an AAC test.

Regards;

Roberto.

[kwanbis]

Well, the anchor rating isn't supposed to be useful to users. It's supposed to be bad (Lame doesn't fit that criteria) and put things into perspective. That's why formal tests would rather use lowpass as anchor

...

What people seem to forget is that the bottom anchor is supposed to fail! It's not there to compare against the actual competitors.

roberto, i respect your idea, but i don't share it ... i mean, and even you said it i think, the test are for the community, as such, for me at least, and a poll could be well be used, it is very interesting to know if the "actual champ", is still up to the task, compared to the "title challenger" ... for me including LAME would be very nice for the test ... it would mean something like "should i bother to re-rip into AAC, or can i still keep my LAME files"? ... lets do a poll

PS: i hoppe we can still go next week to picaso's display at san pablo

EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

This post has been edited by kwanbis: Feb 8 2004, 22:28

[schnofler]

So, if Roberto doesn't want to include LAME here then how can I know whether LAME is better or worse than FAAC?

Including Lame simply to have a comparison Lame/FAAC is not very convincing to me. There are lots of comparisons you have to leave out in every listening test (why not include Xing to see how it fares against compaact?), and I can't see why Lame/FAAC should be so interesting that it would merit an inclusion of Lame in this test (but I'm open for arguments).

Besides, what's wrong with using Lame in many tests? It allows (to an admittedly small amount) inter-test comparability

Inlcuding Lame for inter-test comparability could be interesting but unless results are really clear-cut, you couldn't draw many safe conclusions. For example, if Lame finishes last in the AAC test you could claim with some amount of certainty that all the AAC encoders are better than all the mp3 encoders from the mp3 test. But if, say, Lame finishes last in both the AAC test and the multiformat test (excluding anchors, the latter is to be expected), you will only provoke senseless claims like "Vorbis finished 1.2 points ahead of Lame in the multiformat test, but FAAC only managed an advantage of 0.9 in the AAC test, so Vorbis must be better than FAAC".

this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

That's what the upcoming multiformat test is for.

I'm still against the inclusion of Lame. In my opinion, the disadvantage of increasing the difficulty level of the test outweighs any advantages. You have to be realistic about this. There were on average 16 participants in the mp3 test, and I'm convinced the number of participants depends very much on the difficulty, so if we add another serious contender to this test, we can probably expect even fewer participants (unless Roberto includes a bunch of problem samples). So, even if the results get a bit less interesting without Lame, in my opinion significance beats interestingness any day of the week.

[rjamorim]

EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

The first multiformat test already proved that. The best AAC implementation (QuickTime) was quite better than the best MP3 implementation (Lame)

[QuantumKnot]

I second the idea of using Xing (ver 1.5) as an anchor. It seems to be one of the most popular mp3 encoders used by consumers out there. It's kinda outdated, not tuned since 1999, doesn't use block switching, etc. So it's not going to compete with any of the AAC coders, yet represents the consumer standard

[kwanbis]

EDIT: this way, it would also help to confirm, or refute the idea of "AAC is way better than MP3".

The first multiformat test already proved that. The best AAC implementation (QuickTime) was quite better than the best MP3 implementation (Lame)

right ... i didn't remembered that test ...

[rjamorim]

Well, I see there is too much discussion about what codecs to support and what anchor to use. I think the best and most fair way to settle the issues is creating a poll.

A poll to select the anchor would be easy, but codec will prove complicated since there's no way to set IPB to allow users to choose 5 options.

So, I will use codec packs as options. Something like this:

1. - iTunes, Nero, Faac, Compaact and Winamp
2. - iTunes, Nero, Faac, Real and NCTU
...

Since there would be several different possibilities involved, I would like to see if everyone agrees on a list of codecs that MUST be present.

1 - iTunes: Because it was the winner of the first AAC test and it's hugely popular
2 - Nero: Because it's Nero (heh) and it's popular.
3 - Faac: Because it's the only open source and really multiplatform alternative, and because I believe it's quite popular as well.

From there, I would create different packs featuring Compaact and/or Winamp and/or NCTU and/or Real.

As an added bonus, I will get rid of accusations of favouring this or that codec.

Do you agree?

Regards;

Roberto.

This post has been edited by rjamorim: Feb 9 2004, 03:03

[paradynamic]

Why not use Fhg AAC encoder (Sorenson?) for low anchor? Would be interesting to see progress of recent aac development.

[quackquack]

I think Compaact is essential as right now it is the only VBR-able competition to Nero. I know it has undergone a number of quality improvements as well, and it would be interesting to see how it stacks up.

- Matt

[kwanbis]

only question is ... haw -APS would compare agains AAC?

[rjamorim]

Why not use Fhg AAC encoder (Sorenson?) for low anchor?  Would be interesting to see progress of recent aac development.

Because it might be better than other codecs.

I think Compaact is essential as right now it is the only VBR-able competition to Nero.

Ehm... FAAC?

only question is ... haw -APS would compare agains AAC?

Please, don't tell me you are suggesting I test different bitrates...

[ff123]

Probably the choice of anchor could be a separate poll in itself. But to reiterate a couple of points made by others in this thread, the anchor

1) should be significantly worse than any of the other codecs.

2) should sound bad enough to be readily identifiable. Yes, that will compress the ratings, but it will take a lot of the burden off the listener, who now need concentrate on one less codec. Also, it may encourage more people to participate, who might otherwise be scared off. I think lame and audioactive are too good, but the other mp3 codecs might work.

l3enc might work, but I would probably encode the samples with it to listen to how bad it is. It might be too good as well. In fact, it's probably a good idea to listen to the encodes of any prospective anchor.

Small listening pre-tests are a good idea, in my opinion.

ff123