The excellent performances of newer Nero AAC encoder (2.9.998 “fast”) lead me to compare it with other strong competitors. Unfortunately, there are not so many potential challengers.

• lame MP3: I’m using it every day, and though I like his new VBR mode (-V 5), I’m sure it’s not strong enough with my music to fully compete with the new AAC VBR encoder.

• wma9 PRO: I’m fond of this encoder, working very well with classical music. I don’t know how perform the newest 9.1 encoder. Unfortunately, two years after the first release of WMA9PRO, there’s still apparently no dedicated portable player supporting it (except maybe telephone or PDA). The industry is still manufacturing electronic components and players for standard WMA, which is not a very competitive format.

• ogg vorbis: since Aoyumi works, vorbis improved a lot (last year, with classical music, quality was behind lame ABR for my taste). aoTuV encoders (and now official 1.1) have drastically reduced the hiss/coarseness problems, which annoys me a lot with classical music (maybe other genres are affected too, I can’t really say). Unfortunately, the problem is only reduced, and not fully corrected. Nevertheless, I consider vorbis as one of the most robust format. Latest aoTuV beta 3, announcing an even lower coarseness level and a better pre-echo performance, could be a potential champion. I’d personally consider it a priori as the most dangerous challenger for strong AAC encoders.



I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself.

Therefore, I’ve restricted the comparison to vorbis and Nero AAC. As encoder, I had some choice (official 1.1, megamix, aoTuV beta 2/3). I’ve privileged aoTuV encoder, because Aoyumi’s works are mainly focused on the most important vorbis problem (coarseness). I hesitated a moment between beta 2 and beta 3. I’ve finally opted for beta 3 without preliminary investigations: this test is a good occasion to see how reliable the new beta is, and if potential problems will handicap the format during this test, they will at last be detected, and probably quickly corrected.
I don’t know which vorbis setting would be the fairest to compete with Nero –internet. –q 4,00 is the roundest, and probably one of the most popular. In order to prevent (or at least to limit) some excessive difference, I’ve used –q 4,25. With my computer, I can’t obtain quickly a full and reliable bitrate table. But the comparison based on the tested samples is very satisfying:
- Nero encodings: 136,2 kbps
- aoTuV –q 4,25 encodings: 136,5 kbps
…hard to find closest settings :·)


The testing conditions (hardware, software, methodology…) are the same as those described in the global AAC comparison. Samples are also the same. One difference: I didn’t searched to ABX the difference between the encoded files and the reference ones (already done with all AAC encodings in previous test), but focused my efforts to a direct comparison between the two encoded files. Again, I’ve fixed a dual limit to the number of trials: eight if easy, sixteen trials if less easy. I have respected these limits, without any exception. First second of each sample was systematically discarded from listening.
Now, results:



vorbis aoTuV beta 3: Nice quality, but still coarseness issue. It’s fortunately limited, and with some samples, I simply can’t detect it (problem was systematically audible with older vorbis encoder). Very limited pre-echo problems: Aoyumi has apparently solved most issues in this area. However there is a deep ringing artefact with the organ (Bruhns) sample. I’ve in addition noticed a very strange problem, occurring with solo violin on two different samples (Avison first, and then Mozart-Sonata): it sounded like a noise-reduction processed recording, but additional hiss was audible on the same time. This paradox also happened to the old experimental Quantum Knot’s encoder (v. 3.5 aka vorbis “classical edition”). I don’t know: maybe Aoyumi did similar tuning to aoTuV beta 3? I had to investigate further…

compared to Nero AAC: better with harpsichord (note that aoTuV allocates even more bits for this instrument than Nero AAC!) on Couperin and Bach (excellent encoding) samples. I’m badly surprised by poor performances on Bayle sample: I’d expect from a VBR encoder to detect the difficulty (micro-attack). With aoTuV, both bitrate and quality are low… Noise/hiss/coarseness are like a signature with most samples, and often betray vorbis presence.


STATISTICS

You could try with the following table:


Conclusions are the same with ANOVA and Tukey parametric:

« NERO is better than AOTUV ».


• ABX logs are available here
• SAMPLES (11 MB) are available here. Keep them: I can’t keep them very long time online.

In addition to this test, I’ve compared the performance of aoTuV beta3 to the previous beta 2 on two problem samples: Avison and Mozart-Sonata. To complete the investigation, I’ve also add QK 3.5 “classical edition” at –q 3,4 (this encoder need to lower the –q value for matching bitrate values).

Here are the results:





Conclusion: aoTuV beta3 introduces similar artefacts on these two samples when compared to beta 2. The experimental “classical edition” have similar issue, but at higher level. Of course, it doesn’t necessary mean that beta 2 would be a better competitor than beta 3 against Nero AAC, but just that recent changes to aoTuV encoder have some negative consequences. I’ve already experimented regression of coarseness with beta 3 compared to beta 2.

The fast mode uses so called Scalefactor estimation mode, which hopefully will become the default very high quality mode after it has been refined.
After SFE and some other tweaks are working at full power, well, the LC-core should be extremely good, that is our goal..

Juha Laaksonheimo alias JohnV
Ahead Engineer

Congratulations, JohnV, HA seems to have been a very fertile recruiting ground for Ahead!!!

Again, thanks for the insightful tests, guruboolez. They give Vorbis developers the perfect 'reality check' on their work.

It seems classical music continues to be a weak point for Vorbis. *sigh*

Just as a sidenote, I found some free-time this week playing with some wavelet-based transient analysis (for block switching and dynamic impulse adjustment) for Vorbis 1.1. Nothing concrete to release yet, but the results look quite interesting. Expect an alpha of QKTune 2 sometime in the new year.

@guruboolez,
Thank you for a test. I checked the clear problem with some samples.
I already recognize the cause. And I will release an experimental version in the near future, in order to solve or reduce this problem.

Care to eleborate? ;-D

"I didn’t include wma9PRO in the following test. I don’t know that much this encoder: Is there one VBR profile close to ~130 kbps? Which one? Is it reliable, or is CBR/VBR2-pass a better solution? Answers are needed before including this format in a comparison test, and I don’t have time enough to answer them by myself."

1P CBR, 2P CBR, and 2P VBR have 128kbps modes. For this test, you should use either 2P CBR or 2P VBR.

Very interesting test. Thanks Guru.

Now we need a test comparing all musical styles. Who will be brave enough to stand up and conduce such test? (anyone who answers rjamorim will get socked in the eyes by yours truly )

Roberto Amorim should do it. (I did NOT say "rjamorim"! )

Nah, joking... I would love to, but I don't have that much experience.

Well, unrefined as it may be, it seems its quality is already higher than high.

There are still issues which Guru's test don't reveal, that's why it's not default yet.

Still early times but the results are interesting. That's all I'll say for now. Note that I am not working on a quasi-Vorbis II. It is 100% pure Vorbis I.

btw. who else at HA has the experience (and time) to hold a listening test?

great stuff, neros aac encoder really needed this

Nero's AAC did really really well here. This was also a good test for aoTuV b3 as it has for the most part gone untested so far (but it is reall really new of course). I'm actually really impressed with Nero's AAC here.

I've just discovered the great performance of newer Nero AAC. I've encoded some additionnal tracks in order to perceive additional issues. Is it worth to send samples (to Garf, Ivan, Menno or you)? Or is it better (for you) to wait for the final release?

Hello,

First of all, guruboolez, many thanks for your tests - as always they are extremely useful to all of us developing codecs (please send any clips that you think that might be useful to us)

Also, I would like to clear out some unknowns about the "fast" mode -

"Fast" mode is using direct estimation of the scale-factor values - and it turned out that this method gives better results than traditional "two pass iteration loop" algorithm found in older generation of encoders.

This mode is still "beta" as many improvements are currently being done to both LC and SBR algorithms that should, we hope, result in significant improvement in quality of our codec.

I am also sorry for delay of the planned "V3.0" release of Nero AAC encoder - as we are all extremenly busy with the Nero Digital development - I can't say much now, but we are planning couple of big surprises that will be announced soon

Once again, thank you all for the big support!

Hello Ivan,
first: BRAAAVOOO! You (and other people) did an impressing job with Nero AAC. Honestly, I didn't think that such progress could be possible in less than one year. And thanks for the explanation.

second: here is a sample, probably the worse I heard for Nero VBR 'fast' internet profile with regular music. The artifact is very common (noise is missing and fluctuant), and occurs mainly on the second part of the sample. The problem is audible on regular listening volume. I've found some samples showing this behaviour, but this one is the worse I've heard.

P.S. There's also (small but real) problems with pre-echo. But you probably know this.

If you need some help for tests or opinions about it, count on me (if I'm not too busy).

Many thanks.

I made some ABX test with guruboolez's samples compressed ~128 bitrate and...
for my ears NERO AAC not better than Aotuv beta3 аnd can even worse??
tested on Avison, Mozart - sonate and Bruhns - prelude.

Avison AAC 8/10
Avison Ao Beta3 7/10

Mozart AAC 7/10
Mozart Ao Beta3 7/10

Bruhns AAС 6/10
Bruhns Ao Beta3 7/10

I thank guruboolez for interesting sample.

Strange results for Bruhns.wav: it's one of the most wounded IMO by aoTuV.
Anyway, thanks for additionnal tests

sorry, typing error
I have corrected

I've just noticed:
results below 8/10 are not significant.
8/10 -> pval = 0.055
7/10 -< pval = 0.172

http://ff123.net/abx/abx.php

Yeah, I would only choose 8 trials if you're reasonably sure that you'll be able to detect a difference.

ff123

Ivan, in your opinion - will the improvements coming in 3.0 be of such magnitude that you would recommend waiting on re-encoding any MP3 LAME 3.96.1 PS files to Nero AAC Normal (Fast)?

It seems like with Nero's latest codec release, there is finally reason to switch from a MP3 3.96.1 -V 'X' or iTunes AAC to a Nero VBR setting. Good stuff!

Ahead AAC at internet quality VBR seems to have a lowpass filter of 16 kHz. It may prove to a better comparison to lowpass Vorbis at the same cutoff, and increase the q-level to reach the same average bitrate that was achieved in the test (ie q-level of about 4.75).

It is always good advice, of course, to steer clear of changing the default lowpass value for any given q level for vorbis but, for a test of this nature, I think it would make for a fairer test. Vorbis at q-level 4.25 seems to have a lowpass around 19 kHz.

3.0 should be a significant advancement over the 2.x series, yes.

I'd currently disrecommend encoding files with fast mode despite good results on some tests - it *does* still have known bugs. 2.x non-fast may not be always the best but it's well tested and should have no very bad failure cases.

You should request it to vorbis developers themselves.
From my experience, lowering lowpass doesn't change anything to vorbis quality: bitrate doesn't seriously drop, other problems like hiss/coarseness are still there, and this lowpass seems to be more damaging than positive (on harpsichord for exemple, I always find 16Khz a bit annoying, and such lowpass will lower the very good notation obtained by vorbis with this instrument).

IMHO, OggEnc aoTuV b3 - best choise for home music collection.

I agree completely. Even at low bitrates, I much prefer Vorbis' output with the lowpass raised.

What do you base this on? Are you somehow trying to invalidate the findings of guru's test simply by saying that you think aoTuV b3 is the best? Why would you do that?

music_man_mpc,
I based on more things:
1. Supporting with portable players.
2. High quality on low and normal bitrates
3. Open Source, many tunes (now actual Floggy, aoTuV, Archer)
4. Ogg Vorbis is the most popular after MP3 and WMA
and many other things.

For guru's best choice is MPC, only after 224kbps.

And AAC:

1/ is supported on the most popular portable player
2/ (much) better quality on low bitrate
3/ much better multichannel efficiency
4/ audio standard, supported by hundred industrials (MPEG-4) and some serious industrial projects (DVD audio)
5/ no buggy hardware players (iRiver's usual bitrate limitation... for VBR encoding: very useful, thanks)
6/ open-source projects
7/ much more development (faster progress)


Anyway, I've created this topic for sharing my experience about Nero and latest aoTuV beta 3 QUALITY. Feel free to post your results, they're welcome. But honestly, your basic opinions about open-source superiority are totally useless.

I'd personally like to believe in the superiority of open source, but guruboolez does indeed have very valid points. AAC is so good at low, low, bitrates that it just plain *excites* me!

Ogg is certainly superior to MP3 in most ways, except for universality of support. But to me, at least, it seems that AAC support is even more lacking in most software and hardware. Hopefully this situation will change, and also, hopefully more free encoders and SDKs will become available for it. Only then will it truly be better than Ogg Vorbis, in my books. (And hopefully lose the image of its seemingly inextricable association with Apple)

Is it likely that these things will happen in the next year or two, for those in the know?

There are 'free' de/encoders and then there are free de/encoders. Last I looked into the licensing issues, AAC was covered in patents up the wazoo. There may be limited patent grants for certain classes of free de/encoders but it's pretty stiff for commercial vendors. I think this is quite a drag for widespread support.

8/ is heavily patented

Anyway, to all: please stay on technical consideration, not political.

I thought it was patented, but certain things people have said has made it seem like a moot point, and that there is open development of AAC going on.

Does the patenting of AAC result in the same issues of licensing and usage flexibility that Fraunhoffer's MP3 had/has? This could be a problem. But in any case, it doesn't seem to have been a big preventative agent for MP3 catching on in a big way.

@PatchWorks: Sorry... I guess this is political more than technical, but if it in any way inhibits development and/or adoption of the codec, then that would have a technical impact on AAC, I'd think.

Yes, very similar.



MP3 caught on in a big way BEFORE Fraunhoffer started kicking up a licensing fuss (sceptically, that was a shrewd business move).

But it's certainly NOT of irrelevance when comparing codecs to also compare the licensing of those codecs. It's vital. A codec can be the best thing in the world, but if the cost (in money or freedom - I'm not a free software zealot but I think that license education is incredibly important) of using it in a certain context is prohibitive then its worth in the comparison immediately becomes nil for those classes of uses.

n.b. If interested, do a google search for 'via aac patent pool'. You'll find a link to the licensing costs (for both encoding and decoding) of HE AAC. I think that's enough reading material for people interested in practical deployment, and most probably my last comment on such on this thread.

http://pessoal.onda.com.br/rjamorim/Dolby.txt

Damn... that's not so good

Given all of that, I'm surprised how much activity there is here in regards to AAC development. I'm guessing that is because many of the key people here get paid for it. Not that it's a bad codec, but if something like Ogg Vorbis is capable of achieving similar capabilities, why is there so little progress made in it? No money in free software?

??? So how is it that you are still carrying AAC encoders and decoders at RareWares?

Back then, that e-mail was only FUD, since the files were hosted in Brazil and we don't recognize foreign software patents here.

Later Dolby stopped licensing AAC in favour of their VIA Licensing subsidiary, and the guys at VIA seem much nicer, as they never sent me threats. That's why I moved the files to the US RareWares server.

If I ever get threats again, I'll just move the files back to Brazil.