HE-AAC v.1 & v.2 comparison, Winamp vs Helix vs Nero Digital Options

[guruboolez]

The very new Winamp “surround edition” (5.1) and its own encoding solution for low-bitrate is a good occasion to make a comparison between different HE-AAC encoders. Preliminary comments posted on this board were clearly enthusiastic, especially for aacPlus v.2 quality. I’d like to contribute to the evaluation of this new encoder and as usual to post a complete report of my personal listening test here.

From now I will be brief.



Samples:

I used 2 series of samples I already used in the past:
• 40 samples mixing “classical” (25 samples) and “various music” (15 samples) (serie used here for all pools.
• 6 samples coming from DVD ripping (48000 Hz files transcoded from original AC3 files) (full serie already used here.

Encoders:

• Coding Technologies “aacPlus v.2” : 48 kbps with SBR and “Parametric Stereo”
• Coding Technologies “aacPlus v.2” : 64 kbps with SBR and “Stereo” (not parametric)
• Helix “Producer” v. 11.0.0.1897 HE-AAC: 64 kbps with SBR
• Nero Digital aacenc32 v.3.2.0.20 HE-AAC: 64 kbps with SBR

Software & Hardware setting:

• Creative Audigy 2
• ABC/HR for Java 0.5 alpha 5
• faad 2.1 MPEG-4 AAC decoder

Important to note:

• the test consists on pure ABCHR evaluation
• offsets were removed and gain was always corrected (difference could reach 1.5 dB!)



OVERALL RESULTS

CODE

WINAMP WINAMP HELIX NERO
48 kbps 64 kbps 64 kbps 64 kbps

A02 3.3 3.5 3.5 2.5
E06 3.0 4.0 4.0 1.8
E15 3.0 5.0 4.8 4.3
E22 3.5 4.0 4.0 3.0
E26 2.7 3.0 3.0 2.0
E31 3.0 3.7 3.7 1.0
E40 3.0 3.5 3.5 2.5
E51 2.5 3.5 3.5 2.5
E53 1.5 2.5 2.5 1.8
S03 4.0 4.5 4.5 3.5
S08 2.5 3.5 3.5 1.5
S12 1.5 1.8 1.8 1.0
S17 3.0 2.5 2.5 1.0
S27 1.2 2.0 2.0 1.7
S38 3.0 3.5 3.5 1.5
S50 2.0 3.5 3.5 1.0
S54 2.5 3.0 3.0 2.0
V02 3.8 3.8 3.8 4.0
V07 3.9 3.7 3.7 3.0
V10 3.5 3.5 3.5 3.5
V15 2.0 3.0 3.0 2.0
V19 3.7 4.2 4.2 4.5
V20 3.0 3.5 3.5 1.0
V24 1.5 4.0 4.5 1.5
V27 3.5 2.0 2.0 1.0
---------------------------------------------
CLASSIC (MEAN) 2.80 3.39 3.40 2.20
---------------------------------------------

41_30sec 4.0 2.5 2.5 1.5
ATrain 3.0 3.5 3.5 1.5
DaFunk 3.5 4.2 4.0 1.5
death2 2.5 2.5 2.5 1.5
EnolaGay 2.3 3.0 3.0 1.0
experiencia 3.5 4.0 4.0 1.5
getiton 3.5 4.3 4.3 2.0
kraftwerk 2.0 1.8 2.5 1.0
LifeShatters 2.8 3.0 3.0 2.0
NewYorkCity 3.0 3.5 3.5 2.5
OrdinaryWorld 4.0 2.0 2.0 1.0
Quizas 3.5 4.0 4.0 1.5
rosemary 3.0 4.0 4.0 1.5
SinceAlways 4.0 2.5 2.5 1.0
trust 1.0 2.0 2.0 1.5
---------------------------------------------
VARIOUS (MEAN) 3.04 3.12 3.15 1.50
---------------------------------------------
Alien 4 3.5 4.0 4.0 1.5
Farinelli 4.5 4.8 4.8 2.0
Come Drink... 5.0 5.0 5.0 5.0
Ran 3.5 4.0 4.0 3.7
BlackAdder 3.0 3.2 3.2 1.0
Pulp Fiction 3.5 3.3 3.3 2.7
---------------------------------------------
DVD (MEAN) 3.83 4.05 4.05 2.65
---------------------------------------------

=============================================
46 SAMPLES 2.89 3.29 3.31 1.94
=============================================

ABCHR logs are available here; I stopped to comment the notation after ~20 tests.



• Helix & Winamp 64 kbps are highly similar. It might be possible to differentiate them with an ultra-careful listening, but my notation is only based on an attentive but fast evaluation. I suppose that Helix HE-AAC encoder is also coming from Coding Technologies. Apart quality, both have the same offset and a very similar gain (volume) deviation.

• 48 kbps HE-AAC with Parametric Stereo is inferior but not very far from 64 kbps HE-AAC with “regular” stereo coding (I guess it’s joint-stereo). It’s a very good performance I would say. At such low bitrate every bit is precious and by removing 16 kbps to a 64 kbps we could expect a huge drop in quality. But here, the quality stays close from the 64 kbps encodings (and is sometimes clearly better – see below).
I quickly noticed unusual issues with PS encodings. I can distinguish three different variants:
- weird and high-pitched noise
- stereo distortion: ping-pong effect (rare) and fake-stereo effect (reverberation, echo….)
- “equalizing issue” (unbalanced sound, often hollowed, lack of bass, flattened image, etc…), which is the most common problem I noticed.

• Nero Digital AAC was the first publicly available HE-AAC encoder (released in summer 2003), but the latest version is now obviously outdated and not competitive anymore. On 46 samples this encoder finished last 36 times and first 2 times only! The quality is often really poor compared to all other contenders, including the 48 kbps encodings made with Coding Technologies/Winamp encoder.
Nero Digital usually suffers a lot from what I’m used to call “SBR artefacts”: a grainy sound, smeared impulse and also a lot of small but highly irritating noise packet/aggregation. I heard these issues for the first time with mp3PRO and found them again with the first (and only tested) HE-AAC (Nero Digital). Problems are audible even at the highest available bitrate: 96 kbps and also with forced 128 kbps. That’s why I always thought that all these issues were consubstantial to SBR technology. Now, by testing other HE-AAC implementation, I must completely change my mind. What I called “SBR defect” is rather a specific Nero Digital flaw. I don’t mean that Coding Technologies & Helix encoders are free of these problems: they’re all more or less perceptible (it really depends on the sample). In other words, there are still “SBR artefacts”. But both encoders have apparently a better control of the side effect of SBR. Pre-echo on strong impulse is also much stronger with Nero Digital (see Appendix) when compared to Helix/CT.
I’m now thinking to my 80 kbps listening test I made on the beginning of the summer. I tested 185 samples with a HE-AAC implementation which appeared to be far from Vorbis aoTuV quality, and surprisingly similar to Apple LC-AAC and not that better than Lame MP3. I’m now pretty sure that Coding Technologies “aacPlus” would perform much better and could maybe compete with aoTuV at 80 kbps (for 64 & 48 kbps, I fear that it would be hard for aoTuV to stay competitive against the SBR beast)… I read somewhere that Nero 7 is announced for October. I guess that the new generation of Nero Digital HE-AAC will be released with the new Nero package and I’m sure that Ivan had built a competitive encoder.




DETAILED RESULTS




• classical group : parametric stereo doesn’t work very well with several samples tested here. The 64 kbps are clearly more enjoying (but are of course bigger). Nero Digital didn’t perform very well, but at least it’s not so far from 48 kbps PS encodings. The Winamp/Helix duo gets a better note with “classical” samples than with “various music” ones. This is another surprise for me: I really expected from SBR to work (much) better with louder music.

• various music group : Parametric Stereo is working very well here, and at 48 kbps the results are very close to 64 kbps encodings without PS. But a closer analysis reveals that these good results are contrasted. Most often 48-PS encodings are worse than 64-JS; not too worse but worse. But with three samples 48-PS encodings are better than 64-JS: not only better but much better (to my ears – see Appendix for illustration). The incriminated samples are 41-30, OrdinaryWorld & SinceAlways. What happens? The Parametric Stereo encodings are free of nasty artefacts audible with 64-JS, located on cymbals mostly, and which consists on aggressive and sandy noise (except for 41-30 which has another problem).
Second surprise: Nero Digital results, worse again compared to competitors, but also worse with “various music”. Nero Digital was used to have problems with classical music; here it’s the opposite. Most often encodings produced by latest Nero Digital encoder are simply unlistenable. These unbalanced results in favour of “classical” samples are confirming the results I got after the 80 kbps LT I made this summer.

• DVD transcoding : there are too few samples to make any strong conclusions. First comment: notations are higher (for all four encoders) with this group of sample. This could be partially explained by the presence of one mono-encoding which sounded transparent with all encoders (which obtained as consequence 5 points for this sample). Helix & Winamp quality at 64 kbps are really excellent. I’m not fond of DVD ripping but I think I will consider HE-AAC again (I was very disappointed by my previous tests, all made with Nero…). Nero Digital performs less badly than with music encoding, but is still far from all other competitors, including the 48 kbps “aacPlus” encoder. It confirms my previous experience with HE-AAC and DVD ripping: poor. The usual artefacts of Nero are also audible, altering voice as well as music.



APPENDIX

• three different statistical analysis of the complete results are available here:
http://audiotests.free.fr/tests/2005.09...ots_mixedx3.png


• Illustration of Nero Digital ultra-smearing issues with Kraftwerk.wav sample:

http://audiotests.free.fr/tests/2005.09...1/kraftwerk.gif
http://audiotests.free.fr/tests/2005.09...ftwerk_Helix.wv
http://audiotests.free.fr/tests/2005.09..._NeroDigital.wv


• Illustration of Nero Digital grainy sound with S50.wav sample:

http://audiotests.free.fr/tests/2005.09/AACHE/01/S50.gif
http://audiotests.free.fr/tests/2005.09...01/S50_Helix.wv
http://audiotests.free.fr/tests/2005.09..._NeroDigital.wv

• Illustration of Nero Digital grainy sound (other samples):

http://audiotests.free.fr/tests/2005.09...01/E06_Helix.wv
http://audiotests.free.fr/tests/2005.09..._NeroDigital.wv
http://audiotests.free.fr/tests/2005.09...semary_Helix.wv
http://audiotests.free.fr/tests/2005.09..._NeroDigital.wv


• Illustration of Parametric Stereo distortions (three samples):

http://audiotests.free.fr/tests/2005.09...ereo_issues.zip
Try also rosemary.wav (beginning)


• Illustration of SBR “sand” reduction with PS encodings:

http://audiotests.free.fr/tests/2005.09...ld_Winamp_48.wv
http://audiotests.free.fr/tests/2005.09...ld_Winamp_64.wv
http://audiotests.free.fr/tests/2005.09...ys_Winamp_48.wv
http://audiotests.free.fr/tests/2005.09...ys_Winamp_64.wv
http://audiotests.free.fr/tests/2005.09...27_Winamp_48.wv
http://audiotests.free.fr/tests/2005.09...27_Winamp_48.wv


• Illustration of a minor artefact audible with Winamp and not with Helix:


http://audiotests.free.fr/tests/2005.09..._blip_Winamp.wv
http://audiotests.free.fr/tests/2005.09...k_blip_Helix.wv


P.S. All uploaded samples are already decoded (in order to any avoid biased evaluation due to possible playback with non-SBR compatible AAC decoders). They’re consequently very short – maybe too short to be convincing. Try if possible to download the full samples and to encode them with the required settings.

This post has been edited by guruboolez: Dec 29 2005, 22:40

[kl33per]

Again, wow guru. I don't know how you find time to do this, but thank you.

[guruboolez]

Again, wow guru.  I don't know how you find time to do this, but thank you.

That's simple: a sleepless night before joining my workplace

[dand]

Guru, great work again!
It's strange how you missed HE-AAC encoder from CodingTechnologies in your 80 kbps test. Since CT standardized SBR as technology, it was obvious that it will take time until someone else develops a HE encoder close in quality to CT's. Even Nero, who started first a couple of years ago, didn't get there yet. Also, as SBR is pretty complicated, there won't be too many new HE implementations, at least I don't expect much in the near future, companies will just licence it from CT (as is the case now).

[guruboolez]

It's strange how you missed HE-AAC encoder from CodingTechnologies in your 80 kbps test. Since CT standardized SBR as technology, it was obvious that it will take time until someone else develops a HE encoder close in quality to CT's.

Was it really obvious that CT implementation should be considered as better than Nero? On this board, I can't remember anyone making such test or even such assumption. Remember Sebastian Mares and his project to organize a collective listening test at 64 kbps: he considered Nero Digital and the upcoming Apple's implementation as the two most interesting ones. Most people on this board (including me) were convinced that Nero Digital is a better encoding solution (compared to CT).
That's why I didn't consider Coding Technologie for my test (I regret it, believe me...).

[optimus]

Very nice article! I really learned something!

[Ivan Dimkovic]

I just want to inform you all that the new HE-AAC encoder from Nero is being finalized as we speak (current version uses very outdated SBR tools) - and it will be a huge (and I mean huge improvement (as already found out by internal tests) over the current version that Guru tested.

So please stay tuned

This post has been edited by Ivan Dimkovic: Sep 5 2005, 10:47

[bond]

wow very interesting! thx a lot guru!

seems we need to get a closer eye on the afaik freely available helix aac encoder!

also its great to see how close 48kbps ps-aac comes close to 64 he-aac with ac3 sources!

[Garf]

I am pretty sure Helix AAC is just the same as aacPlus (CT), and the results show this as well.

[Gecko]

I am pretty sure Helix AAC is just the same as aacPlus (CT), and the results show this as well.

Apart quality, both have the same offset and a very similar gain (volume) deviation

Judging from that last bit, they are not identical. I'd guess different versions.

Thanks for the test, guruboolez!

[IgorC]

This test was very informative for me.
were AC3 samples 48 khz or 44.1 khz?

And what about db HE-AACv2. It has some issue with 44.1 khz but at least it works fine with AC3 48 khz.

I also noticed that on some samples parametric stereo of Winamp 5.1 and db HE-AAC2 sound worth than simple stereo. Maybe that's why Nero is delaying their HE-AAC2 due to unstability of parametric strereo.

Test was quite short. But anyway thank you

This post has been edited by IgorC: Sep 5 2005, 14:41

[guruboolez]

were AC3 samples 48 khz or 44.1 khz?

48000 KHz.

[Garf]

Maybe that's why Nero is delaying their HE-AAC2 due to unstability of parametric strereo.

You can tune PS and SBR for ages. Actually, just LC AAC is probably far from it's end yet, either.

But yes, it's tempting to just keep tuning... :-P

[bond]

it seems coding technologies released the sourcecode for their ps-aac encoder as opensource for the 3gpp reference encoder here:
http://www.3gpp.org/ftp/Specs/html-info/26410.htm

now my questions:
- is it sure that the 3gpp reference is from coding technologies?
- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

would be happy if someone knows an answer to this

[dand]

- is it sure that the 3gpp reference is from coding technologies?

Definitely yes.

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

Probably with minor differences.

[Garf]

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

Probably with minor differences.

Some experimenting will quickly tell they're "not so minor" IMHO...

[bond]

well i also read it here and there, but any proof that they are (not) the same or that its from coding technologies?

apart from that: the helix he-aac codec is indeed from coding technologies as has been stated by karl_lillevold here:
http://forum.doom9.org/showthread.php?t=68245

[dand]

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

Probably with minor differences.

Some experimenting will quickly tell they're "not so minor" IMHO...

Maybe... Where do I get their commercial version?

[Garf]

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

Probably with minor differences.

Some experimenting will quickly tell they're "not so minor" IMHO...

Maybe... Where do I get their commercial version?

I am really the *worst possible* person to ask that. You should really buy Nero instead

[slippyC]

Guru, thanks for tests.

[Oki]

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

Probably with minor differences.

Some experimenting will quickly tell they're "not so minor" IMHO...

Maybe... Where do I get their commercial version?

Sorenson Squeeze 4.1 is using the commercial CT's HE-AAC encoder. I do not know if it is the HE-AAC v2 or just v1 encoder.

[bond]

as mentioned by guruboolez the coding technologies encoder is available in winamp5

[shrinkmail]

Thank you, Guru. For the last three days i have beeing going through your tests, and i have found them all edifying.
Qs: So at the current state of development, what would be the use of HE-AAC v2 for an audiophile, and if so what bitrate would be transparent? Again, one can't possibly use Winamp for secure rips...

[nyaochi]

Your articles are always something exciting for me. Thanks a lot, guruboolez.

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

As far as I tried, the source code does not support 44.1kHz input. We can change the parameters to support 44.1kHz though (see this thread). PS will be activated when the target bitrate is 32kbps or lower. 44.1kHz 48kbps SBR (w/o PS) that I tried with this source code was actually good.

I found many parameters in the source code waiting for someone to tune...

[bond]

- is the 3gpp reference the same or a different encoder than the one coding technologies sells?

As far as I tried, the source code does not support 44.1kHz input. We can change the parameters to support 44.1kHz though (see this thread). PS will be activated when the target bitrate is 32kbps or lower. 44.1kHz 48kbps SBR (w/o PS) that I tried with this source code was actually good.

I found many parameters in the source code waiting for someone to tune...

thx for the info!

is there any technical reason for this exclusion of 44.1?