AAC (Apple, Nero), Vorbis and MP3 (rock, metal), Personal listenting test at ~64/~80 kbit/s Options

[IgorC]

Comparisons:
1. HE-AAC: Apple vs Nero (64 kbps)
2. Apple LC-AAC vs Apple HE-AAC vs Vorbis (80 kbps)
3. LAME (130 kbps) vs others.

Encoders
1) Lame 3.98.2 -V5.9 (~130-135 kbps). Mainly as high anchor.
2) Aotuv b5.7[20090301] -q1.05 (~80-85 kbps)
3) itunes 9.0.0.70 (constrained VBR, 44100 Hz, stereo):
3a) LC-AAC 80 kbps
3b) HE-AAC 80 kbps
3c) HE-AAC 64 kbps
4) Nero 1.3.3.0 -q0.23 (~64 kbps)

[Shifting] Bitrates
This bitrate table based on 10 full albums (12 CDs)

Observations of bitrate table:
1. A general criteria was that the difference between bitrates should be less than 2%. An exception was Apple LC-AAC that exceeds +2.1% in bitrate comparing to Apple HE-AAC. iTunes hasn't -q values to shift bitrates. It's not big deal for me about +/-2%.
2. Aotuv was slightly shifted from q1.0 to q1.05. I know that it's very light shift comparing to the bitrate variation per each album but Aotuv -q1.0 had almost always (slightly) inferior bitrates comparing to Apple LC-AAC 80 kbps. -q1.05 fits well to be compared to both (Apple LC- and HE-AAC)
3. LAME was shifted from -V5 to -V5.9 to produce around 130 kbps on rock&metal albums. idem Nero.

Samples (original and encoded) with ABC/HR logs
.... will be uploaded soon.

Setup
Headphones Sennheiser HD 447. Soundcard Audigy SE 24/96.

Results

Importante observation:
1.Sometimes Apple HE-AAC at 64 kbps was ranked higher than the same encoder but at 80 kbps. I performed few times ABX tests and the results were the same. In past some people reported the similiar issues around Nero and Lame codecs.
2. Nero is working on its new encoder. So Nero encoder is here only as reference to show if Apple did any progress on low bitrate area.
3. I've also included a geometric mean score. I tend to beleive that the geometric mean is more realistic when it's come to judge the results when desviation between average score and some of particular results is something to be considered.

Ranking:
Average score. Geometric mean score is in the parentheses.

CODE

1. LAME 130 kbps    - 3.87 (3.80)
2. Apple LC-AAC 80  - 3.35 (3.26)
3. Vorbis 80        - 2.97 (2.84)
4. Apple HE-AAC 80  - 2.85 (2.79)
5. Apple HE-AAC 64  - 2.67 (2.61)
6. Nero HE-AAC 64   - 2.38 (2.30)

Conclusions
Before to do any conclusion it's worth to mention that it is personal test and on rock/metal music only (despite some classic music, progressive stuff, music from different cultures ...etc.. are included).

1. LAME at 130 kbps is still better than newer generation codecs at 64-80 kbits/s
2. Vorbis doesn't shine at 80 kbits/s but it's still on par with Apple HE-AAC and not that far from Apple LC-AAC. Vorbis presents very clear audible distortion at this bitrate. From my experience Aoyumi's Vorbis encoder is an excelent and well tuned for bitrates >=96 kbits/s.
3. Apple LC-AAC 80 kbits/s. Its performance is good for this kind of bitrates.
4. Apple HE-AAC 80 kbits/s. Many times I ranked HE-AAC 80 the same way or even worse than HE-AAC 64. Mainly because a SBR'es efficiency is limited at this bitrate. All test papers (mainly Coding Techs) indicate comparisons where SBR was design to bring the best advantage comparing to non SBR AAC at 48 kbits/s. There are no mentions about official direct 64 kbit/s comparisons. It's still good at 64 kbits/s but not that good for 80 kbits/s.
5. Apple HE-AAC 64 kbits/s. Overall it's slightly better than Nero.
6. Nero HE-AAC 64 kbits/s. I tested almost outdated version of Nero encoder. New encoder might be better.

Any thoughts and suggestions are welcomed.

This post has been edited by IgorC: Sep 14 2009, 05:39

[sauvage78]

Personnaly I would have liked to see Nero LC-AAC 80Kbps just for my own curiosity.
I am convinced that nero lc beats Aotuv b5.7 on the whole bitrates range, but I would have liked to see how it deals with Itunes which is a codec I never use & also see the opinion of someone else on nero lc vs. aotuv (afterall maybe you would prefer aotuv which would be a surprise to me).
Low bitrate is not my personnal area of interest, but it was nice to read anyway.
It seems to give some favor to Itunes LC AAC but without Nero LC AAC, I fell it's incomplete IMHO.

This post has been edited by sauvage78: Sep 14 2009, 06:39

[IgorC]

Mathematicians think that 2+2=4. Physicists are diasgree with them.
And engineers think that mathematicians and physicists are idiots (all of them).

This post has been edited by IgorC: Sep 14 2009, 09:11

[C.R.Helmrich]

Thanks for your work, IgorC! It's interesting to see that on average, you prefer AAC LC over HE-AAC even at 80kbps.

Chris

[muaddib]

Was there any problem with Nero that occurred often? Or was it many different problems among samples?

[IgorC]

There were two kinds of artifacts on Nero HE-AAC. One is constant almost for all samples and other is variable per sample.

1. The constant artifact was present on many samples. It's like Nero resamples to low frequency more than Apple's. It's very similar to 22khz resampling.

2. The other was exaggerated stereo loss that is most pronounceable on "Since Always" sample. Itunes doesn't present this problem.
http://ff123.net/samples/SinceAlways.flac

This sample is aslo easy to ABX at -q0.5 (Nero 1.3.3.0):

CODE

foo_abx 1.3.4 report
foobar2000 v0.9.6.9
2009/09/14 16:25:29

File A: C:\MULTI\15 Since Always Stereo Loss\SinceAlways.wav
File B: C:\MULTI\15 Since Always Stereo Loss\SinceAlways.mp4

16:25:29 : Test started.
16:25:40 : 01/01  50.0%
16:25:44 : 02/02  25.0%
16:25:47 : 03/03  12.5%
16:25:51 : 04/04  6.3%
16:25:55 : 05/05  3.1%
16:25:57 : Test finished.

----------
Total: 5/5 (3.1%)

[IgorC]

Personnaly I would have liked to see Nero LC-AAC 80Kbps just for my own curiosity.
I am convinced that nero lc beats Aotuv b5.7 on the whole bitrates range, but I would have liked to see how it deals with Itunes which is a codec I never use & also see the opinion of someone else on nero lc vs. aotuv (afterall maybe you would prefer aotuv which would be a surprise to me).
Low bitrate is not my personnal area of interest, but it was nice to read anyway.
It seems to give some favor to Itunes LC AAC but without Nero LC AAC, I fell it's incomplete IMHO.

The test is done. I didn't test Nero at 80 kbps mostly because new encoder should come soon.
I will perform test at bitrate >128 kbps (maybe around 160-180 kbps). It will be more difficult test and I will spend more time and put more care on it. I'm looking for encoder to be transparent on my collection.

Talking about Vorbis. I get opposite results to yours (even on sample from your test). I found a lot of samples where Nero -q0.5 doesn't performs enough well while Vorbis is transparent or near transparent. Samples like Fatboy, Emese, Since always and other.

Thanks for your work, IgorC! It's interesting to see that on average, you prefer AAC LC over HE-AAC even at 80kbps.

I felt if I did something wrong when I saw that final score for LC is better. But I found similar test performed by Guruboolez. He also prefered LC. That's good symptom. http://www.hydrogenaudio.org/forums/index....c=35438&hl=

This post has been edited by IgorC: Sep 15 2009, 04:12

[Zarggg]

Isn't comparing ~64kbps AAC to ~130kbps MP3 a bit like comparing V0 MP3 to lossless? I thought a better rule of thumb was ~100 AAC = ~130 MP3.

[sauvage78]

Zarggg:
... mp3 is the high anchor, it is a reference not a competitor.

[antman]

Great work. I enjoy reading your tests.

What I'd like to see is Coding Technologies' (Winamp) HE-AAC encoder thrown in there. It always gets left out of the AAC discussion.

[/mnt]

Awesome test!

Seems like LAME sill struggles greatly with the harpicods on Slayer's Spill The Blood. Or is it a different track with the same name?

This post has been edited by /mnt: Sep 16 2009, 01:25

[IgorC]

The samples of test with ABC/HR logs can be downloaded from here www.mediafire.com/?mmztlvzr4mi

Awesome test!

Seems like LAME sill struggles greatly with the harpicods on Slayer's Spill The Blood. Or is it a different track with the same name?

It's the same sample.

[IgorC]

Great work. I enjoy reading your tests.

What I'd like to see is Coding Technologies' (Winamp) HE-AAC encoder thrown in there. It always gets left out of the AAC discussion.

You're right. CT encoder was always the last AAC encoder to test. 6 codecs were tested and I performed only ABC/HR test ( didnt't full ABX test). I wouldn't stand for one more codec. Set of 6 codecs required a lot of time . There was no need in ABX test at these low bitrates. The artifacts are very audible and different per each codec to judge quality fairly.

I prefered to concetrate on difference of artifacts between different codecs instead of useless ABX lossless vs lossy. The concentration is lost when you waste a lof of time on ABX lossless vs lossy before you will start to compare differences between codecs.

However I will perform ABX (for lossless vs lossy. Also codec A vs codec B in doubtful cases) for high bitrate test and will spend much more time on it.

This post has been edited by IgorC: Sep 16 2009, 18:09

[Zarggg]

... mp3 is the high anchor, it is a reference not a competitor.

Oh duh, I missed that. Carry on.

Edit: Ah, I think seeing it listed in the Comparisons section threw me off.

This post has been edited by Zarggg: Sep 17 2009, 15:40