Nero vs Apple AAC encoders at ~90...95 kbits, Personal Listening Test Options

[IgorC]

Encoders:
LAME 3.98.2 -V 5
Nero 1.1.34 -q 0.325
Nero 1.3.3 -q 0.31
Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings):
Actually Apple AAC with true VBR at 96 kbit/s produces ~90-95 kbit/s.
http://img254.imageshack.us/my.php?image=qt1dq3.png
http://img254.imageshack.us/my.php?image=qt2kl8.png

Bitrate table
This bitrate table based on 8 full albums (mostly pop, rock, heavy metal and alternative)


Samples (original and encoded) with ABC/HR logs
If somebody want to see it.
http://www.mediafire.com/?x1d8vzzkfbw

Setup
Headphones Sennheiser HD 447. Soundcard Audigy SE 24/96.

Results

Apple AAC - 4.70
LAME 3.98 V5 - 4.50
Nero 1.1.34 - 4.16
Nero 1.3.3 - 4.03

Any comments are welcomed.

This post has been edited by Canar: Nov 2 2008, 02:00

[Canar]

I cleaned up the silly imageshack ad-links or whatever from under your results. Thanks for sharing your hard work. Would you consider running an additional test with LAME at 90-95 kbps, just for comparison? "Thundersuck"?! HAHAHA! Awesome. I feel that way about that song too.

This post has been edited by Canar: Nov 2 2008, 02:04

[IgorC]

I tried LAME at 96 abr before. http://www.hydrogenaudio.org/forums/index....c=54967&hl=
Quality drops drastically to 3.2 average score.

[muaddib]

10 kbps (~10%) difference can make big quality difference.

[M]

10 kbps (~10%) difference can make big quality difference.

Absolutely, and based on the bitrate difference alone Apple's performance isn't much of a surprise.

What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

- M.

Edit: Fixed a typo.

This post has been edited by M: Nov 3 2008, 16:00

[halb27]

10 kbps (~10%) difference can make big quality difference.

This just means that QT AAC was so wise to increase bitrate as necessary - at least better than Nero did in these cases.
What counts is the bitrate table IgorC gave in the beginning where he tried to give all contenders roughly the same chance. Here QT AAC even had a bitrate roughly 5% below Nero. So the procedure IMO is okay in principle though can be improved because 8 full albums of a similar genre are representative only for people who mainly listen to this kind of music.

[muaddib]

What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

We always tune our encoder for the "average" listener. It might happen that IgorC prefers 1.1.34.2 over 1.3.3.0. It might also happen that it is like that only on this samples set (we tune on much larger set of samples). And difference of 0.13 is not statistically significant.

Anyway it is nice to see that people are testing new encoder and everybody is welcome to do that

[guruboolez]

Interesting tests, thanks to IgorC.
I second halb27 comment: Apple's AAC doesn't benefit from any bonus bitrate ; Nero AAC encoders are both using 5% more bits than its contender.
The pertinent bitrate table is the first (album-based) one ; the second table (sample-based) is also important: it tells us that the choice of sample isn't fully representative of the full albums… which is usual in listening tests.

@muaddib « We always tune our encoder for the "average" listener. »
I'm pretty sure that every audio encoder are tuned that way

This post has been edited by guruboolez: Nov 3 2008, 16:39

[muaddib]

My apologies because at my first post I overlooked the first (bitrate) table.

[IgorC]

We always tune our encoder for the "average" listener. It might happen that IgorC prefers 1.1.34.2 over 1.3.3.0. It might also happen that it is like that only on this samples set (we tune on much larger set of samples). And difference of 0.13 is not statistically significant.

Anyway it is nice to see that people are testing new encoder and everybody is welcome to do that

I think talking about "average" in this case is totally wrong. (very carefully speaking).
70% of these samples are here already for years for ABX tests!
25% are also available on HA since enough time . And only 1 sample (AC/DC Thunderstruck) is new.
They are not average? They are most used in every test. You could simply tuned your codec on them if you wanted to cheat. Add to it that Nero used 4% of extra bitrate. But Nero wasn't even close to contender.
The bitrate table is also based on some very popular (if you want "average") CDs.

Speaking about my "averageness":
I don't pretend to be an average Joe but from previous experience of public tests my preference is very close to public results. For example you can compare my particular results with public results. They are too close. http://www.listening-tests.info/mf-64-1/mi...aneous/results/ .

This post has been edited by IgorC: Nov 4 2008, 02:15

[Tyashki]

Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings):
Actually Apple AAC with true VBR at 96 kbit/s produces ~90-95 kbit/s.

Wait a minute. Every time I have used Apple AAC True VBR it has prompted me for a "quality setting" from 0 to 127, nothing about kbits.
So what settings DID you use exactly?

[halb27]

70% of these samples are here already for years for ABX tests!

Yes, and that's what impresses me very much which your test. It made me download iTunes8 last night and listen for myself (with 'my' samples and at a higher bitrate), and the results are real good.

BTW - because I couldn't see a special quality option with iTunes, I just chose the bitrate and VBR (nothing like 'true' VBR) - did you use iTunes or Quicktime (possible with the free download?) for AAC conversion?

This post has been edited by halb27: Nov 4 2008, 09:15

[IgorC]

I used Apple QuickTime Player Pro 7.5.5 : File -> Export -> Export to Quicktime Movie
It's paid version.

Settings:
1. I disable streaming option for real time. It's not necesary at all.


2. Click in Options on sound layer - "Ajustes..." in Spanish. It will open this window


3. I choosed "true variable bitrate" option. There are 4 options:
Velocidad media - ABR
Velocidad de bits variable - (true) VBR
Velocidad de bits variable limitada - limited VBR (it's same in iTunes)
Constante - CBR


4. Quality option - Optimal (maximal)


Then I remuxed all tracks from .mov to .mp4

This post has been edited by IgorC: Nov 5 2008, 02:40

[halb27]

I see, QT pro offers some more features like true VBR and optimum quality which iTunes doesn't.

[nao]

Every time I have used Apple AAC True VBR it has prompted me for a "quality setting" from 0 to 127, nothing about kbits.

QT true VBR encoder can be configured with quality 0-127, but actually it has only 11 quality steps internally. So QT's user interface shows 11 typical settings like this :

Quality 0-6 : 48 kbps
Quality 7-20 : 56 kbps
Quality 21-31 : 64 kbps
Quality 32-44 : 72 kbps
Quality 45-57 : 80 kbps
Quality 58-69 : 96 kbps
Quality 70-82 : 112 kbps
Quality 83-95 : 128 kbps
Quality 96-107 : 144 kbps
Quality 108-120 : 160 kbps
Quality 121-127 : 192 kbps

[IgorC]

I'm just curious about plots.
I will appreciate if anybody will make a plots or where/how I can do it. I'm reading right now http://www.rarewares.org/rja/ListeningTest.pdf and some other sources but I haven't time at all.
Here is excel sheet with results http://www.mediafire.com/download.php?w4gzmzuluol

[dbAmp]

Interesting results. Thanks for all the hard work!

I would love to see a comparison including the iTunes 7.4.3.1 / QuickTime 7.2 AAC codec. IMHO, every subsequent version of the Apple encoder has been of lower sound quality.

[DigitalDictator]

What? Are you sure the quality has gone down after iTunes 7.4.3.1? Yeah, I'd like to see that version included please!

edit: typo

This post has been edited by DigitalDictator: Nov 7 2008, 09:42

[uart]

What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

... And difference of 0.13 is not statistically significant.

Indeed, just doing a quick and dirty calculation stddev/sqrt(n) is approx 0.1 so the difference between the two nero encoders here is definitely not statistically significant. (there is slightly more than a 20% random chance of this magnitude of discrepancy with this number of samples even if the two encoders were identical).

[IgorC]

I wonder why I was so deaf
After some intensive artifacts listening training and some experience with on-head positions of the same headphones I started to listen a lot of difference. The samples witch were rated by me as transparent now I rate something between 2.5 and 4 points. I even started to enjoy more music.
When I listened the sound I made a lot of contact between ear and earphone. I thought it was better.
That maked low frequencies to have more amplitude while middle/high freqs were (some kind of) masked .
I won't rip my music at such low bitrate as for AAC 90-100 kbps anymore

This post has been edited by IgorC: Feb 8 2009, 08:12

[rpp3po]

Interesting results. Thanks for all the hard work!

I would love to see a comparison including the iTunes 7.4.3.1 / QuickTime 7.2 AAC codec. IMHO, every subsequent version of the Apple encoder has been of lower sound quality.

What is this opinion based on? 7.5.5 has been excellent. The only flaw remaining was some a kind of spatial fuzzyness, what they finally got fixed in 7.6. QT 7.6 ist what I had always been waiting for. I even dump my lossless files for true VBR Q127.

This post has been edited by rpp3po: Feb 8 2009, 18:27

[C.R.Helmrich]

I wonder why I was so deaf
After some intensive artifacts listening training and some experience with on-head positions of the same headphones I started to listen a lot of difference. The samples witch were rated by me as transparent now I rate something between 2.5 and 4 points. I even started to enjoy more music.
When I listened the sound I made a lot of contact between ear and earphone. I thought it was better.
That maked low frequencies to have more amplitude while middle/high freqs were (some kind of) masked .
I won't rip my music at such low bitrate as for AAC 90-100 kbps anymore

Welcome to the club, Igor! Yes, being trained to coding artifacts definitely changes your judgment of audio quality. I've been working on audio codecs for five years now. After more than 100 listening tests, I cannot rip MP3s at less than 192 kbps any more When I heard my first MP3s about 10 years ago, even 128 kbps sounded great (even though most encoders were clearly worse than today).

But this is an interesting test. Makes me curious. I think I will do a similar blind test at the same bit rate soon, probably CBR with FhG MP3, Lame MP3, iTunes AAC, Nero AAC, FhG AAC. That should be enough candidates

[TechVsLife]

I think I will do a similar blind test at the same bit rate soon, probably CBR with FhG MP3, Lame MP3, iTunes AAC, Nero AAC, FhG AAC. That should be enough candidates :)

--you'll be testing them against quicktime aac (true vbr) at highest quality? I look forward to seeing more test results on it.
http://www.hydrogenaudio.org/forums/index....showtopic=60117


This post has been edited by TechVsLife: Feb 14 2009, 00:44

[C.R.Helmrich]

You mean the VBR mode which averages around 192 kbps? I can test that seperately by ABXing it to the umcompressed audio. Since the 96 kbps items will most likely not be transparent, I will probably conduct a MUSHRA test for them. Including QT's highest VBR setting in that test would not make much sense.

[TechVsLife]

You mean the VBR mode which averages around 192 kbps? I can test that seperately by ABXing it to the umcompressed audio. Since the 96 kbps items will most likely not be transparent, I will probably conduct a MUSHRA test for them. Including QT's highest VBR setting in that test would not make much sense.

I meant to suggest testing true vbr quicktime aac (latest version: 7.6) -- at whatever produces the comparable bitrate to whatever else you're testing:

Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings)

This post has been edited by TechVsLife: Feb 14 2009, 03:53