Encoders:
LAME 3.98.2 -V 5
Nero 1.1.34 -q 0.325
Nero 1.3.3 -q 0.31
Apple QuickTime 7.5.5 AAC ( true VBR at 96 kbits, highest quality settings):
Actually Apple AAC with true VBR at 96 kbit/s produces ~90-95 kbit/s.
http://img254.imageshack.us/my.php?image=qt1dq3.png
http://img254.imageshack.us/my.php?image=qt2kl8.png

Bitrate table
This bitrate table based on 8 full albums (mostly pop, rock, heavy metal and alternative)


Samples (original and encoded) with ABC/HR logs
If somebody want to see it.
http://www.mediafire.com/?x1d8vzzkfbw

Setup
Headphones Sennheiser HD 447. Soundcard Audigy SE 24/96.

Results

Apple AAC - 4.70
LAME 3.98 V5 - 4.50
Nero 1.1.34 - 4.16
Nero 1.3.3 - 4.03

Any comments are welcomed.

I cleaned up the silly imageshack ad-links or whatever from under your results. Thanks for sharing your hard work. Would you consider running an additional test with LAME at 90-95 kbps, just for comparison? "Thundersuck"?! HAHAHA! Awesome. I feel that way about that song too.

I tried LAME at 96 abr before. http://www.hydrogenaudio.org/forums/index....c=54967&hl=
Quality drops drastically to 3.2 average score.

10 kbps (~10%) difference can make big quality difference.

Absolutely, and based on the bitrate difference alone Apple's performance isn't much of a surprise.

What is an initial surprise is the fact that 1.3.3.0, despite averaging 1.4 kbps more than 1.1.34.2, apparently performed worse than its predecessor at the approximate targeted bitrate. Or is that simply a factor of better tuning on the part of the -q settings (since 1.3.3.0 was restricted to -q031, while 1.1.34.2 had the advantage of -q0325)?

- M.

Edit: Fixed a typo.

This just means that QT AAC was so wise to increase bitrate as necessary - at least better than Nero did in these cases.
What counts is the bitrate table IgorC gave in the beginning where he tried to give all contenders roughly the same chance. Here QT AAC even had a bitrate roughly 5% below Nero. So the procedure IMO is okay in principle though can be improved because 8 full albums of a similar genre are representative only for people who mainly listen to this kind of music.

We always tune our encoder for the "average" listener. It might happen that IgorC prefers 1.1.34.2 over 1.3.3.0. It might also happen that it is like that only on this samples set (we tune on much larger set of samples). And difference of 0.13 is not statistically significant.

Anyway it is nice to see that people are testing new encoder and everybody is welcome to do that

Interesting tests, thanks to IgorC.
I second halb27 comment: Apple's AAC doesn't benefit from any bonus bitrate ; Nero AAC encoders are both using 5% more bits than its contender.
The pertinent bitrate table is the first (album-based) one ; the second table (sample-based) is also important: it tells us that the choice of sample isn't fully representative of the full albums… which is usual in listening tests.

@muaddib « We always tune our encoder for the "average" listener. »
I'm pretty sure that every audio encoder are tuned that way

My apologies because at my first post I overlooked the first (bitrate) table.

I think talking about "average" in this case is totally wrong. (very carefully speaking).
70% of these samples are here already for years for ABX tests!
25% are also available on HA since enough time . And only 1 sample (AC/DC Thunderstruck) is new.
They are not average? They are most used in every test. You could simply tuned your codec on them if you wanted to cheat. Add to it that Nero used 4% of extra bitrate. But Nero wasn't even close to contender.
The bitrate table is also based on some very popular (if you want "average") CDs.

Speaking about my "averageness":
I don't pretend to be an average Joe but from previous experience of public tests my preference is very close to public results. For example you can compare my particular results with public results. They are too close. http://www.listening-tests.info/mf-64-1/mi...aneous/results/ .

Wait a minute. Every time I have used Apple AAC True VBR it has prompted me for a "quality setting" from 0 to 127, nothing about kbits.
So what settings DID you use exactly?

Yes, and that's what impresses me very much which your test. It made me download iTunes8 last night and listen for myself (with 'my' samples and at a higher bitrate), and the results are real good.

BTW - because I couldn't see a special quality option with iTunes, I just chose the bitrate and VBR (nothing like 'true' VBR) - did you use iTunes or Quicktime (possible with the free download?) for AAC conversion?

I used Apple QuickTime Player Pro 7.5.5 : File -> Export -> Export to Quicktime Movie
It's paid version.

Settings:
1. I disable streaming option for real time. It's not necesary at all.


2. Click in Options on sound layer - "Ajustes..." in Spanish. It will open this window


3. I choosed "true variable bitrate" option. There are 4 options:
Velocidad media - ABR
Velocidad de bits variable - (true) VBR
Velocidad de bits variable limitada - limited VBR (it's same in iTunes)
Constante - CBR


4. Quality option - Optimal (maximal)


Then I remuxed all tracks from .mov to .mp4

I see, QT pro offers some more features like true VBR and optimum quality which iTunes doesn't.

QT true VBR encoder can be configured with quality 0-127, but actually it has only 11 quality steps internally. So QT's user interface shows 11 typical settings like this :

Quality 0-6 : 48 kbps
Quality 7-20 : 56 kbps
Quality 21-31 : 64 kbps
Quality 32-44 : 72 kbps
Quality 45-57 : 80 kbps
Quality 58-69 : 96 kbps
Quality 70-82 : 112 kbps
Quality 83-95 : 128 kbps
Quality 96-107 : 144 kbps
Quality 108-120 : 160 kbps
Quality 121-127 : 192 kbps

I'm just curious about plots.
I will appreciate if anybody will make a plots or where/how I can do it. I'm reading right now http://www.rarewares.org/rja/ListeningTest.pdf and some other sources but I haven't time at all.
Here is excel sheet with results http://www.mediafire.com/download.php?w4gzmzuluol

Interesting results. Thanks for all the hard work!

I would love to see a comparison including the iTunes 7.4.3.1 / QuickTime 7.2 AAC codec. IMHO, every subsequent version of the Apple encoder has been of lower sound quality.

What? Are you sure the quality has gone down after iTunes 7.4.3.1? Yeah, I'd like to see that version included please!

edit: typo

Indeed, just doing a quick and dirty calculation stddev/sqrt(n) is approx 0.1 so the difference between the two nero encoders here is definitely not statistically significant. (there is slightly more than a 20% random chance of this magnitude of discrepancy with this number of samples even if the two encoders were identical).

I wonder why I was so deaf
After some intensive artifacts listening training and some experience with on-head positions of the same headphones I started to listen a lot of difference. The samples witch were rated by me as transparent now I rate something between 2.5 and 4 points. I even started to enjoy more music.
When I listened the sound I made a lot of contact between ear and earphone. I thought it was better.
That maked low frequencies to have more amplitude while middle/high freqs were (some kind of) masked .
I won't rip my music at such low bitrate as for AAC 90-100 kbps anymore

What is this opinion based on? 7.5.5 has been excellent. The only flaw remaining was some a kind of spatial fuzzyness, what they finally got fixed in 7.6. QT 7.6 ist what I had always been waiting for. I even dump my lossless files for true VBR Q127.

Welcome to the club, Igor! Yes, being trained to coding artifacts definitely changes your judgment of audio quality. I've been working on audio codecs for five years now. After more than 100 listening tests, I cannot rip MP3s at less than 192 kbps any more When I heard my first MP3s about 10 years ago, even 128 kbps sounded great (even though most encoders were clearly worse than today).

But this is an interesting test. Makes me curious. I think I will do a similar blind test at the same bit rate soon, probably CBR with FhG MP3, Lame MP3, iTunes AAC, Nero AAC, FhG AAC. That should be enough candidates

--you'll be testing them against quicktime aac (true vbr) at highest quality? I look forward to seeing more test results on it.
http://www.hydrogenaudio.org/forums/index....showtopic=60117

You mean the VBR mode which averages around 192 kbps? I can test that seperately by ABXing it to the umcompressed audio. Since the 96 kbps items will most likely not be transparent, I will probably conduct a MUSHRA test for them. Including QT's highest VBR setting in that test would not make much sense.

I meant to suggest testing true vbr quicktime aac (latest version: 7.6) -- at whatever produces the comparable bitrate to whatever else you're testing:

Sorry. Yes, I can include that. However, I will use the latest iTunes (8.0.2.20) for encoding since that's freely available and supports 96 kbps VBR, as I just realized. But it doesn't allow choosing the encoding quality level.

note that iTunes doesn't support a true VBR mode (it supports vbr only in a constrained mode, with a floor), and there's been some indication here that the aac true VBR mode which is available in quicktime pro might be significantly better (in terms of quality per bitrate). also, unlike iTunes aac, quicktime vbr aac has not yet been in public listening tests.

But it's true that quicktime pro is not freely available, and it's also not practical to use (or test) the true vbr mode except on a mac. I wouldn't have confidence in it until apple makes the interface more easily accessible (maybe it will be added to iTunes following the release of QuickTime X?).

Correct me if I am wrong but doesn't iTunes use QuickTime for decoding and encoding? I know that iTunes just used to be a graphical front-end for QuickTime. It is my understanding that iTunes uses QuickTime for AAC encoding. It uses the CBR and VBR_constrained options though not allowing for full VBR encoding. Either way that should mean that one can test QuickTime pro's AAC encoding quality by using iTunes so long one doesn't mind using the CBR or constrained VBR settings.

It would be interesting to test QuickTime pro's true VBR setting against the iTunes constrained VBR setting. My guess is that the true VBR mode would provide higher quality but I don't know. I too am not going to test QuickTime's true VBR setting simply because it is a pain to use it on Windows.

The problem is bitrate.
True VBR 96 produces around of 92 kbps (for 8 albums of the first post in this topic) while constrained VBR produces 100-105 kbps. It makes incomparable them.
In the past I performed comparison between true and const. VBR. Both were very tied, maybe constrained was a little bit better because it doesn't go too down in bitrate occasionally

But it still will be good idea to test them to see how true VBR is efficient.

I never understood why there are so many low bitrate comparisons. Especially here at HA there won't be many using bitrates below 128kbs. Results at low bitrates don't say anything about a codecs capabilities at high rates. Musepak is the best example: it would lose any comparison at small rates, but left the competition behind at high rates for years.

The reason is probably that it's easier to ABX at low rates. People think they can take a low rate comparison's winner and just add a good chunk of headroom to get the best codec for transparency at high rates. But that equation is too simple. Low bitrate tuning can even have an adverse effect on high rates.

Trying to ABX at target rates using killer samples is probably a better approach. Nero and Quicktime have become so good that real "killer" samples are nowadays hard to find, though.

Yes--quicktime is doing the playing & encoding for the itunes media organizer & storefront. I meant the quicktime true VBR mode vs. the other quicktime modes. (It's unlikely to be much better since it's buried from view, but otoh there's been no public listening tests on it.)

@rpp3po:
I also am more interested in the "high but not insane" end of lossy encoding (128 to 192 kbps), though painful to test. But as soon as mp3 players (or USB drives) get sufficient solid-state storage, perhaps another year more, I'll switch to all lossless. It's interesting if there can be no further improvements and music can't be compressed further without losing quality for average listeners. Presumably there can't be a mathematical proof of this, the way one could establish there is a definite limit to the greatest maximum lossless compression.

Are any of today's popular encoders like musepack was? (worse than other encoders at low bitrates, but better at high bitrates)

I'd suppose it's because at 128kbps+ the differences between most modern codecs are not great enough to make a practical difference, even if they're not actually statistically negligible. IOW, it doesn't really matter what you use.

I can't recall an HA listening test over the last two or three years that hasn't concluded with something along the lines of "there's no difference really worth noting here, so let's move on to lower bitrates".

One would have to wonder if the tests were becoming an end in themselves. However, I suppose there are people with very large collections looking to carry a good percentage of what they've got around with them. Combine that with the move to flash memory in DAPs and I guess there's something of an incentive to clip filesizes as much as is feasible.

Nevertheless, by "ringing the changes" on playlists one can more-or-less solve that problem, and in truth I think the tests are more an end in themselves than not for people who enjoy that.

ITunes is using QuickTime - absolutely. You've only got to rip and encode something in iTunes to see that. Do that, do a GetIfo on a track in the "Music" listing pane, and choose the "Summary" tab. Under "Encoded with ...." you'll see something like this:



However, I think you're mistaken about its being "the quicktime true VBR mode vs. the other quicktime modes". It seems there are modes available on OS X in CoreAudio (on Windows, I don't know) that are not accessible through iTunes's GUI. Stephen Booth's Max can access no less than five different VBR modes - Maximum, High, Medium, Low, Minimum. AFAIK, that's because Apple made switches available for that in CoreAudio - but, nevertheless, chose not to make those accessible from iTunes's. In iTunes, VBR is either "On" or "Off" - "that's all, that's all, that's all". And the "new" so-called "True VBR" mode is over and above those five.

Well, the true VBR mode has been recommended for CD ripping by skuo on this board, who's one of the developers of the codec. Also remember that the constrained VBR mode was "buried from view" quite some time in QT Pro before it made it into iTunes. I wouldn't be surprised if the true VBR mode finds its way into iTunes some time in the future. Some speculate they haven't included it since it would be too confusing for the end user, but I don't quite buy that. If you look at the mp3 encoder in iTunes, there's tons of settings to chose from, including setting the quality level for the VBR mode.. They could easily have made a similar type of interface for the true VBR mode too.

<off topic>
LAME is
Well, almost; it's not discernibly better than other lossy codecs at "higher" bitrates.
But heck, 99.9% (100%?) of hardware players supports it, and to 99.9% of listeners, it sounds just as good as anything else.

That pretty much sums up the relative lack of interest in 128+ kbps listening tests here, or anywhere; imho, there's little excitement to the kind of pedantry whether Vorbis, Musepack, WMA or AAC deserve a 4.8 or 4.9 score at a given bitrate, if MP3 sounds at least as 4.8ish AND is supported anywhere and anytime. That would be statistical philosophy with little ties to real life.
</off topic>

That said, I'd like to thank IgorC for his interesting test results. I've expressed personal interest in a multi-format 96k listening test on numerous occasions before, which is why I've been following this thread with above-average interest. As coined before, it's just a shame that Apple's AAC codec is not as straight-forward to use (i.e. command-line on a Windows platform). Apart from that, 96k AAC seems to hit the sweet spot I'm looking for in lossy encodings; near-to-transparent quality at very portable bitrate with fairly to very good player support, especially on the Apple player hardware I'm partly using.

Edit: if you compare LAME V5's 4.5 score at 154 kbps to Apple AAC's 4.7 at 103 kbps in IgorC's test results, "near-to-transparent quality at very portable bitrate" might even be quite an understatement.

I don't know about "confusing", but over-complicated did sound plausible - inasmuch as Apple sometimes does seem to prefer "clean" interfaces.



I hadn't looked. I see you're right. How peculiar! So they did that for MP3 but not AAC. Perhaps looking for the "why" is mistaken - perhaps they haven't got a coherent strategy.

I agree with Polar and Nick E that testing modern encoders at 192 kbps or more is not very exciting. Or put differently, it's very tiring, simply because all current popular coders have become very good at such bit rates. It's hard to find outliers. I'm working on Fraunhofer's AAC encoder, and I've heard only about half a dozen extremely critical test items which are not transparent at 192 kbps AAC. MP3 is a different story, though.

That's why I'm also more interested in bit rates around 128 kbps maximum, not some VBR which, as seen in the last public tests, leads most coders to switch to, for example, 200 kbps on the Fatboy Slim item. In fact, except for the anchor, all coders averaged at least 139 kbps. That's not really a 128 test...

I think a 96 VBR test would be interesting. Or 128 CBR, but I think that has been done.

Tuning a codec at low bitrates (especially CBR coding) can certainly improve the size efficiency (but not necessarily quality) of the same codec at higher-bitrate/quality VBR settings.

By "different story," do you mean that a recent mp3 encoder did significantly worse than aac at 192? In what appears to be the last listening test of mp3 vs aac vs wma pro at 128 (back in 2005), they were all tied, and I thought that was the consensus view here for such "safely" high bitrates as 192. I don't believe Fraunhofer AAC was ever tested (at ha)--and it's no longer available for download. Will an encoder/decoder become available? (In the alternative, perhaps encodings that show several examples of marked superiority to mp3 at 128 or higher kbps could be posted, along with the originals?)

http://www.iis.fraunhofer.de/EN/bf/amm/dow...mpeg4/index.jsp

Yes, I meant that MP3 at 192 kbps is not transparent on more critical items than AAC at the same bit rate. MP3 has some problems with very tonal items. This is a technological limitation. But again, these are very critical items which do not necessarily represent your daily music.

No, a free Fraunhofer encoder will not become available again. And sorry, I don't think I'm allowed to post Fraunhofer encoded demo AACs. But you can test e.g. Lame vs. iTunes at 128 kbps. For a start, try this item (Google, my friend , found it for me):

http://www.informatics.indiana.edu/donbyrd..._tcm6-12548.wav

There was a discussion about something similar to this either earlier in this thread or in another thread. Basically people came to an agreement that a bitrate increase of ~10kbps wasn't that big of a deal even if encoders produce worse results while staying closer to the target bitrate. The reason being is that these worse performing encoder weren't "smart" enough to increase the bitrate when needed. Either way I would consider 139kbps to be a 128kbps test. After all, working with VBR is not an exact science and I would hate to enforce a bitrate ceiling (and/or floor) on an encoder instead of letting it perform optimally.

The last 128kbps VBR test showed us quite a bit; mainly that each encoder produced transparent results at those settings (there really isn't big enough of a difference between 4.0 and 4.1-4.2). I agree that a 96kbps VBR public listening test would be nice. That way we have something to reference aside from IgorC's extensive tests (thanks Igor!).

Since your May 2007 Rapidshare spreadsheet doesn't really mention the eventual bitrate the samples averaged to, would you mind unleashing your above 8 reference albums at:

• LAME -V7
• LAME --abr 96
• Vorbis aoTuV Beta 5 -q 2
• (whatever you think would produce the highest quality) WMA (10 Pro 96k-ish CBR/VBR?)
• ... anything else making any sense comparison-wise, which imho Musepack or HE-AAC are not.

I'm quite eager to get the interest up for an unprecedented public 96k multi-format listening test in this forum. I'd be more than happy to host the test files once more. So in order to get at least some frame of reference to start a new pre-test discussion thread, I'm looking forward to those average album bitrates of yours. Tia!

I'm very busy now. Maybe later.