That's pretty cool stuff. One probably costs more then what I'll make in an entire year, but its damn cool stuff.

Too bad we can't encode with those codecs. It would be interesting to see where "satellite radio" falls in the group.

- Agent 86

This particular type of ANOVA does take into account the varying scales. That's because I treat each listener as a "block." Here is a general description of blocked ANOVA:

http://www.basic.nwu.edu/statguidefiles/on...ay_b_anova.html

ff123

I updated the overall graph, following a suggestion by ff123 and Garf. Now, there's no more results being "eyeballed"

As you will notice, Faac got a little worse in this updated overall, and Nero is now almost perfectly tied to Psytel AACenc. Sorenson also got closer to QuickTime (or maybe QuickTime got ranked lower).

For those feeling nostalgic of the old plot:
http://rarewares.hydrogenaudio.org/test/aa...t/plot10old.png

Regards;

Roberto.

Thanks again for organising this most useful test Roberto, and thanks to the others who helped with the setup, and post analysis.

Based on these results, which AAC encoder will be the "champion" to go up against the next round of contenders, namely vorbis, mpc and mp3?

It concerns me a little, that a particular AAC encoder will be chosen as the representative, but in reality, it may not be the most logical choice for the average user. For example, I don't think QT will be used as often as some of the others by Mr/Ms Joe Windows-User in its current state, due to the crappy "one file at a time" user interface. Even Sorenson may be a great encoder, but its cost may keep people away?

I for one will always favour an encoder that allows me to access it from the command line, and feed it from a range of front ends of my choosing depending on what I need at the time, ie foobar, EAC, CDex, Frontah, home made batch files, "insert you favourite media player/front end/ripping/compression package here"...

If all the command line optioned encoders sound like complete crap, I would be forced to use one of the superior encoders as the difference will be worth it, but I suspect from these results, Sorenson, Quicktime, Nero and Psytel all have some potential to compete against the others at ~ 128 kbit, especially if they are allowed to go VBR. If you choose QT to be the AAC representative, I will still be keen to be involved, and see the results, but if it is close to the other non-AAC formats in quality, I'll have this nagging question in the back of my head. "I wonder how Psytel would have gone..."

I appreciate that these listening tests can not accomodate everyone's wishes, and I suppose ultimately you need to pick the best performing encoder, regardless of its expense or convenience...

Just my 2 Australian cents.

Den.

Add WMA to that list.

And atrac3 is another possibility, but that's only a speculation right now. I don't even plan to take this decision now, I'd rather go take a looong nap after I finish dealing with the test results. Preferably at the beach.

After I'm sufficiently rested, I'll start thinking about codecs, bitrates, settings, etc.

Anyway, ATM there are two options for the AAC: Nero and QuickTime

QuickTime because it kicks ass. Nero because Ivan reportedly has been improving his codec, spurred on by this test results. If he can deliver an improved codec on time, it might be considered for the extension test.



Right, I don't think people will be letting go 400 hard earned smackers just so that they can encode their audio tunes to AAC.

Besides, I think Sorenson suffers from the same problem as QuickTime, of "one file at a time". Gotta check that later.



I understand you are talking about AACenc, but including AACenc wouldn't be a good idea. From the development point of view, AACenc is now dead, since it can't be developed furthermore. Nero has the downside of being GUI-only, but at least it's being actively developed, more so these days.



Well, by far the most important aspect is the quality, but price and convenience also have their importance. For instance, I would hardly pick Sorenson for the extension test even if it had won, because it's an encoder few people have access to, it's damn expensive and you can only encode one file at a time.

QuickTime, on the other hand, is cheaper, widely available and reportedly you don't even need to register it if you access the encoding component directly through another program. (maybe a CLI program )

Thanks a lot for your remarks.

Regards;

Roberto.

If you use Squeeze 3.5, you can "Open Watch Folder" to do a batch encode. It's very odd name for batch encoding until you realize that Squeeze will remain open and "watch" for any new WAV files you drop into the Watch Folder and immediately start encoding them according to your previous settings. It's weird, but kinda cool in the right situation. Squeeze will also separate the WAV files and MP4 files into separate sub-folders called "Work" and "Done", which may or may not drive you insane.

We can consider the possibility, for the next test, to include two AAC codecs :

- VBR one, in fast development (don't forget HE-AAC), easy to use, pretty GUI (I can transcode my MPC to MP4 with Nero)
- Winner of this test : CBR QT 128, for people that want a very good quality, without consideration of time encoding, GUI annoyance, etc...

Other codecs may be mpc, vorbis (official), wma (standard, pro ?), mp3 (lame). Maybe ATRAC3...
But that's maybe too much challengers (12 samples x 6-7 codecs = 72...84 files).
(don't forget that you need a CLI decoder, if you want to reduce the package size with a .bat file : is there any WMA9 or ATRAC3 separate decoder available ?)


Problem is : what's the purpose of this test ?
Is it a practical one : what is the best audio format for portable use ? MPC is actually useless here...
Is it a theorical one : what is the best audio format at 128 kbps, regardless of any portable available, or GUI, or encoding speed ?
Is it a mix of the two ?

We need to clearly answer to these questions in order to make the codec/setting choices.

I was talking with Roberto earlier today exactly about this. We have a plan how to proceed, but I don't know when Roberto wants to make it public.

I'm also wondering if MPC is necessary for low-medium (around 128kbps) vbr test. But Roberto claims that Musepack users will be very upset if it's not included. I'm not so sure about this, because people already know that Musepack is very good at higher bitrates, and I'm not sure if MPC users are even interested in the bitrates this low.

Most current Musepack users (audiophiles/HA-whores ) are definetly not interested in anything below --standard, but it has been mentioned before that it will probably do a lot better than people would expect it to at around 128kbps, so this might bring some clarity to the situation and possibly make Musepack an option for a new group of users.

dev0

Excluding MPC makes no sense - it's definetely a contender at 128kbps.

I'm partially agree with you. I did some tests long time ago, and I found mpc --radio to be the most pleasant format to my ears : PsyTEL -streaming was second, followed by LAME ABR. I didn't like Vorbis (rank #5 if I remember correctly).
Nevertheless, if the new test has for purpose PORTABLE abilities, musepack is a contestable choice... It may be a better choice to include ATRAC3 (MiniDisc portable) or WMA9, even if theses competitor are worse than mpc.

There are some problems with mpc too. Accoding to my experience, the amplitude of musepack is the higher I saw on 'common' music. A piano Goldberg Variations at --radio is ~100 kbps (Gould 81 for exemple) ; on harpsichord, it's ~170-180 kbps. Some isolated tracks are ~90 kbps with piano, and ~200 with harpsichord. Difference isn't so big with other codecs : it's easier to reach an approx. 130 kbps, or fit them to an exact 128 kbps without losing too much in quality.
Here are the bitrate for each samples of the previous test :
http://membres.lycos.fr/guruboolez/AUDIO/a...pack_report.htm
(--radio, but don't remember if it's 1.14 or 1.15r)

Do you know anybody who uses MPC at these bitrates? Do you think thousands of new people will start using MPC even if MPC wins that test? Imo MPC is clearly a niche format, not even targeted at masses. The (sad) fact is MPC does not have mass appeal, starting from its website...

I think many people here already know that MPC is surprisingly good at average 128kbps. Not sure what would it serve to include it, simply because even though MPC is very good, it is generally unknown format used by small niche group and lacks the appeal ouside hifi quality high bitrate use.

Sure, maybe MPC gets few dozen of new users who will start using it at higher bitrates, but I'm not sure if it makes any difference regarding MPC's position as a format.

I don't have anything against the inclusion of MPC, on the contrary, but if we have to chop some codecs, imo we should start from those codecs which have overall the least interestingness globally.

I don't get it. Why should we include Vorbis then? What's the point of the test?

Xiph tries to make Vorbis appeal to masses. It's relatively good at all bitrates, it has multichannel encoding, it has directshow filters, it is used in movie encoding, it has an order of magnitude bigger userbase than MPC, Vorbis has had quite a lot of media coverage (MPC once introduced in c't), there are several broadcasters streaming vorbis etc. etc.
In short, Vorbis is trying to be in the same league with the big formats, when MPC is not even trying and it isn't in the same league, except quality wise. I'm not sure if the knowledge that MPC is good at 128kbps is very relevant for the masses outside HA, and most HA people interested in MPC know that MPC is good at near 128kbps already, so why include MPC?

Well, I for one would like to know how good or bad it is.

But you did not really answer my question, what is the use of the second test? Is it to compare AAC to it's direct competitors, and see who has the best quality? In that case, MPC would not be a competitor indeed.

Umm, I don't think it's the idea that the cross-format test is AAC vs others. Rather I'd see it as comparison of the big players against each others - the formats which are the most widely used in PC, so that the test has as much significance as possible for as many people as possible.

I'm not deciding anything here. Besides, I sure will be talking a lot about this with Roberto. I just think we should consider carefully which formats will be included.

Just to chime in with Garf... MPC is used by exactly as many portable devices as Vorbis. There is only those who say they are working on it and that it will be released later - that goes for Vorbis as well as MPC. So from that is definately not an argument for excluding one from the test and keeping the other.

If the test is setup to answer which is the best choice for portables both WMA8 and ATRAC3 have to be included. Neither Vorbis nor MPC has anything to do in such test.

As far as I can see I didn't say anything about portables at all.
Portable usage is one aspect which should be considered also though.

guruboolez did.

In case it will be narrowed down to portable use you could afford to have two AAC codecs in the test as well.
- QT
- Nero VBR
- LAME ABR
- WMA
- ATRAC3

More than five codecs is too much IMO.

The objectives of this test are not that clear, at least to me anyway.

Despite being an ATRAC3 user (dodges rotten tomatoes) , I'm not sure if it belongs here. I hate to generalise, but most if not all ATRAC3 users do so because they have no other choice, ie they own an ATRAC3 only Sony/Minidisc product. OK, so Sony now have these Discmans with ATRAC and MP3, but I'd bet my left one that savvy owners would use MP3/LAME alt presets. You simply have more control over your encode...

I would expect in the longer term that ATRAC will be left to die when Minidisc eventually goes away...

I suggest we drop ATRAC3 from the list, unless you really need another codec as a control, or comparison for the test.

I think WMA, Vorbis, Musepack and the omnipresent MP3 should stay in. In fact LAME and FHG should probably be tested as some here swear (substantiated or otherwise) about FHG being better at 128 kbits than LAME. They all offer creditable, albeit sometimes unused performance at this bitrate, even if they lack of portability. While Vorbis and MPC have similar, virtually non-existant portable support, it will probably come in the future, even if only on PocketPCs, etc (Vorbis is there now.)

With Musepack in particular, I came away pleasantly surprised with mpc -radio during recent tests. It isn't regularly used or recommended by HA, but that could be from habit rather than the facts. (In my tests, I actually preferred it over Vorbis...)

Just another 2 Australian cents wirth (which currently buys a lot more than it did 12 months ago...) B)


Den.

Wow. I wake up, and there's almost a format war going on.

Well, people, first, I'd like to repeat that I'm not willing to make decisions ATM. We have a full month ahead until the next test starts, it's now time to chill out. B)

Some misconceptions that happened here:

1) Where the heck did you guys came from that this test is intened to address formats with portable support? I never wrote it, either at posts or at the test pages. The main purpose of these tests is verify what format performs best at quality. And only that. Other issues like usability, popularity and availability might be taken in consideration, but quality weights in about 90% in my decisions. Only when two formats/codecs are tied in quality that I will consider other features.

2) I definitely don't plan to allow a format to be represented twice in the extension test (I.E, QT and Nero). I remember how ff123 suffered from his decision of representing vorbis twice on his 64 test, and I don't plan to go the same way.

3) atrac3 is only speculation ATM. There'll be lots of things to consider.

4) It's likelier that MPC will be at the test instead of atrac3

5) No Lame vs. FhG MP3 at this time. I decided that this can be handled in a separate test.

Now, the idea JohnV and I had was to conduce a fast listening test (5 days, 8 samples, 2/3 codecs) at the beginning of July, QuickTime 128 vs. Nero -streaming (and maybe a 3rd codec kept in secrecy ATM). If Nero wins, it'll be used on the extension test. If QT wins, it'll be used and God help us all.

Again, it's only an idea. It's not definitive

And this is my planned schedule:
June 28th to July 3rd: The quick AAC test
July 16th to July 27th: The extension test

And it's only a plan. Don't go planning your schedule based on that, and don't start complaining about the dates now. You'll have time to do that later.

As a bottom line, the formats that are already guaranteed at this test are Vorbis (official CVS version), AAC (QT or Nero), MP3 and VQF. I mean, WMA v9.

Regards;

Roberto.

What about MP3Pro ??

There isn't much point testing mp3pro at 128kpbs...

I'd like to better understand the purpose of this test, too. I know Roberto said "the purpose is to find what encoder performs best encoding AAC at 128kbps", but that still leaves me with some questions.

CBR was chosen because some codecs are incapable of making VBR audio, and a level playing field was desired, correct? Well, this isn't a friendly, sporting match between AAC implementations, this is a codec war! Let every codec employ its best tools for the job, I say, and that would mean using VBR wherever possible. I know there was a discussion back with the 64kbps test about setting the Vorbis q-value so that the samples' bitrates averaged to 64kbps, and I think ff123's conclusion was that q0 should be used simply because it's the most likely setting for a typical user to choose. But that decision was not a quality-based one, and since this test is all about quality, the question should be re-visited. Or maybe you've already dealt with the issue by planning a quick AAC test at the end of the month. In that case... never mind.

Another question: I don't recall reading why 128kbps was chosen. Is it just a convenient number chosen for historical reasons? Are the codecs optimized for this specific bitrate? Different bitrates can be used for streaming, file distribution, digital radio, portable playback, and archival. Is 128kbps considered to be the best compromise for all these applications? Or are people just more likely to pay attention to the results of a 128kbps test?

Roberto, I'm not trying to discount your listening test. I still think it was (and will be) very well-designed. It has already given us valuable information, and I know it will continue to do so.

MP3pro isn't considered good above 96kbps. The SBR codec starts degrading the audio quality more than mp3 alone would do.

Here's a comment from the masses:

http://groups.google.com/groups?dq=&hl=en&...ob.news.rcn.net

It warms my heart.

ff123

Sorry, Is there any one try to compare ODG of these AAC Codec ?

I want to do this. but I dont know how to let QT output as wav file ?

By the way, where can I download the samples ?

I just see samples**.zip.torrent but I dont have bitTorrent..


Thanks a lot

It would be nice to see MPC @128kbps added to the test, so that we can validate those comments saying that "subband can rule at 128kbps too!". 2 main problems I see:

1. Have we already hit our rule-of-thumb limit of maximum 6 test examples?
2. How do you control the MPC bitrate? In previous listening tests @128kbps, the performance of MPC depended on the resulting bitrate. MPC clearly won the first dogies test , but that can be attributed to bitrate bloat to 143kbps. For rawhiles and fossiles, MPC was at 136 and 138 and there were no significant results. For wayitis, MPC was tied at the top with Ogg Vorbis, but needed 140kbps to do that.

The result of the test has been posted here as well...

http://www.cdfreaks.com/news/7355

I like some of the comments below....

You can use FAAD to decode the QT files, I believe. I haven't used QT, but I think that's how Roberto decompressed the test samples. I think FAAD can be downloaded at RareWares.

The test files are no longer available for download, since the test is over. You can still get the names of the samples, and then search for available copies. HA and ff123 still host some, if not all, of the samples used in the test.

As for ODG... isn't that the EAQUAL measurement standard? An EAQUAL comparison would be academic, not practical, since previous discussions have shown that its quality rankings are not very precise, nor are they appropriate for codec comparison.

"I hope they are talking about Astrid/Quartex AAC, others AAC sux..."

Ladies and gentlemen: Hydrogen Audio, the ivory tower of psychoacoustic compression.

Radio@AOL is using the now famous Dolby AAC encoder. More information can be found in the following link:

http://www.dolbylabs.com/DolbyAAC/

Given Dolby's comment:



it seems that they don't simply use the FhG codec modified for speed but they must have done some work in order to improve the quality. This makes sense after the listening test results.

This article is not entirely correct because the Sorenson encoder was voted second in quality and not the one in Nero.

Nice board

I would just like to point out to anybody interested, is that iTunes 4 does use the same encoder as QuickTime 6.3, it's just that in the 4.0 release it was accidently hardcoded to the fastest setting. 4.01 has corrected this and it is now hard coded to highest quaiity (as seen by the extreme decrease in encoding speed on my powerbook (13.9x down to 7.2x)).

I am 99.99999% sure on this, but maybe someone with better ears than me could verify it.

I used FAAD.

QT has some issues and can't decode AAC files wrapped with mp4creator (Psytel and FAAC, in the test)



I know I should reply, talking about Pac and about stolen libraries. But I really don't feel like it. >_<

I mean, what the heck, since the days of the VQF forum people know Astrid was stolen from CelestialTech's AudioLib 1.0

In order to clarify the challengers for the second (part of the) test, I did a quick blind test, and opposed WMA9 called 'PRO' to WMA9 standard. Results are here.

WMA9 PRO seems to be the best encoder, and should be prefered to the standard (and portable-compatible) one.

I head about that at the 3ivX forum too. But they aren't sure either.

I'm waiting for some confirmation before I update the results page.

One way to confirm would be to encode a file with iTunes, then with QT in high mode, and compare them.

Regards;

Roberto.

I will encode a file using QT6.3 (Highest) and iTunes 4.01 and compare the encoded files using a hex editor. I will report the results back to this thread.

Tested (quickly) GT2 vs Post CVS release of ogg vorbis, here.
I changed my mind about GT2 virtue : final version is maybe more 'dirty' (hiss, noise added), there are less annoying artifacts (and size is a bit smaller), at least, on difficult samples.

That won't work because iTunes adds extra metadata. The files will be slightly different sizes. Someone will have to do a listening test I think.

Hrm... maybe extracting the AAC tracks with MP4creator for Mac, and then comparing the AACs?

Your reply appeared on google groups:
http://groups.google.com/groups?dq=&hl=en&...52dfa%404ax.com

Ok, I'll do that then and report back.

Ok, I encoded an AIFF file with QuickTime 6.3 and then with iTunes 4.01 at 160Kbps. I then used mp4creator to extract the audio track and... the iTunes encoded file does not match any QuickTime 6.3 encoded file at any quality setting

I also checked the tracks in a hex editor and they are clearly different. The sizes are also different i.e.

QT6.3 High - 1994056 bytes
QT6.3 Better - 1994053 bytes
QT6.3 Good - 1994087 bytes
iTunes 4.01 - 1995001 bytes

I then tried AAChoo (front-end for QT) and the encoded tracks were the same to those produced by QT for each quality setting, although the .mp4 files were slightly different in size because of the addition of extra tags.

I am confused... Is iTunes passing some special parameters to QT?

i did a quick encode of one 6+ minute track in itunes 4.0.1 @ 160 and qt 6.3 @ 160 "best" -- itunes took 24 seconds, qt took 44 seconds. dunno if itnues is more altivec or mp savvy, tho. so . . . fwiw . . . might not be worth much.

What options did you select with QT?

iTunes only lets you set bitrate but QT has a load of other options. Surely the setting of those would have an effect? In particular there are options for optimizing the file for streaming.

Now I would, on first thought, switch them all off but one of the major features of iTunes 4 is LAN streaming (with internet streaming to specific clients due to return in a future update according to rumour) so it makes sense for iTunes to use these settings if it doesn't massively affect encoding time.

Optimize for streaming only affects the MP4 container. If you strip the AAC file from inside, there should be no difference.

It might make it easier to compare if you decode to WAV.

ff123

I encoded the file using QT and each quality setting without any other special parameters. The option to hint for streaming was disabled but that does not affect the encoding. I then extracted the tracks using mp4creator and I could not match any of them with the extracted track from the iTunes 4.01 encoded file. I managed to match them with encodings using AAChoo which is a front-end for QT (like iTunes for AAC). The iTunes 4.01 speed of encoding was faster than the QT one in high quality.

My guess is that iTunes calls the QT encoder with some special parameters to optimise the speed resulting in a file which cannot be generated by simply using the QT interface.

Hmm, I wonder what the quality is like tho. Maybe in the next test have an iTunes encoded AAC file and a QuickTime encoded one. Who knows. I just wish iTunes had a quality option so I could be sure.