I'd like to announce that my first listening test successfuly ended.

The results page is available here:
http://rarewares.hydrogenaudio.org/test/aa...st/results.html
(veteran forum members will notice that page resembles to someone else's)

So far, you only get results for each sample, and overall results.

Tomorrow, I'll add fancy things like user comments, bitrates, etc.

If someone notices flaws or typos, please report in this thread or through PM. Thanks a lot.
(Also, info on the 41_30sec sample is welcome)

For those really hurried, here is the overall plot:


Once again, thanks to everyone that participated. Hopefully, there'll be more in a month.

Kind regards;

Roberto Amorim.

wow
QT really shines wonder how it compares to psytell -streaming.

Just a little typo in the last line: "Sorenson is good, but it's price is prohibitive." It should be "its" which is possessive.

Interesting test. I guess we shouldn't be too surprised to see how well the QuickTime's AAC did. I was a little surprised to see FAAC getting beaten so badly though. Would you say it's still a better alternative to 128kbps MP3? (I assume we're using this bitrate for the purpose of portables use.)



This may have been addressed a million times elsewhere, but how is AACenc illegal? Is this the encoder used with PsytelDrop? I have it but don't recall where I downloaded it. Didn't realize it was illegal. (Warez?) Or is it just a licensing issue?

the sample is from the song "#41" by the dave matthews band from the album crash.

allmusic link: http://www.allmusic.com/cg/amg.dll?p=amg&u...l=A4c6tk6dxqkrf

regards; ilikedirt

Very interesting, especially considering how bad Dolby did in the 64kbps test.
Overall I was quite suprised by the high quality of all codecs except for FAAC, but there definetly is still room for massive improvments. None of the codecs was transparent to a degree where I'd consider using it for archival purposes, but for portable/casual use, which is the main field of usage at 128kbps anyway, AAC looks very interesting.

Thanks to Roberto and everyone else, who helped and participated, for investing time and work on this test. I can't wait for the next test to see how AAC compares to Vorbis, Musepack and MP3...

dev0

I am also interested to see how aacenc -streaming compares to QT. I was very surprised when I was informed which samples were which encoder after I submitted my test results. Psytel -streaming in my own tests has been quite impressive for the bitrate, and to have it fixed as CBR in this test was not really letting it work at its best. I also understand why this testing decision was made however, for the sake of fairness.

As for FAAC vs mp3 128 kbit, I haven't done a direct back to back comparison on identical samples, but some of the FAAC samples in this test were really bad, as can be seen from the charts. I'm relatively new to codec listening tests, but in some FAAC samples, the voices/instruments actually sounded different in their tonality from the others, let alone pre-echo or other more typical codec introduced artifacts.

Den.

There is a ongoing improvement to Ahead Nero "streaming" preset as well, some changes already were made (still not out) as long as some grouping improvements and bug fixes. This will probably be available in the next web version of the Nero.

Interesting results. Does this test say anything about how the decoders are doing with VBR files in that bit rate area? (i.e. the encoders that are able to produce VBR files). I'm thinking if 128 CBR isn't good enough for portable use, maybe ~128 VBR is?

I did some preliminary tests, and I compared blindly QT 6.3 to PsyTEL & Nero -streaming. Tested on 6 samples of this test.
Difference is impressive. Most audible one is the higher lowpass, which provide a richer sound than QT (appreciable on Atrain and 41_30), and less artifacts in some case. Now, PsyTEL & Ahead Nero AAC encodings are close to QT (for my taste), and sometime better than QT (41_30 for exemple). Unfortunately, I didn't have too much time last week to test it further. And I broke my headphone. The new one is here now, and I have some time the 2 next days

My results are here :

http://membres.lycos.fr/guruboolez/AUDIO/a...128/tableau.txt

and for VBR :
http://membres.lycos.fr/guruboolez/AUDIO/a...tableau_vbr.txt

minor nitpick with track info:

It should be "You've Got the Love" by The Source feat. Candi Staton (Although a popular dance tune and available on many compilations I don't know if the track was ever featured on an actual album by The Source)

-- a happy Mac user B) off to harass Apple for VBR capabities

What I find strange is the difference between the Sorenson (FhG Pro codec) and the QT (Dolby codec) results. I have the impression that the Dolby codec is based on the FhG codec with optimisations for speed rather than quality. Still, according to my experience, QuickTime Pro 6.3 and Sorenson Squeeze 3 take almost the same amount of time to encode an audio file in AAC, when the quality setting for QT is set to the highest setting. Could it be that this setting disables all options which reduce the quality of the encoding in favour of speed?

Another interesting point is the following link from Dolby Labs:

http://www.dolbylabs.com/DolbyAAC/FAQ/

which contains the following note:



So, what is Dolby's AAC codec? An improved FhG codec for both quality and speed, as the listening test results indicate?

We didn't have the same experience. According to my timings, QT is one of the fastest AAC codec, and Sorenson the slowest with PsyTEL :

http://membres.lycos.fr/guruboolez/AUDIO/a...ac128/speed.txt

Hmm... I am a MacOS X user and on that platform QT6.3 (with highest quality setting) and Sorenson take almost the same amount of time to encode at 128Kbps.

Interesting. Therefore, for 95% of the users in this world, QT is three time faster than Sorenson (and a lot cheaper).
What is interesting to note is that QT is the absolute winner of the test, and in the same time, one of the fastest codec (FAAC is faster on my Duron, but quality isn't as good). Unfortunately, the interface is not the prettiest I saw... Hope this will change, by implemeting soon the same codec in iTunes.

Nice test, Roberto.

A few comments:

1. I think classical and jazz could have been better represented.

2. It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez. You might link to his listening results.

3. The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test.

4. You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used. The similar presentation format might make people think that all the analysis was identical. The difference is mainly one about risk. The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs. On the other hand, it's more sensitive than the Tukey HSD.

5. I'm still uncomfortable with the squishy way that a summary graph is constructed. But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.

6. In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result. Also, I'd like to see some mention of the results you threw out for rating the original less than 5.


Some ideas for a future test:

1. Perhaps another call for samples -- classical and jazz samples -- would be profitable.

2. You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA. This can be done with a small filesize penalty using Sox. That would help to keep the ratings in perspective.

3. Verifying VBR average bitrates: I think that this task could be split up among several people, each encoding whole albums with all codecs.

ff123

Edit: Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.

Edit2: The next test you'll probably want to be sure to check for level (volume) differences too.

ff123,

What do you think about 'eyeballing' the results to get the ranks. Wouldn't a straight ranking be more solid and not necessarily less powerfull? I see the eyeballing was done in the 64kbps test also.

What about a bootstrap analysis of the results. Possible?

This is true. The QuickTime interface does not allow the batch processing of files and therefore it is not easy to use it for mass encoding of CDs. Also note that on MacOS X the iTunes application does not allow the use of the highest quality setting of the AAC codec and therefore the only option left for someone is to use the Ovolab AAChoo front-end

http://www.ovolab.com/

for batch encoding at any quality setting. This is what I use for my iPod encodings and it works well. Still, I hope that the next version of iTunes will allow the setting of the encoding quality. I also hope that the next QuickTime revision will support VBR AAC encodings...

The problem I was having with just using the data directly to get an overall summary is the idea that one sample might have a greater influence on the overall results than another. For example if one sample had a clear winner with a mean of 4.5, but another sample had a different clear winner with a mean of 2.5, I was thinking that the first sample would be given more weight in the overall results.

However, perhaps that's not really an issue when using a "blocked" analysis, which is supposed to take care of such things. Different listeners have varying internal quality scales. So I suppose that's analagous to the different samples having varying difficulty levels.

In that case, I would just try the exact same ANOVA / Fisher LSD on the data directly.

Problem with bootstrap is that it doesn't easily give you the nice 95% confidence intervals for the graphs. Also, it's similar to the Tukey HSD in being more conservative with risk (and less sensitive).

ff123

Edit: Roberto, don't forget to include the ATrain and Layla numbers when you do this analysis

Maybe compare AAC and Ogg Vorbis (both GT3b1 and latest CVS) @ 128kbps ?

Learn to read! It was mentioned in the original announcement, that the winner of the AAC test will compete against MP3, Musepack and Vorbis. I'd probably try GT2's 128kbps mode too.

dev0

I would definetly _not_ use any of the tuned Vorbis encoders for 128kbps. Just plain 1.0 -q4 should be best.

I'm not agree. On most samples (i.e. not problematic), Ogg GT2 128 seems to be more transparent, due to the hiss/noise issue of 1.0 release.
With killer or problem sample, 1.0 may win (not sure).

Yeah I'd Never Expect QT to come in as top cuz i thought there audio support was crap. The whole domination over movie trailers is pretty good though (QT trailers dominate WMP or REAL). Do you think Apple Store rips their albums using Quicktime? And which version and up supports AAC ripping (obviously Pro)?

Yes, but doesn't GT2's 128kbps mode give more than 128k on average?

GT2 128kbps is Vorbis RC2 128kbps with lossless channel coupling

I would be surprised if that's still state-of-the-art as far as Vorbis at the most common bitrate goes.

Don't really know. I haven't played a lot with vorbis, especially at this bitrate.
I encoded the 12 samples with GT2 (bitrate is of course not representative of a general behaviour) :




==> 136.75 kbps (according to foobar2000)

I can only support Guru's comment: GT2's 128kbps mode seems to be the most transparent Vorbis encoder at around 130kbps at least in the few tests I've done. Maybe there should be some extra testing done about GT2 vs. Post-1.0-CVS Vorbis at 128kbps before deciding which one to use for the big 128kbps codec shoot out.

dev0

I can't imagine a big listening test in july 2003, including an early codec, dated from november 2001. Most people will use a CVS or PostCVS (never heard anything different) version for their ~128 encodings.

Nevertheless, a dedicated listening test between GT2 and 1.0 should be started. If conclusions are in favor of the old codec version, it may convinced Xiph people of some flaws in their 'final' codec. The noise issue is acceptable at low bitrate (50-90 kbps), and I prefered some hiss that metallic distorsion. But at ~120 kbps, I prefer some occasional distorsion than a continuous noise, perceptible on most music. Especially when I tested QT encondings, hiss-free and with very few artifacts.

AACenc = Sorenson ?

PsyTEL AACenc.exe version 2.15

ahh, ok thanks

also, would interresting to se if/how much worse psytel fastenc is compared to the normal psytel...

Quick and dirty new analysis, in response to complaints about the "eyeball" method of ranking

Substituting all 4.5's for the two samples in which means are not shown, I quickly came up with the following graph for the overall results, using ANOVA / Fisher LSD instead of the "eyeball" ranking method:

http://ff123.net/export/aac.png

The rankings are about the same as before, with QT the clear winner, FAAC the clear loser, and the rest tied. However, this method has the advantage of showing how close the codecs are to the reference score of 5.0

In a future test, I recommend using an anchor (filtered version of the original) to put things in better perspective.

Edit: the graph label should be corrected to read "Rating" instead of "Ranking"

Back to some results.

I'm really surprised to see BeautySlept notation. I tested this sample in two test, this one and a preliminary one.


* Ahead MP4 Codec was an older version
** FAAC was not tested previously

I systematically found all encoding, except QuickTime, more than annoying. And I can't bear Ahead codec distorsions ! Even with VBR (-streaming) , Ahead was rated 2.5 and PsyTEL 3.5 (QT = 5.0 on this test !).
Can't really explain why. It is because I'm most familiar with harpsichord, and can't bear any distorsions ? Or maybe headphone issues ? In fact, I tried to perform this test with a set of poor earbuds (Sennheiser MX-500), began with BeautySlept sample, and found all codecs, including Ahead & FAAC, to be near transparency ! I gave up on this result... With a good set of headphone, the distorsion was not only easily perceptible, but awfully annoying.

P.S. According to the general conclusion, BeautySlept is one of the two samples where FAAC provide a better sound than Nero encoding (3.81 vs 3.92 - second one is LifeShatters, 3.88 vs 3.89)

Thanks to Roberto and know that my admiration continues to grow. I think this is fascinating.
1) Any speculation as to why ATrain and Layla were so easy for AAC? Surely they were chosen because the samples were challenging to MP3, Ogg and MPC. (I'm guessing the next test will address issues of what samples are handled relatively better or worse by the codecs.)
2) What encoder does AOL use for the AAC tracks streamed via Radio@AOL?
3) One guesses that Apple is devoting considerable resources to further development. Are Ahead and/or Sorenson doing sufficient work to make one expect substantial improvements in their offerings? [Afterwards addition: OK - Ivan's quick response and comments suggest Ahead is -- tnx, Ivan!] Is there another developer readying an offering? What prevents Dolby from doing so?
4) Do any or all of these codecs provide enhanced capacities for "digital rights management" vs. what wma already has and what mpc and ogg can offer? Or is that a matter dependent solely on the OS platform?
Adding an off-topic rant, I wish there were a way to build support for MPC. "As flies to wanton boys are we to the gods, they kill us for the sport," is my impression of the corporate Olympians' treatment of the public. "The public be damned, we'll tell them what they want," is the way media and software conglomerates are handling the evolution of codecs.

Guruboolez, ask Menno for the internal NERO plug-in update - I think I found the reason for the BeautySlept problem, and it should be much better now -

File was ended up heavily undercoded at 112 kbps - clearly a CBR bug (which is being fixed now)



Atrain has relatively low perceptual entropy, making it easy to encode at 128 kbps for AAC - also, most passages are "noisy" which gives more masking abilities for the psych model. Same goes for layla (noisy)

Fixed. Thank-you very much.



Only licensing issues. It's not warez, or pirate, because the copyright owner (Ivan) allowed it to be distributed.

Thanks.

2 more pieces of info: who submitted it (if it's known), and what's the style?

Fixed (i think)

Thanks a lot.

It's a FhG codec improved for speed. And, miraculously, it also improves quality while at it. :B

[quote=ff123,Jun 16 2003 - 08:08 AM] 1. I think classical and jazz could have been better represented. [/quote]
Well, Lizst and Bach were removed because people guessed they would be too easy, and therefore not significative.

Besides, we needed to make space for problem samples, and JohnV wouldn't accept a test with more than 12 samples.

[quote]2.  It should be noted somewhere, probably in the recommendations section, that this was a CBR test only, and that Nero and Psytel also have VBR modes, which perform better, according to Guruboolez.  You might link to his listening results.[/quote]

Addressed. Thanks.

[quote]3.  The crack about people advertising for FAAC is unneccessary. and doesn't help you win over a certain enthusiast to participate in your next test. [/quote]

He didn't participate in this one already. :-P

[quote]4.  You mention that you used an ANOVA analysis, but maybe you should also mention that this is different from what the 64 kbit/s test used.  The similar presentation format might make people think that all the analysis was identical.  The difference is mainly one about risk.  The ANOVA / Fisher LSD method is more at risk for falsely identifying differences between codecs.  On the other hand, it's more sensitive than the Tukey HSD.[/quote]

Addressed

[quote]5.  I'm still uncomfortable with the squishy way that a summary graph is constructed.  But since I can't think of a better way, and people have a need to see things in one, concise picture, I suppose it must be that way.[/quote]

Well, I have no clue about statistics. If you have any idea to fix that, please inform me.

[quote]6.  In the more detailed pages to follow, I'd like to see some mention about how a time misalignment of only 25 msec spoiled at least one result.  Also, I'd like to see some mention of the results you threw out for rating the original less than 5.[/quote]

OK, I will do that.

Of course, I won't make public the names of the participants that rated the reference. That wouldn't be nice to them, I believe.

In your test, you had people rating the reference too?


[quote]1.  Perhaps another call for samples -- classical and jazz samples -- would be profitable.[/quote]

Well, that's OK, but I planned to use the same test suite, even with the samples that ended up "too transparent". Else, if I change the suite too much, whiners will say there's no significance between the first test and the extension. "Who knows if QuickTime would win in this new suite? neener-neener"

[quote]2.  You might think about adding at least one anchor sample -- a lowpassed version of the original, a la MUSHRA.  This can be done with a small filesize penalty using Sox.  That would help to keep the ratings in perspective.[/quote]

OK, I think I'll do that. Thanks for the idea.

[quote]3.  Verifying VBR average bitrates:  I think that this task could be split up among several people, each encoding whole albums with all codecs.[/quote]

Yes, I plan to start a taskforce on this later this month.

[quote]Edit:  Oh, and if iTunes doesn't use the same codec that you used for this test, I would make some mention of that fact too.[/quote]

Done

[quote]Edit2:  The next test you'll probably want to be sure to check for level (volume) differences too.[/quote]

OK, I will.

Regards;

Roberto.

I don't know who submitted it, but the style would be something along the lines of jazz-rock. Dave Matthews Band is very hard to categorize... (Incredible live band, by the way.)

Nick

That argument simply doesn't work at all.

If you believe another set of samples would produce a significant performance difference between the AAC codecs compared to what we've seen here, it follows directly from that the next test would be highly unfairly biased towards AAC if you did use the same samples.

Cradle Of Filth's harpsichord naturally comes from a synthesizer so there may be differences, whether surprising or subtle, with real harpsichord music.

I have to ask before I lose my mind, did the BeautySlept sample originate from me (in this thread) or is it a funny coincidence

No, they were chosen because they were featured in ff123's 64kbps test.

I will probably ditch them for the extension test though (thanks for the clarification, Garf)



No idea.



There's another developer, that actually offered to send me a demo encoder that I can use in some future listening test, but I don't know if I can decline informations now.

What prevents dolby from doing what?



There's no standardized DRM system for AAC. And no, none of these encoders offer the option of DRM'ing your AACs, differently from WMA/Windows Media Encoder.

Regards;

Roberto.

@NickSD: Thanks for the info.

I was usually embarassed by the synthetic consistency of this harpsichord (and by the metronomic performance of the player). Thank you for this confirmation
Here is a real harpsichord sample. Bitrate is higher than with BeautySlept (265 kbps with --standard and 1.15r). It's a real killer too for most codec, if you want to reach 128 kbps and not more. I strongly suggest anyone to test it with WMA9 (PRO), incredibly transparent at 128 kbps compared to all other format/codecs I heard.

http://membres.lycos.fr/guruboolez/AUDIO/s...chpsichord.flac

I think the best way is to just use the same procedure that you did for each sample. But you'll have to calculate the mean scores for ATrain and Layla first (easiest to just use Excel for that).

ff123

Regarding 41_30sec.wav


Originally submitted by Filburt

style: rock

Does ANOVA take into consideration the varying scales that people use when they grade the test samples? For example someone who thinks that the goal of the codec is to achive total transparancy may grade everything between 1 and 3 when someone else who doesn't care that much would give grades between 4 and 5 to the same set.

So could someone very briefly explain how ANOVA works (or post some good links)? Which types of data sets is it possible to use ANOVA on?

Does anyone know anything about the AAC codec that XM is using to broadcast its satellite radio?

XM AAC Information

It apppears as though it is heavily customized, but I wonder if it originally came from anything used in this listening test.

- Agent 86

XM is probably using gear from Thales
http://www.thales-bm.com/
Probably this baby.
It's the same gear most DRM stations are using.

Thales uses AAC encoder from FhG and SBR encoder from CT. Probably integerized versions.

Very interesting. I'm quite disappointed in Nero, yet another reason I just keep re-using their trial. :roll:

I'm starting to wish my player supported AAC now.