Personal evaluation at ~130..135 kbps, 200 samples, AAC (iTunes, Nero) - MP3 - Vorbis aoTuV Options

[yulyo!]

Hy guys,
I feel terrible.
I feel that i am the one who started those ...crazy discussions. With only one or two replies (and one mistake )
I think we should wait for the next 128kbs test and let Ivan and Garf improve their work.
Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY and with guruboolez's ears and equipament ONLY.
Yes, is surprising that Nero AAC is fighting to beat Lame and not Vorbis, but this is it. For now.
So, in conclusion, i think we should wait a few days for Sebastian's test and then start those contradictions again

[guruboolez]

Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY

It's precisely ONE thing you can immediately forget...
http://audiotests.free.fr/tests/200...results_gr5.png

Take a look on results before commenting them, please


This post has been edited by guruboolez: Dec 29 2005, 23:01

[lexor]

Also, we don't have to forget ONE thing: this test was performed on clasical music ONLY and with guruboolez's ears and equipament ONLY.

did you read the write up?

/EDIT: oops, I guess I should read replies before posting mine, guru got there first.

This post has been edited by lexor: Nov 19 2005, 21:56

[clintb]

Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME. Is that correct?

[smz]

Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME.  Is that correct?

My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

Quite embarassing for a strong LAME supporter (like myself) and an an allergic to apples (like the very same myself).

Sergio

[clintb]

Ok, maybe I'm dense, but barring that, my ability to interpret the sea of numbers presented in the graphs is telling me that iTunes is coming super close to LAME.  Is that correct?

My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

Quite embarassing for a strong LAME supporter (like myself) and an an allergic to apples (like the very same myself).

Sergio

And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod. Might have to try out 160K VBR iTunes and get back space for photos and videos.

[smz]

And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod.  Might have to try out 160K VBR iTunes and get back space for photos and videos.

I hope you kept your FLACs, like I keep my WavPaks...

[QuantumKnot]

Garf, Nero AAC may to some extend be your baby.... however, a certain kind of reaction from devs towards listening tests (i'm primarily not talking about the "what" but the "how") torpedoed musepacks public image on ha.org not so long ago.

I agree. I think it is esp. good for devs to welcome listening tests by other people who volunteer to do quality-testing for you. I know Aoyumi follows these tests very closely and I believe it benefits his Vorbis development.

I just wanted to make a comment about iTunes/QT AAC. As guru pointed out in a previous reply, I am somewhat amazed at how its ABR-like nature can give such great quality. In fact, I've made the comment long ago about iTunes AAC on castanets compared with Vorbis at 128 kbps. iTunes/AAC barely moved in bitrate, yet produced much lower pre-echo than vanilla Vorbis with inflated bitrates. When I did those ABX tests last night, I noticed that the iTunes files had very minimal bitrate fluctuation, yet I was unable to ABX it against the original. Is this magic or something? The devs at Apple certainly are doing a great job, esp when they are giving their AAC encoder away for free.

[ErikS]

guruboolez great test, you trully have the patience of Job.

Also, Garf claimed several times that "most people" would perceive new encoder as better, is he referring to the old (Roberto and Co.) listening tests or is there another study I've missed? Anyone has a link to the study in which those "most" people participated?

Nobody has so far participated in a listening test proving that "most people" are considering the new encoder as better than the previous one. Not even me. Not on HA.org, Doom9 or any other known board or website.
He's only referring to his own beliefs, disguised as valid and general claims.

I bet he listened to it several times himself and also let other people at Ahead listen. Just that those listening tests are not public...

[clintb]

And I just did a FLAC>3.97b1 -V 2 --vbr new conversion on all my music for the new 5th Gen 60GB iPod.  Might have to try out 160K VBR iTunes and get back space for photos and videos.

I hope you kept your FLACs, like I keep my WavPaks...

Oh man, after all that EAC ripping to cue/single files, proper case for the titles, checking date and genre...yeah, the FLAC files are kept and doubly so. Once, on the primary/usage drive, secondly on an external drive and soon on DVD.

BTW, anybody looking for good DVD media to backup, check Micro Center (www.microcenter.com) for their in-house brand "WinData"....it's TY. Right now, they're $8.99 for a 50pk spindle. That's right, $8.99 for 50 Tayo Yuden 8x DVD+R.

[clintb]

Garf, Nero AAC may to some extend be your baby.... however, a certain kind of reaction from devs towards listening tests (i'm primarily not talking about the "what" but the "how") torpedoed musepacks public image on ha.org not so long ago.

I agree. I think it is esp. good for devs to welcome listening tests by other people who volunteer to do quality-testing for you. I know Aoyumi follows these tests very closely and I believe it benefits his Vorbis development.

I just wanted to make a comment about iTunes/QT AAC. As guru pointed out in a previous reply, I am somewhat amazed at how its ABR-like nature can give such great quality. In fact, I've made the comment long ago about iTunes AAC on castanets compared with Vorbis at 128 kbps. iTunes/AAC barely moved in bitrate, yet produced much lower pre-echo than vanilla Vorbis with inflated bitrates. When I did those ABX tests last night, I noticed that the iTunes files had very minimal bitrate fluctuation, yet I was unable to ABX it against the original. Is this magic or something? The devs at Apple certainly are doing a great job, esp when they are giving their AAC encoder away for free.

This brings up a good question: Who's doing the development on the iTunes AAC encoder? Anybody that frequents this board, I wonder?

This post has been edited by clintb: Nov 20 2005, 02:57

[Nayru]

I asume i'm not alone with this impression when i say that AAC has been in the making for a LONG time, it has been sold to us as the "next big thing" and "the future of lossy encoding".... and when during the previous years the tests didn't look good, we were told over and over "well, its still new and requires some more tuning and more research, then it will rock"..... with nero AAC, this is what we've been told for almost EVERY upcoming version: "yes, there are some errors, but the next version will be much better and probably fix this and that"....

Oh, Nero claims much more than that. For a good laugh read: http://ww2.nero.com/nerodigital/enu/Nero_D...highlights.html

"CD quality stereo at 48 kb/s" "Transparent quality at 128 kb/s" "MP3 quality with 50 % of the space."

It's not surprising that customers are upset when they find out it's not true.

Now, when after years in the medium bitrate arena quicktime AAC can barely compete with vorbis, which does NOT have many of the patented toys available to AAC...... and nero AAC can barely beat LAME-MP3.... the format which supposedly is "obsolete"...... then maybe it's no surprise that people aren't very euphorous about the performance of AAC, especially nero AAC? Or is this encoder only gonna be useful for narrowband-scenarios? (question is intentionally worded in a provocative way)

Both AAC and Vorbis use quantized MDCT with two different block sizes, followed by Huffman coding. I would be surprised if there was a substantial difference in coding efficiency between the two formats. What are the patented toys available to AAC? SBR? LTP? SSR? It'd be interesting to know how much improvement those actually make...

[rjamorim]

This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?

Yes. But he prefers to remain anonymous because of some scary Apple policies.

[clintb]

This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?

Yes. But he prefers to remain anonymous because of some scary Apple policies.

He? It's not one person, is it? If so, hat's off to him for a fine job.

[Maurits]

This brings up a good question: Who's doing the development on the iTunes AAC encoder?  Anybody that frequents this board, I wonder?

Yes. But he prefers to remain anonymous because of some scary Apple policies.

And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple.

[de Mon]

Both AAC and Vorbis use quantized MDCT with two different block sizes, followed by Huffman coding.  I would be surprised if there was a substantial difference in coding efficiency between the two formats.  What are the patented toys available to AAC?  SBR?  LTP?  SSR?  It'd be interesting to know how much improvement those actually make...

http://www.hydrogenaudio.org/forums/index....showtopic=34075

[rjamorim]

He?  It's not one person, is it?  If so, hat's off to him for a fine job.

It's one person (that I know of) that is member of this forum. But I suspect he's not alone working on the AAC encoder...

And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple. 

I bullied him already :B

He said he wouldn't mind including it in the encoder, but it doesn't depend only on him. It also depends on the QuickTime division, the iTunes division, the iPod division... For that reason, it'll probably only happen when a decision comes from above.

[Maurits]

And probably because he'd be bullied by hundreds on this board about when gapless playback is going to be supported by Apple. 

I bullied him already :B

He said he wouldn't mind including it in the encoder, but it doesn't depend only on him. It also depends on the QuickTime division, the iTunes division, the iPod division... For that reason, it'll probably only happen when a decision comes from above.

I can imagine that being the problem. I think it would be relatively easy to implement it in the encoder but harder (more complex) to implement it in the decoders in iTunes and iPod. It would be odd when the files made by iTunes support gapless but iTunes wouldn't be able to playback gapless. Or what if the iPod could play gapless but iTunes couldn't, or the other way around, or...or...

I believe Apple said they found gapless playback irrelevant. That's usually Applespeak for "We're going to introduce it very soon..."

[sTisTi]

BTW, anybody looking for good DVD media to backup, check Micro Center (www.microcenter.com) for their in-house brand "WinData"....it's TY.  Right now, they're $8.99 for a 50pk spindle.  That's right, $8.99 for 50 Tayo Yuden 8x DVD+R.

Completely OT here, but are you sure these are not so-called "fake" TY? There has been a lengthy discussion about this at cdfreaks, you might want to check it out before trusting your data to these media.

[guruboolez]

Would it be possible for you to create some small webpage pointing to your tests, like the one of rjamorim, so its easy to see whats going on and compare the results aso... atm it seems more to me your results are vanishing in the depths of the forum
if you cant make a own page, maybe you can create some thread carrying that info?

Yes, I could.

vorbis GT2 vs. vorbis PCVS - 12 samples
extension of the AAC 128 kbs LT - part 2
http://www.hydrogenaudio.org/forums/index....topic=10555&hl=

WMA9 vs WMA9PRO 12 samples test
extension of the AAC 128 kbps LT.
http://www.hydrogenaudio.org/forums/index....topic=10551&hl=

Personal multiformat listening test at ~130 kbps
based on classical (baroque) music only
http://www.hydrogenaudio.org/forums/index....topic=14091&hl=

MP3 decoders test : MAD isn't so good! (for me...)
MAD vs LAME vs Winamp 5 vs foobar2000
http://www.hydrogenaudio.org/forums/index....topic=17728&hl=

128 kb Multiformat listening test...
...based on classical music samples ONLY
http://www.hydrogenaudio.org/forums/index....topic=16395&hl=

listening test at 160 kbps
pre-echo with aoTuV, GT3...
http://www.hydrogenaudio.org/forums/index....topic=22495&hl=

another lossless performance comparison
...but on classical music only
http://www.hydrogenaudio.org/forums/index....topic=28441&hl=

lame 3.90.3 vs lame 3.96.1 at ~130 kbps
ABR and VBR
http://www.hydrogenaudio.org/forums/index....topic=29422&hl=


MPC vs OGG VORBIS vs MP3 at 175 kbps
listening test on non-killer samples
http://www.hydrogenaudio.org/forums/index....topic=23355&hl=

AAC: Ahead vs Apple (end 2004)
one year of progress
http://www.hydrogenaudio.org/forums/index....topic=29924&hl=

lame 3.97 alpha 5 testing thread
tests & results
http://www.hydrogenaudio.org/forums/index....topic=30547&hl=

Vorbis quality – wrong direction?
RC3 against post-final encoder
http://www.hydrogenaudio.org/forums/index....topic=18359&hl=

Ahead AAC VBR vs Vorbis aoTuV beta 3
at ~130 kbps with classical music
http://www.hydrogenaudio.org/forums/index....topic=29925&hl=

Short re-encoding blind listening test
wavpack - mp3 - mpc - aac - vorbis
http://www.hydrogenaudio.org/forums/index....topic=32440&hl=

Ogg Vorbis and Nero AAC
microattacks & microdetails VBR 140 kbps
http://www.hydrogenaudio.org/forums/index....topic=32080&hl=

1.01j encoder vs 1.15u
listening test inside
http://www.hydrogenaudio.org/forums/index....topic=34911&hl=

80 kbps personal listening test (summer 2005)
AAC MP3 Ogg Vorbis WMA
http://www.hydrogenaudio.org/forums/index....topic=35438&hl=

lame 3.98 alpha 2 testing thread
vbr new & athaa-sensitivity test at V5
http://www.hydrogenaudio.org/forums/index....topic=37973&hl=

HE-AAC v.1 & v.2 comparison
Winamp vs Helix vs Nero Digital
http://www.hydrogenaudio.org/forums/index....topic=36868&hl=

MPC vs VORBIS vs MP3 vs AAC at 180 kbps
2nd checkup with classical music
http://www.hydrogenaudio.org/forums/index....topic=36465&hl=








____
LAME ALPHA
http://www.hydrogenaudio.org/forums/index....81&#entry300681


DIRECT LINKS:


• alpha 5 -V4 - 20 samples and alpha 5 -V5 - 6 samples
• alpha 6 -V2 - 20 samples
• alpha 7 -V4 - 54 samples
• alpha 8 -V3 - 20 samples and alpha 8 -V2 - 20 samples


Lame test version - may 2005
http://www.hydrogenaudio.org/forums/index....ndpost&p=300065

Lame test version (June 2005)
http://www.hydrogenaudio.org/forums/index....ndpost&p=304757

wouldnt it be more interesting to compare helix/ct to apple in the future and not always nero over and over again? (i know nero is well established (again is used in sebastians public test), but this doesnt mean that there arent better codecs)

Winamp (Coding Tech & Dolby) AAC implementations are less interesting than Nero Digital. It has nothing to do with quality, but with progress. Nero Digital is updated more often. To give you a simple fact: Winamp still embbed Dolby AAC 1.0 for two years now! No update since...
And Coding Tech AAC encoder is still handicaped by a 15 KHz lowpass, making this encoder too easy to detect on ABCHR evaluation. You can test it if you want. But in my opinion, CT AAC encoder (Winamp, Helix) is more interesting for HE profile than LC one.

I'm testing Nero Digital because it's often updated. It's also interesting because Ivan is present on the forum; he could comment the test and propose some improved encoders. Exactly like LAME developers and Aoyumi. There were also a lot of changes during three years. The first encoders I've tested were really poor with classical (see my first tests); the lastest ones are much better. Too bad that the Nero Digital team has released an old generation encoder with a version number corresponding to a major release (aacenc32 v.4 and aac.dll v.3). The current new generation encoders shows interesting improvements compared to the one I've tested here:
- better handling of very tonal signal
- no ringing anymore on low volume part
- much less distortions with harpsichord.

And compared to my beloved old aacenc32 v3.xxx "fast", the new encoder has no bloated bitrate anymore with some kind of sample/music and is able to produce the same kind of high quality.


I'm currently sick with listening test. I must force myself to finish Sebastian's 18 samples. I can't make extensive listening test of any encoder now, including latests lame alpha or newest Nero encoder. I can only perform quick and small comparison. It's enough to appreciate some improvements of latest Nero Digital, but not enough to perform a complete and rigorous evaluation.

[Gambit]

The highly interesting off-topic discussion was split here.

[bond]

well my proposal for guru to create a small webpage with his tests and to maybe use ct in future comparisons was split too, i hope guru might still consider it

[eltoder]

Fantastic job, guru. Don't you have some kind of HA award yet?

The good surprise comes from LAME MP3, which get the best mark (3,95)

Am I missing something, or plot says that it's Vorbis who get 3,95 and LAME get 3,94?

[pepoluan]

Yes, I could.

vorbis GT2 vs. vorbis PCVS - 12 samples
extension of the AAC 128 kbs LT - part 2
http://www.hydrogenaudio.org/forums/index....topic=10555&hl=

... and so on...

Whoa... a treasure trove of listening tests...

Guruboolez, if you'll be kind enough to indicate the date of each listening test you did, I'll gladly add them to the ever-expanding list of listening tests at this HA Wiki page.

Oh and PM me when you do that. So I know when to recheck this thread :-)

[evereux]

Check the date of the original posts?