IMPORTANT WARNINGS

• this test is unfinished
• this test is about “classical music” samples, nothing else
• choosen VBR settings are pertinent and fair with a given set of samples or musical genre (in fact, classical music) but are unlikely to work with different musical genres
• one person and only one contributed to this test. For collective listening tests, go on Roberto Amorim and Sebastian Mares websites

INTRODUCTION

My latest fad is to ask Santa Clauss one of those big iPod (160 GB) and then fit my whole musical collection and access to every CD of mine everywhere and without any computer help. My plan is to have everything at ~130 kbps which should be enough in most situation.
My last evaluation at this bitrate is two years old now. There wasn't any revolution in the meantime but all tested encoders are now outdated. That's why I decided to evaluate current encoders a second time and to restrict the test to iPod compatible formats: MP3 and AAC. Excellent formats such as Vorbis or WMAPro are for this sole reason NOT included in this 2007 evaluation.

(Un)fortunately new encoders were released or announced before the test completion (which needs almost 2 full weeks to be performed): a new LAME beta version, a new QuickTime AAC on Macintosh system. I wouldn't spend so many hours to test the 80 remaining samples with encoders that are not up-to-date anymore. I therefore cancelled the test. Some results are nevertheless very interesting - that's why I decided to post them on the forum.

TEST SETTINGS

As previous tests of mine, 150 classical music samples are supposed to feed my curiosity. I tested the half.
Hardware settings are: Terratec DMX6Fire 24/96 soundcard, Onkyo R-A5 Amplifier, Beyerdynamic DT-531 headphones. Testing rythm: ~10 samples per day.

CHOICE OF ENCODERS AND POOLS

I decided to not close the door to any modern encoders. That's why Fraunhofer MP3 iTunes MP3, Helix [Real] MP3, Coding Technologies [Winamp] AAC encoders are competing here. But all couldn't be seriously compared each others and accurately ranked. Not 150 times at least. A pre-selection was needed and that's why I started the test with smaller MP3 and AAC pools (25 samples in each).

---- In MP3 pool:
• Fraunhofer v.1.4 (encoder date: 2007.05.18 ; package from 2007.07.11): VBR -m3 -q1
• Helix MP3 v.5.1: VBR -V65
• iTunes v.7.4.3.1 MP3: VBR 128 highest
• LAME v.3.98 beta 5: VBR -V5



No real surprises here. LAME is better than any other competitors with a confidence >95% and will therefore compete with AAC in final test. iTunes and Fhg performances are similar and aren't globally bad; but both are more unstable (or less robust against encoding difficulties) than LAME. Helix is considerably worse and finishes last.


---- In AAC pool

I exempted iTunes from the selections due to its excellent past performances. It will be opposed in the global evaluation with the winner of this pool. I know two serious challengers: Coding Technologies and Nero Digital. The latter is represented in the pool by two releases: the last one and the previous one. I'm very suspicious about the last Nero's encoder since I discovered problems with it. This test and the direct comparison with the previous Nero Digital release is a good occasion to confirm or infirm my suspicions.

• Coding Technologies[encoder built in Winamp 5.5 beta, bit-identical output to final version]: CBR 128
• Nero Digital v.1.0.7.0 (February 2007): VBR -q 0,45
• Nero Digital v.1.1.3.4 (Aug 2007): VBR -q 0,44 (higher bitrate and better performance than -q0.45…)



This test confirms the existence of a quality gap between Nero Digital 1.1.34.2 and the previous version. This gap is really huge for my taste (and with classical music I recall). Something bad happens with last version and I hope it will fixed. Coding Technologies AAC's performances are rather good, especially for a CBR encoder. Nero 1.0.7.0 wins and will be opposed to iTunes AAC and LAME.

The final test will therefore be: iTunes AAC + Nero Digital 1.0.7.0 + LAME 3.98b5 + 2 anchors



CHOICE OF ANCHORS

For once I decided to be more practical than “scientifical” in the choice of anchors and take some risks.

As low anchor I used 96 kbps AAC. It's a (small) risk because this setting might give similar results to MP3 at 130 kbps. It's unlikely but not impossible. I was also very curious to check how would perform ~100 kbps by itself, and when compared to MP3. If low anchor appear to be “good enough” why not using it to fill a digital jukebox? iTunes can't be used here because last version forces 32000 Hz sampling rate (too easily noticeable for my taste on most situation). Nero Digital is configurable and offers VBR 96 kbps encodings. For obvious reasons I avoided last Nero implementation.

As high anchor I used 160 kbps MP3 (LAME -V4). The risk is higher IMO and I wouldn't be really surprised if any AAC implementation would reach or surpass even with a penalty of 30 kbps. But if MP3@160 would outperfom AAC@130 I would seriously consider the most compatible and universal format as final encoding choice for my future chritmas present.
• Nero Digital 1.0.7.0 -lc -q0,25
• LAME 3.95b5 -V4


The following evaluation can be read as:
- Nero AAC vs iTunes AAC vs LAME MP3
- Nero AAC at 100 kbps vs Nero AAC at 130 kbps
- LAME -V5 vs LAME -V4
- 100 kbps AAC vs 130 kbps MP3
- 130 kbps AAC vs 160 kbps MP3




RESULTS

Note: only plots are posted here. As unfinished test I won't spend too much time to format all results individually as I did in previous tests (like here and here).


I. CLASSICAL: 5 electronic/artificial samples



5 samples aren't enough to get statistically pertinent conclusions. With confidence we can say that LAME at 160 kbps is better than Nero AAC at 100 kbps (what a surprise...). I can also add that none of current AAC implementation is able to give good anything better than average results on these critical samples. I regret aoTuV Vorbis performance which was able to match (and even surpass) LAME -V2 at 128 kbps (nominal) (see here to get 2 years old plot).


IIa. CLASSICAL: 30 orchestral & chamber samples on modern instruments



With this group of 30 samples we have what people would consider as “classical” music: chamber and orchestral various compositions on modern instruments.
And here the first big change since my last evaluation: while iTunes still performs well (average mark is in progress) it's now clearly and statistically inferior to Nero Digital. The latter offers similar results to LAME at 160 kbps and is the only AAC implementation that appear to be more efficient than MP3. I must also recall that Nero Digital -q0,45 doesn't necessary offers ~130 kbps with other musical contents.


IIa. CLASSICAL: 30 orchestral & chamber samples on period instruments



Early instruments are usually harder to encode properly. It's confirmed here. LAME performances are significantly worse here. It's strange: 2 years ago LAME 3.98 alpha 2 was more constant and offered similar results between modern and period instruments recording. At least with -V5... With higher settings LAME also showed stronger regression in the past (-V2 with alpha 2 and now V4 with beta 5). LAME is here statistically tied with low anchor.
iTunes AAC is very constant. Nero Digital is now statistically tied with iTunes and with high anchor but ends this group with the best mark. It confirms on 30 additionnal samples how great it is.



VERY SMALL CONCLUSION


I haven't tested the whole “vocal music” group (30 samples) and most samples of the “solo instruments” (60). But intermediate results (20 samples on 60) of the later group are confirming previous ones:

LOW ANCHOR: 3.1
iTUNES AAC: 4.1
LAME MP3: 3.2
Nero AAC: 4.4
HIGH ANCHOR: 4.0

In other words, Nero Digital 1.0.7.0 -q0.45 looks like a jewel: better than iTunes 7.4.3.1 at the same bitrate, as good as MP3 at 160 kbps. Now it would be more than interesting to compare it to the newest QuickTime version (only available on Mac OS 10.5 ATM) and also expecting from Nero Team to fix current issues on 1.1.34.2.
… and hope from Apple they will correct all iPod classic (good name) issues with a new firmware.



___
appendix

bitrate table (150 full tracks, 16 hours of music from 150 classical music CD):
http://audiotests.free.fr/tests/2007.11/iPodTEST_bitrate.ods

2 gif files illustrating worrying ringing artefacts (+ agressive lowpass/ATH value) with Nero 1.1.34.2:
• E22 sample (0-22000 Hz)
• V19 sample (10000-19000 Hz zoom)

A great listening test. This shows that iTunes mp3 encoder is not that bad and Helix carries the Xing imfamus legacy . I have found that the latest verison of Nero to be poor compared to the Feb 2007 version a few months ago with some AAD mixed samples such Iron Man by Black Sabbath which produces a annoying metallic noise in the background.

Thanks.
I guess it would be better to wait for Sebastian's MP3 test (more listeners, different samples, but probably lower VBR setting) before stigmatising Helix.
iTunes also surprised me. It reacted differently from Fhg's encoder but finally ended with the same overall quality. It's worth noting that I used -m3 as VBR setting for Fraunhofer (which seems to be closer to 160 kbps than 130 kbps with most popular musical material - cf. AlexB bitrate table on the thread dedicated to the upcoming listening test). Would -m4 be as good as iTunes encoding? Maybe iTunes MP3 will appear to be the second best MP3 encoder in Sebastian's test.

It sure is better to wait for Sebastian's since some encoders are likely to perform better at different genres and samples, but am suprised that iTunes mp3 performed well since Apple would spend more time devloping their AAC encoder.

Made me re-think the use of Nero AAC 1.1+ instead of using 1.0.7.0. Think I will go back to using the older Feb 2007 release over Aug 2007 release. Good test by the way, well done.

EDIT: Its use for Classical Music that is.

I've merged all 85 scores and analysed it with friedman:



Graphical results looks like this:




It is safe (for me) to say that:
• Nero Digital (Feb 2007) is better on a wide selection of classical music than iTunes AAC and LAME MP3
• AAC@130 kbps (Nero only, classical music) may offer similar (and even slightly better) quality than top-MP3@160 kbps
• iTunes AAC is better than LAME -V5
• AAC@100 kbps is clearly lower quality than top-MP3@130 kbps.
• Nero@130 kbps > Nero@100 kbps
• MP3@160 > MP3@130 (trivial conclusion but would it appear on a collective listening tests involving several “non-critical” listeners?)

=> Nero Digital AAC (Feb 2007) is what I'll use first, though it's not perfect at this quality setting (-q0,45).

great test guru! wish i hadn't deleted my copy of Nero Digital v.1.0.7.0

Here ya go!

http://www.brothersoft.com/mp3_audio/rippe...udio_59677.html

It's more fun to get the upcoming new Apple's vbr aac encoder in this test in the near future. With all fixes of the regressions found maybe the next version of Nero digital would get much better, which is also worth waiting.

Using Nero Digital v.1.0.7.0 (February 2007): -q 0.45 I get bit rates around 155k, noticeably higher than 130k. Am I getting unusual results?

I don't think I have enough material to build a valid bitrate for non-classical music; from what I've tested (some old heavy metal stuff) it's indeed closer to 160 kbps than 130 kbps.
You finding are probably more a rule than an exception. I mentioned it in the test: the choosen VBR settings are based on average bitrate obtained with classical music, which is often (but not always) bitrate-friendly compared to modern or popular (i.e louder, less dynamical) music.

A small sample of classical tracks was generating values around 145k, but more testing brought the average closer to your range. Rock tends to be higher.

Great testing

Its a shame that Nero Digital v.1.0.7.0 has a track length bug that effects playes such as Quicktime, GStreamer FAAD and XMMS FAAD plugins which causes the player to cut off at the last few secends. Which has been fixed on Aug 2007 and also listed on their release notes.

I stay with lossless and wait for an update because in some very silent parts (also classical here) the encoder chooses very low bitrates even at q6, resulting in kind of digital hiss artifacts.

I just noticed it, in real listening conditions and not during a listening test. It occurs with quiet piano recordings with -q 0,45. Bitrate is low (I expect this from a VBR encoder), between 80 and 100 kbps; but quality is sometimes low too (I don't expect this from VBR which is supposed to maintain a constant quality)
I reached the best size/quality ratio with LAME -V5: ~100 kbps with this material and close to transparency files (but not always transparent). The most reliable VBR LC-AAC encoder in this situation seems to be iTunes which isn't (unfortunately) the most efficient one.
I'm now mixing the encoders for my growing M4A library.

Guruboolez, have you tried the new AAC encoder in iTunes 7.5 / QuickTime 7.3 yet? I've encoded about 20 CDs from various genres to do comparisons... and I'm finding that at the 128 VBR setting, I'm getting lower bitrate and lower quality files than I did with the same settings in iTunes 7.4 / QuickTime 7.2. I'm not entirely sure what the changes are... but this new encoder seems to be a step backward... especially when you consider that upping the settings to 160 VBR creates a significantly higher bitrate and larger sized file.

No, I haven't try yet and I don't think I can test it before christmas.
But as I plan to use iTunes soon for encoding several discs (piano music for which Nero Digital lowers the bitrate as well as the sound quality; harpsichord discs too for which iTunes@160 sounds better than Nero@~170) it would probably wise for me to check for possible issues with the newer encoder, especially after your interesting report.

There's actually four encoding mode with the latest QuickTime aac codec: true vbr, cbr, abr, constrained vbr. I'm not sure which mode is the default vbr mode for iTunes.

guruboolez, I suggest you checking all 3 vbr modes when you do the test.

I will only use and therefore test what I can use in batch encoding. An encoding mode available in QuickTime only is pointless for CD ripping / batch encoding purpose - unless someone will provide a tool alla iTunes_encode to exploit it in third-party apps.
I would be disappointed if iTunes didn't include the most interesting (true VBR) encoding mode available in QuickTime.

I'm fairly certain that iTunes is and always has used the constrained VBR setting.

It's low anchor settings. Doesn't default -q 0.34/0.35 produce 96-100 kbit/s LC?

I'm fairly certain that iTunes is and always has used the constrained VBR setting.
I share this. The new iTunes uses a constrained VBR setting too; it's confirmed here.

It's low anchor settings. Doesn't default -q 0.34/0.35 produce 96-100 kbit/s LC?
I recall that I'm not using -q,025 but -q0.25 -lc. The -lc or -he settings forces the use of a different bitrate scale.
I tried to reach ~96kbps with the regular scale (it was indeed with something near to -q0.35) and the encoder forces the use of HE-AAC - which is pointless on an iPod compatible format setting. That's why I had to find the adequate -q level with the forced -lc mode.

Christmas is coming up, and so is that iPod of yours ;-)
Any progress on your interesting effort, Guruboolez?

is the test finished now ?

Has only the newest version now, here's a working link for the older version:

http://download.chip.eu/de/Nero-AAC-Encode...7.0_155834.html

Thanks for your test, guru!

Hi...
I want to know about the: MP3 Test: 128KBIT-CBR-Standard Stereo with latest FhG & LAME encoders.
Is this possible?

Using an Ipod now, so it would interest me a lot to hear any updated version of this test, to reflect the newest Itunes-QT, the newest Nero Digital, etc.

For instance, is the older Nero Digital still better than the current new version? (If so, has anyone contacted Nero to tell them about the defects in the newer versions? It shouldn't get worse.)

It would also be interesting to hear how the current Itunes stacks up against the current version of Nero Digital. Itunes is certainly more convenient to use. (Do you use some kind of GUI with Nero Digital? Or straight command-line usage?)

I think you wrote the Itunes encoding was VBR. Isn't the default encoding in Itunes CBR? How do you change it to VBR? Also, I read in another thread that Itunes VBR is constrained VBR? Is that still the case in the newest version? Could that be the cause of not winning against Nero?

Thank you for your insight.

Of course we are aware of the problem and trying to fix it.
See this thread: http://www.hydrogenaudio.org/forums/index....=14&t=61724
So far there were not many reports with provided samples. Only 2.
We ask everybody who can find a sample please to provide it in that thread.


For example Exact Audio Copy can be used with Nero AAC Encoder. Nero AAC Encoder can of course be used in Nero Burning Rom - there is an option in Extras "Save Tracks" where, among other formats, Nero Digital Audio is available. (There is a limitation that only VBR AAC can be used in Nero 8 and it will be fixed in Nero 9 allowing also CBR and ABR)

I just re-encoded some tracks to see what I got. With the current Apple encoder I get higher bitrates on the sample I tried -- recordings of Clifford Curzon playing piano. With an older version of Apple's AAC I had got: 165, 164, 163, 162, 161, 160, 160. With the latest I get: 175, 173, 170, 170, 168, 168.

I didn't make a note of when I made the original encoding. In both cases the encodings were made using Max, which uses Apple's Core Audio. The interface in Max allows one to access options that are in Core Audio but not revealed in iTunes GUI. (Presumably, Apple, like the GNOME people, consider a simpler UI with fewer options is better.) Anyway, in both cases the encoding chosen was 160 kbps + VBR + a "Maximum quality" switch. (Core audio, apparently, has -- had? -- five different switches for VBR, although you can only either enable or not enable VBR in iTunes.)

In general, I'd say Core Audio's AAC was giving me bitrates less than I seem to be getting with the latest version with the settings I use. With Nero's AAC encoder (latest version) at q 0.5 I often get bitrates of 145 or 150 with quiet stuff -- particularly guitar music -- it can drop down to 130 or less. Steve Hackett's latest album "Tribute", where he plays various Bach pieces and so forth gets encoded by Nero at around 110 kbps.



Ouch! That's a very interesting observation. I noticed something like that the other day and wondered if it were on the original CD but didn't get around to checking. This was on an encoding I'd made with Ogg Vorbis for my Linux box. I hadn't recalled hearing hissing when listening to the CD (or the AAC version on my Mac/iPod), but, as I say, when I noticed the hiss I more-or-less assumed it must be on the CD. perhaps not, then. I'll have to look into that.



That's another very interesting comment to come out of this thread. I guess the choice between reliability and efficiency isn't an obvious one at all -- specially, if space is not an issue.

AFAIK Max can use different profiles than iTunes to encode, i.e. Max, through Core Audio, has access to the "true" VBR profile in QuickTime, while iTunes uses the "VBR_Constrained" profile. Therefore I would expect very different results using Max than iTunes, even using the same version of QuickTime.

Since a vast majority of iTunes users are on Windows, expect most listening tests of "iTunes" to use the "VBR_Constrained" profile. The "true" VBR used by Core Audio may sound better... but that won't do me much good unless I purchase a mac.

All that being said... it seems like guru has been mysteriously absent for some time... I hope all is well...

Maybe. But I think these five options in Max may all be different adjustments you can make to the "VBR_Constrained" mode. I know there have always been more options available in Core Audio (and accessible from the CLI or from something like Max) than were revealed in iTunes, but the "true" VBR is quite new. And actually, I'm not even sure that the current version of Max can access the "true" VBR: there's not been a release version of Max for quite some time.

According to a thread here XLossless Decoder can access the true VBR mode:

http://www.hydrogenaudio.org/forums/index....w=&st=&

Here's the program:

http://tmkk.hp.infoseek.co.jp/xld/index_e.html



That's a good point: it's not of so much interest if most users can't access it.

But I'm not sure it would sound better, anyway -- perhaps, to go back to those earlier terms, it also would be more "efficient" but not more "reliable". And if most people can't access it and can't use it how would Apple get any feedback on its performance?

I want to check the bitrate of nero aac encoded files. How can I check this?
I curious about the bug of 3.1.0.2 version (http://www.listening-tests.info/mf-128-1/miscellaneous/nero.txt) exists in the February 2007 release? If the encoded files sounds better at the beginning, then it's possible. Anybody care about the full length files quality?

---

I was sceptical about the full lenght stability of bitrate so I make a test.

I created a 10 sec testloop and repeated in wave-editor (without any smoothing of course) 12 times, the loops marked with an impulse.
After encode-decode process, the second and last loop extracted, inverted and mix-pasted to first loop.
The difference of noise-spectrum of two result is insignificant.

I repeated this with cutting out from some jazz recording and find that a banding effect appears in the difference of first and second loop spectrum, not between in first and last loop differences. It's looks like harmonics, the difference is -1 dB at 2.7kHz, 5.5kHz, 8.2kHz, 11kHz, 13.8kHz ... Maybe the encoding bitrate is changing in the process?

OT- have we heard anything about the release of this? I'd love to know