MPC vs VORBIS vs MP3 vs AAC at 180 kbps, 2nd checkup with classical music Options

[LoFiYo]

Great work Mr Guru. Your occasional listening tests alone make it worth visiting this website every day. Thank you very much.

[guruboolez]

Thanks to you for all support

There's always QuickTime Pro 7 and its VBR encoder:-)

True
Is it available on Windows? It's also not certain that one of the VBR preset would correspond to the desired bitrate. And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible. I hope that a VBR encoding mode with iTunes will be released soon.

It is normal though - independently of the psymodel - that transform codecs do have a big encoding advantage on these highly tonal samples (1 sine wave = 1 coefficient). This puts Musepack at a big disadvantage - but still, it is very very interesting to read, especially about Vorbis 

Two notes:
1/ on 18 samples I would call "highly tonal" two samples only: sample_05 (organ) and sample_03 (chorus). Most other samples are not especially tonal (except maybe the trumpet and the bassoon ones).
2/ musepack is rather disadvantaged on efficiency than on quality: the encoder could allocate a lot of bits on those moments in order to maintain the quality. Extreme example with this short accordion passage: mpc reaches 300 kbps with standard! On the extreme opposite we found WMAPro which encode the same sample at 100 kbps only with VBR90 (~200 kbps in average)...

Would be interesting to test hard rock samples some day, to see if Vorbis beats the others there too 

I'd like to see such test too

I'm not sure about it, but I heard that WMA Pro is meant for high bitrates, so this test should be ideal to compare it to others. Would it be too difficult to ad these formats?

I've already tested WMAPro at 128 kbps and it performed amazingly well on classical samples (and didn't performed badly on Roberto's collective test based on various musical genre). It implies that WMAPro is not only meant for high bitrate and has a great potential (considering the greenness of this encoder) at lower bitrate.
Unfortunately, WMAPro offers few VBR modes (six only) and none of these would correspond to the desired bitrate. WMAPro also has poor pre-echo performance and suffers from the same bad issues than musepack audio on low volume tracks (low bitrate -> severe ringing/noise eradication).

[rjamorim]

Is it available on Windows?

Yes.

And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible.

It has always been impossible.

Yopu will share the pain I endured while preparing listening tests before iTunes was released

I hope that a VBR encoding mode with iTunes will be released soon.

Hopefully in v5, whenever that is released...

I'd like to see such test too

Me too. About time someone shows up to replace me in testing dictatorship.

[guruboolez]

And correct me if I'm wrong: it's seems that batch encoding with current beta of QuickTime is impossible.

It has always been impossible.

I'm not that sure. There was once a small app with an unpronoucable name (qtibaocas, qutibolas, qbtbicolous, qtboacoas ) used to drive QT encoder from an external application. dBPowerAmp (fortunately easier to articulate) also has this feature.

You will share the pain I endured while preparing listening tests before iTunes was released

Hey! I also tested QT AAC encoder before iTunes first release on Windows (see link on my previous poest). And I also have to encode 150 full tracks in order to know the corresponding bitrate for each preset. Without any batch mode, it's an herculean task.

[rjamorim]

I'm not that sure. There was once a small app with an unpronoucable name (qtibaocas, qutibolas, qbtbicolous, qtboacoas  ) used to drive QT encoder from an external application. dBPowerAmp (fortunately easier to articulate) also has this feature.

Ah, sure. I meant that QT itself never offered a batch feature.

Hey! I also tested QT AAC encoder before iTunes first release on Windows (see link on my previous poest). And I also have to encode 150 full tracks in order to know the corresponding bitrate for each preset. Without any batch mode, it's an herculean task.

It is indeed. If I was home, I would help you out by downloading the samples and testing at least a handful of them.

Oh well... :/

[PabUK]

Excellent test, as always. Thanks.

[Sunhillow]

Wow! A great test, Guru!

[user]

Chapeau Guruboolez !

Congratulations to the progress of Vorbis.

Regarding MPC, the result is split, somehow for classical samples there seems to be a degration in quality and size-effectivity (bitrate boost), comparing MPC v1.14b with MPC v1.15v
I cannot dare to ask Guru, to compare this sample set again between 1.14b and 1.15v to get a ranking of this new multiformat test including not only 1.15v but also 1.14.

Lame - MP3

The overall impression of reading the forum, and now some proof by Gurus tests (low bitrate 80 kbit/s and mid-to-high bitrate 185 kbit/s) I would recommend nowadays new Lame 3.97 over the 3.90.3.
Does the Lame - MP3 recommended settings thread need a (complete, or some major) rewrite ?








My personal point of view nowadays:

low bitrates, ca. 128k abr/vbr, target portable usage in more or less noisy environments

I use still MP3 format, will change from Lame 3.90 to lame 3.97
For portability reasons, MP3_Lame is still uptodate, as this ("free !" format is not only available in every cheap player, from car-CD-player to 1 GB USB sticks and so forth. No need to buy certain expensive portable brand players. Warranted quick 'n' (but! in quality terms) dirty exchangebality of some music with friends.
Due to the memory increasements eg. in USB sticks, the quality/bitrate issue comparing other modern formats around 128k is nothing too important to worry about, like saving 20k.



Music for high-end, Archiving, listening in living-rooms etc.

Lossless, as wavpack (-x -m) (or from previous archivings flac (-8)), due to DVD+-R or HD, no storage-size problems.
Lossy high-quality format MPC 1.14/1.15v at high bitrate (230 - 280 kbit/s, --quality 7 - 8 , --ms 15 , --xlevel) is also suitable for this goal. MPC at this quality level allows easier transport of high quality music eg. on laptop or cheap backup archive solution (on DVD+-R or HD).
Though Vorbis could now be considered ok for same goals, at quality settings 7-8 also.
Both lossy HQ formats have limited prebuilt-purchaseable "hardware" support, for clarification: MP3 has unlimited prebuilt-purchaseable "hardware" support in my definition.





*The series of multi-format tests from 80 - 185 kbit/s, carried out by Guruboolez is so impressive, I have to repeat it here again. The clear descriptions, systematically, the way of elaboration and thoughts before testing, the documentations, I cannot imagine a further improvement, or should i say: Perfect as always ?!

This post has been edited by user: Aug 22 2005, 13:10

[Aoyumi]

Thank you for a test and a detailed report, guruboolez.
This result is glad also for me. It proves what the direction of my tuning was not mistaken in.

>Lyx

What is the cause of vorbis' noise issue? Most here do know that it was introduced between RC3 and 1.0final. Now Aoyumi seems to have succeeded - at least in high-bitrates - to almost nullify this problem, so he obviously has an idea where the problem is/was. So what was it? Also, it does not seem to be an on/off-issue... it was slowly "fixed" over the course of time - from that it would seem to me that the "reason" for this problem is/was a fundamental vorbis problem. If this is the case, then i would wonder "how can such fundamental changes happen between a release candidate and a final release?".

About a noise problem, it does not depend on a single cause. The change for an improvement was made through aoTuV beta2, beta3, and beta4.

The code added by beta2 rectifies change of audio energy (this brought about a big change especially). The code added by beta3 reduces the noise of an attack portion more efficiently as compared with megamix etc. More comprehensive tuning is performed in beta4. Probably, change of ATH, noise/tone masking, and noise compander etc. is useful to the improvement of roughness.

Magic does not exist there.

[echo]

Once again I am impressed by your professional listening tests guruboolez. Thank you for all your efforts.

And also more congratulations to aoyumi! His tunings have probably made vorbis the best encoder for almost all bitrate ranges!

[arman68]

Very good and comprehensive test. Thank you.

It confirms the impression I recently had of Vorbis. Until a couple of months ago, I would not touch Vorbis with a bargepole, due to my own testing I did about 2 years ago, which revealed lots of artifacts that hurt my hears.

I was sold on MPC.

However, I recently retested Vorbis aoTuV beta4, and was amazed to find all the artifacts gone. I quickly did some ABX tests to see how far down I could push the bitrate for use on my portable, and I found out that Vorbis outshone every other codec! I was not sure about higher bitrates, as my ears are not good enough, but your tests confirmed what I suspected.

Vorbis is now my codec of choice.

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog Good to know when planning to use a portable which only supports AAC and MP3.

Oh, and congratulations to Ayoumi for his outstanding work on Vorbis. I bow to you.

[Zurman]

Impressive !

And quite surprising (vorbis>mpc and lame close to mpc...)

Only one regret : wma is not in the test... ;( (for bitrates reasons I know)

[Ivan Dimkovic]

Hi,

First of all, thanks Guru for your very hard work.

As far as the Nero AAC is concerned, it has been already known that "High Quality" codec demonstrates some issues with classical music, and that "Fast Mode" has much better performance in that arena (as it has been proved in earlier Guru's tests) - as Guru already noted, fast mode was not regarded stable enough, and in the current version - it is still not.

I have some news for the HA users - at the moment we are testing new SBR optimizations internally - that will improve SBR quality (16-80 kbps) a lot - and as soon as SBR optimizations are done, work will be continued in making "fast mode" of LC-AAC stable for non-classical music items as well.

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.

Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.

[Mr_Rabid_Teddybe...]

I'm now really happy I recently purchased this little baby:



I used to use MPC on my computer, but now that I own a H340 I was switching to Vorbis anyway. And as this test shows it has caught up with MPC qualitywise and as the SSE optimized build Lancer encoding speeds are as fast as Musepacks there's no reasons for regrets....

Thanks, guruboolez!

[Corsair]

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.

Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.

I don't think it's really possible to call all classical samples 'a particular genre' (in terms of samples), since those samples can be quite diverse (like they are in guruboolez's test). You have, for example: solo instruments, chamber recordings (a couple of instruments), voices (with or without instruments playing), orchestras in small or large concert halls... note that there is also a wide range of instruments that can be used.
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).

[HotshotGG]

Probably, change of ATH, noise/tone masking, and noise compander etc. is useful to the improvement of roughness.

Magic does not exist there.

I was digging through the code before just to try and get a better understanding of how things work? what exactly is noise companding and noise biasing? I mean what specific roles do they play? I understand the ATH and noise/tone masking routines using the FFT and MDCT, but I don't really understand the other two that well. It's my understanding that when you are changing the impulse_noisetune if you are using advanced-command line switch you are actually adjusting the noise bias?

[Ivan Dimkovic]

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog wink.gif Good to know when planning to use a portable which only supports AAC and MP3.

Please note that this is true for a particular music genre (classical) - particular codec configuration (non-SFE mode of Nero AAC), particular bit rate range (VBR ~170 kbps) and limited to a single audiophile listener.

I don't think it's really possible to call all classical samples 'a particular genre' (in terms of samples), since those samples can be quite diverse (like they are in guruboolez's test). You have, for example: solo instruments, chamber recordings (a couple of instruments), voices (with or without instruments playing), orchestras in small or large concert halls... note that there is also a wide range of instruments that can be used.
What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).

During the experiments with Nero AAC algorithms it has been found out that most problems with 2-pass loop (high quality) encoder came from ATH and noise fluttering of the S (side) channel in the range between 11-20 kHz.

This particular effect was noticeable only with tracks that have silent "background" in that particular frequency range and very high interchannel correlation, so M/S mode was triggered on - and most "classical" (I also hate the term, trust me) recordings fall in that category.

This effect could even affect L/R mode in some cases, but only when the original input is very silent - replaygaining that AAC output to higher dynamic range actually amplifies the problem.

The reason why SFE (scale factor estimation) aka "Fast" mode was found to be so much better by Guru in the previous test was direct noise injection which did not have any of the M/S related ATH issues.

[Digisurfer]

Time to play devils advocate. What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music. Everyone is different and needs to test for themselves which is better to them, since everyone hears a little differently. I could do the same test, present the results in the same fancy manner, and come up with completely different results. The other option is an average based on a group of blind testers which makes much more sense, doesn't it?

That said, I'm not surprised by the results since they fall almost along the same lines as my own ABX testing. I use Vorbis (aotuvb4) for my portable an think it's fantastic, but if that didn't exsist I would use AAC over MP3 simply because I find the artifacts less annoying, which emphasises why people should do their own testing rather than choosing based on someone elses tests.

Hopefully I haven't offended anyone. I found this post very interesting and want to thank guruboolez as well for putting in such a huge effort. Great work!

[germanjulian]

thank you for this great test.

[guruboolez]

Regarding MPC, the result is split, somehow for classical samples there seems to be a degration in quality and size-effectivity (bitrate boost), comparing MPC v1.14b with MPC v1.15v
I cannot dare to ask Guru, to compare this sample set again between 1.14b and 1.15v to get a ranking of this new multiformat test including not only 1.15v but also 1.14.

There are two different problems in my opinion:

- bitrate consumption: mpc --standard has increased by more than 10 kbps compared to former release of mppenc (I can post a full bitrate table if you want). 1.15 series is by far the version which presents the highest bitrate (not only with classical: people listening different kind of music have confirmed it - but how much is something I can't say). But this inflation is not necessary a problem: some users don't really care about few additionnal kbps, and it could bring higher quality (higher bitrate doesn't necessary mean lower efficiency).

-quality (with classical): I'm not sure that 1.15 presents regression compared to 1.14 beta. The problem maybe occur earlier. I compared 1.15u to a much older release of mppenc (1.01j) which clearly revealed issues with the latest version of mppenc (+ inflated bitrate). Now I can't tell when exactly the problem happened, or if the quality (with classical) has slowly decreased with version > 1.0 stable.

Thank you for a test and a detailed report, guruboolez.
This result is glad also for me. It proves what the direction of my tuning was not mistaken in. 
(...)
Magic does not exist there. 

As usual, congrats to you Aoyumi. I have tested three times your encoder (at 80, 96 and now 180 kbps) and each time I discovered the results with a deep sigh... of astonishment. I probably have to cease my listening tests before people start to suspect me from zealotry Congrats! Your job is near to reconciliate me with high bitrate lossy encodings.

Interesting that lame 3.97 outperforms AAC. There is still life in the old dog    Good to know when planning to use a portable which only supports AAC and MP3.

I don't want to defend Nero AAC, but keep in mind that LAME -V2 and Nero Digital -streaming won't probably lead to similar music with most musical genre. I don't have the material to build a precise bitrate table with anything else but classical, but from whay I read in the past it's Nero AAC -normal which produce a similar bitrate to LAME --preset standard. And -normal should sound better than -streaming tested in my comparison. It's very important to remind that -besides the samples- the fairness of presets used in my test is not universal.

(...) as Guru already noted, fast mode was not regarded stable enough, and in the current version - it is still not.

I'd like to ask you what does the unstability issue of 'fast' encoder consist. I know two serious issues:
- smearing (it seems that 'high' encoder has better pre-echo handling).
- bloated bitrate on some occasions. I made a graphical comparison which illustrate this point. Bitrate are based on 150 full tracks (> 16 hours of music):



As you can see both curves are parallel for 80% of the samples. But for the 20% remaining one, the -fast encoder start to use more and more bits and encodes the lasts samples with more than 250 kbps (275 kbps for the last one). -high is less hysterical and it stays under 200 kbps (199 for the biggest).

Are there other known issues with -fast encoder?

Anyway, congrats for your work I'm pretty impatient to see you working more intensively on the LC core.

What I'm trying to say is that, unlike most other genres, classical samples can be very different and so when you have a thorough test like this one, you can get a pretty good overall picture of each codec (encoder).

I'm fully agree with you: 'classical' is a generic term which doesn't really mean anything and which wrongly encompasses most written instrumental/lyrical music composed before the 20th century.
About lossy encoding: I would also believe that any encoder able to handle perfectly all situation encountered with 'classical' would be a champion for every musical genre. But such encoder don't exists and my current listening test is also far to represent all possible situations happening with classical. That's why it would be nice to see tests focusing on other problems than those tested here.
My results can't of course be generalized and extrapolated to other musical genre as well as it would be excessive to believe that results could be totally different with a another set of samples.

This post has been edited by guruboolez: Dec 29 2005, 22:45

[sh1leshk4]

What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music. Everyone is different and needs to test for themselves which is better to them, since everyone hears a little differently. I could do the same test, present the results in the same fancy manner, and come up with completely different results. The other option is an average based on a group of blind testers which makes much more sense, doesn't it?

If so, then what's the relevance of having most af the personally ABX-ed listening tests to be shown as a proof that a codec is better (at some bitrates and/or genres) from another?
And it's not like we're seeing listening tests done in a group everyday.
Plus, with your reasoning, then results coming from a group of blind testers probably won't make sense as well, since it's probably valid only for that certain group.

I guess personally conducted listening tests by others can't be used as something to back up some claims and using them might be against TOS #08...

To guruboolez, thank you for conducting the test.
To Aoyumi, thank you for helping Vorbis to be a better codec one leap at a time.
To Ivan, I'm eager to see the outcome of the next version of Nero AAC encoder.

[Lyx]

The other option is an average based on a group of blind testers which makes much more sense, doesn't it?

And how many users on this board would be able and willing to take such a demanding test? IIRC even on ha.org, the amount of people who are able to do tests at near-200kbit is very low - which is part of the reason why people are grateful for guru's test. And then one would have to get those few together simultaneusly to participate in a test. Sure, of course it would be better if it could be done - but that may be a little difficult to achieve.

This post has been edited by Lyx: Aug 22 2005, 22:09

[Axon]

While I don't doubt that this is a very significant study, I also agree that more work is necessary before we can bless 3.97 stable as the recommended HA version. I think that is eventually going to happen though.

It's very tempting to try to confirm guruboolez's results myself, as a starting point, but I don't think I can scrounge up the 10+ hours of intense ABX time I'd most likely need.

This post has been edited by Axon: Aug 22 2005, 22:12

[ching-3]

Great work guru, a very good test.

I always believed that Aoyumi tuned vorbis very well

[guruboolez]

Time to play devils advocate. What is the point of this test? All it shows are results that only apply to guruboolez's hearing, and only with classical music.

Exactly.
But every listening test presents the same kind of limitation. None listener can't go beyond its own subjectivity.
What I can do is posting objective analysis: graphical illustration showing lowpass, mix-pasted comparisons, or eaqual benchs. But you know what people think about them and how much the penalty is for using such tools for claming anything about quality
We're doomed to our subjectivity. I can also post all samples and expect that other users will try to perform the same test; by the way I always uploaded sample to that purpose. Feedback = 0. It's also understandable: I won't be very interested either to perform a full listening test on music I don't listen to. But you'll also notice that almost nobody has posted in the last years any listening comparison based on their favorite samples.

EDIT: two mistakes.

This post has been edited by guruboolez: Aug 23 2005, 08:24