Split from here. This topic has been discussed to death in the threads related to the multiformat listening tests.
If you want to add something here, re-read these threads first. Only post here if you have something new to say, please
- tigre
-----------------------------------------------------------------------------------------------------------------------

There is something else that I think would give all codecs a fairer chance. Even though using VBR whenever possible it would be helpfull trying to achieve similar filesizes.
(I remember the multiformat 128kpbs test where Musepack was a winner but at the expense of files 11,7% larger than the close second QT AAC.)

Good thing you said a winner. We wouldn't want Roberto to pass away from banging his head against a wall before the test starts

ARGH!! The Musepack files were only 11.whatever percent larger for those samples because those samples were mainly problem samples, so MPC, like any good VBR codec would, allocated more bits than usual! Over a large number of files MPC *at that --quality value* would average 128kbit/s. Do you understand??

Do you see now why I'm retiring in May?

Yes, but then the set is not chosen optimally (balanced). The test gave an advantage to highly variable encoding modes. I'm quite sure, that the results would look completely different, if all samples had been low volume classical pieces (which MPC often underbits and which then sometimes sound terrible at this quality setting).

I hope you can find a good compromise between total average bitrate and average bitrate for the sample suite.

This is a mighty hefty claim. Can you post some samples and ABX results please. I normally don't like throwing TOS #8 at people, but with 400+ posts I think that you really should know better.

@ music_man_mpc:

Even though my post count may cause you to think otherwise, I'm very familiar with the VBR concept (I've been encoding MPEG4 video since the times of the first Nandub SBC releases). So I will thank you immediately for not regarding my posts as noobarrhea.
Perhaps I'm not so familiar with which audio codecs support VBR. I assumed Quicktime AAC, WMApro & Blade were CBR. But even though I always encode at CQ (constant quality), so always VBR, I guess it's very important for me to know that even though as you correctly have put it:


they were also problem samples for QuicktimeAAC, not just for MPC, BUT it still managed to achieve closely the same quality (overall), even though it used a lower bitrate.

My point is that the day such a codec like Quicktime AAC implements CQ (VBR) encoding, it will use a lower bitrate to achieve the same quality. Clear?
This is a 128kbps test for sound quality, right? So let's isolate the bitrate and test for quality!
Otherwise we'll all be encoding at --braindead because we don't need these tiresome ABX tests...

Ah yes, very clear. Thank you very much for the clairification.



What does this mean, though?

It means that even though we are not as restricted as in "We have to fit 2h of video into 700MB", the lower bitrate to the same quality the better.
Or: the better the quality for the same bitrate the better.
There is a relationship of almost equivalence to these sentences, that is why I'm lobying for the test to resemble the second as close as possible.

Cheers,
I'm gone! See you guys tomorrow.

I didn't know that this was surprising for anyone. The choice of samples has a deep impact on the results. I noticed this effect when I made a small private classical music listening a few months ago, MPC (and Vorbis BTW) had problems with some low volume pieces, more so than Lame. Sadly, I lost my results in a Partition Magic failure, but I can give you at least one example:

Billy Budd.flac (0:25, 1.31 MB): Sample is part of the last track of a recording from 1998 of Benjamin Britten's Billy Budd (under Kent Nagano, singer is Anthony Rolfe Johnson).
I compared
1. Lame 3.90.3 -alt-preset 128. Bitrate is 124 for selection, 125 for whole track,
2. MPC 1.14 --quality 4. Bitrate is 100 for selection, 107 for whole track.

Lame is much better to my ears!
(Out of interest I tried Lame ABR 105 as well. Quality is comparable, but I still find Lame preferable.)

OT: Partition Magic is the source of all evil.

I (and the German magazine c't) recommend Acronis Partition Expert.

I have already perform similar private tests, and made the results available.
I think that Roberto selected wisely the samples used in tests:
- there are all globaly difficult (that's why average bitrate for VBR encoder is higher than the targetted value), in order to make ABX easier and accessible for most people. Hard to encode, but not total killer.
- the samples are covering a lot of musical genre. That's why "classical" or simply instrumental/vocal music, recorder with hi-fi principle (dynamic range > 5 dB), are only a short part of the sample suit.


Problems with this test is the wrong interpretation of results. You can't answer to a people asking for "the best encoder for classical music at 128 kbps" to look at Roberto's test results. It would be a terrible mistake. First, because individual subjectivity may differ from the overall one; second, because only 2 or 3 samples (on 12) are assimilable to classical music.

But you have a point when you're saying that the sample suit is lacunary. Some good encoders might encounter bad problems with quiet musical samples. I know that MPC have problems at --radio setting with a lot of material, performing better with Metallica latest album than with a Mozart' Symphony. I've already noticed that on extreme cases, MPC --standard have troubles with very very quiet musical parts (removing a lot of noise, details and life in the background). Of course you have to increase the volume in order to perceive the problem, but the missing details are then audible (but is it a problem for most people?).

That's an interesting thought! Yes, it made the complete test easier. But it also gave a slight edge to more variable codecs (in particular MPC). Maybe, if the files would average to 128, then MPC's rating would be a tad worse with each file?


Luckily I have more than 10 partitions on my harddisk, and the one affected wasn't the most important one (although it was the largest ).

It was the only problem I ever had with PQMagic, and I used it quite frequently in the last years. I might look at Acronis the next time, thanks for the recommendation.

The problem in bitrate calculation: is it meaningul to measure it on very short samples? Isn't it more pertinent to measure the bitrate on (at least) the full track (including quiet passage, silence, difficult and easier part to encode) or (better) the full album?

I thought that's how it has been done? The average bitrate was calculated over a large collection of various music styles.
My point is, that the chosen sample suit should be (more) representative of this collection, and contain some "easy" parts as well.

Maybe a low volume piece should be featured in the next test?

It definitely should. But from experience, I know that low volume pieces are too easy for most encoders, so people give up trying to test it and I get very few results for that specific sample. :/

I understand. But I fear that results wouldn't be usable.
I'm maybe wrong, but it seems that Roberto want to throw the "polonaise" sample away because it's too hard for most people to hear a difference. And "polonaise" (piano) is precisely the sample that fit your description (quiet sample, on the first half)...

I plan to replace polonaise with Fossiles

I like this sample

Did you try this another piano sample? I think there's more problems with this one than with polonaise (strong and sharp attacks).



P.S. In order to answer to you question asked yesterday, I think that the third Symphonie of Mahler (track06) is one of the harder sample of the series. Brass instruments are terrible to reproduce when they're are well-recorded.

For other people, nice classical samples (I used them for my own personal listening tests):
ftp://ftp2.foobar2000.net/foobar/
(link won't be permanent)

With Roberto's permission I am running background compressions - that is 500 lossless audio files (mixed) are being converted to see how close to 128Kbps the compession comes out at using the VBR AAC codecs. There should be no question of codecs being x% over.

Yes, thank-you very much, Spoon