Discussion related to multiple format tests' resul
Reply #11 – 2004-02-14 20:22:35
I didn't know that this was surprising for anyone. The choice of samples has a deep impact on the results. I noticed this effect when I made a small private classical music listening a few months ago, MPC (and Vorbis BTW) had problems with some low volume pieces, more so than Lame. I have already perform similar private tests, and made the results available. I think that Roberto selected wisely the samples used in tests: - there are all globaly difficult (that's why average bitrate for VBR encoder is higher than the targetted value), in order to make ABX easier and accessible for most people. Hard to encode, but not total killer. - the samples are covering a lot of musical genre. That's why "classical" or simply instrumental/vocal music, recorder with hi-fi principle (dynamic range > 5 dB), are only a short part of the sample suit. Problems with this test is the wrong interpretation of results. You can't answer to a people asking for "the best encoder for classical music at 128 kbps" to look at Roberto's test results. It would be a terrible mistake. First, because individual subjectivity may differ from the overall one; second, because only 2 or 3 samples (on 12) are assimilable to classical music. But you have a point when you're saying that the sample suit is lacunary. Some good encoders might encounter bad problems with quiet musical samples. I know that MPC have problems at --radio setting with a lot of material, performing better with Metallica latest album than with a Mozart' Symphony. I've already noticed that on extreme cases, MPC --standard have troubles with very very quiet musical parts (removing a lot of noise, details and life in the background). Of course you have to increase the volume in order to perceive the problem, but the missing details are then audible (but is it a problem for most people?).