[span style='font-size:14pt;line-height:100%']3.97a7 TEST #1: -V 4 & -V 4 --vbr-new full test
[/span]
Some literature first…
1/ SAMPLES[/u]
During previous listening tests, I’ve used a selection of various music (ff123’s samples libraries) and in order to complete it I added two problem samples (castanet2, Orion II) and one classical music (macabre.wav). For the following test, I’ve decided to change this, and to break the previous unity into three separated categories:
• GROUP1 = various music
• GROUP2 = classical music only
• GROUP3 = problem samples
— To build GROUP1, I’ve used the same samples than before (ff123 12+4+1 samples + Dev0’s “Since Always”). I’ve discarded the five “classical music” samples included in ff123’s ensemble (LizstBMinor, BachS1007, Macabre, fossiles, and BeautySlept) and put them in GROUP2. Total = 13 samples. I replaced them with 5 additional samples used in previous collective listening tests; they must be still available on rarewares (thanks to Roberto for this). The 5 samples are: 41_30sec; experienca, NewYorkCity, Quizas, Scars. TOTAL = 18 samples.
— For GROUP2, I’ve began with my previous classical selection (already used for various listening tests I’ve published in the past), which contains 15 samples. I’ve add the five samples excluded from GROUP1 (LizstBMinor, BachS1007, Macabre, fossiles, and BeautySlept) : total attains 20 samples. Three samples were removed: Bruhns and Bayle (problem samples which will join GROUP3) and the harpsichord sample called BeautySlept (I’ve already two harpsichord samples: Couperin and BachBWV1034). I preferred to this last one a guitar sample, called Murcia - Fandango. TOTAL = 18 samples.
— GROUP3. Instead of using one sample representing pre-echo (castanet2) and another one which reveal problems with micro-attacks (Orion II), I’ve used this time 6 samples representing each problem. Hope it will help to have a better idea of real performance of each codec/setting with this kind of issues. There’s also a third issue I’d like to test. It’s a problem I’ve noticed some years ago with Miles Davis’ trumpet: the signal is very tonal, and despite of this, lame introduce severe distortions. With VBR encodings, the bitrate drops to unusual low bitrate. It occurs with all modern lame builds. With time, I’ve collected more samples which suffer from this issue. I’ve therefore put in this group 6 samples representing this “highly-tonal” problem. TOTAL = 6+6+6 = 18 SAMPLES.
In summary: 3 different groups for three different kind of samples; 18 samples for each group; 54 samples as total.
2/ SETTINGS[/u]
-V 4 (aka preset medium) is not as easy to test as 128 kbps encodings, especially with common music. It needs more attention, and tests are therefore quickly exhausting. With 54 samples, I was forced to drastically limit the number of challengers. Comparing -V 4 with alpha 6 and alpha 7 was the real priority. It could be enough. Nevertheless, I’ve noticed previously so strange disparities between old-school VBR mode and the --vbr-new routine (in favor of this one) that I’ve decided to add alpha 7 --vbr-new in this test, which doesn’t apparently changed since alpha 6 (see bitrate table below)
In summary: 3 tested settings: alpha 6 V4 // alpha 7 V4 // alpha 7 V4 --vbr-new.
3/ BITRATE[/u]
[span style='font-size:14pt;line-height:100%']
A complete bitrate table is available here.[/span] Summary:
alpha6 alpha7 alpha6 alpha7
-V 4 -V 4 -V4 vbr-new
GR1 176 162 [-8%] 164 163
GR2 172 165 [-3%] 161 162
GR3 158 162 [+3%] 163 162
• Difference is really important between a6 and a7 with -V 4. The biggest difference happened in GROUP1: bitrate is much lower. With metal tracks, the bitrate diminution exceed 15% (<=> 30 kbps)! With classical music (GROUP2), same tendency, but with lower amplitude. It might be interesting to note that bitrate could be seriously higher (+5…+10%) on some tracks with alpha 7. Four tracks are in this situation. These four tracks share the same characteristic: very low energy in high-frequencies (which doesn’t necessary imply low replaygain value). The situation is entirely different for GROUP3: the new alpha7 produces now higher bitrate encodings (+3%) on problem samples than alpha 6.
• There are quite no difference between a6 and a7 with -V 4 --vbr-new.
• With alpha 7, there are no serious difference between old VBR mode and the new one - vbr-new - (at least with V4). Except maybe for classical music (GROUP2), for which --vbr-new produces slightly lighter encodings.
Of course, bitrate doesn’t tell anything about quality. Now, results :-)
In summary: biggest change occurs with -V 4 between old and new alpha. --vbr-new looks unchanged.
4/ RESULTS[/u]
4.1/ GROUP1: various music
a6V4 a7V4 a7V4_new
3,23 3,97 3,99
COMMENTS:
Much less problems with alpha7 than with alpha6. Encodings are more robust, with no serious ringing issues anymore. It’s a nice progress, especially if we keep in mind that bitrate is 15 kbps smaller on average! Difference between -V4 and -V4 --vbr-new is now marginal: some samples are better with default VBR mode, some others sound better with --vbr-new. Layla.wav is the only exception: I’ve ABXed with success the benefit of defaulted VBR compared to vbr-new.
4.2/ GROUP1: classical music
a6V4 a7V4 a7V4_new
3,14 3,97 4,19
COMMENTS:
Again, alpha6 V4 is worse than alpha7, and even worse when compared to --vbr-new. I think it could be safe to say that ATH issues occur more often with classical. That’s maybe why --vbr-new performed better on average than defaulted vbr mode. I’ve ABXed with success this positive difference on 6 samples. It’s not that much... But even small, it’s still a real good point for --vbr-new. Other good points are: smaller bitrate (-2%) and of course faster encoding (+100%). Maybe some additional tweaking on ATH could fill the gap between the two VBR mode?
4.3/ GROUP3: problem samples
a6V4 a7V4 a7V4_new
1,53 2,51 2,68
COMMENTS:
— micro-attacks samples: ANOVA analysis: alpha 6 [2.00] < alpha 7 [3.00] < alpha 7 --vbr-new [3.67]. It’s the only ANOVA analysis which conclude on --vbr-new superiority for the WHOLE test.
— highly tonal samples: alpha6 (1.17 / 5) is clearly worse than alpha7 encodings. Marginal difference between -V4 and -V4 --vbr-new with alpha 7 (2.20 & 2.30).
— transient samples: alpha6 is worse again. Results were so bad than I wasn’t able to find difference between alpha7 -V4 and -V4 --vbr-new, except on the marimba sample, which sounded sharper with default VBR mode [ABXed 16/16].
5/ GENERAL CONCLUSIONS
• ATH modifications made by Gabriel on -V4 are highly positive: bitrate is much smaller with ‘modern music’, slightly smaller with ‘classical music’ and slightly higher with problem samples; quality is now much better, whatever the samples’ category. Efficiency and usability is therefore much greater. Some problems are nevertheless still audible.
• As consequence of this double progress, the --vbr-new mode is less attractive than it appeared to be during previous tests. Bitrate is similar, quality is comparable, but with a tiny advantage for --vbr-new with classical music and with ‘micro attacks’ samples. Moreover --vbr-new has a big and uncontestable advantage on encoding speed. I wouldn’t say anymore than defaulted mode is “more secure” than --vbr-new or that this one could increase the bitrate. On contrary, for this test, average bitrate is (very slightly) smaller, quality is (very slightly) higher and speed twice higher :-)
ABX log files are here.
Topic: Lame 3.97 Alpha 7 Testing Thread (Read 84986 times)
previous topic - next topic