The 12 samples correspond to the ff123'samples suit selected for the 64 kbps listening test.
www.ff123.net

Encoders and settings:
• lame 3.90.3 | John33 compile | --alt-preset 128
• lame 3.96.1 | John33 compile | --preset 128
• lame 3.97.a5 | John33 compile | --preset 128 -X 10,10

Hardware and software configuration
• Audigy2 soundcard
• Onyko R-A5 FM/AM Tuner Amplifier
• Beyerdynamic DT-531 headphone
• ff123's ABC/HR 1.1 beta 2

Personal mood
• tired... I didn't spent too much time for this test, and I have only ABXed important things (the two best encoders, and only if needed).

RESULTS




COMMENTS
• most often, lame 3.96.1 appeared to be clearly behind lame 3.90.3 and lame 3.97.a5
• I had some troubles to differenciate lame 3.90.3 and lame 3.97.a5. I often changed the notation. Both are close (according to my experience), and it wasn't easy for me to tell which sounded worse or better, even when a difference was audible and ABXable.
• newest alpha had apparently serious troubles with LisztBMinor.wav sample: background noise was severly wounded, removing precious musical information. Difference could be checked through a frequency editor: it's really eloquent. IIRC [proxima] had also reported problems for this sample and recent lame builds.

ANALYSIS

• ANOVA ANALYSIS




• TUKEY PARAMETRIC ANALYSIS



=> 3.96.1 is the worst; 3.90.3 and 3.97.a5 are tied.



EDIT: log files are here

This is what X mode 10 does...in quantize.c

Yes, this problem of altered background noise (with also musical information) is noticeable with even more samples for recent lame builds. Sometimes the removed background noise is replaced with a HF ringing. Both artifacts are really annoying for me , this is the only reason i still prefer the old 3.90.3 in these cases. At this time, with 3.97a5, the following samples are affected: obscured, Atom_heart_mother, FloorEssence, LisztBMinor, spahm, rebel, Queen+-+your+my+best, gbtinc, applaud, amnesia, Bayle - Etching. Often the artifacts are even visible as dropouts with a spectral analysis. Of course, i can upload the remaining samples on request.

I looked at the output of rebel on a spectrogram. There's a definite dropout in the 10-11KHz region. Maybe the encoder is not devoting bits to this sfb because it (falsely) thinks the tones are masked by the much stronger tones in the lower midrange (guitar). Just a guess.

Tested Dev0's sample from here.

3.90.3 --alt-preset cbr 128
3.97a5 -b 128 -X 10,10

I liked 3.97a5's CBR128 better than 3.90.3's on this particular sample even though they were both noticeably distorted.

ABC/HR Version 0.9b, 30 August 2002
Testname: 3.97a5 vs 3.90.3

1R = C:\My Test Samples\giveup\3903cbr128.mp3.wav
2L = C:\My Test Samples\giveup\397a5cbr128.mp3.wav

---------------------------------------
General Comments:
The stereo separation (not sure if this is the correct term) on both files sounds broken for the guitar. On the original, the guitar is at mid-left (no echo from the right channel), but on the mp3s, the guitar is echoed from the right.
---------------------------------------
1R File: C:\My Test Samples\giveup\3903cbr128.mp3.wav
1R Rating: 2.5
1R Comment: This sounds rougher and dirtier. The incorrect stereo separation/added echo from the right channel is more noticeable than File 2.
---------------------------------------
2L File: C:\My Test Samples\giveup\397a5cbr128.mp3.wav
2L Rating: 3.0
2L Comment: The distortion is less annoying than File 1. Sounds slightly cleaner.
---------------------------------------
ABX Results:
Original vs C:\My Test Samples\giveup\3903cbr128.mp3.wav
10 out of 10, pval < 0.001
Original vs C:\My Test Samples\giveup\397a5cbr128.mp3.wav
10 out of 10, pval < 0.001
C:\My Test Samples\giveup\3903cbr128.mp3.wav vs C:\My Test Samples\giveup\397a5cbr128.mp3.wav
10 out of 10, pval < 0.001

Thank you

If I may be so bold, was there a reason why you chose the --preset 128 for the 3.96.1 as opposed to -V 5 ?

Regards
Randy

Cause 3.96.1 had ABR/CBR issues, which Gabriel worked on and need testing.

Unfortunately no. The psymodel thinks that this area needs some bits. However, it appears that sfb10 is empty on those parts, which is in contradiction with the psymodel requirements, and even with the computed distortion which indicates no distortion there.

edit: 3.90.3 seems to exhibit the same dropouts on the Rebel sample. Does 3.90.3 sounds better on this sample?

It seems that lowering the ath is reducing those dropouts (at least in the Atom case).

Would you mind testing "-X 10,10 --athlower 10"?
If this solves the problem, it would be very helpfull the determine which athlower value is leading to satisfactory results.

The rebel sample is only slight better with 3.90.3, i don't think rebel is a problem case for the new aplhas. Even 3.90.3 has distorsions.

I will test when possible the setting with the lowered ATH but i strongly believe this is the way to go because i've already noticed improvments in the past. Now, for VBR, i believe that "--athaa-sensitivity 1" reduce significantly ringing because VBR can choose to lower the ATH even more than the default. When i proposed the "--athaa-sensitivity 1" you seemed to dislike this way of tweaking because of too high ATH adjustment range for a 128 preset. Maybe i have misunderstood something but it seems that now you're going for the "lower ATH" solution even with CBR/ABR. Maybe a lower ATH (or a higher ATH sensitivity) is the key for all presets (CBR, ABR,VBR) that targets below ~150 Kbps (medium included).

I'd prefer to use a lower base level for ath but not increasing the adjustement range too much.
I think that I should also enable ath-aa for cbr/abr.

96 kbps ABR & VBR TEST

Samples:
The 12 first samples correspond to the ff123'samples suit selected for the 64 kbps listening test.
I've also added eight more samples:
- the 4 samples used by ff123 for the 128 kbps collective test (Dogies.wav, Fossiles.wav, Rawhide.wav, Wayitis.wav)
- macabre.wav (full orchestra sample, uploaded by ff123)
- SinceAlways.wav (recently uploaded by Dev0)
- castanets2.wav for testing sharpness & pre-echo
- Orion II.wav (brass instrument) for testing micro-attacks with a real instrument.

Encoders and settings:
• lame 3.90.3 | John33 compile | --alt-preset 96
• lame 3.96.1 | John33 compile | --preset 96
• lame 3.97.a5 | John33 compile | --preset 96 -X 10,10
• lame 3.97.a5 | John33 compile | -V 7
• lame 3.97.a5 | John33 compile | -V 8 [see NOTE ABOUT ENCODINGS for details about these choices]


Hardware and software configuration:
• Audigy2 soundcard
• Onyko R-A5 FM/AM Tuner Amplifier
• Beyerdynamic DT-531 headphone
• ff123's ABC/HR 1.1 beta 2





NOTE ABOUT VBR ENCODINGS:

There's no VBR preset corresponding to 96 kbps for 'general' music. -V7 seems to produce the closest bitrate value on most samples, but -V8 is needed with some kind of music (I've noticed it in the past with metal/hard rock, and it was also reported here by other members). On the other side, even –V7 could be inferior to 96 kbps. It happens here with four samples, with a terrible deviation for the two "classical music" (and low volume) samples: 67 & 72 kbps for BachBWV1007 and LisztBMinor. Of course, -V7 will logically produce bloated bitrate with the first category of music: the seven biggest files of this test were encoded with this VBR setting.

I had therefore two serious possibilities: first one was to discard VBR from this test. This solution is the most opportune, but IMHO not the most useful one. I'd like to see how will perform lame VBR at such bitrate, and wonder if it could outperform ABR/CBR, at comparable bitrate of course. Second valid possibility: introducing BOTH -V7 and -V8 encodings in this test. Then, everybody could choose its own comparison's strategy (comparing ABR/CBR to a fixed VBR preset or comparing with the preset matching with 96 kbps for each situation). I've finally decided for the second option, despite of difficulty (it supposes for each sample an unnecessary file to be tested...)

NOTE ABOUT SAMPLING RATE.

All settings lead to a resampling process (-> 32000 hertz), except one: lame 3.96.1.


RESULTS


click for log files


COMMENTS:

• VBR encoding at low bitrate can’t apparently be recommended. Bitrate fluctuates too much (that’s normal), but quality too (that’s not normal). VBR should provide constant quality for fluctuating bitrate whereas ABR/CBR should conduct to constant bitrate and erratic quality. At low bitrate, ABR is clearly more robust, despite its limited bitrate allocation. VBR suffers too much from ringing and tons of other artefacts. –V 8 is most often awful; -V7 can’t avoid some difficulties (on LisztBMinor for example).
• ABR at 96 kbps with lame 3.96.1 is a complete tragedy. The lack of resampling could maybe explain a part of this disaster.
• There’s no winner at the end of the 3.90.3 / 3.97a5 competition. Overall results are identical (2.85 vs 2.84!), but this final result contains many difference. Each encoder has specific reaction. With micro-attacks [Orion II.wav] 3.90.3 is apparently clearly better. It could be confirmed with other tests already performed by other persons on comparable samples [fatboy.wav and Bayle – Etching.wav]. With pure pre-echo sample, the new alpha is apparently better. 3.90.3 has also less ringing problem (but problem is not totally absent). Nevertheless, 3.97a5 is now close to 3.90.3 in my opinion, but future progress is of course welcome.


STATISTICS:

If some people would play with statistic tool, just copy and paste the following table:



• ANOVA Analysis:





• TUKEY PARAMETRIC Analysis:





• Bitrate table:

160 kbps ABR & VBR TEST [including –VBR-NEW vs default VBR mode]

Samples:
20 samples - same as 96 kbps listening test above.

Encoders and settings:
• lame 3.90.3 | John33 compile | --alt-preset 160
• lame 3.97.a5 | John33 compile | --preset 160 -X 10,10
• lame 3.97.a5 | John33 compile | -V 4
• lame 3.97.a5 | John33 compile | -V 4 –vbr-new [see NOTE ABOUT ENCODINGS for details about these choices]


Hardware and software configuration:
…they didn’t change.





NOTE ABOUT ENCODINGS:

• Modern encoder at 160 kbps should all reach near-transparency state to my ears. Tests are therefore more difficult. In order to avoid unnecessary exhaustion, I’ve tried to limit the number of challengers. 3.96.1 ABR, which clearly appeared as buggy during previous tests, was consequently removed.
• Like previous test, the main epistemological issue was to legitimate the choice of a VBR setting. –V 4 is the closest setting from 160 kbps value… but often higher. Average bitrate for the 20 samples reach for example 171 kbps, with 197 and 134 kbps for extreme samples. I can’t introduce this time a lower setting: the full sample suit drop to 135 kbps with –V 5 setting, and testing it would be completely absurd.
I had another idea at that moment: using the alternative VBR engine, aka --vbr-new, commonly used with --alt-preset fast routines. I’ve noticed in the past that --vbr-new encodings are slightly smaller than default VBR mode. Other people have also reported the same fact. I’ve therefore encoded the 20 samples with –V 4 --vbr-new, and average bitrate reached 160,4 kbps. Nice, isn’t it ;-)
I didn’t choose between –V 4 and –V 4 fast, and simply put both in the arena. It will be a good occasion to compare the performances of these two VBR engines.

P.S. I did a big rest during this test, and resumed it after 8 hours of sleeping.


RESULTS


click for log files


COMMENTS:

• On average, 3.90.3 ABR was slightly inferior to 3.97.a5 ABR. Both are very close, and similar, except on speed: 3.97a5 is obviously faster.
• VBR comparison is more interesting. First, --vbr-new engine produces with most sample better results than default VBR mode. Statistically (see above), it only appears using friedman.exe tool with tukey parametric with –s 0.1 option (10% confidence, instead of defaulted 5%). But it’s probably more interesting to differentiate results. VBR NEW was much better with LisztBMinor, Dogies, Fossiles, SinceAlways, Wayitis and Orion II. For strange reasons, the defaulted VBR suffers a lot from ringing: background noise becomes irregular, and the distortions also infect some precious musical information (especially at low volume). I’d like to illustrate this problem with a frequency analysis, confirming by eyes the seriousness of this problem:
http://audiotests.free.fr/tests/200...r_vs_vbrnew.gif
On the other side, --vbr-bew routine produced a clearly worse result with the harpsichord sample (BeautySlept.wav). It’s not a surprise I must say: I’ve noticed it two years ago [it must be with lame 3.92 or 3.93]. But if we except this issue (very specific, but unfortunately very annoying for me), I didn’t found any other situation in which --vbr-new really suffered compared to default mode.
• -V 4 --vbr-new is not obviously better than lame 3.97 alpha 5 ABR 160 (final bitrate were the same). It’s a bit problematic: Shouldn’t we expect a real difference between ABR and VBR? Is VBR clearly better than ABR/CBR? And for what situation? Are those VBR/ABR similarities something structural (e.g. we can’t expect from well-tuned MP3 encoders a real quality margin between two modes) or something purely accidental (e.g. lack of tuning of current VBR mode compared to well-tuned ABR settings)? We have some elements of answers: we saw first than ABR outperformed VBR at 96 kbps, and then that existing differences at 160 kbps are really minor, at least with common music (situation could differ with killer sample). On the other side, I’ve found –V 5 --athaa-sensitivity with lame 3.96.1 really better than lame 3.90.3 in a recent past. There are dissimilarities between elements of answer I’ve gathered. Therefore, I think we should try to find a real answer in the future (and temporary forgot our current beliefs) to this fundamental interrogation.



STATISTICS:

If some people would play with statistic tool, just copy and paste the following table:



• ANOVA Analysis:
my results can’t lead to any differentiation.

• TUKEY PARAMETRIC Analysis [-s 0.1]:




• Bitrate table:

DVD RIP test: AC3 transcoding at ~96 and ~128 kbps

Samples:
I’ve tried to see how will perform a comparison with DVD Video transcoding. DVD Video are very different from CD: we have on one hand variable quality AC3 (which introduce a lot of quantization noise, and sometimes strong lowpass), 48000 hertz sampling rate, and always high dynamic soundtracks including spoken and ambient parts; on the other hand, CD is 44100 hertz, original PCM quality (with infinitesimal quantization noise) and most often limited dynamically (thanks to loudness race).
For this test, I had to build myself all samples. There are only six samples: the conclusions couldn’t be anything else than leads for further investigations. I always used AC3 as source (no DTS nor PCM). I’ve selected native stereo AC3 encodings when possible; for one sample, I had to downsample myself to stereo. Decoding, downsampling and transcoding were performed directly with foobar2000. Samples are:
• Jean-Pierre Jeunet — Alien 4 Resurrection: Jean-Pierre Jeunet presents the DVD edition in English with pronounced French accent. Native stereo AC3 encoding at 192 kbps.
• Rowan Atkinson — Blackadder IV (“Captain Cook”). English speaking with public’s laughs. Native stereo AC3 encoding at 384 kbps.
• King Hu — Come Drink With Me (L’Hirondelle d’Or). Quiet music with water. Mono (two channels) AC3 encoding at 192 kbps.
• Gérard Corbiau — Farinelli. A morning: Horses, birds… then woman voice on diner in French language. Native stereo AC3 encoding at 224 kbps.
• Quentin Tarentino — Pulp Fiction. Ezechiel and gunshots… 448 kbps multichannel AC3 encoding, downsampled to stereo.
• Akira Kurosawa — Ran. A hunt, and dramatic music (flute & percussion). Native stereo AC3 encoding at 448 kbps (!).



PART I: 96 kbps encodings


Encoders and settings:
• lame 3.90.3 | John33 compile | --alt-preset 96
• lame 3.96.1 | John33 compile | --alt-preset 96
• lame 3.97.a5 | John33 compile | --alt-preset 96 -X 10,10
• lame 3.97.a5 | John33 compile | -V 7 --vbr-new
• lame 3.97.a5 | John33 compile | -V 8 --vbr-new

Hardware and software configuration:
…same as before

NOTE ABOUT ENCODINGS:

I explained before (see 96 kbps encoding test) the reason for maintaining two VBR settings in the test. This time, I used --vbr-new engine, which apparently perform better than defaulted mode, especially on low volume signal (and soundtracks are mainly built with low volume parts).


RESULTS


click for log files


COMMENTS:

• lame 3.90.3 is slightly better on average than 3.97.a5 (use this statement with caution: it can’t be confirmed by friedman.exe analysis). The latest alpha had slight problems with ringing (it also appeared on previous test with the same setting but CD encoding). Difference is not dramatic, but I’d use 3.90.3 in order to maximise quality at this setting (or better: resume the test with more sample).
• lame 3.96.1 is bad, but MUCH BETTER here than during previous 96 kbps test.
• VBR encodings are another time not reliable at this low bitrate. Using the alternative VBR engine is not a solution for all audible problems: ringing first, and many other artefacts. –V8 is pathetic (despite of high bitrate!); -V7 better, but still inferior to ABR for higher bitrate.


STATISTICS:

If some people would play with statistic tool, just copy and paste the following table:



• ANOVA Analysis:


• TUKEY PARAMETRIC Analysis [-s 0.1]:




• Bitrate table:






PART II: 128 kbps encodings


Encoders and settings:
• lame 3.90.3 | John33 compile | --alt-preset 128
• lame 3.96.1 | John33 compile | --alt-preset 128
• lame 3.97.a5 | John33 compile | --alt-preset 128 -X 10,10
• lame 3.97.a5 | John33 compile | -V 5
• lame 3.97.a5 | John33 compile | -V 5 --vbr-new

Hardware and software configuration:
…still the same

NOTE ABOUT ENCODINGS:

This time, I’ve compared –V 5 and –V 5 --vbr-new: bitrate are totally different, and it’s a good occasion to see if --vbr-new engine is also better at ~128 kbps compared to defaulted VBR mode.
Important note: this time, --vbr-new doesn’t lead to lower bitrate, but to much higher one (102 vs 138 kbps). Differences could be amazing. Best example: with the 100% spoken sample (Alien 4), -V 5 encoding = 90 kbps and –V 5 --vbr-new = 171 kbps. [The very end of the sample was encoded at 320 kbps, which is probably excessive for near-silence…].



RESULTS


click for log files



COMMENTS:

• lame 3.90.3 is slightly better on average than 3.97.a5 (again, it can’t be confirmed by friedman.exe analysis). It’s an important change, because with CD encoding at the same setting, lame 3.90.3 sounded slightly worse. But 6 samples are probably not enough to be sure about it. Still ringing (slight but existing) issues with 3.97 alpha 5 (I repeat that 3.90.3 is not entirely free of ringing).
• lame 3.96.1 is not as terrible with AC3@48000 than with PCM@44100. But it can’t be recommended.
• VBR –V5 is inferior again to –V5 --vbr-new, on all samples! Extensive tests should be done to confirm it.
• VBR –V5 --vbr-new and ABR 128 are tied. Difference is really marginal (but bitrate is 10 kbps higher with VBR). Again, we should question the theoretical superiority of VBR compared to ABR, and its usefulness. Especially when we have in mind the bloated bitrate which occurs with (apparently) innocent samples. It could be problematic with some movies.


STATISTICS:

If some persons would play with statistic tool, just copy and paste the following table:



• ANOVA Analysis:


• TUKEY PARAMETRIC Analysis [-s 0.1]:
no reliable conclusion

• Bitrate table:





PART II: 160 kbps encodings

I’m K.O. Use original AC3 instead


EDIT: all 6 samples are available HERE (limited availability).

Thank you very much for those extensive results.

I've forgot this one, from Dev0 at 128 kbps :



No real difference between ABR/CBR for the same encoder.
3.96.1 < 3.90.3 < 3.97a5

Wow. guruboolez, I have to thank you for your professional efforts.

Right now I do not need additionnal feedback in the 96-165kbps range.
All those results are very informative to me, and I will adjust parameters according to them.

A new alpha will be available soon:
*vbr is unchanged
*cbr/abr are now using the X 10 mode (remapped to X9), ath levels changed.

I think that this version should reduce dropouts introduced by 3.96 in cbr/abr.

I think I speak on behalf of everyone (pompous thing that I am) when I say I am *really* happy reading this thread. Huge thanks to Guru for testing, and a huge thanks to Gabriel for continuing development. Thanks

To finish with this alpha:
I've tested yesterday the --athlower setting, using different value (from -5 to -15). With LisztBminor.wav, I've noticed the biggest progress using --athlower 10 (compared to --athlower 9 and lower value). --athlower 15 lead to slight additional progress (one artifact was removed).
I've also tested with some other samples. --athlower 10 is apparently a good way to reduce ringing (but not to totally remove it). But I have to precise that I had to increase the listening volume to hear it (mp3GAIN could also reveal some problems inaudible on 'normal' conditions' — I have often experienced that with my portable player).

If some people are interesed to test, I have three other samples that might be interesting:
ftp://ftp2.foobar2000.net/foobar/ATH_LAME.ZIP

--vbr-new has another problem with controlling bitrate bloat. Guruboolez identified that vbr-new does often use 320kbps frames at the ends of an encoded track (during near silence). However, I have noticed that it also does this on tracks with those notorious "hidden songs", when they'll stick an extra song at the end of the final track with several minutes of silence between the two (in the same WAV).

I encoded a track by Duncan Sheik called "Nichiren" and during the silence vbr-new was using 192 and 224kbps frames. Of course this "silence" is not digital silence but "analog silence". At -85dB, it's probably tape hiss but quiet enough to use 32kbps frames, I'd say.