LAME 3.96 FINAL vs. 3.90.3 Test, Test/Result Thread Options

[tigre]

This is stolen from here mostly, since only few things have changed...
----------------------------------------------------------------------

The purpose of this thread is to (finally) test LAME 3.96 throughly enough to make it the new recommended LAME version and enhance communication between LAME developers and the HA.org community. The recent confusion about the different LAME versions and compiles (-Qrcd trouble etc.) have led to the decision that moving along with LAME development has been postponed for long enough and LAME 3.90.X can hopefully be declared dead very soon.
Please test as many samples as possible according to the following guidelines and post any results here.


1. Use the following LAME compiles (last updated on 2004/04/13):
lame3.90.3
lame3.96 final

2. The focus of the test should be --alt-preset/--preset standard, since it will allow us to make conclusions regarding the overall performance of the 'code level tweaked' VBR presets. Other VBR/ABR/CBR presets are interesting too, but not as important. If problems with --alt preset standard are detected feel free to compare extreme and insane too. Test the following combinations please:

VBR/ABR
(~256kbps) 3.96 --preset extreme vs. 3.90.3 --alt-preset extreme
(~210kbps) 3.96 --preset standard vs. 3.90.3 --alt-preset standard
(~160kbps) 3.96 -V 4 vs. 3.96 --preset 160 vs. 3.90.3 --alt-preset 160*
(~128kbps) 3.96 -V 5 vs. 3.96 --preset 128 vs. 3.90.3 --alt-preset 128*

CBR
(320kbps) 3.96 --preset insane vs. 3.90.3 --alt-preset insane
3.96 --preset cbr <bitrate> vs. 3.90.3 --alt-preset cbr <bitrate>

If you want, you can additionally test VBR/ABR vs. CBR at comparable bitrates

3. You may use any sample you want, as long as you upload a losslessly compressed version of that (or provide a working link), so others can verify your results. The upload thead is here. Using samples that have been used for testing 3.96beta is a good idea to show progress/regressions.

4. Your test results have to include the following:

• ABX results for
  3.90.3 vs. Original
  3.96 vs. Original
  3.96 vs. 3.90.3
• ABC/HR results are appreciated especially at lower bitrates, but shouldn't be considered a requirement.
• (Short) descriptions of the artifacts/differences

Notes:

*
Since ABR presets decrease gain as workarround against audible clipping on decoding of loud tracks, please use replaygain (trackgain) for decoding to .wav / ABXing, ideally using foobar2000 (apply to the original as well to get matching volume, of course!). On quiet tracks --scale 1 can be added after --(alt-)preset <bitrate> on encoding alternatively. Details see here

There won't be a separate discussion thread for now. To make it easy to find posts containing results, please start such posts (only!) with "RESULT". Results matching the requirements that have been posted elsewhere will be added to the "Results" post.

-----------------------------------------------
Edit 1 [2004/04/13]: typo fixed (3.96b2 -> 3.96), thanks to Lyx
Edit 2 [2004[04[14]: Replaygain/--scale related comment added. (-> "*")

This post has been edited by tigre: Apr 14 2004, 19:03

[[proxima]]

I just tested and uploaded two more samples provided by guruboolez in the past: it seems that --athaa-sensitivity 1 makes things better for ringing/high pitched noise with -V 5. Both samples should be very easy, with Bayle sample ringing artifatcs are very obvious and annoying.

ABC/HR Version 0.9b, 30 August 2002
Testname: Bayle

1L = C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_V5.wav
2L = C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_athaa.wav

---------------------------------------
General Comments:
Focising on 0:00 - 0:04 range.
---------------------------------------
1L File: C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_V5.wav
1L Rating: 2.0
1L Comment: ringing, fluctuant HF noise
---------------------------------------
2L File: C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_athaa.wav
2L Rating: 3.0
2L Comment: ringing, fluctuant HF noise is still present but reduced and less disturbing.
---------------------------------------
ABX Results:
Original vs C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_V5.wav
    10 out of 10, pval < 0.001
Original vs C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_athaa.wav
    10 out of 10, pval < 0.001
C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_V5.wav vs C:\LAME_GUI\Bayle - Jeita or Waters' Murmur - Etching_athaa.wav
    11 out of 12, pval = 0.003

ABC/HR Version 0.9b, 30 August 2002
Testname: Avison

1L = C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_V5.wav
2R = C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_athaa.wav

---------------------------------------
General Comments:
ringing + chirping (high pitched notes)
---------------------------------------
1L File: C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_V5.wav
1L Rating: 2.5
1L Comment: ringing with crackles at ~ 2 sec. and after
chirping (high pitched notes) starting from 22 sec.
---------------------------------------
2R File: C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_athaa.wav
2R Rating: 3.5
2R Comment: ringing is still audible but reduced
no chirping at all
all HF artifacts are reduced
---------------------------------------
ABX Results:
Original vs C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_V5.wav
    10 out of 10, pval < 0.001
Original vs C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_athaa.wav
    10 out of 10, pval < 0.001
C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_V5.wav vs C:\LAME_GUI\Track01 (Avison - Concerto Grosso 30 sec)_athaa.wav
    10 out of 10, pval < 0.001

These two samples are not isolated cases, i choose them only because i can hear obvious problems with plain -V 5. According to my tastes -V5 with --athaa-sensitivity 1 is better with Atrain, LisztBMinor, Blackwater, BachS1007, Bayle, Avison samples.