LAME 3.96 vs. 3.90.3 Test, Test/Result Thread Options

[DigitalDictator]

3.93.3 sounds better

I reckon you mean 3.90.3.

[freakngoat]

Oh thanks for catching that. I will go back and edit that...

[LoFiYo]

It looks like no one is testing 3.96b1 vs 3.90.3 any more, but I did spend some time last night trying to ABX Rebel.wav (abr 128 & aps), but failed miserably...

I was able to distinguish between the original and the MP3s in ABC/HR, but couldn't prove it by ABXing.

@[proxima]: Was 3.90.3 the clear winner of the two on this sample, or was it only slightly better? I couldn't tell from your ABX results. If you think it was the clear winner, you must have pretty good ears . Or is it a matter of knowing which section of the sample to focus on?

[Wombat]

@[proxima]: Was 3.90.3 the clear winner of the two on this sample, or was it only slightly better? I couldn't tell from your ABX results. If you think it was the clear winner, you must have pretty good ears  . Or is it a matter of knowing which section of the sample to focus on?

@LoFiYo
Welcome to the club! I have always problems to ABX samples others simply describe as preecho problem samples. Must be an auditory canal defect of a kind

Wombat

[Gabriel]

Must be an auditory canal defect of a kind

By looking at your avatar, probably.

[[proxima]]

@[proxima]: Was 3.90.3 the clear winner of the two on this sample, or was it only slightly better? I couldn't tell from your ABX results. If you think it was the clear winner, you must have pretty good ears  . Or is it a matter of knowing which section of the sample to focus on?

The problem is not only preecho, with --ap 128 the very annoying artifact is chirping/ringing problem. I've redone the test, this time including 3.96b1 vs. 3.90.2 abx results

ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_396b1_128.wav
2R = C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_3902_128.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_396b1_128.wav
1L Rating: 1.0
1L Comment: Chirping/ringing artifacts with high frequencies are very annoying !! The artifacted sound compromise the whole listening pleasure of the sample so that paying attention on other artifacts is very difficult.  Preecho is present for sure too.
---------------------------------------
2R File: C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_3902_128.wav
2R Rating: 2.7
2R Comment: here i can hear the classic preecho artifacts with noise spread before attacks. Chirping/ringing problems absent. This is far better than 1L.
---------------------------------------
ABX Results:
Original vs C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_396b1_128.wav
    12 out of 12, pval < 0.001
Original vs C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_3902_128.wav
    12 out of 12, pval < 0.001
C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_396b1_128.wav vs C:\Documents and Settings\proxima\Desktop\Batch\Nuova cartella\LAME_GUI\rebel_3902_128.wav
    12 out of 12, pval < 0.001

Similar problems are not isolated and can be heard in applaud and campestre sample i already uploaded. 3.90.2 does not suffers of this chirping/ringing issue. I encourage people to confirm this, i find ABXing even a waste of time for such problems. See spectral analisys for rebel.wav, there are droputs with high frequencies above 10 kHz.

[Wombat]

Thanks for the hint Gabriel!

[LoFiYo]

,Mar 25 2004, 09:19 AM]I've redone the test, this time including 3.96b1 vs. 3.90.2 abx results

Thanks for taking the time to redo the test

[tigre]

3.90.3 --aps wins over 3.96b1 --ps for this one: Chanchan1:
The trumpets arround ~ 5.0 sound less clean, like surround/chorus effect added with both, 3.90.3 and 3.96, but 3.96 adds ringing additionally. ABX results:

CODE

foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:38:17

File A: file://N:\lametest\Chanchan\ChanChan1.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_3903_aps.wav

21:40:30 : Trial reset.
21:41:17 : 01/01  50.0%
21:42:20 : 02/02  25.0%
21:42:27 : 03/03  12.5%
21:42:33 : 04/04  6.3%
21:42:41 : 05/05  3.1%
21:43:13 : 06/06  1.6%
21:43:24 : 07/07  0.8%
21:43:44 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
--------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:44:17

File A: file://N:\lametest\Chanchan\ChanChan1.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_396b1_ps.wav

21:45:06 : 01/01  50.0%
21:45:10 : 02/02  25.0%
21:45:14 : 03/03  12.5%
21:45:18 : 04/04  6.3%
21:45:25 : 05/05  3.1%
21:45:29 : 06/06  1.6%
21:45:32 : 07/07  0.8%
21:45:35 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
---------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:45:47

File A: file://N:\lametest\Chanchan\chanchan1.wav_3903_aps.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_396b1_ps.wav

21:47:09 : 01/01  50.0%
21:47:14 : 02/02  25.0%
21:47:18 : 03/03  12.5%
21:47:25 : 04/04  6.3%
21:47:42 : 04/05  18.8%
21:48:56 : 05/06  10.9%
21:49:12 : 06/07  6.3%
21:49:40 : 07/08  3.5%
21:49:57 : 08/09  2.0%
21:50:56 : 09/10  1.1%
21:51:02 : 10/11  0.6%
21:51:08 : 11/12  0.3%

----------
Total: 11/12 (0.3%)

[tigre]

Rebel sample 3.90.3 --alt-preset standard vs. 3.96b1 --preset standard:
3.90.3 won
ABXed 3.0 - 6.0. Noticable pre-echo at drum hits, 3.96 is worse, slightly ringing/metallic sounding
ABX results:

foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:19:33

File A: file://N:\lametest\rebel.wav
File B: file://N:\lametest\rebel.wav_396b1_ps.mp3

22:20:30 : 01/01  50.0%
22:20:34 : 02/02  25.0%
22:20:38 : 03/03  12.5%
22:20:42 : 04/04  6.3%
22:20:45 : 05/05  3.1%
22:20:52 : 06/06  1.6%
22:20:55 : 07/07  0.8%
22:21:01 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
------------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:21:28

File A: file://N:\lametest\rebel.wav
File B: file://N:\lametest\rebel.wav_3903_aps.mp3

22:22:20 : 01/01  50.0%
22:22:24 : 02/02  25.0%
<snip>
22:25:39 : 19/25  0.7%
22:25:47 : 20/26  0.5%
22:25:55 : 21/27  0.3%
22:26:06 : 22/28  0.2%

----------
Total: 22/28 (0.2%)
------------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:26:22

File A: file://N:\lametest\rebel.wav_3903_aps.mp3
File B: file://N:\lametest\rebel.wav_396b1_ps.mp3

22:26:50 : 01/01  50.0%
22:26:54 : 02/02  25.0%
22:27:00 : 03/03  12.5%
22:27:03 : 04/04  6.3%
22:27:06 : 05/05  3.1%
22:27:09 : 06/06  1.6%
22:27:15 : 07/07  0.8%
22:27:18 : 08/08  0.4%

----------
Total: 8/8 (0.4%)

[Jebus]

I prefer 3.90.3 --alt-preset standard to 3.96b1 --preset standard on these samples.
Destitute Rosemary

no ABX results makes your claim entirely useless to us

[PVNC]

I prefer 3.90.3 --alt-preset standard to 3.96b1 --preset standard on these samples.
Destitute Rosemary 

no ABX results makes your claim entirely useless to us

Check his links, man.

[harashin]

I've had some more tests.

Hosokawa___Atem_lied
Original vs 3.96b1aps (10/10)
Original vs 3.90.3aps (13/14)
3.90.3aps vs 3.96b1aps (10/10)

Chanchan1
Original vs 3.96b1aps (10/10)
Original vs 3.90.3aps (15/18)
3.90.3aps vs 3.96b1aps (10/11)

[dev0]

Gabriel released 3.96b2 today. Download it here.
The Minimal bitrate for --preset standard has been 'adjusted' to 128kbps, so all samples where 3.96b1 did worse than 3.90.3 need retesting/validation.

[LoFiYo]

I tested 3.96b2 vs 3.90.3 with MYF_4SEC.WAV. On this sample, 3.90.3 still sounds much better (closer to the original) to me. I did notice some improvement with 3.96b2 over 3.96b1, I think because it no longer drops down to 96kbps, but the distortion at the 3rd blow is still noticeable. 3.90.3 doesn't have that distortion and sounds much cleaner.

WinABX v0.22 (ABA) test report
03/28/2004 14:49:52

A file: C:\My Music\lab\MYF-4SEC\TEST.WAV
B file: C:\My Music\lab\MYF-4SEC\aps396b2.mp3.wav

14:50:37    1/1  p=33.3%
14:50:58    2/2  p=11.1%
14:51:23    3/3  p= 3.7%
14:51:54    4/4  p= 1.2%
14:52:13    5/5  p= 0.4%
14:53:33    6/6  p= 0.1%
14:53:41    7/7  p< 0.1%
14:57:03  test finished

-------------------------------------
WinABX v0.22 (ABA) test report
03/28/2004 14:57:49

A file: C:\My Music\lab\MYF-4SEC\TEST.WAV
B file: C:\My Music\lab\MYF-4SEC\aps3903.mp3.wav

14:59:12    0/1  p=100.0%
15:00:59    0/2  p=100.0%
15:01:10    1/3  p=70.4%
15:01:19    2/4  p=40.7%
15:01:54    2/5  p=53.9%
15:02:30    2/6  p=64.9%
15:02:36    2/7  p=73.7%
15:02:44    3/8  p=53.2%
15:02:55    4/9  p=35.0%
15:03:18   5/10  p=21.3%
15:03:30   6/11  p=12.2%
15:03:49   7/12  p= 6.7%
15:04:01   8/13  p= 3.5%
15:04:15   9/14  p= 1.8%
15:04:33  10/15  p= 0.9%
15:04:43  11/16  p= 0.4%
15:04:53  12/17  p= 0.2%
15:04:58  13/18  p= 0.1%
15:05:04  14/19  p< 0.1%
15:05:10  test finished

-------------------------------------
WinABX v0.22 (ABA) test report
03/28/2004 15:05:12

A file: C:\My Music\lab\MYF-4SEC\TEST.WAV
15:05:18  test finished

-------------------------------------
WinABX v0.22 (ABA) test report
03/28/2004 15:05:19

A file: C:\My Music\lab\MYF-4SEC\aps3903.mp3.wav
B file: C:\My Music\lab\MYF-4SEC\aps396b2.mp3.wav

15:05:49    1/1  p=33.3%
15:05:57    2/2  p=11.1%
15:06:03    3/3  p= 3.7%
15:06:08    4/4  p= 1.2%
15:06:14    5/5  p= 0.4%
15:06:22    6/6  p= 0.1%
15:06:32    7/7  p< 0.1%
15:06:35  test finished

Edit: for clarity & more detail

This post has been edited by LoFiYo: Mar 29 2004, 01:51

[harashin]

I still prefer 3.90.3 on these samples.
Hustlejet
Original vs 3.96b2aps (13/14)
Original vs 3.90.3aps (15/20)
3.90.3aps vs 3.96b2aps (12/14)

Hosokawa___Atem_lied
Original vs 3.96b2aps (10/10)
Original vs 3.90.3aps (13/14)
3.90.3aps vs 3.96b2aps (10/10)

[FatBoyFin]

Only for 3.96b1

Confirming Fatboy sample

3.90.3 --alt-preset 128 Vs WAV
8/8
Heavy Distortion

3.96b1 -preset 128 vs WAV
7/7
Lighter Distortion


3.90.3 --alt preset 128 vs 3.96b --preset 128
6/6

Conclusion: 3.96b1 has far less distortion
3.96b1 wins

[ViPER1313]

v3.90.3 vs 3.96b2 using --preset standard on sample 41_30sec.flac

I would call this test a tie between the two encoders.

ABC/HR Version 0.9b, 30 August 2002
Testname: Lame v3.90.3 vs v3.96b2 --preset standard

1R = Z:\Music\Test Samples\testTemp\Lame3.90.3\41_30sec.wav
2L = Z:\Music\Test Samples\testTemp\Lame3.96b2\41_30sec.wav

---------------------------------------
General Comments:
Both files suffer from pre-echo in the symbols from 0-2.3 seconds. I cant ABX the 2 files with any confidence.
---------------------------------------
1R File: Z:\Music\Test Samples\testTemp\Lame3.90.3\41_30sec.wav
1R Rating: 4.5
1R Comment:
---------------------------------------
2L File: Z:\Music\Test Samples\testTemp\Lame3.96b2\41_30sec.wav
2L Rating: 4.5
2L Comment:
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\testTemp\Lame3.90.3\41_30sec.wav
    10 out of 10, pval < 0.001
Original vs Z:\Music\Test Samples\testTemp\Lame3.96b2\41_30sec.wav
    10 out of 10, pval < 0.001
Z:\Music\Test Samples\testTemp\Lame3.90.3\41_30sec.wav vs Z:\Music\Test Samples\testTemp\Lame3.96b2\41_30sec.wav
    25 out of 44, pval = 0.226

This post has been edited by ViPER1313: Mar 30 2004, 00:22

[plonk420]

i tried my own (non-specifically chosen) samples but i couldn't do jack with them. i think i'll try some of the samples that came in the AAC-HE+PS test and maybe the one before that test, too to save time... (gotta hunt down the codec killer samples i downloaded somewhere on this computer, too)

[tigre]

I repeated (alt)preset standard test with cantwait sample and 3.90.3, 3.96b1 and b2. Result: If there's any change between 3.96b1 and b2, I didn't have enough patience to find and ABX it. ABXing original vs. 3.96b2 and 3.90.3 vs. 3.96b2 was as easy as with b1 before. ABC/HR log:

ABC/HR Version 0.9b, 30 August 2002
Testname: cantwait 3.96b2 / b1 / 3.90.3 standard

1L = N:\lametest\Cantwait\cantwait.wav_396b1_ps.wav
2L = N:\lametest\Cantwait\cantwait.wav_3903_aps.wav
3L = N:\lametest\Cantwait\cantwait_396b2_aps.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: N:\lametest\Cantwait\cantwait.wav_396b1_ps.wav
1L Rating: 2.5
1L Comment:
---------------------------------------
2L File: N:\lametest\Cantwait\cantwait.wav_3903_aps.wav
2L Rating: 3.5
2L Comment:
---------------------------------------
3L File: N:\lametest\Cantwait\cantwait_396b2_aps.wav
3L Rating: 2.5
3L Comment:
---------------------------------------
ABX Results:
Original vs N:\lametest\Cantwait\cantwait.wav_396b1_ps.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Cantwait\cantwait.wav_3903_aps.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Cantwait\cantwait_396b2_aps.wav
    7 out of 7, pval = 0.008
N:\lametest\Cantwait\cantwait.wav_396b1_ps.wav vs N:\lametest\Cantwait\cantwait.wav_3903_aps.wav
    7 out of 7, pval = 0.008
N:\lametest\Cantwait\cantwait.wav_396b1_ps.wav vs N:\lametest\Cantwait\cantwait_396b2_aps.wav
    6 out of 12, pval = 0.613
N:\lametest\Cantwait\cantwait.wav_3903_aps.wav vs N:\lametest\Cantwait\cantwait_396b2_aps.wav
    7 out of 7, pval = 0.008

Result in short:
No improvement with 3.96b2

[FatBoyFin]

I dont know whether this is needed as it isnt an -aps test but ill post it anyway.
Testing Fatboy with 3.96b2 at 128 ABR.

Fatboy.wav vs 3.96b2 --preset 128
9/9
Some distortion

fatboy.wav vs 3.90.3 --alt-preset 128
9/9
Heavy distorion

3.96b2 vs 3.90.3
9/9
3.90.3 is far worse than 3.96b2

3.96b2 is still better than 3.90.3 for fatboy.wav

[tigre]

FatBoyFin, thanks for your effort, but ...

If I understand Gabriel here correctly, the quality related change between 3.96b1 and b2 is the minimum bitrate for -V2 (= preset standard) and -V1 (extreme). So the results for lower -V settings and abr/cbr presets should be identical. Because of this, 3.96b1 and b2 results for identical samples won't be added separately to the results post for -V 3 and lower and ABR/CBR presets.

Of course testing both can never do a harm, but since there aren't many results for < standard settings that match the test requirements (ABX results etc.) so far anyway, just testing 3.96b2 vs. 3.90.3 in ~ 128-160kbps range without 'looking back' should be enough. Testing -V 5 (and comparing with (alt)preset 128) additionally would be great too.

BTW: FatBoyFin, have you tried to ABX 3.96b1 vs. 3.96b2?

[LoFiYo]

I tested 3.96b2 --preset 160 vs 3.90.3 --alt-preset 160 vs 3.96b2 -V4 using MYF_4SEC.wav. V4 was really bad on this sample, and I think the 2 abr encodes were pretty much a tie, but I felt 3.96b2 was slightly better. I could not ABX 3.96b2 abr160 vs the original, mainly because I think my ears were really tired, but also because it was well encoded. Of course, I couldn't ABX the 2 abr encodes at all.

ABC/HR Version 0.9b, 30 August 2002
Testname: MyF_4sec - V4 vs abr160

1R = C:\My Music\lab\MYF-4SEC\396b2-abr160.mp3.wav
2R = C:\My Music\lab\MYF-4SEC\3903-abr160.mp3.wav
3L = C:\My Music\lab\MYF-4SEC\396b2-V4.mp3.wav

---------------------------------------
General Comments:
#1 and #2 are similar in quality. ABX failed between #1 & #2. #3 is really bad.
---------------------------------------
1R File: C:\My Music\lab\MYF-4SEC\396b2-abr160.mp3.wav
1R Rating: 4.0
1R Comment: ABX failed because of fatigue and also because encode is good.
---------------------------------------
2R File: C:\My Music\lab\MYF-4SEC\3903-abr160.mp3.wav
2R Rating: 3.5
2R Comment: Slight distortion on the first blow but not really bad.
---------------------------------------
3L File: C:\My Music\lab\MYF-4SEC\396b2-V4.mp3.wav
3L Rating: 1.5
3L Comment: Distortion is very noticeable throughout the file.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\MYF-4SEC\396b2-abr160.mp3.wav
    18 out of 32, pval = 0.298
Original vs C:\My Music\lab\MYF-4SEC\3903-abr160.mp3.wav
    7 out of 7, pval = 0.008
Original vs C:\My Music\lab\MYF-4SEC\396b2-V4.mp3.wav
    13 out of 14, pval < 0.001
C:\My Music\lab\MYF-4SEC\396b2-abr160.mp3.wav vs C:\My Music\lab\MYF-4SEC\3903-abr160.mp3.wav
    1 out of 2, pval = 0.750
C:\My Music\lab\MYF-4SEC\396b2-abr160.mp3.wav vs C:\My Music\lab\MYF-4SEC\396b2-V4.mp3.wav
    13 out of 14, pval < 0.001
C:\My Music\lab\MYF-4SEC\3903-abr160.mp3.wav vs C:\My Music\lab\MYF-4SEC\396b2-V4.mp3.wav
    10 out of 10, pval < 0.001

[FatBoyFin]

BTW: FatBoyFin, have you tried to ABX 3.96b1 vs. 3.96b2?

I havent ABX between these but the file sizes are identical so I presume they are exactly the same.

[FatBoyFin]

As there is a new 128Kbps test coming up I redid my Fatboy test to include the -V5 setting:

-V5 ,3.96b2 vs WAV
6/6
Hard to abx

ap128 ,3.90.3 vs WAV
8/8
Heavy Distortion

-V5 ,3.96b2 vs ap128 ,3.90.3
5/5

-preset128 ,3.96b2 vs WAV
7/7
Lighter Distortion

preset 128 vs -v 5
6/6
-V5 is alot better


Conclusion:
-V5 > --preset 128 > --alt-preset128 for FatBoy.wav

PS the bitrate for -V 5 was a huge 220 KBps could someone please check that I have encoded this correctly or should I not be using -V5?