LAME 3.96 vs. 3.90.3 Test, Test/Result Thread Options

[tigre]

Sample : Its me ~ 128kbps tested

Live recording, focussed on problems with applause, details see ABC/HR log
3.96b2 -V 5 won (3.0)
3.90.3 ap 128 was second (2.0)
3.96b2 p 128 was worst (1.5)

ABC/HR log (I tested vorbis codecs with HF reduction as well, to save space here I cut out the related parts in ABC/HR log.)

ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = N:\lametest\Soul_Live\decode\017 Its me.wav_mtb3_4.wav
2L = N:\lametest\Soul_Live\decode\011 Its me.wav_23_4.wav
3L = N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
4R = N:\lametest\Soul_Live\decode\015 Its me.wav_cqk32_4.25.wav
5R = N:\lametest\Soul_Live\decode\010 Its me.wav.wav
6R = N:\lametest\Soul_Live\decode\016 Its me.wav_hfr_4.25.wav
7R = N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
8R = N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav

---------------------------------------
General Comments:

---------------------------------------
<snip>
---------------------------------------
3L File: N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
3L Rating: 2.0
3L Comment: 0.0-3.4: surround-like, warbeling/metallic sound in applause, no hf noise boost -> should be lame.
---------------------------------------
<snip>
---------------------------------------
7R File: N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
7R Rating: 3.0
7R Comment: should be lame; small ringing on right channel, otherwise quite good
---------------------------------------
8R File: N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
8R Rating: 1.5
8R Comment: sounds like 3 with hf ringing problems of 7 added
---------------------------------------
ABX Results:
<snip>
Original vs N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav
    7 out of 7, pval = 0.008
<snip>
Original vs N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    7 out of 7, pval = 0.008
<snip>
N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav vs N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav
    11 out of 12, pval = 0.003
N:\lametest\Soul_Live\decode\012 Its me.wav_3903_ap128.wav vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    11 out of 12, pval = 0.003
<snip>
N:\lametest\Soul_Live\decode\014 Its me.wav_396b2_V5.wav vs N:\lametest\Soul_Live\decode\013 Its me.wav_396b2_p128.wav
    7 out of 7, pval = 0.008

[tigre]

Sample : entierren con rumba ~ 128kbps tested

Salsa with lots of percussion, details see ABC/HR log, results similar to the previous sample I tested
3.96b2 -V 5 won
3.90.3 ap 128 was second
3.96b2 p 128 was worst

ABC/HR log:

ABC/HR Version 0.9b, 30 August 2002
Testname: entierren con rumba lame 128

1L = N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
2L = N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
3L = N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav

---------------------------------------
General Comments:
Guiros (percussion) smeared, clearly noticable (and ABXed) at 2.8-6.0
1 is best, 2 and 3 are similar but 2 has ringing/metallic sound additionally.
After 6.0 it's similar with the additonal precussion instrument (shaker?).
---------------------------------------
1L File: N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
1L Rating: 3.5
1L Comment:
---------------------------------------
2L File: N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
2L Rating: 2.0
2L Comment:
---------------------------------------
3L File: N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
3L Rating: 2.5
3L Comment:
---------------------------------------
ABX Results:
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
    7 out of 7, pval = 0.008
Original vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_V5_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008
N:\lametest\Salsa\decode\entierren con rumba.wav_396b2_p128_.wav vs N:\lametest\Salsa\decode\entierren con rumba.wav_3903_ap128_.wav
    7 out of 7, pval = 0.008

____________________________________________________________________


BTW: IT WOULD BE GREAT IF SOME MORE PEOPLE COULD PROVIDE ~128KBPS RESULTS BEFORE RJAMORIM'S 128KBPS TEST STARTS.

[ViPER1313]

Lame v3.96 final seems to have a serious problem w/ 41_30sec in the 6-8 second range using --preset cbr 128. It sounds like a dropout / skip in the sound. This problem has been seen in Mitok's compile as well as ICL v8.0 and MSVC compiles made by myself. There is no skip using v3.90.3 . Could someone confirm this??

EDIT - Look at the waveforms - the volumes between the wav/3.90.3 and 3.96 are drastically different.

EDIT2 - Lame v3.96b2 does not suffer from this problem either.

Origional WAV:



Lame v3.90.3 --preset cbr 128:



Lame v3.96 --preset cbr 128:



This post has been edited by ViPER1313: Oct 20 2006, 09:22

[SometimesWarrior]

Lame v3.96 final seems to have a serious problem w/ 41_30sec in the 6-8 second range using --preset cbr 128. It sounds like a dropout / skip in the sound. This problem has been seen in Mitok's compile as well as ICL v8.0 and MSVC compiles made by myself. There is no skip using v3.90.3 . Could someone confirm this??

I can confirm it. I tested the 4.5-8.5 second range of 41_30sec, using Mitiok's 3.96 and john33's 3.90.3 and 3.96b2. The 3.96 final version definitely has some nasty dropouts not present the other encodes.

CODE

ABC/HR Version 0.9b, 30 August 2002
Testname:

1L = G:\Filetemp\diskwriter\41_30\41-396.wav
2L = G:\Filetemp\diskwriter\41_30\41-3903.wav
3R = G:\Filetemp\diskwriter\41_30\41-396b2.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: G:\Filetemp\diskwriter\41_30\41-396.wav
1L Rating: 1.0
1L Comment: Pops or dropouts
---------------------------------------
2L File: G:\Filetemp\diskwriter\41_30\41-3903.wav
2L Rating: 2.5
2L Comment:
---------------------------------------
3R File: G:\Filetemp\diskwriter\41_30\41-396b2.wav
3R Rating: 2.2
3R Comment: A bit more swirling than sample 2
---------------------------------------
ABX Results:
G:\Filetemp\diskwriter\41_30\41-396.wav vs G:\Filetemp\diskwriter\41_30\41-3903.wav
   8 out of 8, pval = 0.004
G:\Filetemp\diskwriter\41_30\41-396.wav vs G:\Filetemp\diskwriter\41_30\41-396b2.wav
   8 out of 8, pval = 0.004
G:\Filetemp\diskwriter\41_30\41-3903.wav vs G:\Filetemp\diskwriter\41_30\41-396b2.wav
   11 out of 12, pval = 0.003

This post has been edited by SometimesWarrior: Apr 12 2004, 00:35

[Brainchild4010]

Yes , i also can confirm it.

I test it too with 41_30sec file and at the 4.5 - 6 is definitely a Pop with Lame 3.96 Final.

No Problems with Lame 3.90.3 and 3.96b2.

[ViPER1313]

I have created a gif animation that shows how the volume levels seem to change / drop out in the v3.96 final encoder when using 128kbps and to a lesser extent 160kbps - it can be found here. Take a careful look at v3.90.3 vs. v3.96. More testing on this issue and bitrates compromised by it need to be performed.

This is just a random section (1.6-1.8sec) of the 41_30 clip, and this section is ABX-able (sounds like a click / popping).

This post has been edited by ViPER1313: Apr 12 2004, 23:20

[ff123]

I have created a gif animation that shows how the volume levels seem to change / drop out in the v3.96 final encoder when using 128kbps and to a lesser extent 160kbps - it can be found  here. Take a careful look at v3.90.3 vs. v3.96. More testing on this issue and bitrates compromised by it need to be performed.

This is just a random section (1.6-1.8sec) of the 41_30 clip, and this section is ABX-able (sounds like a click / popping).

Looks like a bug. Does this occur with abr as well (p 128)?

That could be a reason to eliminate 3.96final from consideration in Roberto's upcoming multiformat test.

ff123

[ViPER1313]

From what I have seen and heard it does not seem to occur in ABR or VBR modes, although the encoder could just be throwing bits to where they are needed. The problems shown in 41_30sec seem to disappear using --preset 128, so IMO ABR seems fine.

This post has been edited by ViPER1313: Apr 12 2004, 23:38

[Fr4nz]

OMG! At www.cdfreaks.com have published this new Lame version! Someone should told them to remove the news because of this buggy version.

[damiandimitri]

thank you for the warning.....

i will try to contact an operator at cdfreaks

[Debro]

Cdfreaks has a disclaimer at the bottom of the article warning users of the bug

[Gabriel]

I would like people testing a little more than 1 sample before calling a version "buggy"

edit: additionnaly it could help if you could find other clearly problematic samples where you hear a similar behaviour (problem in cbr but not in abr)

This post has been edited by Gabriel: Apr 13 2004, 10:55

[funkyblue]

I agree with Gabriel, who is to say that Lame 3.96 may break one sample, but fix 25 other samples? Unless it is tested more, you can't judge

This post has been edited by burgerings: Apr 13 2004, 10:05

[Vietwoojagig]

I would like people testing a little more than 1 sample before calling a version "buggy"

edit: additionnaly it could help if you could find other clearly problematic samples where you hear a similar behaviour (problem in cbr but not in abr)

But, on the other hand, it would be good to know, why this sample is problematic with 3.96 and not with 3.96b2. Do you have any guess?

[ViPER1313]

Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final. Tested in the 0-2.2 second region of the song. While there are no clicks, there seems to be a pretty major quality regression between v3.96b2 and v3.96 final, while 3.90.3 still comes out as the winner.

ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset cbr 128

1L = Z:\Music\Temp\3.96 Final.wav
2R = Z:\Music\Temp\3.90.3.wav
3L = Z:\Music\Temp\v3.96b2.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: Z:\Music\Temp\3.96 Final.wav
1L Rating: 3.7
1L Comment: Horrible warble / distortion in the 0-2 second region
---------------------------------------
2R File: Z:\Music\Temp\3.90.3.wav
2R Rating: 4.5
2R Comment: Best of the 3, very clean, no major distortions.
---------------------------------------
3L File: Z:\Music\Temp\v3.96b2.wav
3L Rating: 4.3
3L Comment: Slightly worse than 2, much better than 1, slight warble on the beginning pronunciation of "been."
---------------------------------------
ABX Results:
Original vs Z:\Music\Temp\3.96 Final.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.90.3.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\v3.96b2.wav
    8 out of 8, pval = 0.004
Z:\Music\Temp\3.96 Final.wav vs Z:\Music\Temp\3.90.3.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 Final.wav vs Z:\Music\Temp\v3.96b2.wav
    16 out of 17, pval < 0.001
Z:\Music\Temp\3.90.3.wav vs Z:\Music\Temp\v3.96b2.wav
    18 out of 20, pval < 0.001

[Gabriel]

Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final.

What about abr?

[ViPER1313]

Tested "Waiting" sample at --preset cbr 128 using Lame v3.90.3, 3.96b2 and v3.96 final.

What about abr?

Your wish is my command - "--preset 128" using v3.90.3, 3.96b2 and 3.96 final.

ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset 128

1R = Z:\Music\Temp\3.96 ABR.wav
2L = Z:\Music\Temp\3.90.3 ABR.wav
3L = Z:\Music\Temp\3.96b2 ABR.wav

---------------------------------------
General Comments:
All samples did better than in CBR, and no codec sounded horrible.
---------------------------------------
1R File: Z:\Music\Temp\3.96 ABR.wav
1R Rating: 4.1
1R Comment: Felt this had slightly more warble than 3 and is clearly worse than 2, although  I am not 100% sure that I can ABX it against 3.
---------------------------------------
2L File: Z:\Music\Temp\3.90.3 ABR.wav
2L Rating: 4.6
2L Comment: Better than 1 or 3, sounded cleaner
---------------------------------------
3L File: Z:\Music\Temp\3.96b2 ABR.wav
3L Rating: 4.2
3L Comment: Slight warble, thought it was slightly better than 1 but could not ABX it with 100% confidence - got a 15 of  17, p = 0.001 but with later trials could not ABX - might just be listening fatigue.
---------------------------------------
ABX Results:
Original vs Z:\Music\Temp\3.96 ABR.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.90.3 ABR.wav
    8 out of 8, pval = 0.004
Original vs Z:\Music\Temp\3.96b2 ABR.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 ABR.wav vs Z:\Music\Temp\3.90.3 ABR.wav
    13 out of 14, pval < 0.001
Z:\Music\Temp\3.96 ABR.wav vs Z:\Music\Temp\3.96b2 ABR.wav
    28 out of 42, pval = 0.022
Z:\Music\Temp\3.90.3 ABR.wav vs Z:\Music\Temp\3.96b2 ABR.wav
    10 out of 10, pval < 0.001

[ViPER1313]

At this point in time, I would sum up the whole situation by saying:

• I don't know if 41_30sec is an isolated case or not - I agree that more samples are needed before any judgment can be made.
• I have only seen the problem in --preset cbr 128 - other bitrates/modes don’t exhibit the problem in the same way or only to a degree that cannot be easily heard.
• I would not go so far as to call v3.96 buggy, but more testing needs to be done on the final version.
• v3.90.3 still seems to have higher quality in most samples at most bitrates (with the exception of fatboy and a few others)
• MORE TESTING OF v3.96 IS NEEDED

If other people could do some tests to prove or disprove what I have found, it would be a large help.

[Gabriel]

I don't know if 41_30sec is an isolated case or not - I agree that more samples are needed before any judgment can be made.

I can confirm that there is a failure on this sample on frame 272, 2nd granule (7.11s) when encoding in cbr 128. The output on mp3X is very strange.

[tigre]

OK. I've started a LAME 3.96 FINAL vs. 3.90.3 Test, Test/Result Thread.

I'm a bit short of time right now - I'll add missing results to the related posts ASAP if noone's faster.


Gabriel:

I can confirm that there is a failure on this sample on frame 272, 2nd granule (7.11s) when encoding in cbr 128. The output on mp3X is very strange.

Is this something that is going to be fixed soon (so we can expect 3.96.1 or similar in a couple of days and should better wait with testing 3.96 final) or is it just the price that has to be paid for better performance elsewhere?

-----------------------------------------------------------------
DJED's post split to a separate thread.

[Gabriel]

Is this something that is going to be fixed soon (so we can expect 3.96.1 or similar in a couple of days and should better wait with testing 3.96 final) or is it just the price that has to be paid for better performance elsewhere?

Well, I think that this specific one should be fixed, but I do not know how soon this will be done.
My opinion is that you can test 3.96.

[tigre]

I've just edited all relevant 3.96beta results that were still missing in the results post.

For further testing of 3.96 final as well as for continuing related discussion go here please.

If necessary I can start a separate thread for 3.96beta -> final regression cases and move related posts there. Opinions please?