Lame 3.95 vs 3.90.3, ABC/HR test results here only Options

[AstralStorm]

Starting with mine:

ABC/HR for Java Version 0.4b, 12 styczeñ 2004
Testname: velvet (aps)

1R = C:\Music\LAMEsamples\velvet.3-90-3.wav
2L = C:\Music\LAMEsamples\velvet.3-95.wav

---------------------------------------
General Comments: Original vs Sample 1 - warm up, but still managed to ABX it at 32 tries. (I use techinque of min. 16x, 24x, 32x) Most noticeable artifacti 8.11-9.30
---------------------------------------
1R File: C:\Music\LAMEsamples\velvet.3-90-3.wav
1R Rating: 4.0
1R Comment: Slightly smeared percussion, no ATH problem
---------------------------------------
2L File: C:\Music\LAMEsamples\velvet.3-95.wav
2L Rating: 3.0
2L Comment: Smeared percussion, ATH problems
---------------------------------------

ABX Results:
Original vs C:\Music\LAMEsamples\velvet.3-95.wav
16 out of 16, pval < 0.001
Original vs C:\Music\LAMEsamples\velvet.3-90-3.wav
23 out of 32, pval = 0.01
--

Bad news. 3.95 still worse, noticeably.
I didn't even try to ABX these against themselves, the difference is glaring.

EDIT: Yes, glaring. Glaring PLACEBO. Failed ABX, pval ~0.4 at 32 tries.
Maybe I've got tired of this percussion, I'll redo this tomorrow.

EDIT2: Failed again, similar result.

This post has been edited by AstralStorm: Jan 13 2004, 22:37

[rjamorim]

Nice initiative. Thanks

[AstralStorm]

Oh, I see that there's 3.95.1 out now.
I'll retest with this later. (although the chance is slim the result may be different)
Next sample tomorrow.

Bug report - Lame 3.95.x can't decode at all. (jon33 compile)
Even can't decode its own files!

This post has been edited by AstralStorm: Jan 13 2004, 00:20

[PlazzTT]

Bug report - Lame 3.95.x can't decode at all. (jon33 compile)
Even can't decode its own files!

Verified.

[sony666]

foobar 0.77a ABX, replaygain, no DSP
Sony MDR-P70 headphones, somewhat noisy PC, Terratec PCI soundcard 5 years old

the candidate: fatboy

1) --preset cbr 128
-wav vs. 3.90.3
outrageous, mp3 sounds very "scratchy".. 8/8, 0.4% guess prob.

-wav vs. 3.95.1
much better than 3.90.3, still easy to pick from wav. 8/8, 0.4%

-3.90.3 vs. 3.95.1:
3.90.3 very severe artifacts, 3.95.1 much better. 8/8, 0.4%

clear winner 3.95.1
---

2) --preset 160 (ABR)
wav vs. 3.90.3 (191kbit):
still very scratchy vocals, slightly better than 128CBR. 8/8 0,4%

wav vs. 3.95.1 (204kbit):
needs utmost concentration, scratching is gone 10/16, 22.7% guess prob.

3.90.3 vs. 3.95.1:
easy. 8/8

clear winner 3.95.1
---

3)--preset medium
wav vs. 3.90.3 (247kbit):
scratchy vocals, big letdown for that bitrate. 8/8

wav vs. 3.95.1 (237kbit):
4/16, 98,9% of guessing. transparent for me

3.90.3 (247kbit) vs. 3.95.1 (237kbit):
scratchy vocals in 3.90.3. 8/8

clear winner 3.95.1
---

4) --preset standard
wav vs. 3.90.3 (265kbit):
impossible
wav vs. 3.95.1 (268kbit):
impossible

both transparent to me
---

3.90.3 is very disapointing for this sample at medium bitrates, 3.95 does good
thanks, need a pause now
edit: added bitrates for ABR 160

This post has been edited by sony666: Jan 13 2004, 04:55

[amano]

huh, astralstorm, please edit your post until you came up with a decision.

it is a bit ... biased now.
my feelings are too.

Sony666's results are very promising though. I think, that if there are any major differences, they will be rather in the low bitrate area.
testing preset medium with 3.90.3 is a bit unfair, because that wasn't tuned for 3.90.x.
it was added afterwards and tuned for 3.93, I think.

EDIT: typos

This post has been edited by amano: Jan 13 2004, 03:47

[sony666]

more samples from http://lame.sourceforge.net/gpsycho/quality.html

Metallica - Fade To Black (ftb_samp.wav)

1) --preset cbr 128
-wav vs. 3.90.3:
very small "warbling" effect at ~0.4s, only listenable with headphones, but 8/8

-wav vs. 3.95.1:
11/16 (10.5% of guess.) hmmm.... very hard to diff, if possible at all. need direct comparison vs. 3.90.1

3.90.3 vs. 3.95.1:
8/16 (59.8% guess).. bleh, I could have sworn the guitar attack at very start was more distorted with 3.90.3, but...

Conclusion: both do excellent on this real world sample. I was very surprised with 128k CBR quality higher bitrates useless to abx here..
---


the "infamous" velvet

1) --preset cbr 128
-wav vs. 3.90.3:
obvious.. can't handle the sharp attacks, hissing... 8/8

-wav vs. 3.95.1:
better handling of sharp attacks, still hissing 8/8

3.90.3 vs. 3.95.1:
14/16 (0.2% guess) 3.90.3 has a slight error in first base drum attack

3.95.1 ahead
---

2) --preset 160 (ABR)
-wav vs. 3.90.3 (163kbit):
base drum has obvious errors 8/8

-wav vs. 3.95.1 (171kbit):
base drum errors are gone, trying to diff via hissing... yes. 8/8. hi-hat(?) sounds softer in original wav, but encoding is good here

3.90.3 vs. 3.95.1:
easy, 3.90.3 has erros in base drum that sound a little like clipping 8/8

3.95.1 clear winner for me
---

3) --preset standard
--wav vs. 3.90.3 (231kbit):
took long, long time to figure this out... slight unnatural hissing in encode. 13/16 (1.1% guess)

-wav vs. 3.95.1 (188kbit):
not hard to pick with the training from 3.90.3, slight unnatural sharpness. same as above 7/8 (3.5% guess)

-3.90.3 vs. 3.95.1:
7/8 (3.5%)... 3.95.1 sharp hissing seems a tiny little more obvious. I will dream of this sample tonight.

3.95 better at medium bitrates, --aps.. very hard.. numbers say 3.90.3 but wait for other opinions on that

[ViPER1313]

I used the Waiting sample using the --preset standard command line on both v3.90.3 and 3.95.1 - I found v3.95.1 to be slightly better than 3.90.3 - hard to ABX the two - had to rest, then I got it

ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset standard

1L = Z:\Music\Test Samples\3.90.3\Wavs\001 Waiting.wav
2R = Z:\Music\Test Samples\3.95\Wavs\001 Waiting.wav

---------------------------------------
General Comments:
Tested 3.5 to 5 sec
---------------------------------------
1L File: Z:\Music\Test Samples\3.90.3\Wavs\001 Waiting.wav
1L Rating: 4.4
1L Comment: Smearing, seems worse than sample 2, very close
---------------------------------------
2R File: Z:\Music\Test Samples\3.95\Wavs\001 Waiting.wav
2R Rating: 4.7
2R Comment: Smearing as well, not as bad as 1, not anoying
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\3.90.3\Wavs\001 Waiting.wav
13 out of 14, pval < 0.001
Original vs Z:\Music\Test Samples\3.95\Wavs\001 Waiting.wav
10 out of 10, pval < 0.001
Z:\Music\Test Samples\3.90.3\Wavs\001 Waiting.wav vs Z:\Music\Test Samples\3.95\Wavs\001 Waiting.wav
34 out of 45, pval < 0.001

[ViPER1313]

ABC/HR Version 0.9b, 30 August 2002
Testname: Fatboy --preset standard

1L = Z:\Music\Test Samples\3.90.3\Wavs\Fatboy Audio Quality Test File.wav
2L = Z:\Music\Test Samples\3.95\Wavs\Fatboy Audio Quality Test File.wav

---------------------------------------
General Comments:
Very hard to call which encoder is really better at this sample - 1 seams to have problems with a sort of crackle, but 2 seems slightly distorted throughout - actually had trouble w/ 2 for a sec cause of room noise - i would call 2 the winner cause it doesnt seem as harsh as 1 does in places
---------------------------------------
1L File: Z:\Music\Test Samples\3.90.3\Wavs\Fatboy Audio Quality Test File.wav
1L Rating: 3.9
1L Comment: Slight crackle sound, not bad, close to origional
---------------------------------------
2L File: Z:\Music\Test Samples\3.95\Wavs\Fatboy Audio Quality Test File.wav
2L Rating: 4.3
2L Comment: Less distortion then 1, pre-echo, but no crackles
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\3.90.3\Wavs\Fatboy Audio Quality Test File.wav
10 out of 10, pval < 0.001
Original vs Z:\Music\Test Samples\3.95\Wavs\Fatboy Audio Quality Test File.wav
16 out of 18, pval < 0.001
Z:\Music\Test Samples\3.90.3\Wavs\Fatboy Audio Quality Test File.wav vs Z:\Music\Test Samples\3.95\Wavs\Fatboy Audio Quality Test File.wav
10 out of 10, pval < 0.001

[Gabriel]

Don't forget that there are 2 vbr modes

[sony666]

from http://sound.media.mit.edu/mpeg4/audio/sqam/
harp40_1.wav (harpsichord)

1) --preset cbr 128
-wav vs. 3.90.3: obvious distortions, 8/8
-wav vs. 3.95.1: obvious distortions, 8/8
-3.90.3 vs. 3.95.1: unable to find diff, would be pure guess

2) --preset 160 (ABR)
-wav vs. 3.90.3 (144kbit): less annoying than 128k, still 8/8
-wav vs. 3.95.1 (145kbit): less annoying than 128k, still 8/8
-3.90.3 vs. 3.95.1: 3/8, pure guess

3) --preset medium (VBR)
-wav vs. 3.90.3 (134kbit): 8/8, obvious distortions like 128k
-wav vs. 3.95.1 (111kbit): 8/8, obvious distortions
-3.90.3 vs. 3.95.1: 2/8, both deliver equally bad results for that VBR preset

4) --preset standard
-wav vs. 3.90.3 (159kbit): 10/16, slight distortion at very first note, hard to find
-wav vs. 3.95.1 (144kbit): 16/16 yes..same as above, training from 3.90 helped finding it
-3.90.3 vs. 3.95.1: 10/10, 3.95.1 has slightly more annoying distortion

conclusion: If I was a Harpsichord fan I'd stay away from mp3 . But I dont even know what they look like.

edit:
5) --preset fast standard
-wav vs. 3.90.3 (159kbit): very easily noticeable background noise added during first sec, 8/8
-wav vs. 3.95.1 (165kbit): 15/20 something is not right about background noise during 1st second, hard to find
-3.90.3 vs. 3.95.1: 8/8, 3.90.3 sounds "dirty" during first second, 3.95 clearer

This post has been edited by sony666: Jan 13 2004, 15:06

[ViPER1313]

v3.90.3 seems to edge out v3.95.1 in Waiting using --preset fast standard - read the comments below...

ABC/HR Version 0.9b, 30 August 2002
Testname: Waiting --preset fast standard

1R = Z:\Music\Test Samples\3.95.1 Fast.wav
2L = Z:\Music\Test Samples\3.90.3 Fast.wav

---------------------------------------
General Comments:
Neither clip sounds great, comparing the encoded files to the origional was easy with or without boosted trebble, 2 seems to have a slight edge
---------------------------------------
1R File: Z:\Music\Test Samples\3.95.1 Fast.wav
1R Rating: 4.3
1R Comment: Slightly worse than sample 2, suffers from a loss of high end - seems like background hiss is removed, smearing, sounds harsh in places, such as the during the "at all"
---------------------------------------
2L File: Z:\Music\Test Samples\3.90.3 Fast.wav
2L Rating: 4.6
2L Comment: Very slight loss of clarity, loss of "air" in the file - easy to tell w/ boosted trebble, the "for.....this....moment" part of the song sounds smeared
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\3.95.1 Fast.wav
18 out of 21, pval < 0.001
Original vs Z:\Music\Test Samples\3.90.3 Fast.wav
15 out of 18, pval = 0.004
Z:\Music\Test Samples\3.95.1 Fast.wav vs Z:\Music\Test Samples\3.90.3 Fast.wav
13 out of 14, pval < 0.001

[ViPER1313]

LARGE difference in quality here - v3.90.3 has the clear advantage in this instance.

ABC/HR Version 0.9b, 30 August 2002
Testname: DaFunk --preset cbr 128

1L = Z:\Music\Test Samples\3.95.1 CBR 128.wav
2R = Z:\Music\Test Samples\3.90.3 CBR 128.wav

---------------------------------------
General Comments:
Very supprised between the large difference in qual between 1 and 2 here - 2 very close to the origional while 1 sounds distorted.
---------------------------------------
1L File: Z:\Music\Test Samples\3.95.1 CBR 128.wav
1L Rating: 3.2
1L Comment: Kind of ringing / warble in the background - anoying, much worse than sample 2
---------------------------------------
2R File: Z:\Music\Test Samples\3.90.3 CBR 128.wav
2R Rating: 4.2
2R Comment: Even at 128kbps this sounds decent - the warble that is present in 1 is almost completely absent - actually had to concentrate for a sec to ABX from the origional - crisper than 1
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\3.95.1 CBR 128.wav
10 out of 10, pval < 0.001
Original vs Z:\Music\Test Samples\3.90.3 CBR 128.wav
12 out of 12, pval < 0.001
Z:\Music\Test Samples\3.95.1 CBR 128.wav vs Z:\Music\Test Samples\3.90.3 CBR 128.wav
13 out of 14, pval < 0.001

EDIT - Added encspot info - it seems that v3.95.1 uses many more short blocks than 3.90.3 (On this clip, 3.90.3 uses 7.9% short blocks while 3.95.1 uses 25.3%) - I have seen this in many other clips as well.


v3.90.3 --preset cbr 128

Bitrates:
----------------------------------------------------
128 |||||||||||||||||||||||||||||||||||||||| 99.9%
----------------------------------------------------

Type : mpeg 1 layer III
Bitrate : 127
Mode : joint stereo
Frequency : 44100 Hz
Frames : 791
ID3v2 Size : 0
First Frame Pos : 0
Length : 00:00:20
Max. Reservoir : 486
Av. Reservoir : 281
Emphasis : none
Scalefac : 37.4%
Bad Last Frame : no
Encoder : Lame 3.90

Lame Header:

Quality : 58
Version String : Lame 3.90
Tag Revision : 0
VBR Method : cbr
Lowpass Filter : 17600
Psycho-acoustic Model : nspsytune
Safe Joint Stereo : no
nogap (continued) : no
nogap (continuation) : no
ATH Type : 2
ABR Bitrate : 128
Noise Shaping : 2
Stereo Mode : Joint Stereo
Unwise Settings Used : no
Input Frequency : 44.1kHz

v3.95.1 --preset cbr 128

Bitrates:
----------------------------------------------------
128 |||||||||||||||||||||||||||||||||||||||| 99.9%
----------------------------------------------------

Type : mpeg 1 layer III
Bitrate : 127
Mode : joint stereo
Frequency : 44100 Hz
Frames : 791
ID3v2 Size : 0
First Frame Pos : 0
Length : 00:00:20
Max. Reservoir : 482
Av. Reservoir : 175
Emphasis : none
Scalefac : 1.1%
Bad Last Frame : no
Encoder : Lame 3.95

Lame Header:

Quality : 57
Version String : Lame 3.95
Tag Revision : 0
VBR Method : cbr
Lowpass Filter : 17500
Psycho-acoustic Model : nspsytune
Safe Joint Stereo : no
nogap (continued) : no
nogap (continuation) : no
ATH Type : 4
ABR Bitrate : 128
Noise Shaping : 1
Stereo Mode : Joint Stereo
Unwise Settings Used : no
Input Frequency : 44.1kHz

This post has been edited by ViPER1313: Jan 13 2004, 16:30

[indybrett]

If I'm reading these right, it would seem that 3.90.3 does better on some samples, with 3.95.1 doing better on others (when using APS).

[Gabriel]

Where can I download the DaFunk sample?

[ViPER1313]

http://www.angelfire.com/mt/viper1313/dafunk.flac - you might have to right click the file and select "save-as" - I forget where I dl'ed this sample.....

[ViPER1313]

Just tested the Layla sample at --preset cbr 128 :: v3.90.3 seems to be the distinct winner of this one as well - might try at --preset standard in a few min..... EDIT - tried and failed

ABC/HR Version 0.9b, 30 August 2002
Testname: Layla --preset cbr 128

1R = Z:\Music\Test Samples\Layla 3.90.3 cbr 128.wav
2L = Z:\Music\Test Samples\Layla 3.95.1 cbr 128.wav

---------------------------------------
General Comments:
Started the ABX test focusing in the 3.5 - 5 sec area of the song, used it to compare the 2 files to the original (easy at 128kbps Could not distinguish between the 2 mp3s in that area, focused on the 0-2 sec range and found a large difference.
---------------------------------------
1R File: Z:\Music\Test Samples\Layla 3.90.3 cbr 128.wav
1R Rating: 3.9
1R Comment: Does not exhibit the obvious under water sound of sample 2 in the 0-2 sec range, decent overall, rest of file sounds similar to 2
---------------------------------------
2L File: Z:\Music\Test Samples\Layla 3.95.1 cbr 128.wav
2L Rating: 3.0
2L Comment: Very bad under water sound from 0-2 sec, rest of file sounds less crisp than origional
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\Layla 3.90.3 cbr 128.wav
10 out of 10, pval < 0.001
Original vs Z:\Music\Test Samples\Layla 3.95.1 cbr 128.wav
13 out of 14, pval < 0.001
Z:\Music\Test Samples\Layla 3.90.3 cbr 128.wav vs Z:\Music\Test Samples\Layla 3.95.1 cbr 128.wav
13 out of 14, pval < 0.001

This post has been edited by ViPER1313: Jan 14 2004, 01:39

[amano]

Hmm, is 3.95.1 generally worse at 128 kbps CBR? Viper, can you give 128 kbps ABR a try?

This post has been edited by amano: Jan 14 2004, 01:45

[ViPER1313]

Hmm, is 3.95.1 generally worse at 128 kbps CBR? Viper, can you give 128 kbps ABR a try?

Sure.... results are pretty much the same as CBR, except both clips sound better overall

ABC/HR Version 0.9b, 30 August 2002
Testname: Layla --preset 128 ABR

1R = Z:\Music\Test Samples\Layla 3.90.3 128kbps ABR.wav
2L = Z:\Music\Test Samples\Layla 3.95.1 128kbps ABR.wav

---------------------------------------
General Comments:
Sample 2 sounds distinctively worse in the 0-2 sec range, both samples sound better than CBR - would bet sample 2 is v3.95.1 (sounds similar to CBR)
---------------------------------------
1R File: Z:\Music\Test Samples\Layla 3.90.3 128kbps ABR.wav
1R Rating: 4.1
1R Comment: Not bad sounding, better than sample 2
---------------------------------------
2L File: Z:\Music\Test Samples\Layla 3.95.1 128kbps ABR.wav
2L Rating: 3.6
2L Comment: Warble in the 0-2 sec range, worse than sample 1
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\Layla 3.90.3 128kbps ABR.wav
16 out of 18, pval < 0.001
Original vs Z:\Music\Test Samples\Layla 3.95.1 128kbps ABR.wav
14 out of 15, pval < 0.001
Z:\Music\Test Samples\Layla 3.90.3 128kbps ABR.wav vs Z:\Music\Test Samples\Layla 3.95.1 128kbps ABR.wav
12 out of 12, pval < 0.001

[jkml]

conclusion: If I was a Harpsichord fan I'd stay away from mp3 . But I dont even know what they look like.

They look pretty elegant:

http://www.henrylim.org/Harpsichord.html

[ViPER1313]

Tested the 41_30 sample with both encoders at --preset standard - I'll have to call this one a tie.

ABC/HR Version 0.9b, 30 August 2002
Testname: 41_30 --preset standard

1R = Z:\Music\Test Samples\3.95.1 41_30 --preset standard.wav
2R = Z:\Music\Test Samples\3.90.3 41_30 --preset standard.wav

---------------------------------------
General Comments:
Focused on the 0 - 2.3 sec range of this clip - both encoders suffer from smearing, but I can not ABX the difference between the 2 - I thought I could hear a difference at first, but couldnt ABX it - I'll have to call this one a tie
---------------------------------------
1R File: Z:\Music\Test Samples\3.95.1 41_30 --preset standard.wav
1R Rating: 4.5
1R Comment:
---------------------------------------
2R File: Z:\Music\Test Samples\3.90.3 41_30 --preset standard.wav
2R Rating: 4.5
2R Comment:
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\3.95.1 41_30 --preset standard.wav
10 out of 10, pval < 0.001
Original vs Z:\Music\Test Samples\3.90.3 41_30 --preset standard.wav
14 out of 15, pval < 0.001
Z:\Music\Test Samples\3.95.1 41_30 --preset standard.wav vs Z:\Music\Test Samples\3.90.3 41_30 --preset standard.wav
15 out of 24, pval = 0.154

[AstralStorm]

Fresh new.
Heh, I like Pink Floyd - this one sound like a fragment of a piece from album "Saucerful of Secrets".
Highly psychedelic.

ABC/HR for Java Version 0.4b, 14 styczeń 2004
Testname: main_theme (preset standard)

1R = C:\Music\LAMEsamples\main_theme.3-90-3.mp3.wav
2R = C:\Music\LAMEsamples\main_theme.3-95-1.mp3.wav

---------------------------------------
General Comments: Well well, it seems that without Sensaura on (resamples -> 48 kHz) it is much easier to ABC/HR anything. I didn't even bother to ABX with it on.
---------------------------------------
1R File: C:\Music\LAMEsamples\main_theme.3-90-3.mp3.wav
1R Rating: 4.0
1R Comment: 5-7s: Sharper transition center->right, ABX vs orig is based sollely on this. Failure ABXing with 2 on that, but detected warbling in plates 3-5s
---------------------------------------

ABX Results:
Original vs C:\Music\LAMEsamples\main_theme.3-95-1.mp3.wav
8 out of 17, pval = 0.685
Original vs C:\Music\LAMEsamples\main_theme.3-90-3.mp3.wav
12 out of 16, pval = 0.038
C:\Music\LAMEsamples\main_theme.3-90-3.mp3.wav vs C:\Music\LAMEsamples\main_theme.3-95-1.mp3.wav
18 out of 24, pval = 0.011

[robert]

Fresh new.
Heh, I like Pink Floyd - this one sound like a fragment of a piece from album "Saucerful of Secrets".
Highly psychedelic.

ABC/HR for Java Version 0.4b, 14 styczeń 2004
Testname: main_theme (preset standard)

1R = C:\Music\LAMEsamples\main_theme.3-90-3.mp3.wav
2R = C:\Music\LAMEsamples\main_theme.3-95-1.mp3.wav

I took that sample from "More"

[adlai]

so, from skimming over this thread, I'm guessing that 3.95 is a "meh" release since quality is more or less the same while file sizes have increased

[AtaqueEG]

conclusion: If I was a Harpsichord fan I'd stay away from mp3 . But I dont even know what they look like.

They look pretty elegant:

http://www.henrylim.org/Harpsichord.html

OT:
That Harpsichord is fake! Is made of LEGO pieces!
Although it does look like the real thing


On Topic:
I think that this version of LAME is not going to do very good on Roberto's test