LAME 3.96 FINAL vs. 3.90.3 Test, Test/Result Thread Options

[Lyx]

I didn't do the test with 3.90.3 myself yet. there are two people involved.

The following happened:
- i helped another person (zolder) how to setup EAC with LAME and preset standard.
- we did setup encoder and drive -offset calibration in EAC and used it
- then zolder ripped the CD to wav. Secure Mode, no normalizing, no change to the ripped audiodata.
- He listened to the WAVs with foobar, 16bit non-dithered or waveshaped output. No problems audible. Everything was fine.
- after that, he encoded them the LAME 3.96b1 --preset standard. And we both listened to it again on foobar (same fb2k-settings). At some trackchanges problems were audible.
- we did the same again with LAME 3.96final - same problems
- then, zolder encoded the album with LAME 3.90.3 --alt-preset standard and listened to it with foobar(same settings again), especially to the trackchanges which we learned are notorious for problems. According to him, everything sounded okay with 3.90.3. I have talked alot about gapless playback with him in the previous days, and he knows what to look for.

"How it sounds like": mostly clicks and pops.... like a sudden displacement of the waveform - as if the two waves(of the 2 tracks) aren't aligned correctly at the trackchange. I may be able to get the sample earlier, no guarantees, but i'll try my best.

- Lyx

This post has been edited by Lyx: Apr 15 2004, 16:41

[Gecko]

None of this sounds like a bug, which is what the talk about dropouts made it sound like.  From the initial description, I was expecting something bad like the old Fraunhofer mp3enc 3.1 dropouts.

At this point, I think it's just normal artifacting which would naturally tend to sound worse in cbr than it would in abr or vbr.

Well, I didn't test the range in which the dropout problem occurs and I only tested (alt) preset standard. Iirc, the problem has only been reported for one of the 128k-ish settings.

I don't think you can draw any conclusions about the dropout problem from my test.

[ssamadhi97]

That's TV line whistle - you find that on lots of recordings.

How does this kind of noise make its way into studio tracks anyway? Just wondering..

(sorry 'bout the OT)

[tigre]

RESULTS

Tested settings:
Lame 3.90.3 --alt-preset 128
Lame 3.96 --preset 128
Lame 3.96 -V 5

Tested samples from ff123's 64kbps test:

Blackwater
FloorEssence
Layla

All encoded files were ABXed successuflly (p < 0.01) against the original and against each other (or got equal ratings). For details (abx logs, descriptions of the audible differeces) see attatched .zip file.

Results (ABC/HR ratings):

Blackwater
3.7 Lame 3.90.3 --alt-preset 128
3.7 Lame 3.96 --preset 128
3.9 Lame 3.96 -V 5

FloorEssence
2.5 Lame 3.90.3 --alt-preset 128
2.2 Lame 3.96 --preset 128
3.8 Lame 3.96 -V 5

Layla
3.4 Lame 3.90.3 --alt-preset 128
3.0 Lame 3.96 --preset 128
4.0 Lame 3.96 -V 5

Attached File(s)

 ff123s_samples_part2.zip ( 1.95K ) Number of downloads: 361

 

[ff123]

RESULTS

Tested settings:
Lame 3.90.3 --alt-preset 128
Lame 3.96 --preset 128
Lame 3.96 -V 5

Tested samples from ff123's 64kbps test:

Blackwater
FloorEssence
Layla

It's looking like tigre's results may conflict with mine regarding the comparison between the 3.90.3 and 3.96 128 settings (although that could change with the rest of the samples).

ff123

[FatBoyFin]

RESULT

3.96b2 -V 5 > 3.90.3 --ap 128 :: fatboy :: FatBoyFin :: 1x verified by High Fidelity using 3.96 final

Ive also comfirmed this for 3.96 now with ABX results atleast 7/7.

[tigre]

RESULTS

Tested settings:
Lame 3.90.3 --alt-preset 128
Lame 3.96 --preset 128
Lame 3.96 -V 5

Tested samples from ff123's 64kbps test:

LifeShatters
LisztBMinor
MidnightVoyage

All encoded files were ABXed successuflly (p < 0.01) against the original and against each other. For details (abx logs, descriptions of the audible differeces) see attatched .zip file.

Results (ABC/HR ratings):

LifeShatters
4.2 Lame 3.90.3 --alt-preset 128
4.0 Lame 3.96 --preset 128
4.5 Lame 3.96 -V 5

LisztBMinor
2.5 Lame 3.90.3 --alt-preset 128
1.5 Lame 3.96 --preset 128
1.0 Lame 3.96 -V 5

MidnightVoyage
3.0 Lame 3.90.3 --alt-preset 128
3.6 Lame 3.96 --preset 128
4.0 Lame 3.96 -V 5

Attached File(s)

 ff123s_samples_part3.zip ( 2.29K ) Number of downloads: 283

 

[[proxima]]

RESULTS

Tested settings:
Lame 3.90.2 --alt-preset 128 --scale 1
Lame 3.96 --preset 128 --scale 1
Lame 3.96 -V 5

Layla is a strange case, 3.96 abr has a metallic artifact in the first two seconds. Preecho in all the three but (of course) well handled with VBR encoded sample.
3.90.2 : 3.0
3.96abr: 2.5
3.96_V5: 3.8

LisztBMinor is a low volume sample and VBR is not the best anymore. With all the three samples there are problems with background noise and ringing/chirping. VBR can offer better preecho handling but maybe with such samples is nothing special.
3.90.2 : 3.0
3.96abr: 2.1
3.96_V5: 2.0

Blackwater is very indicative for me. The well known chirping/ringing artifact, even if reduced with the new 3.96 are still present in the two 3.96 encodings, 3.90.2 doesn't ring at all. Again, preecho is better with -V 5.
3.90.2 : 3.2
3.96abr: 2.8
3.96_V5: 2.9

With Waiting critical sample VBR is advantaged. 3.96 --ap 128 manifested a bad ringing in some parts.
3.90.2 : 2.2
3.96abr: 1.5
3.96_V5: 3.3

ABC/HR and ABX results here: http://xoomer.virgilio.it/fofobella/396_1.zip

However 3.96 is not always worse than 3.90.2: with some samples the progress is real. It's a pity for this slight ringing artifact that penalize 3.96 too much for me.
I plan to complete all the 12 samples during this weekend

@tigre: In my previous post i linked a test done with lamex9 slightly in favour of 3.96. I cheched 3.96 vs lamex9 --p 128 --sfscale and they produce the same file, i think that this test could be added to the results even if greyed out.

This post has been edited by [proxima]: Apr 16 2004, 15:57

[ff123]

,Apr 16 2004, 06:48 AM] Blackwater is very indicative for me. The well known chirping/ringing artifact, even if reduced with the new 3.96 are still present in the two 3.96 encodings, 3.90.2 doesn't ring at all. Again, preecho is better with -V 5.
3.90.2 : 3.2
3.96abr: 2.8
3.96_V5: 2.9

Did lame switch back to gpsycho or something?

The ringing artifacting that people complained about in gpsycho was always something that I couldn't hear, because it occurs out of my high frequency range. This was also something that nspsytune seemed to handle better.

I will have a tendency to hear lower frequency artifacting, such as preecho, warbling or fluttering. So this might be a case where there could be an interesting split of opinions.

ff123

[tigre]

RESULTS

Tested settings:
Lame 3.90.3 --alt-preset 128
Lame 3.96 --preset 128
Lame 3.96 -V 5

Tested samples from ff123's 64kbps test:

Thear1
TheSource
Waiting

All encoded files were ABXed successuflly (p < 0.01) against the original and against each other (or got equal ratings). For details (abx logs, descriptions of the audible differeces) see attatched .zip file.

Results (ABC/HR ratings):

Thear1
4.5 Lame 3.90.3 --alt-preset 128
4.5 Lame 3.96 --preset 128
4.5 Lame 3.96 -V 5

TheSource
4.0 Lame 3.90.3 --alt-preset 128
4.0 Lame 3.96 --preset 128
3.8 Lame 3.96 -V 5

Waiting
2.0 Lame 3.90.3 --alt-preset 128
2.2 Lame 3.96 --preset 128
3.0 Lame 3.96 -V 5

Attached File(s)

 ff123s_samples_part4.zip ( 1.63K ) Number of downloads: 290

 

[[proxima]]

,Apr 16 2004, 06:48 AM] Blackwater is very indicative for me. The well known chirping/ringing artifact, even if reduced with the new 3.96 are still present in the two 3.96 encodings, 3.90.2 doesn't ring at all. Again, preecho is better with -V 5.
3.90.2 : 3.2
3.96abr: 2.8
3.96_V5: 2.9

Did lame switch back to gpsycho or something?

I will have a tendency to hear lower frequency artifacting, such as preecho, warbling or fluttering. So this might be a case where there could be an interesting split of opinions.

ff123

Ringing problems, at least for some samples are partially resolved with the changed noise shaping type for --preset 128. 3.96 that defaults to ns 2 is much more better with ringing problems (rebel sample is a very strong proof, even with spectral analisys there are less droputs above 10 kHz). Small problems still remain and i prefer 3.90.2 for some cases.
3.96 seems to be much more "aggresive" with background noise (see blackwater, LisztBMinor) and this could "unmask" artifacts with mid-high frequencies.
Ringing artifact noticeable with blackwater is not audible with 3.90.2 maybe because there is the original background noise.

[tigre]

I haven't compared directly 3.96 beta vs. 3.96 final yet, but I share your observation that there are less ringing problems.

LisztBMinor is a very interesting sample indeed. I just did a quick test with 3.96 --preset standard. There ringing/warbeling added to background noise when the piano starts playing is quite obvious (of course much better then with --preset 128 or -V 5). This seems to be the next case where 3.90.3 outperforms 3.96 with --(alt-)preset standard. (and this is one of the few samples where the bitrate of 3.96 is higher).

I think this sample could be very useful to further improve 3.96 (mainly VBR modes).

[ff123]

For tigre, the results of the 12 samples do not indicate any preference:

CODE

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 12
Critical significance:  0.05
Significance of data: 1.69E-01 (not significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               35          28.68
Testers (blocks)    11          21.23
Codecs eval'd        2           1.11    0.56    1.93  1.69E-01
Error               22           6.34    0.29
---------------------------------------------------------------

Also, when averaged with my results, there is no significant preference (although it is just on the cusp of doing so):

CODE

FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Blocked ANOVA analysis

Number of listeners: 12
Critical significance:  0.05
Significance of data: 5.45E-02 (not significant)
---------------------------------------------------------------
ANOVA Table for Randomized Block Designs Using Ratings

Source of         Degrees     Sum of    Mean
variation         of Freedom  squares   Square    F      p

Total               35          10.92
Testers (blocks)    11           7.08
Codecs eval'd        2           0.89    0.45    3.33  5.45E-02
Error               22           2.95    0.13
---------------------------------------------------------------

ff123

[tigre]

,Apr 16 2004, 04:48 PM] @tigre: In my previous post i linked a test done with lamex9 slightly in favour of 3.96. I cheched 3.96 vs lamex9 --p 128 --sfscale and they produce the same file, i think that this test could be added to the results even if greyed out.

Edit: I found the post you're talking about here

Could you please provide links to the samples used (if I haven't found them already)? I'll add the results to the upload thread once I have links to all samples (editing the post is not very amusing so I'd like to do it in an efficient way). The samples are:

applaud :: from ff123's sample page? If yes, which one?
campestre :: Here?
ct_reference :: I have no idea where to find that one
fall :: Here?
fatboy :: ff123's version or another (longer) one?
preecho1 :: Tell me, please
rebel :: Here?
wating :: Probably the well know one from ff123's page...

This post has been edited by tigre: Apr 16 2004, 22:53

[tigre]

For tigre, the results of the 12 samples do not indicate any preference:
<snip>
Also, when averaged with my results, there is no significant preference (although it is just on the cusp of doing so):
<snip>

Hehe - I've just tried the same. The only thing that gives a significant result is calculating a p-value from our total results for 3.96 -V 5 > 3.90.3 --alt-preset 128

(11+8)/(12+12) = 19/24 -> p = 0.0033

but I doubt that this calculation is correct since every sample is used twice.

[Hoosierdaddy]

"How it sounds like: mostly clicks and pops...like a sudden displacement of the waveform - as if the two waves(of the 2 tracks) aren't aligned correctly at the trackchange."

So this is a 'glitch' which the encoder encounters when trying to encode the interstill BETWEEN tracks on a CD? Hmmm. I thought the 'gap' on a cd was actually 'digital silence'. As I understand it, there isn't any actual audio data there to encode. Just two embedded 'flags' which the signal end of one track and the beginning of another to a CD player. I wonder if the blooper occurs because there really isn't any audio data to encode?

[Shingetsu]

and is there some quality loss betwen 3.90.3 and 3.96 which could be audibly in normal music ? this sounds all so complicated.

[ff123]

CODE

Means:

3.96V5   3.96p128 3.90.3  
 4.03     3.74     3.65  

---------------------------- p-value Matrix ---------------------------

        3.96p128 3.90.3  
3.96V5   0.061    0.017*  
3.96p128          0.547    
-----------------------------------------------------------------------

3.96V5 is better than 3.90.3

Averaging in proxima's current results, it looks like 3.96V5 will be the winner out of these 3 competitors.

and is there some quality loss betwen 3.90.3 and 3.96 which could be audibly in normal music ? this sounds all so complicated.

For the abr mode, it looks like you win some and lose some, and it depends on your ears. If you've got older ears like mine, you might prefer 3.96 abr at 128 kbs.

[ff123]

BTW, have there been bitrate tests of 3.96 -V5?

ff123

[tigre]

BTW, have there been bitrate tests of 3.96 -V5?

ff123

Only with beta 1 AFAIK.

Edit: for -V 5 3.96 beta 1 and final give bit-identical outputs. (Forget 3.90, was a typo.) So no further testing is necessary. (Well, doing some more bitrate tests can't do any harm of course ).

This post has been edited by tigre: Apr 16 2004, 22:20

[Jack Comics]

Only with beta 1 AFAIK.

So @ anyone who wants to help: Could you do some encodingings for 3.90 beta1 AND final with -V 5 and post the results here, please? Thanks in advance.

I'd imagine finding 3.90 beta 1 would be pretty difficult. Why would you want to test it to begin with though?

[tigre]

Jack Comics - you're right. I've already found out and reported earlier in this thread:

VBR (-V 5)
final == beta2 == beta1 (bit-identical)

Hehe - thanks. I'll edit my previous post.

So ff123, -V 5 was tested and the results are "close enough" to 128kbps.

[detokaal]

16 files 779,114 kb wav

Total Time is 75:22

Lame 3.96 -V 5 compresses to total 72,983 kb

Average bitrate from viewing the kbs for these 16 tracks in Winamp is 131.25

Lowest was 117 and highest was 144.

[[proxima]]

Could you please provide links to the samples used (if I haven't found them already)?

Sure. Applaud and fatboy samples from here:
http://lame.sourceforge.net/download/samples
ct_reference and preecho1 uploaded in the apposite thread. You've already found the others

[freakngoat]

RESULTS

Drone_short, aps, 0-1.2 sec

3.90.3 vs. original - 8/8
Easy to ABX.

3.96 vs. original - 8/8
Equally easy.

3.96 vs 3.90.3 - 8/8
3.96 sounds better than 3.90.3 now!

3.96 vs. 3.96b1 - 8/8
Air sounds are quieter in 3.96

Edit: added link to sample and encoding command line used

This post has been edited by freakngoat: Apr 17 2004, 01:59