Personal Listening Test of LAME encoders, 3.97/3.98.4/3.99.5 at ~135 kbps Options

[IgorC]

LAME Encoders & Settings:
3.97 -V 5 --vbr-new
3.98.4 -V 5.7
3.99.5 -V 4.99

Samples:
Almost all of them are from the last AAC public test. All files were decoded by foobar2000 to WAV 44.1 kHz, 24 bits.
http://uploading.com/files/get/mb98d734/LAME_MP3.zip

Hardware:
Audio interface EMU Pre Tracker + Sennheiser HD650 , occasionally HD800.

Results:

Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

Conclusion and observations:
During the whole testing process I had a perception that it was a single encoder with moderately different VBR modes. Somewhere better, somewhere worse.
3.98.4 and 3.99 have only one advantage over 3.97. They have more consistent/constant quality while 3.97 fails a bit more on slightly bigger number of samples (see the graph).
Though all three versions of LAME have practically the same quality.
The setting for LAME 3.99.5 was V 4.99 and not V5 because the last one produces slightly lower bitrate comparing to other versions. Previously I did some tests for LAME 3.99.5 V5 vs 3.97 V5. 3.99.5 V5 was a bit inferior but also has 2-3 kbps less. Now 3.99.5 -V 4.99 has ~3 kbps more bitrate and it's on par with 3.97/3.98.4 in quality terms. Probably it's a useless hair splitting. The quality is the same in the end.


I have used Kamedo2's graphmaker http://zak.s206.xrea.com/bitratetest/graphmaker.htm . It has saved me a bunch of time.

This post has been edited by IgorC: Sep 2 2012, 21:08

[lvqcl]

What track is represented by a dark blue mark (its score is 4.0 for lame3.97, 2.5 for lame3.99) ?

[IgorC]

CODE

3.97 3.98.4 3.99.5
1 4 3.7 2.5
2 2.5 3 3.2
3 3.2 3 3.1
4 2.7 2.8 2.8
5 3.9 3.7 3.8
6 2.5 3 3.1
7 4 3.9 4.3
8 3 3 3.2
9 1.7 2.2 2.3
10 4 4 3
11 3.9 3.8 3.8
12 4.4 4.2 4.1
13 4.2 3.7 3.8
14 2.5 3.5 4
15 4.6 4.6 4.7
16 4.2 4.2 4
17 4.5 4 4.2
18 4.6 4.6 4.7
19 4.1 3.4 3.7
20 4 4.3 4.3

This post has been edited by IgorC: Sep 2 2012, 20:44

[C.R.Helmrich]

Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

And not a single item transparent to you. Interesting! Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test? In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris

[IgorC]

Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

And not a single item transparent to you. Interesting!

I will say true. It's a human psychology. I had two samples where at least one encoder was transparent for me. I haven't include those two samples.
When I don't hear the difference I just press "Close" button. Not so for high bitrates.

Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test?

Apple CVBR - 4.32
Apple TVBR - 4.20

In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris

Sure, without even a little doubt. Apple,FhG AAC/Vorbis/Opus 96 kbps are better than LAME MP3 130-135 kbps. Have tested.

This post has been edited by IgorC: Sep 2 2012, 21:28

[C.R.Helmrich]

I had two samples where at least one encoder was transparent for me. I haven't include those two samples.

I see, thanks for clarifying. So I wonder at which bitrate LAME catches up quality-wise to the AAC encoders you mentioned. 150 kbps, i.e. 1.5 times AAC bitrate?

Chris

[IgorC]

My findings are the same as for Kamedo2

Apple AAC requires 1.5-1.6x less bitrate comparing to LAME
http://d.hatena.ne.jp/kamedo2/20111029/1319840519

[lvqcl]

Apple AAC requires 1.5-1.6x less bitrate comparing to LAME

IIRC MP3 uses huffman compression that is 10-15-20% less effective than newer algorithms ( see http://www.hydrogenaudio.org/forums/index....showtopic=44102 )

Probably it's a useless hair splitting. The quality is the same in the end.

Here is the comparison of the codecs based on your ratings:

3.97 vs. 3.98.4 (LAME 3.97 is a reference: its ratings are 100%). There are several samples where 3.98.4 is worse than 3.97, but there are also several samples where 3.98.4 is better than 3.97.




3.98.4 vs. 3.99.5 (LAME 3.98.4 as a reference). Usually 3.99.5 is slightly better than 3.98.4, but there are a few samples where 3.99.5 is noticeable worse than 3.98.4.



This post has been edited by lvqcl: Sep 3 2012, 16:45

[IgorC]

if You calculate the geometric mean 3.98.4 should have just slightly better score.
While all three 3.97/3.98.4/3.99.5 have the same average score 3.98.4 still should provide a bit more consistent quality as geometric mean indicates the grade of constancy for quality.

Your graphs confirm it.

This post has been edited by IgorC: Sep 3 2012, 18:45

[IgorC]

Personal Listening Test of MP3 encoders (part II)

LAME vs Helix MP3 encoders at 130 kbps.


MP3 Encoders & Settings:
LAME 3.98.4 -V 5.7
Helix v5.1 -X2 -U2 -V60

Samples:
http://uploading.com/files/get/cm5ec93a/He...2Bvs%2BLAME.zip

Hardware:
Audio interface EMU Pre Tracker + Sennheiser HD800.

Results
Correction

CODE

Helix MP3   - 3.86
LAME 3.98.4 - 3.67

Conclusion and observations:
Helix and LAME encoders were tested some time ago in MP3 public test.


I was curious if my personal results would be the same. Well, they are the same. Though there is no statistically valid difference still Helix MP3 encoder has higher average score.
Add to it that Helix is extremely fast. The speed is >100x on average CPU with two cores. AFAIK there wasn't any recent development of Helix encoder since 2005.

This post has been edited by Garf: Sep 10 2012, 16:43

[Rasi]

Interesting... Oldies dont die?

[IgorC]

Helix and LAME were tested in MP3 public test (2008)
http://listening-tests.hydrogenaudio.org/s...8-1/results.htm
http://listening-tests.hydrogenaudio.org/s...128-1/index.htm

Some observations.
Helix has a bit more pronounceable problem with transients while does better on tonality (see graph).

[benski]

Conclusion and observations:
During the whole testing process I had a perception that it was a single encoder with moderately different VBR modes. Somewhere better, somewhere worse.
3.98.4 and 3.99 have only one advantage over 3.97. They have more consistent/constant quality while 3.97 fails a bit more on slightly bigger number of samples (see the graph).
Though all three versions of LAME have practically the same quality.
The setting for LAME 3.99.5 was V 4.99 and not V5 because the last one produces slightly lower bitrate comparing to other versions. Previously I did some tests for LAME 3.99.5 V5 vs 3.97 V5. 3.99.5 V5 was a bit inferior but also has 2-3 kbps less. Now 3.99.5 -V 4.99 has ~3 kbps more bitrate and it's on par with 3.97/3.98.4 in quality terms. Probably it's a useless hair splitting. The quality is the same in the end.

It definitely feels like LAME has reached a bit where any changes are just trade-offs and compromises rather than absolute improvements.