Skip to main content

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Personal Listening Test of LAME encoders (Read 21900 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Personal Listening Test of LAME encoders

LAME Encoders & Settings:
3.97 -V 5 --vbr-new
3.98.4 -V 5.7
3.99.5 -V 4.99

Samples:
Almost all of them are from the last AAC public test.  All files were decoded by foobar2000 to WAV 44.1 kHz, 24 bits.
http://uploading.com/files/get/mb98d734/LAME_MP3.zip

Hardware:
Audio interface  EMU Pre Tracker + Sennheiser HD650 , occasionally HD800.

Results:

Personal Listening Test of LAME encoders

Reply #1
What track is represented by a dark blue mark (its score is 4.0 for lame3.97, 2.5 for lame3.99) ?

Personal Listening Test of LAME encoders

Reply #2
Code: [Select]
	3.97	3.98.4	3.99.5
1 4 3.7 2.5
2 2.5 3 3.2
3 3.2 3 3.1
4 2.7 2.8 2.8
5 3.9 3.7 3.8
6 2.5 3 3.1
7 4 3.9 4.3
8 3 3 3.2
9 1.7 2.2 2.3
10 4 4 3
11 3.9 3.8 3.8
12 4.4 4.2 4.1
13 4.2 3.7 3.8
14 2.5 3.5 4
15 4.6 4.6 4.7
16 4.2 4.2 4
17 4.5 4 4.2
18 4.6 4.6 4.7
19 4.1 3.4 3.7
20 4 4.3 4.3

Personal Listening Test of LAME encoders

Reply #3
Ranking:
LAME 3.97    - 3.64
LAME 3.98.4  - 3.63
LAME 3.99.5  - 3.63

And not a single item transparent to you. Interesting! Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test? In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris
If I don't reply to your reply, it means I agree with you.

Personal Listening Test of LAME encoders

Reply #4
Ranking:
LAME 3.97    - 3.64
LAME 3.98.4  - 3.63
LAME 3.99.5  - 3.63

And not a single item transparent to you. Interesting!

I will say true. It's a human psychology. I had two samples where at least one encoder was transparent for me.  I haven't include those two samples.
When I don't hear the difference I just press "Close" button. Not so for high bitrates.

Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test?

Apple CVBR - 4.32   
Apple TVBR - 4.20


In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris

Sure, without even a little doubt. Apple,FhG AAC/Vorbis/Opus 96 kbps are better than LAME MP3 130-135 kbps. Have tested.

Personal Listening Test of LAME encoders

Reply #5
I had two samples where at least one encoder was transparent for me.  I haven't include those two samples.

I see, thanks for clarifying. So I wonder at which bitrate LAME catches up quality-wise to the AAC encoders you mentioned. 150 kbps, i.e. 1.5 times AAC bitrate?

Chris
If I don't reply to your reply, it means I agree with you.


Personal Listening Test of LAME encoders

Reply #7
Apple AAC requires 1.5-1.6x less bitrate comparing to LAME

IIRC MP3 uses huffman compression that is 10-15-20% less effective than newer algorithms ( see http://www.hydrogenaudio.org/forums/index....showtopic=44102 )

Probably it's a useless hair splitting. The quality is the same in the end.

Here is the comparison of the codecs based on your ratings:

3.97 vs. 3.98.4 (LAME 3.97 is a reference: its ratings are 100%). There are several samples where 3.98.4 is worse than 3.97, but there are also several samples where 3.98.4 is better than 3.97.




3.98.4 vs. 3.99.5 (LAME 3.98.4 as a reference). Usually 3.99.5 is slightly better than 3.98.4, but there are a few samples where 3.99.5 is noticeable worse than 3.98.4.


Personal Listening Test of LAME encoders

Reply #8
if You calculate the geometric mean 3.98.4 should have just slightly better score.
While all three 3.97/3.98.4/3.99.5 have the same average score  3.98.4 still should provide a bit more consistent quality as geometric mean indicates the grade of constancy for quality.

Your graphs confirm it.

 

Personal Listening Test of LAME encoders

Reply #9
Personal Listening Test of MP3 encoders (part II)

LAME vs Helix MP3 encoders at 130 kbps.


MP3  Encoders & Settings:
LAME 3.98.4 -V 5.7
Helix v5.1 -X2 -U2 -V60

Samples:
http://uploading.com/files/get/cm5ec93a/He...2Bvs%2BLAME.zip

Hardware:
Audio interface EMU Pre Tracker + Sennheiser  HD800.

Results
Correction

Code: [Select]
Helix MP3   - 3.86
LAME 3.98.4 - 3.67






Conclusion and observations:
Helix and LAME encoders were tested some time ago in MP3 public test.


I was curious if my personal results would be the same. Well, they are the same. Though there is no statistically valid difference still Helix MP3 encoder has higher average score. 
Add to it that Helix is extremely fast.  The speed is >100x on average CPU with two cores. AFAIK  there wasn't any recent development of Helix encoder since 2005.

Personal Listening Test of LAME encoders

Reply #10
Interesting... Oldies dont die?


Personal Listening Test of LAME encoders

Reply #12
Conclusion and  observations:
During the whole testing process I had a perception that it was a single encoder with moderately different VBR modes. Somewhere better, somewhere worse.
3.98.4 and 3.99 have only one advantage over 3.97. They have more consistent/constant quality while 3.97 fails a bit more on slightly bigger number of samples (see the graph).
Though all three versions of LAME have practically the same quality.
The setting for LAME 3.99.5 was V 4.99 and not V5 because the last one produces slightly lower bitrate comparing to other versions. Previously I did some tests for LAME 3.99.5 V5 vs 3.97 V5. 3.99.5 V5 was a bit inferior but also has 2-3 kbps less. Now 3.99.5 -V 4.99 has ~3 kbps more bitrate and it's on par with 3.97/3.98.4 in quality terms. Probably it's a useless hair splitting. The quality is the same in the end.


It definitely feels like LAME has reached a bit where any changes are just trade-offs and compromises rather than absolute improvements.