IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
Personal Listening Test of LAME encoders, 3.97/3.98.4/3.99.5 at ~135 kbps
IgorC
post Sep 2 2012, 20:28
Post #1





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



LAME Encoders & Settings:
3.97 -V 5 --vbr-new
3.98.4 -V 5.7
3.99.5 -V 4.99

Samples:
Almost all of them are from the last AAC public test. All files were decoded by foobar2000 to WAV 44.1 kHz, 24 bits.
http://uploading.com/files/get/mb98d734/LAME_MP3.zip

Hardware:
Audio interface EMU Pre Tracker + Sennheiser HD650 , occasionally HD800.

Results:

Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

Conclusion and observations:
During the whole testing process I had a perception that it was a single encoder with moderately different VBR modes. Somewhere better, somewhere worse.
3.98.4 and 3.99 have only one advantage over 3.97. They have more consistent/constant quality while 3.97 fails a bit more on slightly bigger number of samples (see the graph).
Though all three versions of LAME have practically the same quality.
The setting for LAME 3.99.5 was V 4.99 and not V5 because the last one produces slightly lower bitrate comparing to other versions. Previously I did some tests for LAME 3.99.5 V5 vs 3.97 V5. 3.99.5 V5 was a bit inferior but also has 2-3 kbps less. Now 3.99.5 -V 4.99 has ~3 kbps more bitrate and it's on par with 3.97/3.98.4 in quality terms. Probably it's a useless hair splitting. The quality is the same in the end.


I have used Kamedo2's graphmaker http://zak.s206.xrea.com/bitratetest/graphmaker.htm . It has saved me a bunch of time. smile.gif

This post has been edited by IgorC: Sep 2 2012, 21:08
Go to the top of the page
+Quote Post
lvqcl
post Sep 2 2012, 20:41
Post #2





Group: Developer
Posts: 3208
Joined: 2-December 07
Member No.: 49183



What track is represented by a dark blue mark (its score is 4.0 for lame3.97, 2.5 for lame3.99) ?
Go to the top of the page
+Quote Post
IgorC
post Sep 2 2012, 20:42
Post #3





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



CODE

3.97 3.98.4 3.99.5
1 4 3.7 2.5
2 2.5 3 3.2
3 3.2 3 3.1
4 2.7 2.8 2.8
5 3.9 3.7 3.8
6 2.5 3 3.1
7 4 3.9 4.3
8 3 3 3.2
9 1.7 2.2 2.3
10 4 4 3
11 3.9 3.8 3.8
12 4.4 4.2 4.1
13 4.2 3.7 3.8
14 2.5 3.5 4
15 4.6 4.6 4.7
16 4.2 4.2 4
17 4.5 4 4.2
18 4.6 4.6 4.7
19 4.1 3.4 3.7
20 4 4.3 4.3


This post has been edited by IgorC: Sep 2 2012, 20:44
Go to the top of the page
+Quote Post
C.R.Helmrich
post Sep 2 2012, 21:05
Post #4





Group: Developer
Posts: 681
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (IgorC @ Sep 2 2012, 21:28) *
Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

And not a single item transparent to you. Interesting! Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test? In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Sep 2 2012, 21:23
Post #5





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



QUOTE (C.R.Helmrich @ Sep 2 2012, 17:05) *
QUOTE (IgorC @ Sep 2 2012, 21:28) *
Ranking:
LAME 3.97 - 3.64
LAME 3.98.4 - 3.63
LAME 3.99.5 - 3.63

And not a single item transparent to you. Interesting!

I will say true. It's a human psychology. I had two samples where at least one encoder was transparent for me. I haven't include those two samples.
When I don't hear the difference I just press "Close" button. Not so for high bitrates.

QUOTE (C.R.Helmrich @ Sep 2 2012, 17:05) *
Let me ask: which average score did you give to the winner of last year's 96-kbps AAC-LC test?

Apple CVBR - 4.32
Apple TVBR - 4.20


QUOTE (C.R.Helmrich @ Sep 2 2012, 17:05) *
In other words, would you say that AAC-LC at 100 kbps is better than MP3 at 130 kbps?

Chris

Sure, without even a little doubt. Apple,FhG AAC/Vorbis/Opus 96 kbps are better than LAME MP3 130-135 kbps. Have tested.

This post has been edited by IgorC: Sep 2 2012, 21:28
Go to the top of the page
+Quote Post
C.R.Helmrich
post Sep 3 2012, 08:33
Post #6





Group: Developer
Posts: 681
Joined: 6-December 08
From: Erlangen Germany
Member No.: 64012



QUOTE (IgorC @ Sep 2 2012, 22:23) *
I had two samples where at least one encoder was transparent for me. I haven't include those two samples.

I see, thanks for clarifying. So I wonder at which bitrate LAME catches up quality-wise to the AAC encoders you mentioned. 150 kbps, i.e. 1.5 times AAC bitrate?

Chris


--------------------
If I don't reply to your reply, it means I agree with you.
Go to the top of the page
+Quote Post
IgorC
post Sep 3 2012, 14:04
Post #7





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



My findings are the same as for Kamedo2

Apple AAC requires 1.5-1.6x less bitrate comparing to LAME
http://d.hatena.ne.jp/kamedo2/20111029/1319840519
Go to the top of the page
+Quote Post
lvqcl
post Sep 3 2012, 16:43
Post #8





Group: Developer
Posts: 3208
Joined: 2-December 07
Member No.: 49183



QUOTE (IgorC @ Sep 3 2012, 17:04) *
Apple AAC requires 1.5-1.6x less bitrate comparing to LAME

IIRC MP3 uses huffman compression that is 10-15-20% less effective than newer algorithms ( see http://www.hydrogenaudio.org/forums/index....showtopic=44102 )

QUOTE (IgorC @ Sep 2 2012, 23:28) *
Probably it's a useless hair splitting. The quality is the same in the end.

Here is the comparison of the codecs based on your ratings:

3.97 vs. 3.98.4 (LAME 3.97 is a reference: its ratings are 100%). There are several samples where 3.98.4 is worse than 3.97, but there are also several samples where 3.98.4 is better than 3.97.




3.98.4 vs. 3.99.5 (LAME 3.98.4 as a reference). Usually 3.99.5 is slightly better than 3.98.4, but there are a few samples where 3.99.5 is noticeable worse than 3.98.4.



This post has been edited by lvqcl: Sep 3 2012, 16:45
Go to the top of the page
+Quote Post
IgorC
post Sep 3 2012, 18:45
Post #9





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



if You calculate the geometric mean 3.98.4 should have just slightly better score.
While all three 3.97/3.98.4/3.99.5 have the same average score 3.98.4 still should provide a bit more consistent quality as geometric mean indicates the grade of constancy for quality.

Your graphs confirm it.

This post has been edited by IgorC: Sep 3 2012, 18:45
Go to the top of the page
+Quote Post
IgorC
post Sep 10 2012, 03:45
Post #10





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Personal Listening Test of MP3 encoders (part II)

LAME vs Helix MP3 encoders at 130 kbps.


MP3 Encoders & Settings:
LAME 3.98.4 -V 5.7
Helix v5.1 -X2 -U2 -V60

Samples:
http://uploading.com/files/get/cm5ec93a/He...2Bvs%2BLAME.zip

Hardware:
Audio interface EMU Pre Tracker + Sennheiser HD800.

Results
Correction

CODE
Helix MP3   - 3.86
LAME 3.98.4 - 3.67






Conclusion and observations:
Helix and LAME encoders were tested some time ago in MP3 public test.


I was curious if my personal results would be the same. Well, they are the same. Though there is no statistically valid difference still Helix MP3 encoder has higher average score.
Add to it that Helix is extremely fast. The speed is >100x on average CPU with two cores. AFAIK there wasn't any recent development of Helix encoder since 2005.

This post has been edited by Garf: Sep 10 2012, 16:43
Go to the top of the page
+Quote Post
Rasi
post Sep 10 2012, 16:37
Post #11





Group: Members
Posts: 158
Joined: 24-August 02
Member No.: 3182



Interesting... Oldies dont die?
Go to the top of the page
+Quote Post
IgorC
post Sep 10 2012, 16:58
Post #12





Group: Members
Posts: 1506
Joined: 3-January 05
From: Argentina, Bs As
Member No.: 18803



Helix and LAME were tested in MP3 public test (2008)
http://listening-tests.hydrogenaudio.org/s...8-1/results.htm
http://listening-tests.hydrogenaudio.org/s...128-1/index.htm

Some observations.
Helix has a bit more pronounceable problem with transients while does better on tonality (see graph).
Go to the top of the page
+Quote Post
benski
post Sep 10 2012, 19:12
Post #13


Winamp Developer


Group: Developer
Posts: 669
Joined: 17-July 05
From: Ashburn, VA
Member No.: 23375



QUOTE (IgorC @ Sep 2 2012, 15:28) *
Conclusion and observations:
During the whole testing process I had a perception that it was a single encoder with moderately different VBR modes. Somewhere better, somewhere worse.
3.98.4 and 3.99 have only one advantage over 3.97. They have more consistent/constant quality while 3.97 fails a bit more on slightly bigger number of samples (see the graph).
Though all three versions of LAME have practically the same quality.
The setting for LAME 3.99.5 was V 4.99 and not V5 because the last one produces slightly lower bitrate comparing to other versions. Previously I did some tests for LAME 3.99.5 V5 vs 3.97 V5. 3.99.5 V5 was a bit inferior but also has 2-3 kbps less. Now 3.99.5 -V 4.99 has ~3 kbps more bitrate and it's on par with 3.97/3.98.4 in quality terms. Probably it's a useless hair splitting. The quality is the same in the end.


It definitely feels like LAME has reached a bit where any changes are just trade-offs and compromises rather than absolute improvements.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 14:45