LAME 3.100 alpha testing Options

[BFG]

Personally, I'm less concerned about bitrate and more concerned about getting the most transparent MP3 possible in a 320kbps package, paying special attention to sharp attacks and tonal issues. I'm guessing that's true for the majority of -V0 -q0 users.

[halb27]

So what we've learnt is:

- For an exact comparison we should match average bitrate closely. So for instance match 3.100a2 -V2 against 3.99.5 -V1.85.

- Especially at high quality levels deviations in average bitrate are of minor concern to many users of these levels. Quality is what counts in the first place to them.

But that's no contradiction. Just let's search first for an average bitrate comparative setting before comparing.

This post has been edited by halb27: Nov 9 2012, 07:50

[Gainless]

Another sample, pre-echos on the kicks with -V 2 (thanks to Igor for finding it btw):

Meduzz Kick Sample

This post has been edited by Gainless: Nov 9 2012, 20:50

[saratoga]

- Especially at high quality levels deviations in average bitrate are of minor concern to many users of these levels. Quality is what counts in the first place to them.

But that's no contradiction. Just let's search first for an average bitrate comparative setting before comparing.

For what its worth, I'm perfectly happy if V2 leads to higher bitrates on difficult to encode samples if it means fewer artifacts. I think the main concern is that tweaks might boost the bitrate on samples that are already transparent, which would be a regression. I would say we should run a set of normal (that is, not known problem samples) and see if bitrates changes on those too.

[BFG]

I would say we should run a set of normal (that is, not known problem samples) and see if bitrates changes on those too.

Anyone have a lengthy monotone sample that could be used? That would be the ultimate "normal" test.
I have something fairly close - the "DJ Alarm" sample at the end of U2's Zooropa CD - that could be used. It's basically a bitone klaxon.

This post has been edited by BFG: Nov 9 2012, 23:33

[halb27]

???
IMO a selection of tracks should be used which you usually listen to, and which are not of a very special kind. The selection isn't very critical, and it doesn't take extremely many tracks. But genre included is, and when talking about average bitrate genre information should be given.
If it's about pop music it's also important to include relatively new recordings for the main part. I changed my test set this year for this reason. Due to my age of 63 my old test set consisted of rather old recordings to a high percentage, recordings from the time before the loudness war. I exchanged most of them for newer recordings (and I increased the number of tracks). Average bitrate went up a few kbps.

[Kamedo2]

I'm considering a listening test of this new LAME 3.100 alpha, along with some other encoders.
What settings should I use? Is there a better setting that I should test?
I'll be using additional 5 easy samples, along with 20 I used in my last test.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

LAME 256k:
lame3.99.5\lame -S -V0.3 %i %o
lame3.100.a2-64\lame -S -V0.4 %i %o

LAME 160k:
lame3.99.5\lame -S -V3.8 %i %o
lame3.100.a2-64\lame -S -V4.1 %i %o

Helix, BladeEnc(low anchor) 160k:
hmp3 %i %o -X2 -U2 -V88
bladeenc -quit -nocfg %i %o -160

Decode option:
madplay -q -b 32 -o %o %i

The test will take 30~60 days(estimated). Gonna be hard.

[Gainless]

I'm considering a listening test of this new LAME 3.100 alpha, along with some other encoders.
What settings should I use? Is there a better setting that I should test?
I'll be using additional 5 easy samples, along with 20 I used in my last test.
http://www.hydrogenaudio.org/forums/index....showtopic=98003

LAME 256k:
lame3.99.5\lame -S -V0.3 %i %o
lame3.100.a2-64\lame -S -V0.4 %i %o

LAME 160k:
lame3.99.5\lame -S -V3.8 %i %o
lame3.100.a2-64\lame -S -V4.1 %i %o

Helix, BladeEnc(low anchor) 160k:
hmp3 %i %o -X2 -U2 -V88
bladeenc -quit -nocfg %i %o -160

Decode option:
madplay -q -b 32 -o %o %i

The test will take 30~60 days(estimated). Gonna be hard.

A test with -V 2 should be a lot more relevant to most people here, as this is the most popular setting. At least for me it would be interesting to know how far this is true at all.

[IgorC]

Kamedo2,

Performing a full test with an early alpha version of LAME would be waste of time and effort. Currently it has only one change that affects the quality of output. One could think of a tests on a few samples at most in this case.

halb27 was working intensively on his functional extension of LAME. I have tried a few samples and it was actually great.

This post has been edited by IgorC: Dec 2 2012, 13:59

[Kamedo2]

Performing a full test with an early alpha version of LAME would be waste of time and effort. Currently it has only one change that affects the quality of output. One could think of a tests on a few samples at most in this case.

halb27 was working intensively on his functional extension of LAME. I have tried a few samples and it was actually great.

Thank you very much. I'll do the test on LAME V+ instead.

[Kamedo2]

A small a priori test of LAME 3.100a2.
3.100VBR V3.6, 3.100CBR q0 160kbps, 3.99.5 V3.4

CODE

% This format is compatible with my graphmaker, as well as ff123's FRIEDMAN.
3.100VBR    3.100CBR    3.99VBR    
4.700    4.100    4.500    
4.500    4.300    4.100    
4.200    3.900    4.400    
5.000    4.200    4.500    
5.000    4.200    4.400    
%samples    Reunion Blues    Jazz
%samples    French    Speech
%samples    undelete    Pops
%samples    Dimmu Borgir    Metal
%samples    Run up    Pops

CODE

3.100VBR    3.100CBR    3.99VBR    
176388    160357    177434
171927    160292    178264
181507    160327    180015
164784    160569    165735
161089    160553    159509
Average bitrate:
171139    160420    172191
Calibrated bitrate:
160742    160044    159764

Samples used:last five samples of http://www.hydrogenaudio.org/forums/index....showtopic=98003
The number of sample is only 5, so it's not a very trustworthy data. Maybe it would be very nice if someone reproduce the results, especially the VBR vs VBR part.

[halb27]

Wow! This looks great!
So despite its alpha status I'd welcome 3.100a2 VBR to participate.

This post has been edited by halb27: Dec 8 2012, 22:03

[IgorC]

Maybe it would be very nice if someone reproduce the results, especially the VBR vs VBR part.

Can I ask You what kind of artifacts do You hear on "Reunion Blues"? I got totally opposite results.
Probably we're listening to different instruments there. While there are some misc. artifacts here and there, the main issue I hear is the distortion on hi-hat.

Well, we are two different listeners with different hardware.

Here's my result for this sample.
http://www.mediafire.com/?cyjfaoam2b9kmhq

CODE

ABC/HR Version 1.1 beta 2, 18 June 2004
Testname:

1L = D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.100a2.wav
2L = D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.99.5.wav

---------------------------------------
General Comments:

---------------------------------------
1L File: D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.100a2.wav
1L Rating: 4.4
1L Comment: The artifacts sound like flush water on hi-hat.
---------------------------------------
2L File: D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.99.5.wav
2L Rating: 4.7
2L Comment:
---------------------------------------
ABX Results:
D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.100a2.wav vs D:\Audio\3.100a2\01 Reunion Blues\01 reunion blues 3.99.5.wav
5 out of 5, pval = 0.031

I've performed this test twice and it's the same.

This post has been edited by IgorC: Dec 9 2012, 17:43

[Kamedo2]

Can I ask You what kind of artifacts do You hear on "Reunion Blues"? I got totally opposite results.
Probably we're listening to different instruments there. While there are some misc. artifacts here and there, the main issue I hear is the distortion on hi-hat.

What I've noticed is a low frequency collapse in left channel, piano, which is sometimes found in libfaac. The artifact is very slight.

I've performed this test twice and it's the same.

I tried to reproduce the result, but failed to do so. This time, I used Pioneer SE-DHP800-2.

CODE

ABC/HR for Java, Version 0.53a, 10 12 2012
Testname: 25rbmp3160k

Tester:

1R = C:\d\autoencode6\sound_out\25rb_.lame3099_v34.mp3
2R = C:\d\autoencode6\sound_out\25rb_.lame3100_v36.mp3
3R = C:\d\autoencode6\sound_out\25rb_.lame3100_160k.mp3

Ratings on a scale from 1.0 to 5.0

---------------------------------------
General Comments: I've noticed a low frequency distortion in piano.
---------------------------------------
3R File: C:\d\autoencode6\sound_out\25rb_.lame3100_160k.mp3
3R Rating: 4.2
3R Comment: Hi-hat distortions and less clear piano and other low freq sound.
---------------------------------------

ABX Results:
Original vs C:\d\autoencode6\sound_out\25rb_.lame3100_160k.mp3
12 out of 15, pval = 0.017
Original vs C:\d\autoencode6\sound_out\25rb_.lame3100_v36.mp3
9 out of 15, pval = 0.303
Original vs C:\d\autoencode6\sound_out\25rb_.lame3099_v34.mp3
10 out of 15, pval = 0.15


---- Detailed ABX results ----
Original vs C:\d\autoencode6\sound_out\25rb_.lame3100_160k.mp3
Playback Range: 00.000 to 30.000
2:01:03 AM p 1/1 pval = 0.5
2:02:29 AM p 2/2 pval = 0.25
2:03:22 AM f 2/3 pval = 0.5
2:03:46 AM p 3/4 pval = 0.312
2:03:58 AM p 4/5 pval = 0.187
2:04:21 AM p 5/6 pval = 0.109
2:04:32 AM p 6/7 pval = 0.062
2:04:43 AM p 7/8 pval = 0.035
2:04:57 AM f 7/9 pval = 0.089
2:05:07 AM p 8/10 pval = 0.054
2:05:39 AM p 9/11 pval = 0.032
2:05:49 AM p 10/12 pval = 0.019
2:06:01 AM p 11/13 pval = 0.011
2:06:24 AM p 12/14 pval = 0.0060
2:06:51 AM f 12/15 pval = 0.017

Original vs C:\d\autoencode6\sound_out\25rb_.lame3100_v36.mp3
Playback Range: 00.000 to 30.000
2:20:51 AM f 0/1 pval = 1.0
2:21:03 AM f 0/2 pval = 1.0
2:21:50 AM p 1/3 pval = 0.875
2:22:21 AM p 2/4 pval = 0.687
2:22:53 AM p 3/5 pval = 0.5
2:23:21 AM p 4/6 pval = 0.343
2:23:51 AM f 4/7 pval = 0.5
2:24:16 AM p 5/8 pval = 0.363
2:24:36 AM f 5/9 pval = 0.5
2:25:14 AM p 6/10 pval = 0.376
2:25:25 AM p 7/11 pval = 0.274
2:25:39 AM f 7/12 pval = 0.387
2:26:04 AM p 8/13 pval = 0.29
2:26:22 AM f 8/14 pval = 0.395
2:27:01 AM p 9/15 pval = 0.303

Original vs C:\d\autoencode6\sound_out\25rb_.lame3099_v34.mp3
Playback Range: 00.000 to 30.000
2:12:19 AM f 0/1 pval = 1.0
2:13:18 AM p 1/2 pval = 0.75
2:13:46 AM p 2/3 pval = 0.5
2:15:01 AM p 3/4 pval = 0.312
2:15:31 AM f 3/5 pval = 0.5
2:15:47 AM p 4/6 pval = 0.343
2:16:03 AM f 4/7 pval = 0.5
2:16:24 AM p 5/8 pval = 0.363
2:16:41 AM p 6/9 pval = 0.253
2:16:54 AM f 6/10 pval = 0.376
2:17:07 AM p 7/11 pval = 0.274
2:17:40 AM f 7/12 pval = 0.387
2:18:06 AM p 8/13 pval = 0.29
2:18:20 AM p 9/14 pval = 0.211
2:18:59 AM p 10/15 pval = 0.15

[IgorC]

Welcome to our world.

Today You clearly hear the difference. Next day You get something different. It's normal.
That's why we need public tests. Still personal listening tests are informative and useful.

This post has been edited by IgorC: Dec 9 2012, 19:12

[Kamedo2]

Today You clearly hear the difference. Next day You get something different. It's normal.
That's why we need public tests. Still personal listening tests are informative and useful.

Yes, the ABC/HR score can sometimes fluctuate. We are humans, not machines.
I plotted how much the deviation can be in a graph.

It says if I scored something 3.5, in 2 months later, I may occasionally score the same thing 3.0 or 4.0. And the standard deviation was 0.154 in the last Opus test.

[Kamedo2]

Yes, the ABC/HR score can sometimes fluctuate. We are humans, not machines.

Oops, I forgot to say that, in the September scores(x-axis) I used RP-HT560, in the November scores(y-axis) I used RP-HJE150.
These two scores were measured by two different headphones. So naturally they sound differently, rather than to say it's "fluctuation".