The purpose of this thread is to (finally) test LAME 3.96 throughly enough to make it the new recommended LAME version and enhance communication between LAME developers and the HA.org community. The recent confusion about the different LAME versions and compiles (-Qrcd trouble etc.) have led to the decision that moving along with LAME development has been postponed for long enough and LAME 3.90.X can hopefully be declared dead very soon.
Please test as many samples as possible according to the following guidelines and post any results here.


1. Use the following LAME compiles (updated!):
lame3.90.3
lame3.96b2

2. The focus of the test should be --alt-preset/--preset standard, since it will allow us to make conclusions regarding the overall performance of the 'code level tweaked' VBR presets. Other VBR/ABR/CBR presets are interesting too, but not as important. If problems with --alt preset standard are detected feel free to compare extreme and insane too. Test the following combinations please:

(alt)presets + VBR/ABR
(320kbps) 3.96 --preset insane vs. 3.90.3 --alt-preset insane
(~256kbps) 3.96 --preset extreme vs. 3.90.3 --alt-preset extreme
(~210kbps) 3.96 --preset standard vs. 3.90.3 --alt-preset standard
(~160kbps) 3.96 -V 4 vs. 3.96 --preset 160 vs. 3.90.3 --alt-preset 160
(~128kbps) 3.96 -V 5 vs. 3.96 --preset 128 vs. 3.90.3 --alt-preset 128

CBR
3.96 --preset cbr <bitrate> vs. 3.90.3 --alt-preset cbr <bitrate>

If you want, you can additionally test VBR/ABR vs. CBR at comparable bitrates

3. You may use any sample you want, as long as you upload a losslessly compressed version of that (or provide a working link), so others can verify your results.

4. Your test results have to include the following:

• ABX results for
  3.90.3 vs. Original
  3.96b2 vs. Original
  3.96b2 vs. 3.90.3
• ABC/HR results are appreciated especially at lower bitrates, but shouldn't be considered a requirement.
• (Short) descriptions of the artifacts/differences

Notes:
This thread is for results only, discuss about the test here, please.
The earlier 'regression examples" thread has been closed and we'll try to add all results from it to the collected test results below. The original thread is still available here.
Please provide direct links to the sample you tested with or upload it here, even if it's a well-known one - it'll make things much easier for others.

Edit by tigre, Mar 31 2004: Updated to 3.96b2 and modified/simplified medium bitrate ABR/VBR test recommendations according to 3.96 -V x mass encode average bitrates.

Test results so far:


320kbps
3.96b1 --p insane < 3.90.3 --ap insane :: Lazy_Jones :: owowo :: 0x verified so far

~ 256kbps
3.96b1 --p extreme < 3.90.3 --ap extreme :: Lazy_Jones :: owowo :: 0x verified so far

~ 210kbps
3.96b1 --p standard < 3.90.3 --ap standard :: myf_4sec :: LoFiYo :: 2x verified by Wombat, harashin
3.96b2 : 1x verified by LoFiYo
3.96b1 --p standard < 3.90.3 --ap standard :: Rebel :: Proxima :: 1x verified by tigre
3.96b1 --p standard < 3.90.3 --ap standard :: (-) Ions :: Westgroveg :: 1x verified so far by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: Hustlejet :: harashin :: 0x verified so far
3.96b2 : 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: trumpets1 :: SometimesWarrior :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Doesnair :: Moitah :: 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: cantwait :: tigre :: 2x verified by harashin, Moitah
3.96b1 --p standard < 3.90.3 --ap standard :: death2 :: SometimesWarrior :: 1x verified by harashin
3.96b1 --p standard < 3.90.3 --ap standard :: Destitute :: harashin :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Rosemary :: harashin :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: drone_short :: freakngoat :: 0x verified so far
3.96b1 --p standard < 3.90.3 --ap standard :: Chanchan1 :: tigre :: 1x verified by harashin
3.96b2 : 1x verified by tigre
3.96b1 --p standard < 3.90.3 --ap standard :: Hosokawa___Atem_lied :: harashin :: 0x verified so far
3.96b2 : 1x verified by harashin
3.96b2 --p standard = 3.90.3 --ap standard :: 41_30sec :: ViPER1313
3.96b1 --p standard > 3.90.3 --ap standard :: spahm :: Pio2001 :: 1x verified by Wombat
3.96b1 --p standard > 3.90.3 --ap standard :: Birds :: Wombat :: 0x verified so far
3.96b1 --p standard > 3.90.3 --ap standard :: hokuscaredpiano :: Moitah :: 0x verified so far
3.96b1 --p standard > 3.90.3 --ap standard :: Lazy_Jones :: owowo :: 1x verified by freakngoat

~ 192kbps
no results so far

~ 160kbps
3.96b2 --p 160 > 3.90.3 --ap 160 > 3.96b2 -V 4 :: myf_4sec :: LoFiYo :: 0x verified so far + missing ABX results*

~ 144kbps
no results so far

~ 128kbps
VBR/ABR
3.96b1 --p 128 < 3.90.3 --ap 128 :: Quizas :: tigre :: 0x verified so far
3.96b1 --p 128 < 3.90.3 --ap 128 :: Doesnair :: Moitah :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: (-) Ions :: PVNC :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Applaud :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Bassdrum :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Blackwater :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Campestre :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Fall :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Iron :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Thewayitis :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 < 3.90.3 --ap 128 :: Tosca :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 --p 128 > 3.90.3 --ap 128 :: fatboy :: sony666 :: 2x verified so far by Wombat, FatBoyFin
3.96b1 --p 128 -q 0 < 3.90.3 --ap 128 :: Velvet :: [proxima] :: 0x verified so far + missing ABX results*
3.96b1 -V 5 > 3.90.3 --ap 128 :: Quizas :: tigre :: 0x verified so far
3.96b2 --V 5 > 3.96b1 --p 128 > 3.90.3 --ap 128 :: fatboy :: FatBoyFin :: 0x verified so far
3.96b2 --V 5 > 3.90.3 --ap 128 > 3.96b1 --p 128 :: Its_me :: tigre :: 0x verified so far
3.96b2 --V 5 > 3.90.3 --ap 128 > 3.96b1 --p 128 :: entierren con rumba :: tigre :: 0x verified so far

CBR
3.96b1 --p cbr 128 < 3.90.3 --ap cbr 128 :: Quizas :: tigre :: 0x verified so far

* If a result doesn't match the minimum requirements (-> 4. in 1st post), the result is 'greyed out' until the missing data is provided or someone else confirms the results. If your result isn't included in this list at all, we're either too slow, or there's too much missing (ABX results, link to the sample, clear statment which version is better, information about lame version/setting used etc.). If you want to provide missing information, do it in a new post to ensure that we notice it.

3.96b1 shows great improvement with fatboy and --alt-preset 128 (ABR)

-wav vs 3.90.3: 8/8, horribly distorted "vocals", "knocking" artifact 1.5-3.0s
-wav vs 3.96b1: 8/8 badly distorted vocals
-3.90.3 vs 3.96b1: 8/8, clearly less distortion in 3.96b1

I tried aps, but failed miserably on both encoders this time.

I used this to test APS 3.90.3 vs APS 3.96b1.

3.90.3 APS was 138kbps; 3.96b1 APS was 116kbps. If you don't have a golden set of ears, I recommend you try this sample. It's very easy to ABX .

ABC/HR Version 0.9b, 30 August 2002
Testname: My Funny Valentine - 3.90.3 vs 3.96b1

1R = C:\My Music\lab\MYF-4SEC\aps396b1.wav
2R = C:\My Music\lab\MYF-4SEC\aps3903.wav

---------------------------------------
General Comments:
#2 is the obvious winner.
---------------------------------------
1R File: C:\My Music\lab\MYF-4SEC\aps396b1.wav
1R Rating: 1.0
1R Comment: Distortion is very noticeable throughout the sample.
---------------------------------------
2R File: C:\My Music\lab\MYF-4SEC\aps3903.wav
2R Rating: 3.9
2R Comment: Much better than #1. Perceptible and almost not annoying.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\MYF-4SEC\aps396b1.wav
8 out of 8, pval = 0.004
Original vs C:\My Music\lab\MYF-4SEC\aps3903.wav
8 out of 8, pval = 0.004
C:\My Music\lab\MYF-4SEC\aps396b1.wav vs C:\My Music\lab\MYF-4SEC\aps3903.wav
8 out of 8, pval = 0.004

Just verified myf_4sec with aps

I could ABX 396b -> original 8/8
and 3903 -> original 5/8

As this sample is new to me i am surprised how obvious 396b suffers here with the third blow of the trumpet!

Wombat

I begin with the easiest samples for me. I included MPC out of curiosity, and WMA9 for reasons that are not necessary to explain here.

Lame 3.90.3 --alt-preset standard
Lame 3.96b1 --preset standard
Mppenc 1.14 --xlevel
WMA9 VBR 135-215 kbps

Sample : Badvilbel
Wav vs 3.90.3 ABX 11/11 Lots of pre echo, loss of transients, drop outs
Wav vs 3.96b1 ABX 8/8 Same as above
Wav vs MPC ABX 7/8 Gurgling on transients, ringing, drop outs, but better
Wav vs WMA9 ABX 8/8 Best, only transient problems, no drop outs at all

3.96.1 vs WMA9 : ABX 7/8
3.96.1 vs MPC : ABX 8/8

Conclusion of 3.90.3 vs 3.96b1 :
Same quality


Sample : Drone Short
3.90.3 : ABX 16/16 toooo easy ! Artifacts during the first half second, and time smearing during the second half.
3.96b1 : ABX 16/16 Somewhat louder artifacts, but not enough to talk about regression, just chance.
90 vs 96 : 11/14 (3% chance of guessing). The artifacts are not at the same place.
MPC : perfect. Maybe a tiny noise ? ABX 3/8, no.
WMA9 : 16/16, unacceptable quality

Conclusion of 3.90.3 vs 3.96b1 :
Same quality


Spahm
3.90.3 : Noise. 16/16
3.96.1 : perfect. Maybe a tiny noise ? 8/8 It seems so...
3.90.3 vs 3.96.1 : 16/16
MPC : Same effect as transients in badvilbel : grungy instead of hissy (mp3) 16/16
WMA : worse. 16/17 (I went too fast and made one mistake).

Conclusion of 3.90.3 vs 3.96b1 :
Big improvement Sample nearly completely solved.


For these three samples, compared to 3.90.3 :
3.96b1 is better one time, equal two times
MPC equal one time, slightly better one time, better one time.
WMA9 is unlistenable one time, worse one time, better one time.

Overall concusion : none for the time being. Results are sample dependant.
More tests to come, but with more difficult samples. It will take longer.

These samples are available on FF123's page : http://www.ff123.net/samples.html

I wrote befor in another thread about the birds sample and aps

3903 -> original 16/16 clear artifact when she sings the first "e" of become
396b only guessing
396b -> 3.903 8/8

396b is the first aps without a problem to me and birds.

The sample itself is now located here:
http://www.hydrogenaudio.org/forums/index....=0&#entry195385

I did 16 this time, for trying how long i can clearly discern. I am not used to use abx tests much.
I have to admit it is not that easy, even when you know the problem in the test file.

Edit: location of the sample file

Wombat

Sample: rebel.wav
Artifact: Preecho.
ABX results (range 0:01-0:03 sec.):

Original vs. 3.90.2 (--aps -Z) 13/14
Original vs. 3.96b1 (--aps) 10/10
3.90.2 vs 3.96b1 10/10

Both non-transparent but 3.90.2 is slightly better.
As already reported in my previous test, the same sample also reveal ringing problem with 3.96b1 (--ap 128).
The sample is available in the apposite thread.

Sample: Quizas
Tested lame settings:
3.90.3 --alt-preset 128 (132kbps)
3.90.3 --alt-preset cbr 128 (128kbps)
3.96b1 --preset 128 (135kbps)
3.96b1 --preset cbr 128 (128kbps)
3.96b1 -V 5 (138kbps)
Decoded with fb2k with trackgain enabled
Ratings:
3.96b1 -V 5 >> 3.90.3 --alt-preset 128 > 3.90.3 --alt-preset cbr 128 >> 3.96b1 --preset 128 > 3.96b1 --preset cbr 128

ABC/HR results:

Could you remove this one? It isn't worse in 3.96, I was able to ABX for 3.90.3 as well (see further down in the original thread).


This sample is almost certainly a regression... I will do more thorough testing to make sure. Can someone else verify this (listen where the snair is hit, and its a good idea to start with --p 128 because the artifact is a lot more obvious)?

Sample: doesnair
3.90.3 --ap standard vs 3.96b1 --p standard



Both aren't transparent vs original, but 3.96b1 is worse.

spahm
This sample makes my ears vibrating somehow after too many tries!

3903 8/8 pretty easy i only listen the first beats and the added noise is pretty obvious
396b 7/8 the same noise but to a much lesser degree


hustlejet

just gave me helpless clicky, clicky and when i thought i heard the problem - Nada!
Not my cup of sample

btw. ff123 did a nice job with abchr!!

Wombat

Sample: hokuscaredpiano (Hoku - I'm Scared)
3.96b1 --p standard vs 3.90.3 --ap standard



I'm hearing a problem on the piano (first chord change, 1.87 secs in), 3.96b1 sounds a bit better .

cantwait

I wanted to use this CD for testing lower bitrates but at 128kbps the artifacts were so obvious that I decided to start with (alt)preset standard.

Both, 3.90.3 --alt-preset standard and 3.96b1 --preset standard make the singer's voice (or it's echo) sound metallic in many places. One easily noticable example is the "s" of the word "understand" at ~ 8 seconds. Other "s" sounds (but also other places) have similar problems. 3.96b1 is clearly worse.

ABXed with fb2k, original vs. mp3 9/9 each time, 3.90.3 vs. 3.96b1 11/12

EDIT:
Edited in Wombat's + Moitah's (and my) results.

P.S.: Can someone verify my findings, please - is it just me or can this be considered as lame problem/killer sample?

That's bad , it seems even ReplayGain didn't fix its clipping issue(I don't notice here though) on your system.

Anyway, I was able to confirm tigre's cantwait sample. Somehow I can't reach my hosting space at CyberQuébec today, I write these results here.

EDIT: The server is back now. Results

Yeah, both are pretty easy to ABX against the original (--(alt-)preset standard):


Didn't have any luck ABXing them against eachother, but I wasn't concentrating as hard as I normally do... I'll try again later.

EDIT: Also, I only tried one part of the sample (the "s" in "understand").

EDIT 2: Took a break and switched to Grado SR225s (was using PortaPros before):


3.96b1 is worse.

Sample used: Lazy_Jones.flac
3.96b1 --preset standard <268kbps> vs. 3.90.3 --alt-preset standard <242kbps>
EDIT: Average bitrates according to Nero6
Encoded with lame.exe/LAMEDrop1.3 - Decoded with Nero6

Comments:
3.90.3 --alt-preset standard; completely chokes on this one
3.96b1 --preset standard; near perfect accuracy

As requested by tigre i have to provide abx results for my listening test even if samples are very easy because there are few people that test a specific sample. I'm sorry if i'm too lazy to abx the two lame versions each other fot all the samples but you can always see ratings in the posted table and comments. For the great majority of the samples 3.90.2 is better. Most samples should be well known to testers, nevertheless i uploaded some of them.

[quote]Original vs applaud396q0.wav
    8 out of 8, pval = 0.004
Original vs applaud_3902.wav
    11 out of 12, pval = 0.003
Original vs applaud_396b1.wav
    10 out of 10, pval < 0.001[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html
[quote]Original vs Bassdrum_3902.wav
    10 out of 10, pval < 0.001
Original vs Bassdrum396q0.wav
    8 out of 8, pval = 0.004
Original vs Bassdrum_396b1.wav
    8 out of 8, pval = 0.004[/quote]
sample uploaded
[quote]Original vs Blackwater_396b1.wav
    8 out of 8, pval = 0.004
Original vs Blackwater_3902.wav
    8 out of 8, pval = 0.004
Original vs Blackwater396q0.wav
    8 out of 8, pval = 0.004[/quote]
sample available here: http://www.ff123.net/samples.html
[quote]Original vs campestre396q0.wav
    8 out of 8, pval = 0.004
Original vs campestre_396b1.wav
    8 out of 8, pval = 0.004
Original vs campestre_3902.wav
    8 out of 8, pval = 0.004[/quote]
uploaded http://xoomer.virgilio.it/fofobella/campestre.flac
[quote]Original vs emmtop_3902.wav
    8 out of 8, pval = 0.004
Original vs emmtop_396b1.wav
    8 out of 8, pval = 0.004
Original vs emmtop396q0.wav
    14 out of 15, pval < 0.001[/quote]
sorry, sample exceeds 30 sec.
[quote]Original vs fall_3902.wav
    16 out of 28, pval = 0.286
Original vs fall_396b1.wav
    8 out of 8, pval = 0.004
Original vs fall396q0.wav
    18 out of 21, pval < 0.001[/quote]
sample uploaded
[quote] Original vs iron396q0.wav
    8 out of 8, pval = 0.004
Original vs iron_396b1.wav
    8 out of 8, pval = 0.004
Original vs iron_3902.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html
[quote]Original vs Rebel396q0.wav
    8 out of 8, pval = 0.004
Original vs Rebel_3902.wav
    8 out of 8, pval = 0.004
Original vs Rebel_396b1.wav
    13 out of 14, pval < 0.001[/quote]
sample uploaded
[quote] Original vs thewayitis_3902.wav
    8 out of 8, pval = 0.004
Original vs thewayitis396q0.wav
    8 out of 8, pval = 0.004
Original vs thewayitis_396b1.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.ff123.net/samples.html
[quote]Original vs Tosca_396b1.wav
    8 out of 8, pval = 0.004
Original vs Tosca_3902.wav
    8 out of 8, pval = 0.004
Original vs Tosca396q0.wav
    8 out of 8, pval = 0.004[/quote]
sample uploaded
[quote]Original vs velvet_3902.wav
    8 out of 8, pval = 0.004
Original vs velvet_396b1.wav
    8 out of 8, pval = 0.004
Original vs velvet396q0.wav
    8 out of 8, pval = 0.004[/quote]
available here: http://www.mp3dev.org/mp3/gpsycho/quality.html

Ticket to Ride by The Beatles

I had to go down to 96kbps, because this one is easy to encode.

lame-3.90.3 --alt-preset 96
vs
lame-3.96b1 --preset 96

Sound Blaster Live! 5.1
foobar2000 ABX comparator
DSP: volume control, Resampler (SSRC) 48000Hz
all files replaygained on track basis
Harman/Kardon HK-395 speakers


(-) Ions --alt-preset 128

wav vs 3.90.3 modified (494 KB)
8/8
0.4%
Comment: mp3 version sounds less sharp when volume of buzzing reaches peak level; mp3 sounds kind of smeared.

wav vs 3.96b1 (520 KB)
8/8
0.4%
Comment: distortions between volume peaks - this affects the lower-volume buzzing; (peak volume buzzing seems to sound like original wav).

3.90.3 modified (494 KB) vs 3.96b1 (520 KB)
8/8
0.4%
Comment: harder to ABX the two encoded versions. I found that 3.96's distortions were easier to differentiate the two than the peak buzzing volume. 3.90.3 is harder to pick out, at least with my ears.


Rock & Roll --alt-preset 128

wav vs 3.90.3 modified (442 KB)
8/8
0.4%
Comment: cymbals and drum solo sound very "smeared." Also I noticed what seemed like an instant of stereo collapse.

wav vs 3.96b1 (449 KB)
8/8
0.4%
Comment: similar artifacts as 3.90.3, with the initial part of the drum solo seeming to suffer from a HF boost, but I did not notice the stereo problem.

3.90.3 modified (442 KB) vs 3.96b1 (449 KB)
17/21
(ABX comparator gives me N/A as a probability)
Comment: This was the toughest. The smearing of cymbals was of no use to differentiate the two, as it occurs in both, and I stopped getting positive results with the supposed HF boost in the drum solo (hence the 4 failed trials). Then I focused on the end of the last chorus and the final vocal, which sounded more distorted in 3.90.3 than in 3.96. That turned out to be the main difference.

I've done some tests to verify results of other members.

death2 myf_4sec doesnair

Sample used: Lazy_Jones.flac
TEST: 3.96b1 (--preset extreme) <287.5 Kbps> vs. 3.90.3 (--alt-preset extreme) <273.9 Kbps>
Average bitrates according to Nero6
Encoded with lame.exe/LAMEDrop1.3 (l3maniac) - Decoded with Nero6

Comments:
3.90.3 (--alt-preset extreme); artifacts, sound like 'hi-hat close'
3.96b1 (--preset extreme); fatal flaw during initial attack during the first few milliseconds, sounds similar to a 'missing frame header' error, dead give-away not present in 3.90.3. Much clearer than 3.90.3, 'hi-hat close' artifacts not nearly as audible, but still present.
_____________________
Sample used: Lazy_Jones.flac
TEST: 3.96b1 (--preset insane) vs. 3.90.3 (--alt-preset insane)
Encoded with lame.exe/LAMEDrop1.3 (l3maniac) - Decoded with Nero6

Comments:
3.90.3 (--alt-preset insane); faint artifacts sound like "hi-hat close"
3.96b1 (--preset insane); fatal flaw during initial attack during the first few milliseconds, sounds similar to a 'missing frame header' error, dead give-away not present in 3.90.3. Much learer than 3.90.3

I am verifying Lazy Jones 3.90.3 aps vs. 3.96b1 ps. I focused on the last 1.5s of the sample:

3.90.3 aps vs. original
8/8 - popping artifacts in the last second of sample, very obvious

3.96b1 aps vs original
unable to ABX - sounds perfect

3.90.3 aps vs. 3.96b1 aps
8/8

edit: unable to ABX 3.96b1 or 3.90.3 at ape

I prefer 3.90.3 --alt-preset standard to 3.96b1 --preset standard on these samples.
Destitute Rosemary

I tested drone_short, 0-1 second:

3.90.3 aps vs. original 0-1s
8/8 - sounds like air being blown

3.96b1 aps vs. original 0-1s
8/8 - much easier to ABX, air sound louder and more noticable

3.96b1 aps vs. 3.90.3 aps
7/8 - missed last test, ears getting fatigued

3.90.3 sounds better

I reckon you mean 3.90.3.

Oh thanks for catching that. I will go back and edit that...

It looks like no one is testing 3.96b1 vs 3.90.3 any more, but I did spend some time last night trying to ABX Rebel.wav (abr 128 & aps), but failed miserably...

I was able to distinguish between the original and the MP3s in ABC/HR, but couldn't prove it by ABXing.

@[proxima]: Was 3.90.3 the clear winner of the two on this sample, or was it only slightly better? I couldn't tell from your ABX results. If you think it was the clear winner, you must have pretty good ears . Or is it a matter of knowing which section of the sample to focus on?

@LoFiYo
Welcome to the club! I have always problems to ABX samples others simply describe as preecho problem samples. Must be an auditory canal defect of a kind

Wombat

By looking at your avatar, probably.

The problem is not only preecho, with --ap 128 the very annoying artifact is chirping/ringing problem. I've redone the test, this time including 3.96b1 vs. 3.90.2 abx results


Similar problems are not isolated and can be heard in applaud and campestre sample i already uploaded. 3.90.2 does not suffers of this chirping/ringing issue. I encourage people to confirm this, i find ABXing even a waste of time for such problems. See spectral analisys for rebel.wav, there are droputs with high frequencies above 10 kHz.

Thanks for the hint Gabriel!

Thanks for taking the time to redo the test

3.90.3 --aps wins over 3.96b1 --ps for this one: Chanchan1:
The trumpets arround ~ 5.0 sound less clean, like surround/chorus effect added with both, 3.90.3 and 3.96, but 3.96 adds ringing additionally. ABX results:

CODE

foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:38:17

File A: file://N:\lametest\Chanchan\ChanChan1.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_3903_aps.wav

21:40:30 : Trial reset.
21:41:17 : 01/01  50.0%
21:42:20 : 02/02  25.0%
21:42:27 : 03/03  12.5%
21:42:33 : 04/04  6.3%
21:42:41 : 05/05  3.1%
21:43:13 : 06/06  1.6%
21:43:24 : 07/07  0.8%
21:43:44 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
--------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:44:17

File A: file://N:\lametest\Chanchan\ChanChan1.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_396b1_ps.wav

21:45:06 : 01/01  50.0%
21:45:10 : 02/02  25.0%
21:45:14 : 03/03  12.5%
21:45:18 : 04/04  6.3%
21:45:25 : 05/05  3.1%
21:45:29 : 06/06  1.6%
21:45:32 : 07/07  0.8%
21:45:35 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
---------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 21:45:47

File A: file://N:\lametest\Chanchan\chanchan1.wav_3903_aps.wav
File B: file://N:\lametest\Chanchan\chanchan1.wav_396b1_ps.wav

21:47:09 : 01/01  50.0%
21:47:14 : 02/02  25.0%
21:47:18 : 03/03  12.5%
21:47:25 : 04/04  6.3%
21:47:42 : 04/05  18.8%
21:48:56 : 05/06  10.9%
21:49:12 : 06/07  6.3%
21:49:40 : 07/08  3.5%
21:49:57 : 08/09  2.0%
21:50:56 : 09/10  1.1%
21:51:02 : 10/11  0.6%
21:51:08 : 11/12  0.3%

----------
Total: 11/12 (0.3%)

Rebel sample 3.90.3 --alt-preset standard vs. 3.96b1 --preset standard:
3.90.3 won
ABXed 3.0 - 6.0. Noticable pre-echo at drum hits, 3.96 is worse, slightly ringing/metallic sounding
ABX results:

QUOTE

foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:19:33

File A: file://N:\lametest\rebel.wav
File B: file://N:\lametest\rebel.wav_396b1_ps.mp3

22:20:30 : 01/01  50.0%
22:20:34 : 02/02  25.0%
22:20:38 : 03/03  12.5%
22:20:42 : 04/04  6.3%
22:20:45 : 05/05  3.1%
22:20:52 : 06/06  1.6%
22:20:55 : 07/07  0.8%
22:21:01 : 08/08  0.4%

----------
Total: 8/8 (0.4%)
------------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:21:28

File A: file://N:\lametest\rebel.wav
File B: file://N:\lametest\rebel.wav_3903_aps.mp3

22:22:20 : 01/01  50.0%
22:22:24 : 02/02  25.0%
<snip>
22:25:39 : 19/25  0.7%
22:25:47 : 20/26  0.5%
22:25:55 : 21/27  0.3%
22:26:06 : 22/28  0.2%

----------
Total: 22/28 (0.2%)
------------------------------------------------------------
foo_abx v1.2 report
foobar2000 v0.8
2004/03/25 22:26:22

File A: file://N:\lametest\rebel.wav_3903_aps.mp3
File B: file://N:\lametest\rebel.wav_396b1_ps.mp3

22:26:50 : 01/01  50.0%
22:26:54 : 02/02  25.0%
22:27:00 : 03/03  12.5%
22:27:03 : 04/04  6.3%
22:27:06 : 05/05  3.1%
22:27:09 : 06/06  1.6%
22:27:15 : 07/07  0.8%
22:27:18 : 08/08  0.4%

----------
Total: 8/8 (0.4%)

no ABX results makes your claim entirely useless to us

Check his links, man.

I've had some more tests.

Hosokawa___Atem_lied
Original vs 3.96b1aps (10/10)
Original vs 3.90.3aps (13/14)
3.90.3aps vs 3.96b1aps (10/10)

Chanchan1
Original vs 3.96b1aps (10/10)
Original vs 3.90.3aps (15/18)
3.90.3aps vs 3.96b1aps (10/11)

Gabriel released 3.96b2 today. Download it here.
The Minimal bitrate for --preset standard has been 'adjusted' to 128kbps, so all samples where 3.96b1 did worse than 3.90.3 need retesting/validation.

I tested 3.96b2 vs 3.90.3 with MYF_4SEC.WAV. On this sample, 3.90.3 still sounds much better (closer to the original) to me. I did notice some improvement with 3.96b2 over 3.96b1, I think because it no longer drops down to 96kbps, but the distortion at the 3rd blow is still noticeable. 3.90.3 doesn't have that distortion and sounds much cleaner.



Edit: for clarity & more detail

I still prefer 3.90.3 on these samples.
Hustlejet
Original vs 3.96b2aps (13/14)
Original vs 3.90.3aps (15/20)
3.90.3aps vs 3.96b2aps (12/14)

Hosokawa___Atem_lied
Original vs 3.96b2aps (10/10)
Original vs 3.90.3aps (13/14)
3.90.3aps vs 3.96b2aps (10/10)

Only for 3.96b1

Confirming Fatboy sample

3.90.3 --alt-preset 128 Vs WAV
8/8
Heavy Distortion

3.96b1 -preset 128 vs WAV
7/7
Lighter Distortion


3.90.3 --alt preset 128 vs 3.96b --preset 128
6/6

Conclusion: 3.96b1 has far less distortion
3.96b1 wins

v3.90.3 vs 3.96b2 using --preset standard on sample 41_30sec.flac

I would call this test a tie between the two encoders.

i tried my own (non-specifically chosen) samples but i couldn't do jack with them. i think i'll try some of the samples that came in the AAC-HE+PS test and maybe the one before that test, too to save time... (gotta hunt down the codec killer samples i downloaded somewhere on this computer, too)

I repeated (alt)preset standard test with cantwait sample and 3.90.3, 3.96b1 and b2. Result: If there's any change between 3.96b1 and b2, I didn't have enough patience to find and ABX it. ABXing original vs. 3.96b2 and 3.90.3 vs. 3.96b2 was as easy as with b1 before. ABC/HR log:
Result in short:
No improvement with 3.96b2

I dont know whether this is needed as it isnt an -aps test but ill post it anyway.
Testing Fatboy with 3.96b2 at 128 ABR.

Fatboy.wav vs 3.96b2 --preset 128
9/9
Some distortion

fatboy.wav vs 3.90.3 --alt-preset 128
9/9
Heavy distorion

3.96b2 vs 3.90.3
9/9
3.90.3 is far worse than 3.96b2

3.96b2 is still better than 3.90.3 for fatboy.wav

FatBoyFin, thanks for your effort, but ...

If I understand Gabriel here correctly, the quality related change between 3.96b1 and b2 is the minimum bitrate for -V2 (= preset standard) and -V1 (extreme). So the results for lower -V settings and abr/cbr presets should be identical. Because of this, 3.96b1 and b2 results for identical samples won't be added separately to the results post for -V 3 and lower and ABR/CBR presets.

Of course testing both can never do a harm, but since there aren't many results for < standard settings that match the test requirements (ABX results etc.) so far anyway, just testing 3.96b2 vs. 3.90.3 in ~ 128-160kbps range without 'looking back' should be enough. Testing -V 5 (and comparing with (alt)preset 128) additionally would be great too.

BTW: FatBoyFin, have you tried to ABX 3.96b1 vs. 3.96b2?

I tested 3.96b2 --preset 160 vs 3.90.3 --alt-preset 160 vs 3.96b2 -V4 using MYF_4SEC.wav. V4 was really bad on this sample, and I think the 2 abr encodes were pretty much a tie, but I felt 3.96b2 was slightly better. I could not ABX 3.96b2 abr160 vs the original, mainly because I think my ears were really tired, but also because it was well encoded. Of course, I couldn't ABX the 2 abr encodes at all.

I havent ABX between these but the file sizes are identical so I presume they are exactly the same.

As there is a new 128Kbps test coming up I redid my Fatboy test to include the -V5 setting:



PS the bitrate for -V 5 was a huge 220 KBps could someone please check that I have encoded this correctly or should I not be using -V5?