Lame 3.95 vs 3.90.3, ABC/HR test results here only Options

[indybrett]

so, from skimming over this thread, I'm guessing that 3.95 is a "meh" release since quality is more or less the same while file sizes have increased

On my music, filesizes decreased when using APS.

[Digga]

so, from skimming over this thread, I'm guessing that 3.95 is a "meh" release since quality is more or less the same while file sizes have increased

On my music, filesizes decreased when using APS.

on the samples I tried (mostly hip-hop), the size gets smaller too. no wonder though I guess, with bitrates on aps going now down to 96 kbps.
but I think we should respect the title of this trhead, shouldn't we?!
"ABC/HR test results here only", and no meh-stuff.

[fairyliquidizer]

Bug report - Lame 3.95.x can't decode at all. (jon33 compile)
Even can't decode its own files!

Verified.

The decoder is only there for the development frontend to allow analysis during testing. It's not intended for wider use.

Cheers,
Fairy

[phwip]

so, from skimming over this thread, I'm guessing that 3.95 is a "meh" release since quality is more or less the same while file sizes have increased

On my music, filesizes decreased when using APS.

on the samples I tried (mostly hip-hop), the size gets smaller too. no wonder though I guess, with bitrates on aps going now down to 96 kbps.

I have just encoded a random selection of 30 tracks from my hard drive all from different albums with both 3.90.3 and 3.95.1, both --preset standard. The songs are in the main rock and pop but there are a couple of classical and a couple of jazz tracks in there and the rest is a whole mixture of styles.

The total size of the wav files is 1.17 GB. With 3.90.3 this compresses to 171 MB and with 3.95.1 to 164 MB. 21 of the 30 tracks were smaller when using 3.95.1.

but I think we should respect the title of this trhead, shouldn't we?!
"ABC/HR test results here only", and no meh-stuff.

You are right. Having done the test I thought I might as well post the result. But perhaps a moderator could move this whole offtopic section to a different thread?

[indybrett]

It seems that in the lower bitrates that 3.90.3 is winning most of the ABX tests. If the two are found to be of equal sound quality with APS, then the resulting filesize would probably be the determining factor.

There is probably no point in discussing filesize any further until there is a clear winner (or tie) with APS.

[Vietwoojagig]

Hi,

some people mentioned this before, but:

Don't you think that comparing 3.90.3 and 3.95.1 needs a developer-moderated and coordinated testing? I don't know how many tests they have made on which samples, but a little hint, which of the test-cases in the test-case-archive have been checked and which not, would make it more easy for the people around not to waste their time on already proved samples.

And: Only with a moderated and coordinated tesiting, everybody in this forum will accespt the results of the tests and (in the end) will accept, that 3.95.1 is better or worser than 3.90.3.

Thanks for your attention.

[sld]

OT:
That Harpsichord is fake! Is made of LEGO pieces!

It is real, and playable.
OT, but had to clarify to all.

[AstralStorm]

so, from skimming over this thread, I'm guessing that 3.95 is a "meh" release since quality is more or less the same while file sizes have increased

Not necessarily true, the filesizes of the samples I've tested (bytes, preset standard):

275 565 main_theme.3-90-3.mp3
286 648 main_theme.3-95-1.mp3

345 709 velvet.3-90-3.mp3
281 130 velvet.3-95-1.mp3

And the quality isn't same.

Don't you think that comparing 3.90.3 and 3.95.1 needs a developer-moderated and coordinated testing? I don't know how many tests they have made on which samples, but a little hint, which of the test-cases in the test-case-archive have been checked and which not, would make it more easy for the people around not to waste their time on already proved samples.

It isn't a waste of time, different people have different hearing...

EDIT: I'll edit the subject to say "(and filesizes too)"
EDIT2: Blah, it isn't possible, now you can flame Invision Board devs.

This post has been edited by AstralStorm: Jan 16 2004, 17:17

[dev0]

Don't you think that comparing 3.90.3 and 3.95.1 needs a developer-moderated and coordinated testing? I don't know how many tests they have made on which samples, but a little hint, which of the test-cases in the test-case-archive have been checked and which not, would make it more easy for the people around not to waste their time on already proved samples.

I asked about this on lame-dev and tried to contact Gabriel, but have always been given some kind of "Well... whatever... test it." answer.
Of course a developer-moderated and coordinated testing would be more effective, but it seems like the lame-dev team in not really interested in quality and obviously nobody can force them to be.

[Gabriel]

but it seems like the lame-dev team in not really interested in quality

On this board you have to back up your claims with facts. Can you provide facts?

It isn't a waste of time, different people have different hearing...

This is very true. If I say "this and this samples are better when using those settings", will you believe me? I am not sure about it.

[kennedyb4]

I asked about this on lame-dev and tried to contact Gabriel, but have always been given some kind of "Well... whatever... test it." answer.
Of course a developer-moderated and coordinated testing would be more effective, but it seems like the lame-dev team in not really interested in quality and obviously nobody can force them to be.

As an avid observer over the last several years, I have to say that the opposite appears to be true, at least to me.

[indybrett]

Time to split the thread off...maybe?

[Atlantis]

If I say "this and this samples are better when using those settings", will you believe me? I am not sure about it.

Hi Gabriel,
I agree with dev0 that "a developer-moderated and coordinated testing would be more effective".

I know that your time (and other developer's one) dedicated to developing LAME can be limited (everybody needs a life ) but you (LAME developers) should really consider to coordinate the testing process.
I think that many people (me included) will gladly help you if asked for (let's say) a test about a sample using two different lame version or something like that.
I follow the lame-dev mailing list, but still I don't exactly know what is the impact of the changes added by 3.95x encoder: so maybe it could be good to have more "inverse feedback" (please smile at this definition) by you on test done by HA users.



Last but not least, thank-you for working on LAME: it is of great help for me.
Regards

[Gabriel]

I think that many people (me included) will gladly help you if asked for (let's say) a test about a sample using two different lame version or something like that.

When there is a specific change that needs to be tested, I agree. But for 3.95 against 3.90, there are many changes, and so nearly everything could be impacted.

[[proxima]]

Of course a developer-moderated and coordinated testing would be more effective

I perfectly agree with you and i've pointed to this point many times in my posts. I like doing listening tests but sometimes, i feel something like i'm wasting my time because lack of organization.
In this community there are for sure a lot of people interested in the future of LAME but what miss is a good organization: listening tests should be "guided" by someone with artifacts trained ears and with a little coding skills so that he can directly tweak the code. This role was covered very well by Dibrom in the past but now he lost interest and has very little time. Dibrom himself stated many times that the alt-presets are a sort of "hack" of LAME 3.90 and there is always the danger of modified behaviour with newer versions, even a different compilation switches seem to have altered slightly the behaviour of alt-presets !! So we have "our stable version" that is 3.90.2/3.90.3 but (even if very good in quality) now is quite old and we need to unblock this situation if we want to get advantages from future LAME progresses.
Alt-presets as they are now seem to have a too high sensitivity to slight encoder changes, IMHO what we need is to "stabilize" the alt-presets in the sense that they must be considered no more as "hacks" but as an integral part of the LAME code.
Of course, at this time, i don't see any people that could do this and i don't want to accuse developers, i perfectly understand the priorities and the limited time.

but it seems like the lame-dev team in not really interested in quality

Sure, there was some hurried releases marked as "stable" (3.93 for example). Maybe the alt-presets behavior of some "post 3.90" stable releases is not perfect but i think that what you stated is not literally true.

[Atlantis]

But for 3.95 against 3.90, there are many changes, and so nearly everything could be impacted.

Agreed.

In my first post I was just talking about (like you correctly understood) the "usual" development process (alpha => beta => release).

Regards,
Alberto

This post has been edited by Atlantis: Jan 16 2004, 21:09

[dev0]

,Jan 16 2004, 08:55 PM] Alt-presets as they are now seem to have a too high sensitivity to slight encoder changes, IMHO what we need is to "stabilize" the alt-presets in the sense that they must be considered no more as "hacks" but as an integral part of the LAME code.
Of course, at this time, i don't see any people that could do this and i don't want to accuse developers, i perfectly understand the priorities and the limited time.

From what I've gathered from the changelogs that's exactly what Gabriel is trying to do, there's only some more testing needed to verify that his changes and the new version provide an overall higher quality than the currently recommended version with Dibrom's original tunings.

A lot of fine tuning will have to be redone when LAME4 is approaching completition and will hopefully introduce us to a whole new level of MP3 quality.

[Wombat]

At first, i don´t abx much samples at this time.

One of the old samples i always hear a problem with aps is sophia2.
As i already mentioned in another thread the sophia2 sample has an artifact plop with 3.95.1 and has a sandpaper noise with 3.90.3 but no plop.
I prefer the sandpaper over the artifact but the best out of the mix would even be perfect
So someone beeing able to change the preset directly to test some circumstances will be the key like proxima already wrote.
The potential of lame curing this sample is there obviously.

Atm Gabriel seems to be the only one being able to this.
So i have to thank Gabriel, the developer, for his great work till here and wish him good luck finding the final tweak!


Wombat

[Wombat]

One thing to think about:
Even if lame 3.95.1 is behaving equal in the end with the testsamples it wouldn´t tell much!
It took about 2 years now to find all these problem music. It would take another 2 years to
find problems with 3.95 introduced with the modifications. There is no proove at all to set it
equal to 3.90.3.

There is no sense for using aps with any newer build than 3.90.3 until many testsamples "are really cured"

Isn´t it?

Wombat

[AstralStorm]

Actually, there is a reason: it is about 2x faster with similar quality as 3.90.3 most of the time.
It also doesn't produce larger files on (my) average, but it really depends on the type of music.
It's only up to the admins to determine if it's good enough to replace 3.90.3 as recommended version,
but to do this they need test results, hence this thread.
It seems like the recomended version needs a refresh - but when is it going to happen, we shall see.
Maybe just not now.

Anyway, there's no perfect psychoacoustic compression yet and there won't be any method soon.
MusePack is much closer to this than LAME. For True Quality ™, use the lossless compression.
We're trying here to just save some space with acceptable (possibly artifact-free) quality.
It's just the difference in tradeoff (perceived quality vs space vs speed).
One may even find 128kbps MP3s produced by l3enc good enough.
(shame on him, but we don't need to listen to that)

Enough discussion, please test further.

This post has been edited by AstralStorm: Jan 17 2004, 03:30

[LoFiYo]

Here is my test result using my plastic (as opposed to golden) ears .

CODE

ABC/HR Version 0.9b, 30 August 2002
Testname: Polonaise 3.90.3 v 3.95.1 - ABR128

1L = C:\My Music\lab\Polonaise\Polonaise-ABR128-3.95.1.wav
2L = C:\My Music\lab\Polonaise\Polonaise-ABR128-3.90.3.wav

---------------------------------------
General Comments:
3.90.3 vs 3.95.1 (ABR 128kbps mode = alt-preset 128)
---------------------------------------
1L File: C:\My Music\lab\Polonaise\Polonaise-ABR128-3.95.1.wav
1L Rating: 3.7
1L Comment: After around 15.5 sec, noticeable distortion is detected.
---------------------------------------
2L File: C:\My Music\lab\Polonaise\Polonaise-ABR128-3.90.3.wav
2L Rating: 4.0
2L Comment: Distortion is still noticeable, but clearly better than the other encode.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\Polonaise\Polonaise-ABR128-3.95.1.wav
   15 out of 17, pval = 0.001
Original vs C:\My Music\lab\Polonaise\Polonaise-ABR128-3.90.3.wav
   13 out of 14, pval < 0.001

[Continuum]

LoFiYo, try an ABX-test between the two encoded samples as well.

[LoFiYo]

LoFiYo, try an ABX-test between the two encoded samples as well.

I gave it a try just now, but failed miserably. Actually I started out really well (up to the 7th or 8th trial), but ended up something like 9/13. It seemed that toward the end, my ears (and brain) got so tired that they stopped caring about the difference. I guess to me with this sample, the difference wasn't that big after all...

[LoFiYo]

I tried a personal sample file (the first 30 seconds of It's a Sin by Pet Shop Boys). I remember reading somewhere that newer versions of Lame should be a little better at lower bitrates than 3.90.3, so I tried a very low bitrate this time (ABR80).

CODE

ABC/HR Version 0.9b, 30 August 2002
Testname: ABR 80 - 3.90.3 v 3.95.1

1L = C:\My Music\lab\sin\ABR80-3.90.3.wav
2L = C:\My Music\lab\sin\ABR80-3.95.1.wav

---------------------------------------
General Comments:
The fist thiry seconds of "It's a Sin" by Pet Shop Boys
---------------------------------------
1L File: C:\My Music\lab\sin\ABR80-3.90.3.wav
1L Rating: 3.3
1L Comment: Though the distortion is still noticeable, overall this sounds much better than the other encode.
---------------------------------------
2L File: C:\My Music\lab\sin\ABR80-3.95.1.wav
2L Rating: 2.3
2L Comment: Distortion/chirpiness is a little bit more noticeable throughout the sample.
---------------------------------------
ABX Results:
Original vs C:\My Music\lab\sin\ABR80-3.90.3.wav
   8 out of 8, pval = 0.004
Original vs C:\My Music\lab\sin\ABR80-3.95.1.wav
   8 out of 8, pval = 0.004

I couldn't ABX between the two encodes, but when ABXing the original vs 3.95.1, it was much easier to notice the difference than the original vs 3.90.3.

[ViPER1313]

ABC/HR Version 0.9b, 30 August 2002
Testname: BeautySlept --preset standard

1R = Z:\Music\Test Samples\testTemp\3.95.1 BeautySlept --preset standard.wav
2R = Z:\Music\Test Samples\testTemp\3.90.3 BeautySlept --preset standard.wav

---------------------------------------
General Comments:
I cant ABX this sample with either codec..... I think that sample 1 has somthing wrong with it but i cant ABX it with 100% certainty
---------------------------------------
ABX Results:
Original vs Z:\Music\Test Samples\testTemp\3.95.1 BeautySlept --preset standard.wav
17 out of 26, pval = 0.084
Original vs Z:\Music\Test Samples\testTemp\3.90.3 BeautySlept --preset standard.wav
4 out of 10, pval = 0.828
Z:\Music\Test Samples\testTemp\3.95.1 BeautySlept --preset standard.wav vs Z:\Music\Test Samples\testTemp\3.90.3 BeautySlept --preset standard.wav
4 out of 7, pval = 0.500