lame 3.98 alpha 2 testing thread, vbr new & athaa-sensitivity test at V5 Options

[guruboolez]

I must be brief.
Here are the results of a blind listening tests performed on 150 samples, classical music only. I planned to use other samples, but my week-end is not unlimited.

• 3.98 alpha 2 -V5
• 3.98 alpha 2 -V5 --vbr-new
• 3.98 alpha 2 -V5 --vbr-new --athaa-sensitivity 1


Results:









ABX logs:
http://audiotests.free.fr/tests/2005.10/LAME_398_V5.7z

Bitrate table:
http://audiotests.free.fr/tests/2005.10...Q%20Analysis.7z



I can comment it (I'm very sorry about, but there some troubles about some weird usage of internet here at my working place, and internet is currently "under control").


• --vbr-new is apparently same to use with -V5 profile
• --athaa-sensitivity really help to lower the level of annoyance (ringing mostly). It's maybe not the right answer to current quality problems occuring with "pure" V5 preset, but the fix is very helpful and doesn't cost too much (~6 kbps on average).

This post has been edited by guruboolez: Dec 29 2005, 22:36

[Synthetic Soul]

Wow, your result formatting just gets better and better. Your tests have always been exemplary of course.

I thought --athaa-sensitivity had gone. I know you were querying the decision.

Does this mean it's back in for 3.98 or was never out?

As a -V5 user, thank you very much for your time.

Edit: OK, discussions mentioned above are here. A spinoff thread is here. It is out in 3.97b1 but remains in alpha versions.

My re-reading of these previous threads leads me to believe that this test is to keep this discussion alive, and to attempt to find a resolve for the "requirement" for --athaa-sensitivity with -V5.

Good luck to guruboolez and Gabriel.

This post has been edited by Synthetic Soul: Oct 17 2005, 15:17

[shadowking]

An Xcellent read as always. What exactly are the changes from 3.97b1 > 3.98b2 ?

[Gabriel]

I thought --athaa-sensitivity had gone. I know you were querying the decision.

Does this mean it's back in for 3.98 or was never out?

As other experimental switches, it is available in alpha and debug compiles.

[Synthetic Soul]

Thanks Gabriel. I found that out myself as well. I decided I should do as I wish others would do sometimes, and actaully test instead of just asking for the answer.

Is 0.59 a marked improvement when talking about ABX? I remember guruboolez once stating that a 4.0 is "perceptible but not annoying". What is 3.0 please? "perceptible and annoying"? NB: I have tried to find the answer to this one.

I seem to remember seeing before that, if the two lines do not overlap, it can be said that there is a definite difference. They don't overlap in any of the subtotals. I guess this is why guruboolez makes the brief comments he has.

[stephanV]

5.0 = imperceptible
4.0 = perceptible but not annoying
3.0 = perceptible and slighly annoying
2.0 = annoying
1.0 = very annoying

[guruboolez]

Few words about my notation principles.

When a small difference was found= +/- 0.5 point
When a serious difference was heard = +1 point or more
When a very small or unsure difference was heard = < +/- 0.5 point.

Therefore, don't take too much attention on the absolute notation (i.e. it's not totally linked to ITU reference scale).


What matters is:
1/ that --athaa-sensitivity 1 bring more improvement than --vbr-new over defaulted VBR mode
2/ that --athaa-sensitivity could highly reduce severe audible flaws (high level of ringing/noise reduction with -V5/-V5 --vbr-new).

[Synthetic Soul]

Thanks stephenV.

Argh. Just as we get a new HA recommendation and I consider re-encoding all my MP3s (a shocking amount of which are Blade and Xing) up pops this development.

3.97b1 or 3.98a2 --athaa-sensitivity 1? Or do I wait until this anomaly has been properly diagnosed and ironed out?

I think I'm going to have to wait and see how this one progresses...

[robert]

@guruboolez
is it possible to get one or two of your test samples?

[Gambit]

@guruboolez
is it possible to get one or two of your test samples?

Not sure if those are the used ones, but some of guru's samples are here:
ftp://ftp2.foobar2000.net/foobar/

[guruboolez]

@guruboolez
is it possible to get one or two of your test samples?

The 150 samples are all downloadable:
EDIT: links have changed

This post has been edited by guruboolez: Dec 29 2005, 22:37

[guruboolez]

Some stats:

CODE

-V5 --vbr-new with and without --athaa-sensitivity 1

when difference is from: -1.0 to 0.4 point    (70 samples)    +2,84 kbps  in average with --athaa..1
when difference is from:  0.5 to 0.9 point    (29 samples)    +4,17 kbps  in average with --athaa..1
when difference is from:  1.0 to 1.4 points    (23 samples)   +7,70 kbps  in average with --athaa..1
when difference is from:  1.5 to 1.9 points    (13 samples)   +8,69 kbps  in average with --athaa..1
when difference is from:  2.0 to 2.4 points    (09 samples)   +14,44 kbps in average with --athaa..1
when difference is from:  2.5 points and more    (05 samples)    +19,00 kbps in average with --athaa..1

CONCLUSION: the perceptual quality improvment of --ahtaa-sensitivity 1 has a direct relation with the amount of bit added by this command line. When --ahtaa-sensitivity 1 has no impact on bitrate it has no impact on quality; when --ahtaa-sensitivity 1 "bloats" the bitrate it also "bloats" the quality.

10% of the tested samples reveal a tendency to "bitrate bloat" (+15 kbps or more). For these 14 samples, the average notation is going from 2.3 points (without --ahtaa-sensitivity 1) to 4.3 points (with --ahtaa-sensitivity 1).

____
Appendix:
RATIO

CODE

[average notation diff in points] / [average bitrate diff in kbps]

-0.01 / 2.84  =-0.0020
0.53 / 4.17  = 0.1264
1.03 / 4.70  = 0.1333
1.56 / 8.69  = 0.1796
2.00 / 14.44 = 0.1384
2.70 / 19.00 = 0.142

1

This post has been edited by guruboolez: Oct 18 2005, 09:59

[Brigstocke]

Well, it looks like there's very very few instances of regression and that any increase in bitrate is well justified. Since I haven't done any testing myself, I can only say thank you very much for doing so much testing!

This post has been edited by Brigstocke: Oct 19 2005, 06:30

[mad_arab]

3.97 was recommended while still in alpha stage. Are there any significant improvements that would make 3.98 a good choice yet?

[Cartoon]

No, 3.97 was not recommended until it hit beta. Even that caused some noise, as in principle it should be GA code. But as the 3.x tree is pretty mature now, it is pretty safe.

Besides, it is just a recommendation, not the law