Lame 3.97 Alpha 8 Testing Thread Options

[Gabriel]

Updated V3 and V2 presets (vbr-new is unchanged)

What I am interested into is mainly results of V3 and V2 (especially V2) against either 3.96.1 or 3.97a7.
You can test it against 3.90.3 if you want, but I'd like to know the evolution from previous version first.

Results from 3.97a7:
http://www.hydrogenaudio.org/forums/index....showtopic=31255

[rjamorim]

Here's a quick and dirty compile (untested) done in MSVC6, without NASM optimizations.

Please don't link directly to it from elsewhere, it'll just stay there until John XXXIII comes up with a properly done compile.

Edit: look down

This post has been edited by rjamorim: Mar 6 2005, 23:42

[lotr]

ABX results on trumpets sample.

a8 -V3 vs a7 -V3: 10/10 (0.1%) //a8 sounded worse.

a8 -V2 vs a7 -V2: 10/10 (0.1%) //a8 sounded worse.

3.97a8 -V3 vs 3.96.1 -V3: 10/10 (0.1%) //a8 sounded worse.

3.97a8 -V2 vs 3.96.1 -V2: 10/10 (0.1%) //a8 sounded worse.

In all four tests the latest, 3.97a8 version, sounded more "hoarse" during some parts of the sample, than the version it was compared with.

[katharsis]

Does anyone know why I get a 30% faster lame (with preset standard) when I compile it myself?
I've uploaded it to www.geoshock.com/lame.zip , maybe it doesn't work as it should.

bye,
Michael

This post has been edited by katharsis: Mar 6 2005, 20:48

[john33]

Does anyone know why I get a 30% faster lame (with preset standard) when I compile it myself?
I've uploaded it to www.geoshock.com/lame.zip , maybe it doesn't work as it should.

bye,
Michael

If you're referring to Roberto's compile, it's because it's (a) MSVC which is slower than ICL, and (b) mostly because it doesn't include the nasm assembler routines in the compile.

Anyway, new build now at Rarewares.

[Acid Orange Juic...]

I still can ABX this sample very easy:

aps_Killer_sample.wav; from here

My ABX results:

Settings: Lame 3.97a8 -V 2: ( 10/10 ) Very easy to ABX.

Settings: Lame 3.97a8 -V 2 --vbr-new: ( 10/10 ) Very easy to ABX again.

[Gecko]

Angelic sample 397a7 vs 397a8.

-V 2
ABX 6/10 --> no difference to me.

-V 3
1st run: ABX 9/10 (very hard, need longer pauses between trials)
But damnit, I pressed cancel when foobar asked me to save the log because I wanted to jump back and try something, but it quit the ABX dialog and now I don't know which file was which!

2nd run to find out which is which: ABX 5/5 and to my ears 397a8 sounded slightly worse than 397a7. There was more high pitched "sparkle".

Total: 14/15

[jaybeee]

My badvilbel.flac abx results:

Settings: Lame 3.96.1 --preset standard: ( 8/8 ) Very easy to ABX.

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/08 18:33:34

File A: file://C:\My Downloads\Encoders\Problem_audio_samples\badvilbel.flac
File B: file://C:\My Downloads\Encoders\Problem_audio_samples\3.96.1--aps\badvilbel.mp3

18:33:35 : Test started.
18:35:36 : 01/01 50.0%
18:36:41 : 02/02 25.0%
18:37:54 : 03/03 12.5%
18:39:37 : 04/04 6.3%
18:40:49 : 05/05 3.1%
18:41:56 : 06/06 1.6%
18:42:30 : 07/07 0.8%
18:43:04 : 08/08 0.4%
18:43:06 : Test finished.

----------
Total: 8/8 (0.4%)

---------------------------------------

Settings: Lame 3.97a8 --preset standard: ( 8/8 ) Very easy to ABX - although seemed to be slightly better (results do not support this at all though).

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/08 18:45:59

File A: file://C:\My Downloads\Encoders\Problem_audio_samples\badvilbel.flac
File B: file://C:\My Downloads\Encoders\Problem_audio_samples\3.97a8--aps\badvilbel.mp3

18:46:00 : Test started.
18:47:09 : 01/01 50.0%
18:47:31 : 02/02 25.0%
18:47:52 : 03/03 12.5%
18:48:10 : 04/04 6.3%
18:48:27 : 05/05 3.1%
18:48:47 : 06/06 1.6%
18:49:07 : 07/07 0.8%
18:49:25 : 08/08 0.4%
18:49:27 : Test finished.

----------
Total: 8/8 (0.4%)

---------------------------------------

Settings: Lame 3.97a8 -V 2 --vbr-new: ( 8/8 ) Very easy to ABX again.

foo_abx v1.2 report
foobar2000 v0.8.3
2005/03/08 19:08:32

File A: file://C:\My Downloads\Encoders\Problem_audio_samples\badvilbel.flac
File B: file://C:\My Downloads\Encoders\Problem_audio_samples\3.97a8-V2-vbr-new\badvilbel.mp3

19:08:33 : Test started.
19:09:19 : 01/01 50.0%
19:09:43 : 02/02 25.0%
19:10:00 : 03/03 12.5%
19:10:17 : 04/04 6.3%
19:10:34 : 05/05 3.1%
19:10:53 : 06/06 1.6%
19:11:10 : 07/07 0.8%
19:11:26 : 08/08 0.4%
19:11:38 : Test finished.

----------
Total: 8/8 (0.4%)

[lotr]

ABX results on castanets2 sample:

a8 -V3 vs a7 -V3: 10/10 (0.1%) // 3.97a8 sounded better than a7. Less pre-echo.

I couldn't hear a difference on any of the following tests:

a8 -V2 vs a7 -V2 (6/10)
a8 -V2 vs 3.96.1 -V2 (6/10)
a8 -V3 vs 3.96.1 -V3 (5/10)

[yong]

LAME ACM bug
This probably is an old bug:
Here's example,
when the MIN bitrate is 8,
MAX is 320,
if i adjust the "step" slider from 16 to 12,
the "LAME MP3 codec:about" window will stop responding.

[guruboolez]

LAME 3.97 alpha 8 TEST #1: -V 3


For the following tests, I tried to obtain the most accurate results. That’s why I didn’t include extra elements (like comparison with --vbr-new, or with an ancient lame version): my full attention was focused on difference between alpha8 and the previous one. For that reason, I put all efforts on ABX direct comparison between both encodings. 12 trials for each sample, no more, no less (except on training). No ABX tests between reference and encodings: unnecessary waste of time.

I’ve used previously 54 samples to test V4 preset. It was a luxury I can’t afford anymore, because my time is limited and my motivation quite over. Therefore, I returned to the general samples’ set used for alpha 5 and alpha 6 (20 samples, melting ‘general’ music, ‘classical’ and two critical samples). I’ve just replaced the cello’s sample (BachS1007.wav, too transparent at high bitrate) by another cello sample (Kodály.wav), more detailed and consequently harder to encode.

Other change: I’m now using the new java version of abc/hr, which is remarkable. It has a precious feature: it automatically saves the playback range. Now, developers could easily link all written comments to the corresponding part of the sample. I’ll precise all tested range in my synoptic table of results.
Other good point for schnofler’s software: it systematically cleans all comments wrote by the tester after each fresh test. ff123’s abchr beta 2 is buggy here, and often keeps previous comments. It’s sometimes funny (e.g. “distorted trumpet” on a solo piano sample…), but it’s often confusing. Comments are now fully reliable. I’ll recap the audible problems in my tables.


RESULTS




ABX log files are here.


COMMENTS

Gabriel’s tuning have the same impact on -V3 than on -V4: it lowers the bitrate, and increase the quality. It’s important to note that the progress are also very similar to the progress noticed by testing -V4: there are less distortions (noticeable on cymbals mostly), no more issues with low volume parts, and less pre-echo (at least on non critical samples: castanets2.wav is still heavily smeared). It’s probably not a coincidence… I suppose that -V3 alpha 8 benefits from similar tunings than -V4 alpha7.
I’ve only noticed two serious regressions (ATrain and MidnightVoyage.wav). But on average, the new alpha 8 appeared to me clearly better than the old one.

This post has been edited by guruboolez: Dec 29 2005, 23:12

[guruboolez]

LAME 3.97 alpha 8 TEST #2: -V 2, aka --preset standard



Contrary to tunings performed by Gabriel on V4 and V3, the modified V2 preset doesn’t lower the average bitrate. It’s now slightly higher than before, and with a great disparity between samples. The most problematic case is thear1.wav: bitrate jumps from 213 (alpha 7) to 247 kbps, and the most annoying thing is the chasm between -V3 (167 kbps) and -V2 (247 kbps): + 48%, for an average difference between two presets of ~15% only. Is it exceptional? People listening to metal should take care.


RESULTS




ABX log files are here.


I was glad to discover that the modified V2 is also better than the previous one. The progresses are similar to those noticed with -V4 and -V3: less preecho, less distortion, some rare ATH (?) issues corrected (no more ringing, or instability in highest frequencies). Two regressions: MidnightVoyage.wav (again), and castanets2.wav (no additional pre-echo, but an additional colouration which isn’t pleasant to my ears).

This post has been edited by guruboolez: Dec 29 2005, 23:13

[guruboolez]

LAME 3.97 alpha 8 EXTRA: comparison between different listening tests


It might be interesting to sum up the results of recent tunings made by Gabriel on defaulted VBR mode.
1/ -V4 was significantly inferior to -V4 --vbr-new (alpha 5), but after tunings (alpha 7), both produces similar quality encodings (with maybe a slight advantage for --vbr-new).
2/ -V2 was significantly inferior to -V2 --vbr-new (alpha 6). I didn’t compared the newer -V2 to -V2 --vbr-new, but the progresses noticed for -V2 alpha 8 are very similar (same genre, same level of improvement) to those noticed between alpha5 V4 and alpha7 V4. Therefore I strongly believe that -V2 and -V2 --vbr-new are now very close, and that 3.97 alpha 8 should reaches the same level of quality than 3.90.3 (see the following table comparison).




Important note: I don’t like comparison between distant listening tests. I have created this table in order to see how much VBR presets improved on average since Gabriel modified them. Progresses are very similar… Please don’t compare single notations. And don’t be surprised by apparent incoherencies with notation: -V2 is not inferior to -V3 because average notation is slightly superior with this one. My notation is relative (to the expected quality from given bitrate), floating (no anchors, low nor high).



Now, I’ll probably make a good break with listening tests. Hope they were helpful. Anyway, a big thanks to all lame developers

This post has been edited by guruboolez: Dec 29 2005, 23:13

[pest]

OT:

@guruboolez

i appreciate your work for lame.
imho lame would not be the same without your listening tests!

best regards
pest

[Gabriel]

Once again, thank you very much for those precious results.

[dev0]

Thanks a lot Guruboolez.
If more people could provide similiar test results (you don't need to test as many samples as Guru), 3.97 could easily become the recommended version once it's released.

[Gabriel]

thear1.wav bitrate at V2 is caused by extreme sfb21 bloating:
-V2 --> 247.3 kbps
-V2 -Y --> 173.1 kbps

[henkersmahlzeit]

@guruboolez
Overwhelming again! You are something of a living treasure!

Maybe a bit too overwhelming for a ABX newbie ... I think, it would be nice to have a small but defined sample-set with specific instructions for starters ... maybe this could motivate some people!

You lame developers are living treasure too

[mithrandir]

Ah, now -V2 and -V3 use x mode 9 instead of 3. This allows the maskingadjust and ath floor values to be set much lower (i.e. less masking and more sensitive ATH). This is essentially a "trading of the bits" and it appears the tradeoff is much better this way if guru's results are any indication.

[indybrett]

These ABX tests were all done with vbr-old, correct?

[crazychimp132]

@guruboolez
Overwhelming again! You are something of a living treasure!

Maybe a bit too overwhelming for a ABX newbie ... I think, it would be nice to have a small but defined sample-set with specific instructions for starters ... maybe this could motivate some people!

You lame developers are living treasure too 

I agree! I wish I could help with testing but I just don't know where to begin. The problem is getting someone who knows what they are doing to actually organize some sort of standard ABX test.

[esa372]

I think, it would be nice to have a small but defined sample-set with specific instructions for starters...

I agree! I wish I could help with testing but I just don't know where to begin.

Cosign.

[lotr]

ABX results on awe32_20sec

3.97a8 -V3 vs 397a7 -V3: 10/10

a8 gave some artifacts that sounded higher in pitch than the artifacts on the a7 encoded sample.


3.97a8 -V2 vs 3.97a7 -V2: 9/10

I didn't hear the high pitched artifacts with a8 -V2. Overall less artifacting than a7, so a8 was better.


3.97a8 -V2 vs 3.96.1 -V2: 10/10

3.97a8 was better.



ABX results on OrionII (2.1)

3.97a8 -V3 vs 3.97a7 -V3: 8/10

Less smearing with a8, a8 was better.


3.97a8 -V2 vs 3.97a7 -V2: 10/10

Less smearing with a8.


3.97a8 -V2 vs 3.96.1 -V2: 3/10

They sounded the same.



ABX results on hihat

wav vs LAME 3.97a8 -V3: 9/10

Each hit on the hihat produced a less precise/more spread sound with LAME.


wav vs LAME 3.97a8 -V2: 10/10

The same alteration here.


3.97a8 -V3 vs 3.97a7 -V3: 9/10

a8 sounded less precise than a7, a7 was closer to the original.


3.97a8 -V2 vs 3.97a7 -V2: 7/10

I'm not too sure here. Sometimes a7 sounded better, sometimes they sounded the same.

This post has been edited by lotr: Mar 13 2005, 01:37

[guruboolez]

These ABX tests were all done with vbr-old, correct?

I've only used -V2 & -V3 command line. The VBR mode isn't --vbr-new.

[Enig123]

Gabriel,

I tried to encode a track with this alpha today in a command line, and found there's a "switch" display between "long" and "short". I've checked the outputed mp3 and couldn't find any mixed-block, so it doesn't mean mixed-block.

I am curious what does this exactly mean.