Upgrade the official HA LAME version?, Now (3.96) or wait until 4.0? Options

[Jebus]

Basically, as Takahiro has noted that 4.0 is still a ways off, and 3.90 is several years old, should we work on testing 3.96 now? If this version tests out as well as 3.90.3, or does with some minor tweaking, then the speed advantages at the very least make for some incentive, don't you think?

Benefits of an upgrade now:

• Faster encodes
• Lower bitrates
• Built-in clipping removal
• Forced --alt-presets when using VBR

This post has been edited by Jebus: Mar 7 2004, 00:05

[atici]

What about: Wait for 3.96, and request developers to focus all further development towards FAAC? MP3 is as good as it gets anyway.

[Pio2001]

I vote for a test of the last version. Especially because the current recommended version has only been tested by Dibrom. I don't remember of any blind test results between alt-presets and custom command lines. Everytime someone wanted to challenge the presets, he disappeared without testing properly.
It would be a good thing to publicly test the presets.

[Teqnilogik]

I think we should test 3.96 and see about making it the recommended version. I tested 3.95.1 and the encoding was noticably faster and the quality seemed quite good with lower bit rates than 3.90.3. If we all put an effort in to test 3.96 I think it could become the recommended version and if we find flaws it would help the LAME developers to correct those issues so eventually we will get a new recommended version.

[westgroveg]

Tight preset tunning would need to be done for 3.96 as was done for 3.90.x by Dibrom.

[Moguta]

I'm all for a 3.96 --preset listening test vs. the 3.90.3 --alt-presets (and an anchor, obviously).

I have a feeling this will be incredibly harder to differenciate than even the recent AAC @ 128Kbps test, though.

[RyanVM]

I'm all for a 3.96 --preset listening test vs. the 3.90.3 --alt-presets (and an anchor, obviously).

I have a feeling this will be incredibly harder to differenciate than even the recent AAC @ 128Kbps test, though.

It seems to me that if that were the case, it would be a good argument for 3.96 being able to take over as the new recommended build

[funkyblue]

Bring on the Testing Would be great to have a new recommended version
Cheers

[bidz]

I second that. People tend to have more interest in AAC/Ogg nowadays, but hey - MP3 IS the defacto standard, and IS the king of the hill, and absolutely nothing will change that atleast in the next 5 years.

And yes, a new recommended version would be wonderful. I refuse to use 3.90.3 even now, i'd rather use 3.95.1.

Bring on the testing!

[kl33per]

There's an old saying that says if it ain't broke, don't fix. Unless LAME 3.96 offers the possibility of the same quality of aps at a reduced bitrate (which it may do, I'm not a LAME dev so I don't know) then I don't see why we should test it for now. However, as I understand it LAME 4.0 will have many new features and I think it will be worth testing.

[westgroveg]

There's an old saying that says if it ain't broke, don't fix.  Unless LAME 3.96 offers the possibility of the same quality of aps at a reduced bitrate (which it may do, I'm not a LAME dev so I don't know) then I don't see why we should test it for now.  However, as I understand it LAME 4.0 will have many new features and I think it will be worth testing.

With that thinking there would never be any progress, comparing the very well tunned 3.90.3 & 3.96 will help improve later versions of lame. If everyone just stopped testing until 4.0 was released I think we will find it to be a very disappointing release.

[kl33per]

I wasn't trying to discourage testing of new releases, just to discourage the massive amount of effort required to replace 3.90.3 as the recommended version.

[jtclipper]

Evolve use 3.96 , the developers keep working on the project, I see no reason to settle for less.

[guruboolez]

There's an old saying that says if it ain't broke, don't fix.  Unless LAME 3.96 offers the possibility of the same quality of aps at a reduced bitrate (which it may do, I'm not a LAME dev so I don't know) then I don't see why we should test it for now.  However, as I understand it LAME 4.0 will have many new features and I think it will be worth testing.
(....)

I wasn't trying to discourage testing of new releases, just to discourage the massive amount of effort required to replace 3.90.3 as the recommended version.

Interesting. But what about other encoders, like musepack, vorbis, or Nero AAC? Are there massive or collective tests? Or are people systematically adopting the latest version as reference encoder? This last behaviour is what happened and will happen with all encoders, except for lame, protected by very strange principles on this board.
Lame 3.96 is not an attempt to "fix" some problems, but a lot of work to improve quality and speed. Exactly what people like Frank Klemm did with musepack or Ivan Dimkovic with Psytel->Nero. I never saw the HA community discouraging people to use these new mpc or aac encoders; the whole community always trust their developers, and believe that new encoders are better, or at least flawless. Can't we trust lame developers? Why?

EDIT: spelling

This post has been edited by guruboolez: Mar 7 2004, 11:34

[ArsonDragon]

I agree...test 3.96 now, then, once 4.0 is ready in a year or two, undergo testing with that. It can't hurt to have a new recommended version in the interim. Personally, I can't wait to see what the addition of IS in 4.0 does. Seeing as I can't successfully ABX anything above 80kbps @ 44.1khz from the original...it will be a good move for me.

But until then, 3.96...7...8...9 will have to do.

Gotta love the Sony MDR-201 headphones

This post has been edited by ArsonDragon: Mar 7 2004, 10:26

[phwip]

I would like to see the latest version tested, and adopted if it is good enough. However, on the few samples I've tried to ABX 3.90.3 --alt-preset standard against the source wav file I have failed completely, and these have been supposedly difficult to encode samples.

This doesn't really surprise me as I don't have headphones and as I understand it --alt-preset standard is supposed to be transparent where possible with lame. But it does mean that I can't see that I have much to offer for the testing. And there is the problem: testing so close to transparency is really only possible for those with very sensitive hearing. So the rest of us can vote for testing but if those people are not interested in mp3 and lame then it is unlikely to ever happen.

[funkyblue]

I mean if a test was setup like AAC 128K we might have a successful test with a clearer view...I'm sure there would be really hard samples out there that would be good for alot of people to test...It would be nice to just be able to recommend the latest version of lame with --present standard....
Cheers

[n68]

gday..


well. am noy that huge mp3 fan.. after i have bee introduced to HE-AAC
and MPC. the only reason i can think of.. using the mp3 format..
would be for compabilety.

the only build i have tested out.. (slightly) is 3.95.1
and i liked what i heard.

if the 3.96 is tested out propperly.. i would say it`s time to upgrade.


btw. is there a changelog aviable for 96..

[guruboolez]

The changelog is available in the archive file on rarewares.
You can also take a look here.

[elmar3rd]

The last option in every poll should always be "I Don't know / Can't decide", depending on the subject.

[evereux]

If a test was to take place (and assuming it's conducted in a similar manner to roberto's tests). What kind of a confidence margin would the newer Lame version (if it was to win) require in order to replace 3.90.3 as recommended Lame version? I doubt the tests would show one to be clearly better than the other, since I bet they're both pretty transparent at these bitrates for most music.

[Jojo]

If a test was to take place (and assuming it's conducted in a similar manner to roberto's tests). What kind of a confidence margin would the newer Lame version (if it was to win) require in order to replace 3.90.3 as recommended Lame version? I doubt the tests would show one to be clearly better than the other, since I bet they're both pretty transparent at these bitrates for most music.

well, since LAME 3.95.1 is MUCH faster and produces a smaller file size in most cases (I replaced almost my entire CD-Collection with LAME 3.95.1 and it was always smaller than LAME 3.90.3 or even LAME 3.92; but some people reported a similar or a slightly higher bitrate on some samples), it would be enough if LAME 3.95.1 or LAME 3.96 would perform as good as LAME 3.90.3 . I'd say, if at least 50% of all tested samples sound better than 3.90.3 it should be updated...

Also, I think it kinda sucks for Gabriel and all the other LAME developers who have put so much time into the new releases and tuning them...since LAME 3.90 is still recommended it looks like if all there work from then was useless

This post has been edited by Jojo: Mar 7 2004, 14:43

[tigre]

IMO this can't be done in a way that is comparable to rjamorims listening tests, because:

- Something like 12 samples is not enough
- We need reliable results (= ABX testing required)
- We need to test several different presets / quality settings
- ...

Here's my suggestion how this should be done:

• Start a thread for reporting results only. The 1st post in this thread can be edited to collect the results.
• To make sure that comparable bitrates are tested (for abr/vbr presets that is), some mass-encoding and comparison has to be performed first (-> separate thread). The results are used to decide what to compare (e.g. 3.90.3 --alt-preset 128 vs. 3.96 ...)
• Everyone can use his own music or known problem samples to compare 3.90.3 vs. 3.96 at the quality setting he wants.
• Posting results in the thread requires:
  • Upload or link to sample
    
  • ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
    
  • Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones
• Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.
• I'm not sure if/how ABC/HR-like ratings should be used for this

The 1st post could look like this (everything is made up as example):
____________________________________________

Results for recommended lame version tests



1. 128kbps CBR, commandlines used: 3.90.3 --alt-preset CBR 128, 3.96 --preset CBR 128
___ samples where 3.90.3 is better,
___ samples where 3.96 is better so far.
Links to related posts: 1, 2, 3
Remarks:
3.90.3 has more problems with warbeling/flanging on cymbal-like sounds
3.96 has more pre-echo/smearing problems
...


2. 128kbps VBR/ABR, commandlines used: 3.90.3 --alt-preset 128, 3.96 ...
...
...


6. 160kbps VBR/ABR, commandlines used: 3.90.3 --alt-preset 160, 3.96: --preset medium
...
...
_____________________________________________

Hopefully there's some way to perform statistical analysis on these results that is able to tell after enough tests have been performed with e.g. > 95% reliability which one is better for a given quality/bitrate setting (ff123?)

[tigre]

Oh - and about ABR/VBR: We would need to know what to test, i.e. what setting is supposed to give best quality at a given average bitrate. IIRC the abr presets like --preset 128 still work with lame 3.95/3.96, but the -V settings have been changed by Gabriel and are supposed to work similar to Musepack/Vorbis -q ... settings. Probably it would be a good idea to test both while we're at it (e.g. --preset 128 and -V ... (whatever gives 128kbps on average)) - unless Gabriel (or someone else who should know) contradicts.

[kl33per]

Interesting. But what about other encoders, like musepack, vorbis, or Nero AAC? Are there massive or collective tests? Or are people systematically adopting the latest version as reference encoder? This last behaviour is what happened and will happen with all encoders, except for lame, protected by very strange principles on this board.
Lame 3.96 is not an attempt to "fix" some problems, but a lot of work to improve quality and speed. Exactly what people like Frank Klemm did with musepack or Ivan Dimkovic with Psytel->Nero. I never saw the HA community discouraging people to use these new mpc or aac encoders; the whole community always trust their developers, and believe that new encoders are better, or at least flawless. Can't we trust lame developers? Why?

You bring up some very good points that I hadn't thought of, however...

Firstly I can't speak about MusePack, because I wasn't on this forum when the major development was happening. Vorbis has seen little official devlopment since 1.0 (again I wasn't here for much pre-1.0 testing or what not) although people like Garf and QuantumKnot have been implementing there own tunnings to the codec. As far as I know (but correct me if I'm wrong, I'm not really sure) GT3b1 had quite a bit of testing before people decided to use it over pre 1.0-to-post1.0 versions of Vorbis. Futhermore, I was under the impression that Ahead had a QA department that made sure that at least nothing got worse in the codec compared to previous versions.

Again, I'm not trying to discourage some testing. If an extensive test is organised, I'll probably try to even be apart of it. I'm just trying to let people know that this sort of testing will be much more difficult then the 128kbps AAC test and only people with very good hearing will be able to provide ABX results. It will also be much longer, with many more samples and many weeks (probably months) of testing before LAME 3.96 (or any other version) can replace 3.90.3.

P.S. I agree that LAME is protected by strange principles. However, that doesn't make it any less of a fact that 3.90.3 is the most tested version of LAME ever, and that at least an equal (almost definately more) amount of testing will be required to replace it.