Basically, as Takahiro has noted that 4.0 is still a ways off, and 3.90 is several years old, should we work on testing 3.96 now? If this version tests out as well as 3.90.3, or does with some minor tweaking, then the speed advantages at the very least make for some incentive, don't you think?

Benefits of an upgrade now:

• Faster encodes
• Lower bitrates
• Built-in clipping removal
• Forced --alt-presets when using VBR

What about: Wait for 3.96, and request developers to focus all further development towards FAAC? MP3 is as good as it gets anyway.

I vote for a test of the last version. Especially because the current recommended version has only been tested by Dibrom. I don't remember of any blind test results between alt-presets and custom command lines. Everytime someone wanted to challenge the presets, he disappeared without testing properly.
It would be a good thing to publicly test the presets.

I think we should test 3.96 and see about making it the recommended version. I tested 3.95.1 and the encoding was noticably faster and the quality seemed quite good with lower bit rates than 3.90.3. If we all put an effort in to test 3.96 I think it could become the recommended version and if we find flaws it would help the LAME developers to correct those issues so eventually we will get a new recommended version.

Tight preset tunning would need to be done for 3.96 as was done for 3.90.x by Dibrom.

I'm all for a 3.96 --preset listening test vs. the 3.90.3 --alt-presets (and an anchor, obviously).

I have a feeling this will be incredibly harder to differenciate than even the recent AAC @ 128Kbps test, though.

It seems to me that if that were the case, it would be a good argument for 3.96 being able to take over as the new recommended build

Bring on the Testing Would be great to have a new recommended version
Cheers

I second that. People tend to have more interest in AAC/Ogg nowadays, but hey - MP3 IS the defacto standard, and IS the king of the hill, and absolutely nothing will change that atleast in the next 5 years.

And yes, a new recommended version would be wonderful. I refuse to use 3.90.3 even now, i'd rather use 3.95.1.

Bring on the testing!

There's an old saying that says if it ain't broke, don't fix. Unless LAME 3.96 offers the possibility of the same quality of aps at a reduced bitrate (which it may do, I'm not a LAME dev so I don't know) then I don't see why we should test it for now. However, as I understand it LAME 4.0 will have many new features and I think it will be worth testing.

With that thinking there would never be any progress, comparing the very well tunned 3.90.3 & 3.96 will help improve later versions of lame. If everyone just stopped testing until 4.0 was released I think we will find it to be a very disappointing release.

I wasn't trying to discourage testing of new releases, just to discourage the massive amount of effort required to replace 3.90.3 as the recommended version.

Evolve use 3.96 , the developers keep working on the project, I see no reason to settle for less.

Interesting. But what about other encoders, like musepack, vorbis, or Nero AAC? Are there massive or collective tests? Or are people systematically adopting the latest version as reference encoder? This last behaviour is what happened and will happen with all encoders, except for lame, protected by very strange principles on this board.
Lame 3.96 is not an attempt to "fix" some problems, but a lot of work to improve quality and speed. Exactly what people like Frank Klemm did with musepack or Ivan Dimkovic with Psytel->Nero. I never saw the HA community discouraging people to use these new mpc or aac encoders; the whole community always trust their developers, and believe that new encoders are better, or at least flawless. Can't we trust lame developers? Why?

EDIT: spelling

I agree...test 3.96 now, then, once 4.0 is ready in a year or two, undergo testing with that. It can't hurt to have a new recommended version in the interim. Personally, I can't wait to see what the addition of IS in 4.0 does. Seeing as I can't successfully ABX anything above 80kbps @ 44.1khz from the original...it will be a good move for me.

But until then, 3.96...7...8...9 will have to do.

Gotta love the Sony MDR-201 headphones

I would like to see the latest version tested, and adopted if it is good enough. However, on the few samples I've tried to ABX 3.90.3 --alt-preset standard against the source wav file I have failed completely, and these have been supposedly difficult to encode samples.

This doesn't really surprise me as I don't have headphones and as I understand it --alt-preset standard is supposed to be transparent where possible with lame. But it does mean that I can't see that I have much to offer for the testing. And there is the problem: testing so close to transparency is really only possible for those with very sensitive hearing. So the rest of us can vote for testing but if those people are not interested in mp3 and lame then it is unlikely to ever happen.

I mean if a test was setup like AAC 128K we might have a successful test with a clearer view...I'm sure there would be really hard samples out there that would be good for alot of people to test...It would be nice to just be able to recommend the latest version of lame with --present standard....
Cheers

gday..


well. am noy that huge mp3 fan.. after i have bee introduced to HE-AAC
and MPC. the only reason i can think of.. using the mp3 format..
would be for compabilety.

the only build i have tested out.. (slightly) is 3.95.1
and i liked what i heard.

if the 3.96 is tested out propperly.. i would say it`s time to upgrade.


btw. is there a changelog aviable for 96..

The changelog is available in the archive file on rarewares.
You can also take a look here.

The last option in every poll should always be "I Don't know / Can't decide", depending on the subject.

If a test was to take place (and assuming it's conducted in a similar manner to roberto's tests). What kind of a confidence margin would the newer Lame version (if it was to win) require in order to replace 3.90.3 as recommended Lame version? I doubt the tests would show one to be clearly better than the other, since I bet they're both pretty transparent at these bitrates for most music.

well, since LAME 3.95.1 is MUCH faster and produces a smaller file size in most cases (I replaced almost my entire CD-Collection with LAME 3.95.1 and it was always smaller than LAME 3.90.3 or even LAME 3.92; but some people reported a similar or a slightly higher bitrate on some samples), it would be enough if LAME 3.95.1 or LAME 3.96 would perform as good as LAME 3.90.3 . I'd say, if at least 50% of all tested samples sound better than 3.90.3 it should be updated...

Also, I think it kinda sucks for Gabriel and all the other LAME developers who have put so much time into the new releases and tuning them...since LAME 3.90 is still recommended it looks like if all there work from then was useless

IMO this can't be done in a way that is comparable to rjamorims listening tests, because:

- Something like 12 samples is not enough
- We need reliable results (= ABX testing required)
- We need to test several different presets / quality settings
- ...

Here's my suggestion how this should be done:

• Start a thread for reporting results only. The 1st post in this thread can be edited to collect the results.
• To make sure that comparable bitrates are tested (for abr/vbr presets that is), some mass-encoding and comparison has to be performed first (-> separate thread). The results are used to decide what to compare (e.g. 3.90.3 --alt-preset 128 vs. 3.96 ...)
• Everyone can use his own music or known problem samples to compare 3.90.3 vs. 3.96 at the quality setting he wants.
• Posting results in the thread requires:
  • Upload or link to sample
    
  • ABX results Original<->3.90.3, Original<->3.96, 3.90.3<->3.96, with detailed description of the difference(s)
    
  • Report about software/hardware used: Soundcard (resampling?), Player/ABXtool, DSPs (shouldn't be allowed, besides resampling to 48kHz and volume reduction/replaygain to prevent clipping <- both a 'must'), Amplifier, Speakers/Headphones
• Results must be confirmed by someone else before they are included in 'official' statistic, p-values must be < 0.05 for at least 2 people.
• I'm not sure if/how ABC/HR-like ratings should be used for this

The 1st post could look like this (everything is made up as example):
____________________________________________

Results for recommended lame version tests



1. 128kbps CBR, commandlines used: 3.90.3 --alt-preset CBR 128, 3.96 --preset CBR 128
___ samples where 3.90.3 is better,
___ samples where 3.96 is better so far.
Links to related posts: 1, 2, 3
Remarks:
3.90.3 has more problems with warbeling/flanging on cymbal-like sounds
3.96 has more pre-echo/smearing problems
...


2. 128kbps VBR/ABR, commandlines used: 3.90.3 --alt-preset 128, 3.96 ...
...
...


6. 160kbps VBR/ABR, commandlines used: 3.90.3 --alt-preset 160, 3.96: --preset medium
...
...
_____________________________________________

Hopefully there's some way to perform statistical analysis on these results that is able to tell after enough tests have been performed with e.g. > 95% reliability which one is better for a given quality/bitrate setting (ff123?)

Oh - and about ABR/VBR: We would need to know what to test, i.e. what setting is supposed to give best quality at a given average bitrate. IIRC the abr presets like --preset 128 still work with lame 3.95/3.96, but the -V settings have been changed by Gabriel and are supposed to work similar to Musepack/Vorbis -q ... settings. Probably it would be a good idea to test both while we're at it (e.g. --preset 128 and -V ... (whatever gives 128kbps on average)) - unless Gabriel (or someone else who should know) contradicts.

You bring up some very good points that I hadn't thought of, however...

Firstly I can't speak about MusePack, because I wasn't on this forum when the major development was happening. Vorbis has seen little official devlopment since 1.0 (again I wasn't here for much pre-1.0 testing or what not) although people like Garf and QuantumKnot have been implementing there own tunnings to the codec. As far as I know (but correct me if I'm wrong, I'm not really sure) GT3b1 had quite a bit of testing before people decided to use it over pre 1.0-to-post1.0 versions of Vorbis. Futhermore, I was under the impression that Ahead had a QA department that made sure that at least nothing got worse in the codec compared to previous versions.

Again, I'm not trying to discourage some testing. If an extensive test is organised, I'll probably try to even be apart of it. I'm just trying to let people know that this sort of testing will be much more difficult then the 128kbps AAC test and only people with very good hearing will be able to provide ABX results. It will also be much longer, with many more samples and many weeks (probably months) of testing before LAME 3.96 (or any other version) can replace 3.90.3.

P.S. I agree that LAME is protected by strange principles. However, that doesn't make it any less of a fact that 3.90.3 is the most tested version of LAME ever, and that at least an equal (almost definately more) amount of testing will be required to replace it.

Nice idea to count test samples that became better or worse with 396b!

I can add here sophia2 is clean the first time with aps!

390.3 -> sandpaper noise
395 -> added plop
396b -> clean

Wombat

I always thought LAME 3.90.3 was recommended for --APS only

LAME 3.90.3 is recommended over other versions for many different settings. APS is the recommended setting for perceptual transparency. Sometimes you need to make a compromise on quality vs. filesize (which means you'll lose transparency). This is why other bitrates also need to be tested.

Is there a special reason to be testing 3.96 rather than the upcomming 3.97?

I'm not trying to sound sarcastic. Maybe the developers consider 3.96 to be a very good version. Maybe they feel it is just an interim release before the real spanking 3.98 comes out.

I'm asking because with every new version people will come and ask if it is recommended over the current one. I just want to make sure that good effort wich should maybe be spent on a later (not so distant) version isn't wasted on this version.

Serious testing should be done, even if lame 3.96 isn't a major version. There are two possibilities:
- lame 3.96 is fine => recommanded version, and bye-bye old 3.90
- lame 3.96 have major problems => this will help developers to improve the next version.

Waiting unfortunately doesn't help. Lame developers need feedback, otherwise, we can't expect major improvements.

maybe we should give Gabriel the chance to state his opinion. However, if you take a look at the change history, there is quite a big time frame between LAME 3.93.1 and the newest stable release 3.95.1 (more than 1 year!).

So I think we have to ask Gabriel. He once told me that there will be of course always a newer release...so it wouldn't make much sense to wait and wait, especially if LAME 3.97 would take another year or so...

Personally, I wouldn't wait to long for test...because AAC is coming much faster than expected...probably because of Apple's: iPod, Music Store, Superior AAC Encoder and iTunes...it's easy to use, free and gives very good quality at low bitrates; no confusion about all the different settings...so what I really want to say is, that a test that takes place in a year or so might not be that interesting for many people, because of other formats that more and more find their way on people's computer's... I mean, iTunes AAC @ 160kbps = awesome

ACH! my post disappeared.

well, anyways, i say wait for 4.0

in my opinion there's very little difference in the quality of --alt-presets from version 3.90 up. it would make more sense to go to all that testing effort with 4.0 rather than a release that it's taken as a given that the difference will be tremendously difficult to spot.

3.96 will probably be promoted to stable.

3.90.X are based on a more than 2 years old release.

You have some choices:
*keep using 3.90.X, and do not bother with new versions. Of course, you fully understand that the current justification used to not test new versions will forever be true, so it means that you will always keep using the same version.

*Try new versions. If they are fine to you, then use them. If they are not fine to you, then report why so it could be corrected in next versions.

By choosing the second option, you will be able to know if you can replace the version you are using, and will help new better versions to appear.
By choosing the first option, you choose to ignore if you can replace your current version, and are also choosing to not help in development.

You also have to keep in mind that 3.97 will be based on 3.96, and 3.98 will be based on 3.97. They will not be based on 3.90.X.

So jump in, or stay where you are. But in this case, you will likely stay there forever.

I hope that you do not have preset standard in mind, because a preset targeted to transparency will always be close to a preset targeted to transparency, beeing in 3.90.3, 3.96, 4.0 or 9.0.

Well the birds sample is cured absolutely also with aps!

Sandpaper noise is gone!

Very promising so far, Gabriel

Wombat

We can't keep the 3.90 release "haunting" us forever...it might be a landmark in the Lame history, but we have to check if things have actually progressed since then. Hydrogenaudio is all about testing and suggesting accordingly. Keeping up-to-date is not always essential, but we have to check v3.96's quality before condemning it as inferior and sticking again with 2 years old code. I don't know if the latest version is better, but I sure want to find out.

I think that's the point.

The developers have invested their valuable time to improve the new versions.
So we, the users should give our feedback.
Keeping 3.90.x as the holy cow forever and ignoring what the developers offer might be disrespectful ...
Maybe it is not necessary to test every new release to the maximum extend, but the ones the developers recommend as worth to be.
Maybe Gabriel or s.o. could give a clue if 3.96 is THE ONE or when the next relase comes out that has the potential to be the successor of the honourable 3.90
...and as Guruboolez stated, this is also a chance to detect the difference between new releases, which could result in further improvements.

Thanks for your answers, Gabriel. The remaining question would be this:
Can the current 3.96 beta be considered stable quality-wise? From what I read from the change-log there haven't been any quality-related changes since 3.95.1 - is this correct? Of course there can always be bugs that affect quality, finding them is one reason for testing - I just mean starting tests with big efford would be pointless if quality-related changes are planned before 3.97 stable. Comments?

Quality related changes are always planned: we are developing in mp3 encoder, and so quality is one of the goals.
So yes, there will be quality related changes, but how is that preventing you from testing the current version?

I hope that you are not waiting for the end of quality related changes before trying a new version , or you will probably never try any new version before the final stop of the Lame development.

you also have to take into account, that even if we now test 3.96 and tune it(if necessary)... and then one day there'll be an 3.97 - then 3.97 will probably take advantage of at least some of the tunings and fixes of 3.96.

So, later versions will take advantage of 3.96 testing anyways. Of course with the implementation of new features, this advantage will slowly fade. Still... even if we dont get the absoluetely-total-optimum-version to do testing, future versions will take advantage of this. In other words - no effort gets lost.

- Lyx

3.96 has some bugs fixed that have been in LAME for over a year. those bugs surely have an effect on quality. we want 3.96 to be as stable as we can get to have some anchor for further development. if there are no big problems (*encoding* problems), end of march we hope to release 3.96 stable. For 3.97 we will probably focus more on the decoder. there is a plan replacing mpglib by hip finally.

Bitrate bloat seems under control now as well. I was using --lowpass 17600 on 3.90.3. This version gives lower bitrates than that setting without even using a lowpass.

I think you misunderstood me. I wasn't saying that we should stick to 3.90.3 forever and testing new versions was pointless.

I believe that the amount of testing required to possibly find a replacement for the current recommendation is very large. A great effort like this will not be made often and the amount of (free) time people are willing to invest should be wisely spent.

So I'm mainly asking if this version is worth it or if maybe you have a not so distant version on your roadmap that would benefit more from such an effort. For example a developer has been working on theoretically significant quality improvements which are going to be merged with the official branch in version 3.97.

I have no insight in the development process, so I was asking about that, without implying that the status quo should remain unchanged forever.

You've said that before, but obviously I wasn't able to ask my question in an understandable way...

My point is: If many people start testing with the current beta, but the next version you're working on right now contains modifications that would change these test results completely, the modifications are already finished, there's just no new beta version released yet ... in this case it would make more sense to test the next version than starting tests right now. Of course - if you want a comparision between current beta and next one, you can ask for tests and people will help, as it has happened here before.


Well, if - as in the example I described above - tests are outdated while they're done, the benefit from the effort is not optimal ...

Anyway, Robert's post answers my question - thanks.

But why do we need a grand test before changing the official recommandation? Couldn't we simply use lame 3.96 for two or three weeks, and then decree 3.96 fully safe if nothing wrong was reported? It makes sense. I don't see anybody keeping Buschman 1.78c encoder, just because Klemm's encoders weren't collectively tested. Same for AAC, same for vorbis, etc... From where come this idea of a big, collective and complete check-up? And why is it so specific to lame? When a new Nero version is released, all people expect a new release of aacenc32.dll; they're not saying to Ivan "sorry, but PsyTEL 2.15 was more tested than Nero 2.6.2.0, I'll consider Nero AAC in two years if stable. Bye."

If people really care about a possible and serious regression, they have to prove it. It's not to other people to prove that the newest version is superior to the 27 months (!!) old 3.90.x.

Your position makes sense, guruboolez.
The burden of the proof lies on the person making the claim. LAME has undergone tweaking and has become faster at encoding. That alone is an improvement that warrants upgrading. If the upgrade hasn't been made, is because there is an implicit claim stating that 3.90.3 is still better than 3.96, despite the slower encoding and its age. Is it so? Prove it. Otherwise, the logical choice is to recommend the newer version, which in theory should be better.

To make it clear: there is no major quality improvment ready to be released just now.

There is no major quality improvement IN 3.96 beta over 3.95?

Or do you mean that there isn't any major quality improvement to be expected for some time AFTER 3.96 (beta)?

3.96b has already been released, so its quality improvments are already released.

TNX for clarification.

I hope, that the HA recommendation can be updated now soon.

TNX for spending time with working on LAME, Takehiro, Gabriel, Robert and the others.