1st test from me to evaluated the new 3.97a11 abr mode in comparison to 3.90.3


128k - preset 128 - 3.97a11 vs 3.90.3


Average result:

3.90 = 2.2

3.97 = 3.0

3.90 has annoying ringing , 3.97 much less but also different distorsions. but overall more stable sounding on many samples.

3.90 --preset abr/cbr around 128 kbps has serious ringing issues, especially on quiet moments. 3.97 solves this, but as a consequence warbling is introduced on some tonal samples.

Anyway, thanks for the test. There are too few people to perform test

I thought it was a good idea to test Lame 3.96.1 versus 3.97alpha11 in --alt-preset standard mode.

As you all know, performing ABC/HR + ABX tests for that bitrate is very difficult, so I was able to test 8 samples only. Currently, my ears are very tired so adding more samples won't make the results more reliable. Maybe I'll add more results in the future.

Because guruboolez reported good results with the --vbr-new switch with the latest alphas, I included this setting to the test.

Test setup

I used the following setup:

PC with E-MU 0404 soundcard > Alecto Pro-147 mixing desk > Sennheiser EH1430 headphones. For the ABC/HR tests I used the program by ff123 (version 1.1beta2). Any ABX tests were done by either that program or foobar2000.

Test samples

As said before, I tested 8 samples. The collection contained 3 'famous' samples: castanets, fatboy and 41_30sec. I've also included 5 samples that I've found myself:

everybody (Jeff Buckley - Everybody here wants you)
My favorite. The snare drums in the intro sound really bad after encoding, warbling all over the place.

nyexcuse (Soulwax - NY Excuse)
The synthesizer sound at the very beginning is critical.

serpentine (dEUS - Serpentine)
The intro includes a high-pitched bell sound, which gets a little bit distorted.

belong (R.E.M. - Belong)
This track got a very nice recorded, natural sounding kick drum, which unfortunately doesn't survive the encoding process.

thinking (Zita Swoon - Thinking about you all the time)
The 'snare rim' sound at the beginning gets a little bit smeared.

Results

The results are presented in the following table:



Conclusions

Just like guruboolez, I think 3.97a11 --alt-preset standard --vbr-new is a winner. The normal aps setting of the latest alpha performs slightly worse than 3.96.1.

Ok, this was tough! I'm going to listen to some smooth jazz now.

Nice test, thanks a lot
Just one comment: the samples tested here are apparently for some of them critical samples: castanets, fatboy (I don't know the others, except 41_30). There's nothing wrong with that, but a listening evaluation based on critical sample isn't necessary representative of common material. An encoder could on one hand handle very well strong transients but on the other hand having a bit more problems with less critical material.
Of course, testing samples that are not very hard to encode is difficult, especially with --preset standard. I did it, but I have to listen carefully to subtle details.

Anyway, I'm pleased and also reassured to see that other people are sharing the same impression about --vbr-new mode

Yes, I agree. It is very hard to test a setting that's supposed to be transparent. Therefore I had to search for samples that I could ABX without getting too tired or losing my concentration too quickly.

Because of this, I tried to search for samples that had different 'problem sounds': snare drum, kick drum, synthesizer, vocoder (fatboy), etc. To not making things too unfair, I mean.

Way to go bug80!

Here's the data in a form you can use on http://www.ff123.net/friedman/stats.html. The default test parameters result in: "397a11new is better than 3961, 397a11". Thank you, bug80!

And thank you for showing that the results are in fact statisticly relevant.

I wonder, what settings lower than -V2 are usefull to test besides aps? V4 or V5 maybe?

My opinion is that the phrase "representative of common material" is a little bit inappropriate, or at least a bit inadequate.

No sample, or small set of samples, can be representative of common material. It just makes me wonder what types of sounds are considered common and by whom? My collection is large & varied enough that to me, anything is common (or nothing is).

Most ciritical samples I've listened to have come from music that is popular, or at least not very obscure. Or from natural sounding instruments like castanets or harpsichord. Why shouldn't these sources be considered common?

I think we use these critical samples because there is an assumption that if an encoder does well at hard-to-encode sounds where they are not masked by other sounds, they will do just as well with mixed sounds.

yes! -V 5 would be great!

Could you possibly try it (the latest LAME alpha) against the Helix encoder? I KNOW that is a bit off topic, but LAME -V 5 and Helix around -V65 (something in that area) should produce roughly the same file size.

What guruboolez means, is that a particular set of problem samples that focusses on one particular problem, for example high transient sounds (sharp attacks), may not reveil other problems with audio compression.

For example: I once tested Ogg Vorbis at different settings. Because I have experience with testing MP3, I focussed on high transient sounds, and found that I heard no major problems at q4. However, when I started focussing on other things I noticed that on one piece of a sample a complete instrument (cello) was gone!

Therefore it is important to have a set of samples that not only consists of so-called "problem samples".

As soon as I have the time I will try V5. Why is it exactly that you want to campare with Helix? File size isn't really important, quality is. Of course the file size of the "new" V5 should not differ too much from the old one, but in the end it's of most interest to see if the new Lame is an improvement, sound quality wise.

My major goal is comparing with older Lame versions, because I think 3.97 deserves a final release as soon as possible. The more people test the current alpha, the better, am I right?

WHY I want to test it against Helix? I know I'm a bit impatient but the Helix hasn't been compared to LAME in that bit range (120-140 kbps). At least not here at HA, AFAIK. That's what I'm using for my portable. My frickin' speakers are kaputt so I have to put my own ABXing on ice. Grrr...

But that's just a suggestion! You do whichever comparison you like, they will nontheless be valuable.

Ok, I will add Helix to the test, if you can point me to the latest compiled version. Could you also give the exact command line I have to use? I have no experience with this encoder

See this thread:

http://www.hydrogenaudio.org/forums/index....=35531&hl=helix

Everything you need to know is to be found there. I listen mostly to metal and I've found that -V65 is pretty clost to LAME -V5 in size. My commandline with Wav2gogo (see thread) is "-V65 -X -U2" (with or w/out -U2).

Thanks. I need hmp3enc.exe (rev11) right?

By the way, I tried to test the -V4 setting today, but it's virtually impossible to ABX 3.96.1 versus 3.97a11 on that setting, the bitrates are very close too. Guruboolez, what do you do, when you encounter problems like this? Do you focus on one very short, specific part of the samples? Have you tried to ABX the latest -V4 setting yet?

It's also possible that my ears aren't trained enough, or that the quality of my headphones is too low.

Yes. See the thread how to set up the encoder. Either with Foobar2000 or Wav2GoGo. (using the latter, you have to rename the encoder to "gogo")

It works (that 16 bit setting is a tricky one )

By the way, for fair comparison I think -V55 is a better setting for Helix. I converted 10 random files and these are the average results:

Lame 3.96.1: 147 kbs
Lame 3.97a11 vbr-new: 151 kbs
Helix (-V55): 148 kbs

Yeah, I have noticed the bitrate vary a whole lot from track to track and from album to album using Helix. In my opinion more so than LAME. Using -V65 I get files from 120 kbps to 150 kbps from the same album! -V55 (Helix) may very well be comparable with LAME -V5 regarding size.

Very interesting. Thank you shadowking for the results.

Thanks for your effort.
Second bad vilbel. Always worth the listen

I did ABC/HR tests with about 5 samples so far and was very impressed by the quality of the Helix encodings. The results look good for Helix, could be coincidence. I have to test more samples to be sure.

Thanks shadowking! "noise dropouts" is something I notice a lot too with the 3.96.1 version. It seems that this has improved in 3.97a11.

Thank you bug80. You are the first one who test Helix mp3 encoder at that bitrate rage here.

It would be nice if this test, this one and further test of LAME 3.97a11 would be merged in a single topic in the MP3 TECH forum, as other topic dedicated to different LAME alphas:

lame 3.97 alpha 5 testing thread
lame 3.97 alpha 6 testing thread
lame 3.97 alpha 7 testing thread
lame 3.97 alpha 8 testing thread
lame 3.97 alpha 10 testing thread
Lame test version - may 2005
Lame test version (June 2005)


EDIT: done, thanks [if another moderator see this message, he can safely delete it]

Unless I am mistaken, -vbr-new maps to the old alt-preset standard fast which always provided slightly lower quality but for increased encoding speed. Any idea why in 3.97 it would now offer higher quality? It does not seem to make sense. Faster and higher quality? Too good to be true right?

Why? Slow encoders are not necessary better than faster ones. Take AAC as example: Psytel is twice slower than Nero AAC which is twice slower than iTunes AAC. But iTunes is better than Nero which is better than Psytel.
Other example? MPC: old Buschmann encoders are slower than Klemm encoders, and I didn't someone claiming that the last ones is worse than the old ones. Still not happy? Vorbis 1.00 RC2/3 are much slower than Vorbis 1.1, but so worse in comparison...
Recently, MPC developers had slightly improved speed without consequence on quality. Vorbis LANCER project offers even a better speed boost.
So, if AAC, Vorbis and MPC could improve speed and quality in the same time, same could happen for MP3.

I can't give you any technical information nor explain why --vbr-new is suddenly as good if not better than defaulted VBR mode, but I can just say that there's no direct and systematical relation between speed and quality.

Sounds good to me!

Simple: vbr-new is not suddendly better than the legacy vbr mode, but rather improved progressively over time. There is no sudden magic here, just long work of Robert.

Regarding speed vs quality:
*dist10 (ISO reference code) was very slow, but Blade managed to produce exactly the same output while beeing way faster.
*Lame is claimed to be of higher quality than Blade, while beeing quite faster.
*not really related, but: if I compile my code in debug mode, it will be quite slower but this will not improve quality.

By the way, please forget my last statement, that "The normal aps setting of the latest alpha performs slightly worse than 3.96.1". When performing statistical analysis, it can not be said with enough probability that this is true.

However, the statement that 397a11new is better than 397a11 and 3961 holds, based on my data.

LAME 3.96.1 vs 3.97a11new vs Helix (quality V5)

I decided to do some testing for Lame alpha11 at quality -V5. By request, I also added the Helix encoder. These are the different settings:

Lame 3.96.1 -V5
Lame 3.97a11 -V5 --vbr-new
Helix rev11 -V55

I choose V55 for Helix because the bitrate will be in the same range as Lame with that quality.
To keep the number of samples low, I only tested 3.97a11 with the --vbr-new switch. I think we can conclude by now that this will give the best results anyway (?).

Test setup

The test setup was (again):

PC with E-MU 0404 soundcard > Alecto Pro-147 mixing desk > Sennheiser EH1430 headphones

Test samples

I used the same samples as in my APS test, plus some extra 'famous' samples. The total number of test samples is 16.

Test results

Below are the ABC/HR results.



Statistical analysis

If I put the following data in the analysis script at http://www.ff123.net/friedman/stats.html:



the result is (with default settings):



and with a tolerance of 10% instead of 5%:



Comments

According to the results, both Lame 3.96.1 and 3.97a11new are better than Helix, with a tolerance of 10%.

On most samples, I wasn't able to ABX both Lame versions. The quality is really close to my ears (and according to the stats, I don't prefer one of the two more than the other).

For people interested in Helix: I run some quick tests with Helix before and I was impressed by it's quality on some of my problem samples (castanets, everybody, fatboy). However, Helix seems to have serious problems with some other samples (the synthesizers in FloorEssence en nyexcuse, the piano in Hongroise for example). I think most problems occur on transient sounds with lots of low frequency content and transient sounds that have a low amplitude.

The speed of the Helix encoder is amazing. I didn't measure it, but I guess it's 2-3 times faster than Lame 3.97a11 on my PC.

Hi

I don't know if this will help but I decided to do a little comparison between 3.96.1 and 3.97 a10 (I'm only using vbr so a11 shouldn't be any different). I'm only using one sample as I haven't done listening test in years and it took me an age before I could spot the encoder errors. First I tried to ABX the original from 3.97 --vbr-new -v4 -q0. Took an age going from A to B but I picked up a "warrble" (my word for a wierd morse code sounding wow in the sound) and then dived into the ABX and got this result:

A file: C:\Documents and Settings\Andrew\My Documents\My Music\ABX\Lame397a10 V settings\Def Leppard - 03 - Love Bites.wav
B file: C:\Documents and Settings\Andrew\My Documents\My Music\ABX\Lame397a10 V settings\defv4.wav

Start position 00:00.0, end position 00:10.4
18:32:21 1/1 p=50.0%
18:37:44 2/2 p=25.0%
18:40:08 3/3 p=12.5%
18:44:20 4/4 p=6.2%
18:55:32 5/5 p=3.1%
19:12:05 6/6 p=1.6%
22:42:12 7/7 p=0.8%
22:44:02 8/8 p=0.4%
22:44:42 9/9 p=0.2%
22:45:21 10/10 p< 0.1%
22:46:00 11/11 p< 0.1%
22:46:21 12/12 p< 0.1%
22:46:56 13/13 p< 0.1%
22:47:53 14/14 p< 0.1%
22:48:45 test finished

(I went out to rent a DVD at 19:15 in case you were wondering) as you can see by then end I pretty much could spot it strait away. Then I went the whole hog and tried 3.97 --vbr-new -v0 -q0 and got

A file: C:\Documents and Settings\Andrew\My Documents\My Music\ABX\Lame397a10 V settings\Def Leppard - 03 - Love Bites.wav
B file: C:\Documents and Settings\Andrew\My Documents\My Music\ABX\Lame397a10 V settings\defv0.wav

Start position 00:00.0, end position 00:10.4
12:00:20 1/1 p=50.0%
12:25:18 2/2 p=25.0%
15:37:34 3/3 p=12.5%
15:38:43 4/4 p=6.2%
15:39:12 5/5 p=3.1%
15:39:31 6/6 p=1.6%
15:40:23 6/7 p=6.2%
15:40:46 6/8 p=14.5%
15:42:02 7/9 p=9.0%

Not conclusive but I've subsequently gone up to 6/6 three times (without any other tries in between) as sometimes I can hear the difference as clear as day and then it just goes and I press A and B and can't hear a difference. Thing is I can't ABX 3.96.1 at the same settings (except no vbr new) at all. Am I just freaky or can anyone else hear it. Link here (but please only download if serious)
http://zaffman.co.uk/sitebuildercontent/si...derfiles/lb2.wv

Thanks

Could you please give me the exact point or range where the artefact occurs?

When using 3.97 a11 with 'preset standard, vbr new' the bitrate drops down to 112 kb/s at times... I personally think this is a bit too low. How do I set 128 kb/s as a minimum?

Problem is not whay you think, but what you hear. Anyway, the answer is -b 128. See Full command line switch reference.

Adding -b128 will LAME tell to use 128 kbps as minimum bitrate, but it would not make much difference, as it affects frame allocation only and the extra bits go into the bitreservoir.

At around 2.3 to 3.7 seconds

i'm sorry for going off topic but this thing interests me

it's good that lame uses the not-need bits for the reservoir. how is the reservoir used if the framesize is above 128? does lame keep a certain amount for problem cases?

the point of my posting is, we all know that mp3 is limited to 320 kbit frames. couldn't the reservoir bits be used if a higher bitrate is needed?
a solution would be to always have a certain amout of bitreservoir, but i think that would be infefficient.
an elegant solution would be a 2-pass mode that scans the file in the first pass to save bits that will be used immediatly after the "easy" part of the audio signal.

i just wanted to write down my thoughts. it would be nice if someone could explain some things about the bit-reservoir and how it's used in lame

I could ABX -V4 --vbr-new. But something is not clear in my mind: you said previously that you did a comparison between 3.96.1 and 3.97a. What I don't understand is: did you ABXed a difference between both encoding? And if yes, what is the setting for 3.96.1 (--vbr-new or not)?
I've tried to ABX 3.96.1 -V4new to 3.97a11 -V4new, and focused on the playback range (2.3 - 3.7), but I couldn't get positive ABX results.

After I ABX'd v4 (which I didn't expect to do) I then tried -V0 on 3.97a10 -q0 to see if the artifact was still present and I have had moderate sucess (6/6) ABXing it (although it understandably takes a long time as its very difficult to hear). I then tried 3.96.1 -v0 (old vbr) and have not been able to ABX that to the original at all leading me to think 3.97 is introducing some form of artifact even at the extreme quality setting. Could be the -q0 switch so if I have the time I'll try that later and it could be fluke on the ABX (although I did it three time but never above 6). All that said I would never notice a thing if I didn't have the original reference so it's not like it's noticable or distracting. I've subsequently tried other samples and unable to ABX (3.97 to original) below v2 on any of them I've not yet had the chance to compare to 3.96.1 on those either.