Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Lame 3.97 alpha 11 testing thread (Read 29715 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Lame 3.97 alpha 11 testing thread

1st test from me to evaluated the new 3.97a11 abr mode in comparison to 3.90.3


128k - preset 128 - 3.97a11 vs 3.90.3   


Average result:

3.90 = 2.2

3.97 = 3.0 

3.90 has annoying ringing , 3.97 much less but also different distorsions. but overall more stable sounding on many samples.
 

Lame 3.97 alpha 11 testing thread

Reply #1 –
3.90 --preset abr/cbr around 128 kbps has serious ringing issues, especially on quiet moments. 3.97 solves this, but as a consequence warbling is introduced on some tonal samples.

Anyway, thanks for the test. There are too few people to perform test

Lame 3.97 alpha 11 testing thread

Reply #2 –
I thought it was a good idea to test Lame 3.96.1 versus 3.97alpha11 in --alt-preset standard mode.

As you all know, performing ABC/HR + ABX tests for that bitrate is very difficult, so I was able to test 8 samples only. Currently, my ears are very tired so adding more samples won't make the results more reliable. Maybe I'll add more results in the future.

Because guruboolez reported good results with the --vbr-new switch with the latest alphas, I included this setting to the test.

Test setup

I used the following setup:

PC with E-MU 0404 soundcard > Alecto Pro-147 mixing desk > Sennheiser EH1430 headphones. For the ABC/HR tests I used the program by ff123 (version 1.1beta2). Any ABX tests were done by either that program or foobar2000.

Test samples

As said before, I tested 8 samples. The collection contained 3 'famous' samples: castanets, fatboy and 41_30sec. I've also included 5 samples that I've found myself:

everybody (Jeff Buckley - Everybody here wants you)
My favorite. The snare drums in the intro sound really bad after encoding, warbling all over the place.

nyexcuse (Soulwax - NY Excuse)
The synthesizer sound at the very beginning is critical.

serpentine (dEUS - Serpentine)
The intro includes a high-pitched bell sound, which gets a little bit distorted.

belong (R.E.M. - Belong)
This track got a very nice recorded, natural sounding kick drum, which unfortunately doesn't survive the encoding process.

thinking (Zita Swoon - Thinking about you all the time)
The 'snare rim' sound at the beginning gets a little bit smeared.

Results

The results are presented in the following table:

Lame 3.97 alpha 11 testing thread

Reply #3 –
Quote
Conclusions

Just like guruboolez, I think 3.97a11 --alt-preset standard --vbr-new is a winner. The normal aps setting of the latest alpha performs slightly worse than 3.96.1.
[a href="index.php?act=findpost&pid=316811"][{POST_SNAPBACK}][/a]


Nice test, thanks a lot
Just one comment: the samples tested here are apparently for some of them critical samples: castanets, fatboy (I don't know the others, except 41_30). There's nothing wrong with that, but a listening evaluation based on critical sample isn't necessary representative of common material. An encoder could on one hand handle very well strong transients but on the other hand having a bit more problems with less critical material.
Of course, testing samples that are not very hard to encode is difficult, especially with --preset standard. I did it, but I have to listen carefully to subtle details.

Anyway, I'm pleased and also reassured to see that other people are sharing the same impression about --vbr-new mode

Lame 3.97 alpha 11 testing thread

Reply #4 –
Quote
Quote
Conclusions

Just like guruboolez, I think 3.97a11 --alt-preset standard --vbr-new is a winner. The normal aps setting of the latest alpha performs slightly worse than 3.96.1.
[a href="index.php?act=findpost&pid=316811"][{POST_SNAPBACK}][/a]


Nice test, thanks a lot
Just one comment: the samples tested here are apparently for some of them critical samples: castanets, fatboy (I don't know the others, except 41_30). There's nothing wrong with that, but a listening evaluation based on critical sample isn't necessary representative of common material. An encoder could on one hand handle very well strong transients but on the other hand having a bit more problems with less critical material.
Of course, testing samples that are not very hard to encode is difficult, especially with --preset standard. I did it, but I have to listen carefully to subtle details.
[a href="index.php?act=findpost&pid=316816"][{POST_SNAPBACK}][/a]

Yes, I agree. It is very hard to test a setting that's supposed to be transparent. Therefore I had to search for samples that I could ABX without getting too tired or losing my concentration too quickly.

Because of this, I tried to search for samples that had different 'problem sounds': snare drum, kick drum, synthesizer, vocoder (fatboy), etc. To not making things too unfair, I mean.

Lame 3.97 alpha 11 testing thread

Reply #5 –
Way to go bug80!
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #6 –
Here's the data in a form you can use on http://www.ff123.net/friedman/stats.html. The default test parameters result in: "397a11new is better than 3961, 397a11". Thank you, bug80!

Code: [Select]
3961 397a11 397a11new
3.0 2.5 3.0
3.5 3.0 4.0
3.5 3.5 4.0
3.5 4.0 4.0
4.0 3.5 4.5
4.0 3.5 4.5
3.5 4.0 4.0
4.0 4.0 4.0

Lame 3.97 alpha 11 testing thread

Reply #7 –
Quote
Here's the data in a form you can use on http://www.ff123.net/friedman/stats.html. The default test parameters result in: "397a11new is better than 3961, 397a11". Thank you, bug80!
[a href="index.php?act=findpost&pid=316825"][{POST_SNAPBACK}][/a]

And thank you for showing that the results are in fact statisticly relevant. 

I wonder, what settings lower than -V2 are usefull to test besides aps? V4 or V5 maybe?

Lame 3.97 alpha 11 testing thread

Reply #8 –
My opinion is that the phrase "representative of common material" is a little bit inappropriate, or at least a bit inadequate.

No sample, or small set of samples, can be representative of common material. It just makes me wonder what types of sounds are considered common and by whom? My collection is large & varied enough that to me, anything is common (or nothing is).

Most ciritical samples I've listened to have come from music that is popular, or at least not very obscure. Or from natural sounding instruments like castanets or harpsichord. Why shouldn't these sources be considered common?

I think we use these critical samples because there is an assumption that if an encoder does well at hard-to-encode sounds where they are not masked by other sounds, they will do just as well with mixed sounds.

Lame 3.97 alpha 11 testing thread

Reply #9 –
yes! -V 5 would be great!

Could you possibly try it (the latest LAME alpha) against the Helix encoder? I KNOW that is a bit off topic, but LAME -V 5 and Helix around -V65 (something in that area) should produce roughly the same file size.
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #10 –
Quote
I think we use these critical samples because there is an assumption that if an encoder does well at hard-to-encode sounds where they are not masked by other sounds, they will do just as well with mixed sounds.
[a href="index.php?act=findpost&pid=316871"][{POST_SNAPBACK}][/a]

What guruboolez means, is that a particular set of problem samples that focusses on one particular problem, for example high transient sounds (sharp attacks), may not reveil other problems with audio compression.

For example: I once tested Ogg Vorbis at different settings. Because I have experience with testing MP3, I focussed on high transient sounds, and found that I heard no major problems at q4. However, when I started focussing on other things I noticed that on one piece of a sample a complete instrument (cello) was gone!

Therefore it is important to have a set of samples that not only consists of so-called "problem samples". 

Lame 3.97 alpha 11 testing thread

Reply #11 –
Quote
yes! -V 5 would be great!

Could you possibly try it (the latest LAME alpha) against the Helix encoder? I KNOW that is a bit off topic, but LAME -V 5 and Helix around -V65 (something in that area) should produce roughly the same file size.
[a href="index.php?act=findpost&pid=316872"][{POST_SNAPBACK}][/a]

As soon as I have the time I will try V5. Why is it exactly that you want to campare with Helix? File size isn't really important, quality is. Of course the file size of the "new" V5 should not differ too much from the old one, but in the end it's of most interest to see if the new Lame is an improvement, sound quality wise.

My major goal is comparing with older Lame versions, because I think 3.97 deserves a final release as soon as possible. The more people test the current alpha, the better, am I right?

Lame 3.97 alpha 11 testing thread

Reply #12 –
WHY I want to test it against Helix? I know I'm a bit impatient but the Helix hasn't been compared to LAME in that bit range (120-140 kbps). At least not here at HA, AFAIK. That's what I'm using for my portable. My frickin' speakers are kaputt so I have to put my own ABXing on ice. Grrr...

But that's just a suggestion! You do whichever comparison you like, they will nontheless be valuable.
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #13 –
Quote
WHY I want to test it against Helix? I know I'm a bit impatient but the Helix hasn't been compared to LAME in that bit range (120-140 kbps). At least not here at HA, AFAIK. That's what I'm using for my portable. My frickin' speakers are kaputt so I have to put my own ABXing on ice. Grrr...

But that's just a suggestion! You do whichever comparison you like, they will nontheless be valuable.
[a href="index.php?act=findpost&pid=316887"][{POST_SNAPBACK}][/a]

Ok, I will add Helix to the test, if you can point me to the latest compiled version. Could you also give the exact command line I have to use? I have no experience with this encoder 

Lame 3.97 alpha 11 testing thread

Reply #14 –
See this thread:

http://www.hydrogenaudio.org/forums/index....=35531&hl=helix

Everything you need to know is to be found there. I listen mostly to metal and I've found that -V65 is pretty clost to LAME -V5 in size. My commandline with Wav2gogo (see thread) is "-V65 -X -U2" (with or w/out -U2).
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #15 –
Quote
See this thread:

http://www.hydrogenaudio.org/forums/index....=35531&hl=helix

Everything you need to know is to be found there. I listen mostly to metal and I've found that -V65 is pretty clost to LAME -V5 in size. My commandline with Wav2gogo (see thread) is "-V65 -X -U2" (with or w/out -U2).
[a href="index.php?act=findpost&pid=316992"][{POST_SNAPBACK}][/a]

Thanks. I need hmp3enc.exe (rev11) right?

By the way, I tried to test the -V4 setting today, but it's virtually impossible to ABX 3.96.1 versus 3.97a11 on that setting, the bitrates are very close too. Guruboolez, what do you do, when you encounter problems like this? Do you focus on one very short, specific part of the samples? Have you tried to ABX the latest -V4 setting yet?

It's also possible that my ears aren't trained enough, or that the quality of my headphones is too low.

Lame 3.97 alpha 11 testing thread

Reply #16 –
Quote
Thanks. I need hmp3enc.exe (rev11) right?
Yes. See the thread how to set up the encoder. Either with Foobar2000 or Wav2GoGo. (using the latter, you have to rename the encoder to "gogo")
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #17 –
Quote
Quote
Thanks. I need hmp3enc.exe (rev11) right?
Yes. See the thread how to set up the encoder. Either with Foobar2000 or Wav2GoGo. (using the latter, you have to rename the encoder to "gogo")
[a href="index.php?act=findpost&pid=316998"][{POST_SNAPBACK}][/a]

It works (that 16 bit setting is a tricky one  )

By the way, for fair comparison I think -V55 is a better setting for Helix. I converted 10 random files and these are the average results:

Lame 3.96.1: 147 kbs
Lame 3.97a11 vbr-new: 151 kbs
Helix (-V55): 148 kbs

Lame 3.97 alpha 11 testing thread

Reply #18 –
Yeah, I have noticed the bitrate vary a whole lot from track to track and from album to album using Helix. In my opinion more so than LAME. Using -V65 I get files from 120 kbps to 150 kbps from the same album! -V55 (Helix) may very well be comparable with LAME -V5 regarding size.
//From the barren lands of the Northsmen

Lame 3.97 alpha 11 testing thread

Reply #19 –
Very interesting. Thank you shadowking for the results.
WavPack 4.31 / LAME 3.98 alpha 3 -V9 -vbr-new

Lame 3.97 alpha 11 testing thread

Reply #20 –
Thanks for your effort.
Second bad vilbel. Always worth the listen

Lame 3.97 alpha 11 testing thread

Reply #21 –
Quote
Yeah, I have noticed the bitrate vary a whole lot from track to track and from album to album using Helix. In my opinion more so than LAME. Using -V65 I get files from 120 kbps to 150 kbps from the same album! -V55 (Helix) may very well be comparable with LAME -V5 regarding size.
[a href="index.php?act=findpost&pid=317026"][{POST_SNAPBACK}][/a]

I did ABC/HR tests with about 5 samples so far and was very impressed by the quality of the Helix encodings. The results look good for Helix, could be coincidence. I have to test more samples to be sure.

Lame 3.97 alpha 11 testing thread

Reply #22 –
Thanks shadowking! "noise dropouts" is something I notice a lot too with the 3.96.1 version. It seems that this has improved in 3.97a11.

Lame 3.97 alpha 11 testing thread

Reply #23 –
Quote
I did ABC/HR tests with about 5 samples so far and was very impressed by the quality of the Helix encodings. The results look good for Helix, could be coincidence. I have to test more samples to be sure.


Thank you bug80. You are the first one who test Helix mp3 encoder at that bitrate rage here.

Lame 3.97 alpha 11 testing thread

Reply #24 –
It would be nice if this test, this one and further test of LAME 3.97a11 would be merged in a single topic in the MP3 TECH forum, as other topic dedicated to different LAME alphas:

lame 3.97 alpha 5 testing thread
lame 3.97 alpha 6 testing thread
lame 3.97 alpha 7 testing thread
lame 3.97 alpha 8 testing thread
lame 3.97 alpha 10 testing thread
Lame test version - may 2005
Lame test version (June 2005)


EDIT: done, thanks  [if another moderator see this message, he can safely delete it]