Do we need to test 3.97 vs. 3.98 in a public listening test? That was never considered in this test. What do LAME developers think?

I am not a Lame dev and I think that its a bad idea for a public test. Too difficult for most people - harder than a normal test 130 k test which is already hard enough for many.

As I said in a previous post - given the fact that Gogo is not developed any longer (at least quality-wise) and that VBR was not mature in 3.88 according to Gabriel if I recall correctly and therefore not recommended, I fancy the idea to test FhG CBR just to have a comparison between high quality encoders (LAME, FhG with Q switch and VBR, iTunes using best quality settings) and fast encoders (Helix VBR and FhG CBR without Q switch).

IMHO, if you don't test the first second of encoded audio, that should be enough to let the encoder adapt itself correctly for Lame-based encoders. Within Lame (up to now), the ABR window is quite short.


I'd say no. People might do some side tests about 3.98 vs 3.97 in order to help Lame's development, but I don't see the purpose within a multi-encoders listening test.

I'd say yes. LAME is one of the most important mp3 encoders ever made, so let's see if 3.98 is a step up from 3.97. This is probably one of the best opportunities to do so. It is a mp3 listening test after all ...

I would like to add, that all modes LAME has to offer use the same information:
- psymodel: uses info from two previous granues (one MPEG-1 frame, two MPEG-2 frames).
- quantization loops: use info from one previous frame
So, ABR doesn't do anything special compared to CBR or VBR here.
LAME's ABR doesn't even observe the actual average bitrate.
LAME's ABR really is working like its CBR, but:
- ABR bit allocation uses:
* base amount from target bitrate, base_bits
* bits from 320 kbps frame plus bitreservoir minus base_bits, depending on PE
- CBR bit allocation uses:
* base amount from target bitrate, base_bits
* bits from actual frame plus bitreservoir minus base_bits, depending on PE

Maybe I did the same mistake that some others have done when they ask to include a specific codec or setting. I asked to include an additional encoder (LAME 3.97) without carefully considering if that would serve the purpose of this test. I just saw a great oppurtunity to actually compare 3.97 and 3.98 in a public test. I would like to make sure that it is safe to use 3.98 instead of 3.97. /mnt's report might indicate that with some samples the quality may be regressed (especially because /mnt used LAME 3.98 with a setting that gives a bitrate advantage. LAME 3.98 -V5 appears to produce higher bitrates than 3.97 -V5 --vbr-new).

In general, I have not seen many personal reports of 3.98 at "about 130-135 kbps VBR" during the development cycle. However, as I already said, it should be easy to me and others to personally test LAME 3.97 using the samples from the public test.


Do you have a reason to believe that FhG VBR -q 1 would provide better quality than FhG VBR -q 0 (aka default) ? My personal test didn't indicate anything like that, but naturally it is only a limited proof.

Personally I am not convinced that adding a CBR encoder would be a good idea, but I am not strictly against it. I have no opinion about FHG v. 4.1 CBR because I have not tested it in the CBR mode.

After all, I would be good with only four VBR contenders and the low anchor. It would make the test easier.

Thanks for the information Gabriel and Robert. I decided to start a poll for the fifth contender having LAME 3.97 vs. FhG CBR (as very fast encoder) vs. Gogo ABR.

And here is the poll: http://www.hydrogenaudio.org/forums/index....showtopic=64446

Yes, it is labled as high quality switch so I have the right to assume that. I am doing it just to be on the safe side since I don't have the ability to contact the developers and ask for whatever setting they recommend.

Unfortunately other users have not posted ABX results. Personally I can only trust my ABX results (TOS 8). Actually, is it impossible to reach the FhG developers? Surely they should be interested in this matter.


From the LAME 3.98 "news" thread:

LAME 3.98 -V5 results 141.7 kbps with my test track set.

Unless Robert and Gabriel suggest something else I think we should proceed in the usual way:
1. Take the contenders that do not allow intermediate quality settings and find the average bitrate of all of these contenders.
2. Test and find the suitable settings for the other contenders. The settings that provide the closest bitrate match with the average bitrate from the "step 1" should be selected.

EDIT

A possible CBR encoder should be excluded from the step 1. Including it would lower the "step 1" average and give slight unfair advantage to the "step 1" VBR encoders.

I think also this is more on topic here:


Hmm... what you mean by "real-life"?

The developers have obviously decided to let the average bitrate of LAME -V5 increase. I can't believe they would have not constantly tested the resulting bitrates. Someone could call this change cheating, but I think they have just redesigned the settings scale. Why should we not use the new scale and do what should be considered fair?

I meant that most people use integer values in practice ....

But right you are, - every listening test has its own fair policy. If this one already has some agreed procedure of choosing encoder settings, it should be kept, regardless of any "integer" values for sure.

I checked the three previous public listening tests:
Ogg Vorbis AoTuV 4.51 Beta -q 4.25 and Nero HE-AAC Jul 20 2007 -q 0.24
have been chosen in similar situations and the used procedure for finding the correct setting was like I suggested now.

If LAME is now supposed to be similarly "stepless" an intermediate setting can be used when it is closer to the test target.

EDIT

The test's "target bitrate" should probably be announced as 130-135 kbps or so.

Another option would be to pick LAME 3.98 -V5 as a reference and try to match the other encoder's settings with it. The test target would then probably be ~140 kbps.

I am very thankful for the time you invested in testing and although your conclusion is that there is no difference between the two settings, I am going to use the quality switch because of the aforementioned reasons. According to the Fraunhofer IIS page, FhG recommends using the maximum quality setting available in the encoder interface:

IMO the bitrate difference should not be bigger than +-2.5 kbps.
I found that the only scenarios that will work are those two:

135-140 kbps, if there should be iTunes VBR and Lame -V5:

1) Lame 3.98: -V5
137 kbps

2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 3 -q 1
138 kbps

3) iTunes 7.6.2.9: http://img77.imageshack.us/img77/9634/itunesmp3nf1.png
135 kbps

4) Helix v5.1 2005.08.09: -V72 -X2 -SBT450 -TX0 -HF2
137 kbps

(lame 3.97)
127 (-V5 --vbr-new); 150 (-V4 --vbr-new)
those settings would disqualify lame 3.97



Or the ~128 kbps range:

1) Lame 3.98: -V5.8
128 kbps

2) FhG 1.4 (Encoder-Library V04.01.00): -br 0 -m 4 -q 1
128 kbps

3) iTunes 7.6.2.9: settings from above but CBR
128 kbps
hitting the 128 range with maximum VBR is not possible due to the lower limit at 128 kbps . Maximum VBR with 112 kbps doesn't work either (bitrate too low).

4) Helix v5.1 2005.08.09: -V62 -X2 -SBT450 -TX0 -HF2
128 kbps

5) Lame 3.97: -V5 --vbr-new
127 kbps


I'm not sure about the Helix settings at that low bitrate though.

As most people so far want to see Lame 3.97 as the 5th candidate yielding 128 kbps seems to be the way to go.

As always, I am going to build a bitrate table containing the average bitrate an encoder gave to all samples. If the difference between the encoder with the lowest average bitrate and the encoder with the highest average bitrate is greater than 10% of the target bitrate, I will think about either exchanging some of the samples or choosing different encoding parameters. The initial parameters will be chosen based on recommendations from developers and based on a large set of music tracks that should average to our target bitrate.

By the way, this means that in Raiden's example, the first scenario with LAME 3.97 -V5 --vbr-new would still be a valid solution. -V4 is also within the 10% range, but in this case, I would choose -V5 in order to compare the same quality setting between 3.97 and 3.98 and also because 127 kbps is closer to the average bitrate of the four oher encoders than 150 is.

Raiden,

What test tracks did you use and what tool did you use for checking the bitrates?

I would like to see lame 3.98 as the new recommended encoder. And to do so, tests has to be done. *vote for lame 3.97 AND 3.98*

ooops... sorry! I should have put that info in the post above.
I haven't got many different genres on my PC (mostly electronic music) so this was just a quick test with a small 'representative' selection (just six albums, 375 minutes): Some electronic music/IDM, ambient, Jazz/NDW, Rock, and a radio play with lots of effects. I guess for modern, compressed music one has to add ~5 kbps to the values.
I've read the bitrate with foobar2000 (after fixing the VBR headers on the FhG files), averaged over time...

So no big conclusions, but there are a few:
- lame 3.98 MP3s are ~10 kbps bigger than their 3.97 counterparts at all quality levels, so direct comparison (i.e. 3.97 -V5 vs. 3.98 -V5) is not so good.
- FhG's VBR scale is not fine at all, whereas with Lame and Helix there is no problem. So FhG would likely be the 'bitrate reference'.
- iTunes VBR is very difficult. 112-VBR results in a bitrate of ~120 kbps, and 128-VBR in ~135 kbps, but I have not found an inbetween VBR setting for 128 kbps.

I'm very curious about bitrates from bigger testcases!

As I said already, if the target bitrate is 128 kbps (which it is), a difference of 12.8 (or let's round to 13) kbps is allowed between contenders (again, on average, not per sample - per sample the difference can be higher).

LAME developers, in case the bitrate difference between LAME 3.97 and LAME 3.98 is too big, is there anything I can do to increase the bitrate of LAME 3.97 without having a negative impact on audio quality?

Alex B, any chance you could post a bitrate table for your samples with LAME 3.97 -V5 --vbr-new to have a direct comparison?

Or else, still include tried 'n' true 3.97 at -V5 --vbr-new, and play around with the new stepless quality settings on 3.98, to encode at - say - V5.5?

Problem is that all other encoders also produce bitrates near the 140 kbps range. The only encoder that (according to Raiden - I didn't had time to test myself that is why I asked Alex B if he can also test) encodes even "lower" than the target bitrate is LAME 3.97.

Personally, I wouldn't mind a bitrate difference of 10 kbps (which is even less than 10%) since I assume the difference to be very low between the two LAME versions and if it's not, it is most likely caused by other factors not only the bitrate.

I have already tested it. LAME 3.97 -V5 --vbr-new produces about 134 kbps with my sample set. This has not changed since I tested 3.97 beta 2 for the 128 kbps multiformat test. As you can see from my previous table also 3.98 beta 5 was quite similar. After that the scale has changed and now 3.98 is different. The obvious solution would be to adjust LAME 3.98 to exactly match 3.97 or the test average, but as you said, we have some tolerance and naturally we should not use a setting that would decrease the quality if the next minor setting "step" should produce clearly better quality.

I would like to ask Robert and Gabriel if there are some fundamental changes when, for instance, the setting is changed from -V5 to -V5.5 or -V5.7. In other words, should the integer steps still be preferred for some reason or is the -V scale now truly stepless? It would be good to have info about that before the encoding setting for LAME 3.98 is decided.

I am going run a few additional tests and post a new table soon. My test set has usually been quite good in representing the "various" genre aka an average of pop/rock/jazz/electronic genres even when it has been compared with big encoded libraries, but in addition, I am gathering a modest set of classical tracks and I'll publish those results in separate table.

Thanks a lot for your efforts and help! Problem with my music is that most of it is progressive rock. I will try to do some tests in the near future (today or tomorrow evening).

If LAME 3.98 appears to be the only encoder that stands out of the rest, that would hopefully be no problem since you can fine tune the -V parameter.

Only yes/no settings in the presets may change on integer steps. All other parameters--except sfb21mod--are linear interpolated between step n and n+1. There is no reason to prefer integer steps.

Small status update:

- Real did not conact me
- Gogo will be replaced by LAME 3.97 -V5 --vbr-new
- LAME 3.97 -V5 --vbr-new produces an average bitrate of 136 kbps according to foobar2000 based on my collection of 1083 files

Since all encoders are close to the 135 kbps rate, we should choose some settings for LAME 3.98 that should average to 135 kbps.

For my (much smaller ) test set: LAME 3.97 -V5 --vbr-new => 136.8 kbps. And LAME 3.98 produces the same bitrate somewhere at -V5.33.

And dependence between -V value and bitrate looks very linear within [5.2, 5.8] range.

Please suggest and / or upload samples here.

FhG released MP3S CLE V1.5, but the download link is dead, they will fix download link soon:
http://www.all4mp3.com/tools/sw_fhg_cl.html
Whats New in Version 1.5:
New features - Encoder

Minor changes and bugfixes.
Now supporting files with 24 bit resolution

Finally, download link is alive, download it here:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip

what you think about lame 3.98 -V floatnumber multi-pass encode to get most close V to 128kbps size ("true abr")?

or this mode is not good for users and should be not tested (need more time for encode, etc)

Afaik there is no multipass switch in LAME.

No the link is broken, here it is: Fraunhofer IIS mp3surround command line utilities.

Also, multipass encodes would have to be done on the whole songs, using it with short samples would produce skewed results. (I think this was discussed before wrt Nero or WMA.)

ok, case closed

Download link works fine:
http://www.all4mp3.com/dev/download.aspx?n...15_20080530.zip
Check your connection.

I've checked once again link. My connection is OK.

These are the links that work for me:

Win32 mp3 command line encoder (v1.5)
Win64 mp3 command line encoder (v1.5)
Mac OS X mp3 command line encoder (v1.5)
Linux32 mp3 command line encoder (v1.5)
Linux64 mp3 command line encoder (v1.5)

Sorry, but I can't help...

Yes, you're right.
I don't know what they doing.
Did you download that?
Also, what's new in this version?

I finally had time to continue my bitrate tests with classsical music as I promised earlier in this thread (a thing called summer got in the way...)

After browsing through my lossless classical library I picked 25 "reference" tracks that should be quite representative. I avoided the extremely low and high lossless bitrates and tried to select tracks that have quite varied qualities.

Apparently iTunes has changed radically since my last test. Back then the 128 kbps VBR setting was suitable, but the 7.7 version uses bitrates in a more relaxed way and the 128 kbps VBR setting produces higher bitrates than before. Fortunately the 112 kbps VBR setting appears to be suitable for our test.

In addition, I retested the "various" bitrates with the latest encoder versions when applicable.

Since I didn't have the old iTunes version installed I couldn't test the "classical" bitrates with it (which would have been unnecessary anyway).

FhG, iTunes and LAME 3.97 have only one suitable VBR setting for this test so I'd suggest to calculate the average bitrate of these encoders and adjust the Helix and LAME 3.98 settings to match this average. Helix -V60 and LAME -V5.7 appear to be pretty close to this average with my test tracks.

Here are the new results:

Summary




Various - table and chart






Classical - table and chart






EDIT

I forgot to mention that if anyone wants to test FhG's bitrate behavior the bitrates must be measured correctly. Most programs don't show accurate bitrate values because FhG doesn't write Xing headers to VBR files.

I used EncSpot Pro's "full scan" option.

Thank you very much for your efforts Alex!

Mmm... there's a curious thing on those graphs, the performance of Itunes respect of the rest in a couple of files:
17 in the Various group, and in 5 and 19 in the Classical one. I guess this could point to flaws (killer samples) for it. Could you try abx'ing one of them?

(the 11 in the Various test is curious too, it shows an increase of bitrate where all the rest do the opposite)

Yeah, displaying the measured values in a graph always shows interesting differences. We must remember that these values are from complete tracks and do not indicate how the encoders adjust bitrates on different parts of each track. Also, the bitrate is only one of the variables that a VBR encoder adjusts.

Since you are interested about the differences I'll check the tracks you mentioned and if they seem to be interesting for quality test purposes I'll post samples.


EDIT

I wonder what is wrong with the quotation system. The quote doesn't appear to show up correctly. I tried to fix it a couple of times.


Moderation: Fixed your bracket problem.

It doesn't like brackets in names because IB.Code tags are also enclosed in brackets. Bug is fixed in latest IPB 2.3.5.

type

"&-#-9-1-;" for a "["

and

"&-#-9-3-;" for a "]"

(but don't include the dashes)

What do you mean by that?

Great Job Alex. I think it's "sad" getting 188 Kbps files while using iTunes 128 VBR !!!

It is a shame. I expressed concerns about not giving iTunes a fair shake last year, but if their 128 setting produces such high bitrates...

IIRC, with iTunes in VBR mode, the selected bitrate is the minimum bitrate.

Well, any news?
It was already one month and half since call for samples. What is situation around the test?