I would like to ask you to submit suggestions what MP3 encoder to include in my upcoming test, as well as suitable settings to produce ~128 kbps MP3 files. I would like to use only one setting per contender, so no suggestions like "LAME with -V 5 and -b 128 to test VBR vs. CBR". If that is really needed, a possible solution could be having -b 128 as contender and -V 5 or a higher setting as high anchor. Low anchor could be Shine, Blade or some very (very) old FhG encoder.

The goal of the test is to find out if fast encoders are really worse than LAME and if yes, how much worse. Some ideas for contenders: Gogo, LAME, iTunes, Helix, some FhG from Nero or Adobe or MMJB (in case I can still download it - heard that Y! stopped developing it) or WMP...

Also, since MP3 supports real CBR, do you think I should favor CBR over VBR for possible portability reasons (iPods not dealing well with LAME VBR)?

Bear in mind that LAME devs are the only persons to ask for recommended settings. I tried getting in touch with Helix folks some months ago but I had no luck (they didn't know what to recommend). Obtaining recommendations from FhG isn't possible neither.

Edit: One more thing - the max no. of contenders should be 4 + 2 anchors.

Helix has a vbr quality mode for 128k, but FHG lowest setting is 150..160k (m3) which rules it out. Alternative for FHG is ABR mode for 135..140k (m4), but would that bitrate be unfair in a 128k test ?. So we can do FHG / Helix CBR 128k :

mp3sencoder -br128000
hmp3 -B64
gogo -b128

ABR:

mp3sencoder -br0 -m4
hmp3 - ?????
gogo -b135 -a

Note that Nero is using multiple MP3 encoders depending on the application. The one from FhG we use in one place is AFAIK the same as what they have as demo on their website (unless that one is somehow limited, but doesn't look like it).

I would like to see AAC 96 kbps CBR LC (used in previous test as high anchor) as a low anchor in this test.
This would be helpful to have some connection of results for 48 kbps and 64 kbps with this test.
But I suppose most people will be against it.

Yes this would be great, also because I hear the phrase "AAC 96k is equal to MP3 128k" thrown around a bit and it would be interesting to see if it's in any way true.

People have the wrong idea regarding this. it depends where the quality bump is. For mp3 there is a big one from 96 > 128k , smaller from 128 > 160k and yet smaller from 160 > 192k. The other codecs are similar , converging more or less @ 128k but drop quality more gracefully @ bitrate < 100 k.

So saying ogg/aac 96k = mp3 128k is a possibility (IMO 96k will be worse than LAME -V5). We can't use these assumptions for high bitrates: 170=200, 256=320 etc etc, because once you pass the big quality bump you get less and less as you bump up the bitrate.

Anyway, I would like to see the mp3s encoder from FhG as we are using this since recently and it would be good to see how it performs compared to LAME. We only use CBR at the moment.

Me too.

96 kbps LC-AAC might be too good as low anchor. Consider that the low anchor has to be significantly worse than the rest.

IMO we should use with any encoder that setting which is a priori supposed to yield the best result. This can be quite disputable but we should try it.

Helix has a good VBR mode, so with Helix I suggest to use -V55 or -V60 which according to a listening test I did some time ago matches 128 kbps on average quite well. According to that (restricted) test we should use the default options (with the exception of -X2).

The problem with Lame is: 3.97 or 3.98b5+? With this only exception I suggest we use both versions because of their practical importance and in order to see what things are like with these versions.

With FhG IMO we shouldn't use VBR. According to my experience FhG's VBR mode is not attractive qualitywise.

Sorry, but I maybe missed previous discussions where it was discussed why low anchor has to be significantly worse. Can you please give a rationale about this?
IMO it is enough that it is worse than almost all contenders in almost all samples. There was also a case in 64 kbps where I would prefer low anchor over one of contenders.

I agree to that one. Considering the results of the 96 kbit/s AAC anchors of the 48 kbit/s and 64 kbit/s multiformat tests, it definitely won't be an appropriate low anchor in this case. I vote for Shine @128 kbit/s as low anchor.

Reason: Shine's an extremely simple encoder, which could be called a reference implementation for people starting to develop own encoders. It would be a good indicator to see how much better optimized encoders perform compared to a very basic one like Shine. Lately I had a few arguments with technically unexperienced people who refused believing me that the quality of MP3s can highly vary, depending on the encoders used for creating them. For these guys MP3 encoding is a fix matter that can't be altered/optimized in any way. I guess that's a major reason for the many 128 kbit/s full stereo Blade MP3s still found throughout the internet.


Judging by the multiformat test results I mentioned there's no clear proof that 96 kbit/s AAC is actually worse than the contenders for this test. In this one it resulted at 4.59, i.e. on many samples it was transparent for a lot of listeners. How should you distinguish a transparent low anchor from the contenders and the high anchor?

If the low anchor is on par with the contenders, or only slightly worse, then there is no need for an anchor at all. An anchor should be clearly distinguishable from the rest.

Now, doesn't this look really sweet: http://www.rjamorim.com/rrw/l3enc.html

what about the mp3 encoder from iTunes ? dunno if it's FhG-based...

I should like to see the iTunes encoder included, not for any technical reason but because it is so prevalent. It would be convenient for me to use it, instead of Max + Lame, and I'd like to know how the quality stacks up -- and I guess I'm not the only one.

I say 3.98 and FHG @ 128 vbr mode is really ABR. The FHG cbr is also like a freeformat.

Here's my two cents' worth,

FhG, Nero or WMP11 ?
- muaddib, please make some in-house queries about the exact details and version of FhG. Has anyone inside Nero actually compared the surround version against the presumably older "standard" version which included in WMP11?

Helix, VBR -- the exact -V value must be tried with a large amount of complete tracks (I can contribute).
- what is the last released source code version? AFAIK, the compile at Rarewares has not changed since it was put online.

LAME, VBR (whatever version and setting the developers recommend)
- 3.97 vs. 3.98 should be tested in private tests. The differences are probably not significant unless 3.98 is seriously regressed because of the developers' mistakes. I can promise to do a private test after this public test using the same samples. It would be relatively easy after doing the public test and finding out which kind of problems and where exactly the selected samples produce (which is going to be tough).

GoGo, ABR ~135 kbps (would produce a similar bitrate with LAME -V5)
- according to my limited personal testing this setting is slightly better than CBR 128. Perhaps I, halb27, shadowking, muaddib and others interested could select a few samples and compare these encoding modes in a quick "semi-public" pretest...

iTunes, CBR or VBR 128 kbps
- must be quite popular, is quite fast and has supposedly lower quality than LAME. Is anyone else interested?
(to kurtnoise: it is not FhG)

Low anchor, Shine 128 kbps
- this worked fine in the previous 128 kbps multi-format test. Even a very old FHG at 128 kbps would probably be too good. iTunes AAC at 96 kbps would most certainly be too good.
(To muaddib: The intention of a poor quality low anchor is to introduce the testers with typical encoding artifacts so that they can more easily try to pinpoint similar, but less pronounced artifacts that the contenders possibly produce.)

High Anchor, a surprise suggestion: FhG, CBR 192 kbps
- if it is good enough (I have been privately testing LAME -V2 variants too much lately and I would like to know how good FhG is at higher bitrates. Would it be transparent for the majority of testers?)

Edit: several typos...

Sounds like a good choice to me, even with FhG already being included as a contender. The two FhG results could be very interesting for the many users out there, who rip their CDs with widely available commercial software, e.g. Windows Media Player. I don't have any figures available, but to me it doesn't look like the majority of people goes for alternative software, that uses a bundled version of LAME for MP3 encoding.

Alternatively, if Nero's/WMP's FhG encoder was used as the High Anchor, we could also go for the MP3 Surround encoder as a contender instead.

Because of the meaning of Lame I still think it's worth while to test 3.97 as well as most current 3.98.

But if I'm the only one who thinks like that I suggest to use 3.98 cause it's the latest version and close to final.

As for iTunes mp3 I also like to see it tested.

I'm pretty sure WMP does not use the new 4.0 series of FhG encoders. Nero does, but I'm not sure since what version (or if we even included that already). FhG claims their 4.x version is completely new compared to 3.x.
As said, I do not think the command line evaluation from the FhG website uses anything different than the SDK Nero uses, but I might be mistaken. If you want me to check it, let me know.
It support CBR and since recently also VBR (not enabled in Nero), there's also a quality/fast switch.

We are also still using the CodingTechnologies MP3Pro encoder in some places. I haven't heard anyone about that one yet (as it also does normal MP3).

The WMP11 encoder is currently this:



The great thing about this codec is that it can used through Nyaochi's "acmenc" command line tool (after a small registry edit). Acmenc can include a LAME info tag style header which contains info for gapless playback. It is easy to configure foobar to encode through acmenc. I have used it as a faster option to LAME when I am in a hurry.

I wonder if Nyaochi could create a modified version which could be used with FhG's exe encoder (for creating a compatible gapless playback info header instead of FhG's quite useless newly introduced header).

*Low anchor:
AAC-LC @96 is not a suitable low anchor for such a test, IMHO. (anchor should not be used as a contender)
Regarding Shine, well, now that everyone seen how lame is Shine I think that it's perhaps a too low anchor. I would rather suggest either Blade or L3enc 2.72

*CBR vs VBR: hardware compatibility is a non-issue now. Every player should be able to properly play mp3 streams (even Apple is reported to having fixed Ipod's buffering issues with vbr)

*Lame version: I'd suggest 3.98 rather than 3.97. 3.98 is representative of current Lame, while 3.97 is from 1 year ago, and is unlikely to be updated/maintained by anyone.

dBPoweramp uses the Fhg IIS 4.0.3 encoder as well. Is this the surround sound encoder? Does Nero use 4.0.3 as well? What versions of Nero offer this encoder? WMP, as someone showed, uses Fhg 3.x.

I'm not sure of any Mac applications that can use Fhg 4.0.3. I asked Fraunhofer and they replied Cubase. Well this "only" costs about $1000!!!

dBpowerAMP could be an option, indeed. However, it seems to support only CBR encoding.

I thought the consensus was that Fhg only does CBR very well and that VBR still left much to be desired? I may be wrong though.

A bit off-topic, but still valid to the concerns of those doing the listening test -

which encoders can do gapless on iTunes and the iPod. Apple's own MP3, iTunes AAC, LAME MP3 can. What about Fhg MP3, Nero AAC?

What do you mean by IIS 4.0.3 ? Is it available somewhere?

The current command line encoder reports this version info:



EDIT

I found that Illustrate advertises the encoder as 4.0.3:
http://www.dbpoweramp.com/codec-central-mp3-fraunhofer.htm

It looks like Fraunhofer does not provide their latest codec version here:
http://www.all4mp3.com/tools/sw_fhg_cl.html

In general, I dont think we have any reliable information about the FhG v.3.4.0 vs. v.4.0.x quality differences. Has Fraunhofer actually done any development work with the standard stereo mode since the IIS 3.4 version?

I guess we can't expect Fraunhofer to answer.


Perhaps Spoon could say something. He must have some insider information.

I don't know the exact version numbering, but we have something that starts with 4 and ends with 10 and VBR was only supported since the version that starts with 4 and ends with 4... IIRC

I just noticed that the command line encoder is 4.01.00 not 4.0.1. So perhaps FhG means 4.1.0.

I'd suggest that we disqualify Fraunhofer because of misleading version numbering.

I pretty much agree here. I'd like to see FhG (no particular version preference), LAME (3.98 preferable over 3.97) and iTunes. Shine seems like it would do well as the low anchor (though perhaps at a lower bitrate than 128kbps -- I'm not familiar enough with Shine to have an idea how it sounds at 128kbps), and FhG at 192kbps sounds like a fitting high anchor. I think it might be more fitting to use both VBR and CBR formats in this test, when appropriate. Naturally, opinion may vary on this to a high degree, but I agree with others that the hardware issues are being eliminated at a fairly brisk pace, and each encoder should use whatever method it's stronger with and/or whatever method is more widely used.

This is what the creator of the Shine encoder says:


According to Gabriel Shine is intended to be the most basic MP3 encoder that can produce standard conforming MP3 files. It is just an educational tool.

I must check how it was ranked in the 128 kbps multi-format test. I don't recall it being too bad, but perhaps I remember wrong.

In any case, I am afraid that even the oldest release version of FhG is not suitable at 128 kbps. It wasn't that bad. I think it already included advanced things like bitreservoir, joint/intensity stereo and short/long block switching. It kind of created the myth of CD quality MP3 at 128 kbps.

Perhaps Blade would be a better low anchor, but even Blade may be surprisingly good with tonal samples that don't have sharp attacks.

In that case, I'll try to do some evaluations with both Shine and Blade tonight versus LAME before forming a real opinion.

Given my understanding of the purpose of the low anchor, so long as it's is easily identifiable, it doesn't truly matter how poor it sounds. According to Sebastian, way back in 2006, in fact, "..a low anchor is supposed to sound really bad, it should be the worst", so Shine would seem suitable (i.e., "really bad") for that if quality at 128kbps is as poor as you say.

In any case, I'll give 'em both a listen and see how I feel after doing so.

Shine was the low anchor in this test:
http://www.listening-tests.info/mf-128-1/results.htm

Instead of wasting your time testing Shine vs. Blade you could try GoGo 3.13 with the following settings and post your findings here.
-b 135 -a -q 2
-b 135 -a
-b 128 -q 2
-b 128

-b 135 -a = an ABR setting for an average bitrate of 133 kbps = about the average bitrate of LAME -V5
-b 128 = CBR 128
-q 2 = a slightly slower and possibly higher quality setting


BTW,

We had a 14-page discussion last year before this 128 kbps MP3 test was postponed:
http://www.hydrogenaudio.org/forums/index....showtopic=47313

During the discussion Sebastian decided to not use a high anchor. Perhaps without a high anchor we could have five contenders (LAME, GoGo, FhG, Helix and iTunes).

What do the others think about the idea of dropping the high anchor?

Well, being an anchor, it wouldn't be that much of a nuisance, would it? FhG @192kbps (maybe even higher, like 224/256) seems like a good choice, since it can serve its goal well and probably showcase the drastic leap in quality between it and LAME.

(Then again, I visit HA so rarely nowadays so my opinion doesn't really matter.)

I think an ancient yet reasonably popular historic encoder like Radium or SoloH could serve well as a potential low anchor even at the same bitrates. I am interested in seeing how far along MP3 has come in the past 7-8 years. I see Blade's been suggested, which isn't bad except it was never realistically considered a high quality solution at any time. It was just free and quickly available after ISO's code release. Radium was *the* MP3 codec for some time, for longer than it had any right to be. I suggest that if there's hesitation about its possible competitiveness with respect to being a low anchor, then we aren't convinced whether MP3 has truly made gains since the late 90s. In any case, I think the choice should be something that mattered at some time, not an artificially lousy choice like Shine. I'll take Xing circa 1998 over that.

I agree with eliminating the high anchor. It's pretty clear from recent tests that its value approaches 5 at these bitrates.

Also, I'm for the exclusion of LAME 3.97 if it bumps an otherwise worthy contender. Far more relevant than possible LAME regression is its performance compared to other encoders in general. I doubt minor issues between point releases would make or break comparisons between fundamentally different software. LAME 3.97 vs. 3.98 is important, but I think this test would be too crowded for meaningful comparisons.

I would agree with foregoing the high anchor. You could have two low anchors, one worse than the other.

Yes, it's 4.1.0 I just checked. This is the version that adds VBR (with VBRI headers) and original file length support.

Gabriel suggested 3.98 so I am going to include that instead of 3.97. Gabriel, do you recommend -V 5 --vb-new or some other setting for ~128 kbps?
iTunes and Helix are also featured for sure. Should I use both in VBR mode?
Fraunhofer is also an encoder I would like to include - same question now: CBR or VBR?
Is Gogo still used or did people switch to Helix / FhG / iTunes for quick encoding?

ff123 and other pros, what do you think about dropping the high anchor and using the five contenders LAME, FhG, iTunes, Helix and Gogo at 128 kbps?

As for the low anchor, I don't see any point in using Blade. I could imagine using Shine because it's very simple, Radium because it was very popular years ago or l3enc because of its age.

I just reread the complete last year's thread. The codecs and settings were decided (almost) and we were talking about the sample choices on the last two pages.

We discussed about the need of a high anchor on the page three. The conclusion was to not include one.
The selected low anchor was FhG l3enc v.1, but I was concerned about its possibly too high quality.

Here are a few selected quotes from that thread:












I don't think much has changed since then. Perhaps we could reconsider the FhG version, but for me it would still make sense to include the WMP11 version, which is the version that most Windows users already have. If FhG 4.1.0 really is better in stereo mode it could be considered. It may also be regressed. I really would like to see some personal test results with ABX logs. It would also be useful to compare the v. 3.4 and 4.1.0 encoding speeds before making the choice. After all we were supposed to compare the faster options against LAME.

Also, wondering if Gabriel changed his mind regarding VBR vs. ABR in 3.98. Using RealPlayer for creating the Helix MP3s is reasonable - saves us from testing the various options given by the CLI.

Does anyone know how quick Microsoft usually is with updating the MP3 libraries?

There seems no mode to produce something around 128kbps using VBR, only much lower (mode 5) and much higher (mode 4).

Why would Gabriel had changed his mind between the versions 3.97 and 3.98 ? Isn't -V5 (--vbr-new) de facto standard that is known to be able to compete with the newer encoders. (If you mean the quoted reply, it was for 3.88.)

I just updated my WMP11, iTunes and Real Player to the latest versions.

WMP11 had no recent MP3 codec updates. It uses the version 3.4 as I said earlier.

The current iTunes version is 7.4.0.28 and the installer is 13 MB bigger than the 7.2 installer that I had previously. I have no idea if its MP3 encoding has changed anyhow.

Real Player is still the version 10.5, but the so called product version number has advanced from 6.0.12.1483 to 6.0.12.1662 since last year.

I am going to run my 25-track bitrate test set with these versions and also check the encoding speeds.

I really like to see GoGo included because it has not been tested against Helix & FhG. For GoGo I would suggest to use -a -b 135 and a q-option that would produce at least about three times faster speed when compared with LAME. The default without a q-switch could be possible. On my PC the default setting produces about 60x speed and -q 2 about 40x, but I have not tested LAME yet. (Joint stereo seems to be enabled automatically at this ABR bitrate so a stereo mode switch is not needed)

EDIT

I'll include the FhG IIS v. 4.1.0 command line encoder in my bitrate & speed test. Perhaps I am able to do some quick quality comparisons too (using the encoded files). I'll post the results during the coming weekend.

Previous listening tests at 128 kbps clearely teached us that transparency is almost reached for several people. Including an high anchor is very important but in this situation it may be pointless. I would go for the one of the following configurations:
• 3 competitors + 2 anchors (low & high)
• 4 competitors + 1 anchor (low)
=> 5 encodings to focuse on (and for most people, 4 only - assuming that low anchor is immediatelly detected)


VBR vs ABR/CBR:

I wouldn't drop VBR. From my bitrate table (based on classical music only - thus with limited validity) we should get three competitors with comparable bitrate : LAME -V5 (~130 kbps); Fraunhofer 4.xx CLI (~130 kbps) and of course Helix which is by far the most flexible tool. I don't know anything about iTunes & gogo.

The remaining question is: is VBR the best choice for HELIX and Fraunhofer (and other encoders). There's only one way to get a valid answer: pre-listening tests. I seriously doubt that people would spend their free time to get a solid experience with these rather unused (here on HA) encoders. I tried once and quickly gave up: I don't see the point of spending hours and hours to analyse encoders that are immediately unusable (for my taste).

Using CBR/ABR for each competitors is another possibility but such configuration won't learn anything to the community (assuming that most HA members are following the recommendations/experience of other members and are using LAME -V5 to get small and HQ MP3 encodings). So I would definitely discard this option.


If I had to make this test alone I would go for:
- VBR only encoders
- it implies to discard all encoders that doesn't allow ~130 kbps VBR encodings and keep the other ones
- unless switches are specifically labelled as 'improve quality' I wouldn't tweak the commandline (there are some dubious ones for HELIX) unless some people could provide valid proof of their positive effects on most situations.

I don't see any reason why every encoder should be used in VBR mode or to exclude competitors with poor VBR quality.
IMO we should use any encoder with settings that are expected to be best from experience though certainly there is no scientific evaluation.
As for additional switches I agree that only when there's strong experience against it we should use other than the default settings.

From Gabriel we know already that gogo should be used with ABR.

From my experience with FhG I can say that VBR mode isn't very robust. So CBR should be used IMO.

With Lame Gabriel and robert can say what setting to use but certainly this will be -V5.

Helix should be used in VBR mode IMO.

I have no idea for the other encoders.

As I suggested, there would be no problem If the test idea was MS, Apple & Real vs. LAME & GoGo.

VBR can and should be used in the iTunes and Real Player options as the developers provide the option and the average bitrate seems to be suitable.

WMP11 has no VBR option so MS has made the choice. If the newer FhG CL encoder (or the same encoder through Nero's UI) is selected instead of WMP11 the CBR option may be the only possibility. It has been said that there is no suitable VBR quality setting. In this case Fraunhofer has made the choice.

GoGo is an exception because it is not maintained anymore and we don't have any developers' recommendations. We should use a setting that a sensible and experienced user would select when LAME -V5 (--vbr-new) is too slow for getting the files ready before they are needed. If GoGo is left out of the test we will never know if e.g. Helix would have been better or worse than GoGo. If GoGo is found to be worse than the others it would work as a good "second low anchor". It may also positively surprise with some of the samples.

I don't see any reason why every encoder should be used in VBR mode or to exclude competitors with poor VBR quality.
To avoid criticism and suspicion... it already happened in the past: if any encoder, set with ABR/CBR, doesn't come on first position, some people (call them zealot or not) are claiming that the whole test is biased. Here on HA and also on other website. It partially ruin the credibility of the whole test (and not necessary for a bad reason).

IMO we should use any encoder with settings that are expected to be best from experience though certainly there is no scientific evaluation.
The lack of "scientific" evaluation lead to the whole internet carnival of opinions. HA.org was born to fight against this whole mess. Listening tests share the same goal. I can't imagine any listening test based on fragile preconceptions. That's precisely why I'm so worried about this idea of starting a MP3 listening test. Unlike multiformat listening tests, we don't know anything solid about most competitors.

As for additional switches I agree that only when there's strong experience against it we should use other than the default settings.
I also said that we shouldn't use the default setting if some specifically "HQ" switches are existing but aren't used by default. It's the case for Fraunhofer 4.xx IIRC.

From Gabriel we know already that gogo should be used with ABR.
As far as I can read he said that LAME 3.88 was likely to be better with ABR. I can't understand the source code of gogo and I can't say if gogo is nothing more than LAME 3.88 with speed optimisation. In this case we would have a valid opinion about gogo. But if GOGO is nothing more than a faster LAME 3.88 what's the point of testing such antiquity?


From my experience with FhG I can say that VBR mode isn't very robust. So CBR should be used IMO.
Perfect. But are there any chance to see detailed results?

IMHO, this whole test has no point unless the tested LAME rivals are sigificantly faster than LAME. That's why I may change my opinion about using a high quality -q switch with GoGo if it reduces the encoding speed too much.

My very modest proof is this:

I reuploaded the "AC_DC___Highway_to_Hell.wv" sample:
http://www.hydrogenaudio.org/forums/index....st&p=514963

Correction, mode 4 seems to produce 132 kbps on a big set of files, so it seems usable.

So should we do anything to avoid people to react strangely? IMO this is not a good target and we won't succeed anyway.

We should do the best according to what we know - well conscious about the imperfections.
You're absolutely right that we don't know a lot about the competitors, and everything is disputable, but that's no reason to me to worry about such a test.
To me any listening test has it's restricted meaning - but still is valuable and adds to experience.