RealPLayer and the command line Helix encoder from Rarewares seem to use quite different VBR settings by default. Helix @ -V60 was 3.6 x faster than RealPlayer in my test.

I have run my bitrate and speed tests and just started to prepare my report. It may take an hour or so. I encoded my usual "25 complete tracks" test set 12 times using the discussed encoders and some possible setting options.

Another "speed" consideration is that encoding potential far outstrips ripping potential, and the disparity will only grow. Looking at Tom's Hardware CPU chart, it looks like just about any new machine purchased in the past three years or so is capable of encoding an entire CD with LAME in five minutes or less, though I don't remember which settings they use.

*Lame settings: for 3.98, I'd suggest "-V5" (vbr-new is the default vbr mode in 3.98), but I'm not sure if it would not be higher than 128kbps.

*High anchor: I agree that it could be dropped, as we are expecting some already high results

*Gogo: is anyone still using it?

*"Radium" crack: I think that the results would be very similar to Audioactive Production Studio

*Lame fast setting: well, I don't think that there is a need for any test to know that current Lame would produce quite lower quality than FhG if we want to compete with speed in mind.

Here are my results. For iTunes and RealPlayer I used the standard GUI programs. For the command line encoders I used Speeks' Batchenc frontend. The WMP11 mp3 codec was accessed by Nyaochi's Acmenc and Speek's Batchenc. The test bench was a 2.8 GHz P4. Before the speed tests I tried to disable all unnecessary background processes. The source files were in wave format on one HD and the target HD was separate. I measured the encoding times with a digital stopwatch. I didn't run several passes that would have been needed for lowering the error margin, but I think the results are accurate enough for this purpose. For measuring the bitrates I used Encspot Pro's "Complete Scan" feature (some other programs may not show correct bitrates with files that don't contain a Xing header).

The test file set consists of 25 selected complete tracks of various genres. The total duration is one hour, 31 minutes and 34 seconds.

The encoding speeds and average bitrates:


The individual bitrates of the VBR files:


A chart of the VBR bitrates:


Some readers may also find the following table interesting (Album = the 25-track test set):



I uploaded the test data here (Excel file & Encspot Pro reports):
http://www.hydrogenaudio.org/forums/index....st&p=515301

May I suggest that LAME ABR 128 be included in the test, aside from -V5 which is not a true 128 setting?

Thanks a lot for your test Alex. I am wondering if the highest iTunes setting (VBR quality 7) would increase bitrate too much. Have to do some testing myself.



No.

By the way, seeing that FhG 4.1 VBR is only 0.8x slower than Gogo ABR, I am wondering if I should test Gogo...

Last night I saw AlexB's sample thread in the uploads section.

Is this the place for sample proposals, or is it just to provide unknown new samples that can be tested?

Well, that is a rather old thread I started last year, but you can submit samples you think would be valuable for this test. Samples discussion will follow once we have the encoder and the settings set.

I don't have any numbers in the quality options in the English version. Perhaps you have the German version and it is different. Mine has: Lowest, Low, Medium Low, Medium, Medium High, High and Highest.


Has anyone verified that FhG 4.1 @ -m 4 & -q 0 would be better than GoGo? (Obviously -m 3 produces too high bitrates for this test.)


Since this encoder discussion seems to not advance I would like to suggest that we discuss about each encoder separately. Perhaps then we could make final decisions. Since you mentioned the iTunes' quality settings we could start with iTunes. I tested some additional iTunes options:



Surprisingly the quality options have almost no effect to the encoding speeds. The very small encoding time differences may be caused by other things as well (I did only one pass). I wonder if these settings do anything, even though there was a very small bitrate increase.

The Smart Encoding Adjustments and Filter Frequencies Below 10 Hz options seem to slightly affect the speed so perhaps they actually do something.

In general I can't see any advantage of using iTunes instead of LAME. In my experience iTunes produces inferior quality and it seems now that it is not significantly faster. Also, it doesn't write a LAME info tag for gapless decoding so even if the quality would be comparable I would prefer LAME. Unless Apple has miraculously increased the quality of this encoder I wouldn't mind if iTunes was dropped from this test. If other forum members insist to have it tested I would suggest using the settings that probably produce the best quality. I think the setting names Apple has chosen suggest that the best quality would be achieved @ 128 kbps VBR, Quality: Highest, Stereo Mode: Joint Stereo, Smart Encoding Adjustments & Filter Frequencies Below 10 Hz: enabled.

Edit: I made a small mistake in the iTunes table. I uploaded a fixed file. Refresh your browser to see it.

Yes, I only have numbers, 3 of them also having a German description:



I think iTunes is an encoder that should be featured in this test and I would drop Gogo if an encoder really needs to be excluded.



Shouldn't we test with -q 1?

I've explained why I don't need its test results. Why would you like to see it tested? Just for completeness' sake? In any case this is not a biggie for me, but I think each choice should be supported by proper argumentation. *)

I really would like to see all iTunes's MP3 encoder fanboys (and girls) to show up with their ABX logs and settings suggestions...


So you don't want to discuss about each encoder separately? We have at least three other debates to solve: Helix vs. Real, FhG versions & settings and GoGo vs. the other fastest encoders (which may be an empty list if Real and FhG in the slow mode is chosen). If we could concentrate on one thing at a time perhaps we could make progress.


EDIT

*) Previously I thought that iTunes was faster. Now I have no personal reason for testing it.

These two comments are contradictory. -q 1 is a lot slower than GoGo. We really would need to try a few samples and compare the FhG 4.1 -q 1, -q 0 and perhaps FhG 3.4 quality differences. But as I said I would like to discuss about FhG separately after we are through iTunes.

OK, you are right about the contradiction - I missed the fact that 71x was with -q 0, sorry.

So here's an important question: do we need to drop any of the 5 contenders LAME, FhG, iTunes, Gogo and Helix? If we add a low anchor to the configuration, there should be 6 encoders.

So here are my reasons for the contenders:

LAME: Recommended encoder here on HA and so far it is said to offer the best quality.

iTunes: Encoder is available for both Windows and OS X and seems to be popular outside HA. People with iPods who prefer to use MP3 over AAC usually choose iTunes to encode. Moreover, some artists who are offering songs in MP3 format seem to use iTunes as encoder (like Bruce Springsteen http://www.radionowheredownload.com/).

FhG: New encoder version which is worth testing because it's also pretty fast.

Helix: Fast encoder.

Gogo: Fast encoder, but on the other hand, quite old and I am wondering if it offers any big advantage over Helix or FhG which at least are actively developed (or is Helix not developed any more?). FhG with -q 1 is much slower that Gogo, though.

Sounds like the speed thematics is still there as a major one - at least to some of us.

Anyway, I second taking each encoder separately one in a row and call for opinions whether it should be included and with which settings.

And as things began to start already to some extent with iTunes mp3:
I personally like to see it tested, with the settings: q highest, js, smart&filter enabled.
In case average bitrate thus turns out ot be significantly higher than that of the competitors, q medium should be chosen IMO.

These are good points. Possibly the test can prove that it is worth of using LAME on a Mac too. It would also be good to be able to show proof to audio techinicians. If iTunes turns out to be better than I expect I would be happy to be proven wrong...

[OT] I have been listening to Bruce's Live 1975-1985 album (3CD Box) for an hour now... [/OT]

Before discussing about each encoder separately we should clarify the exact purpose of this test:
1/ evaluating several MP3 implementations at their best settings
2/ evaluating several MP3 implementations at their fastest settings
3/ evaluating several MP3 implementations set with different principles
4/ ???

Once the goal of the upcoming test totally clarified we should debate about the pertinence of each possible competitor.


I would go for the 1st one. The second one is in my opinion anachronic ; the third one would be biased (as it supposes that we can arbitrary decide that some implementations are only worth for their speed and don't have to be evaluated at their full quality potential).

I asked the same question a few posts ago and it seems that people tend to want 1. However, if the number of contenders is a problem, we need something in order to choose whether or not encoder A should be preferred over B. In such a situation, I think speed and popularity could be good criterions.

I go with 1) as well.

I'm also interested in the behavior of different settings, but as most people do I see that this is not easy to do and we better don't.


5 contenders and 1 low anchor - is that too much?

I agree: popularity seems a good criterion (though it's sometimes hard to evaluate).

I'm also interested in the behavior of different settings, but as most people do I see that this is not easy to do and we better don't.
Do you mean testing different settings of the same encoder? It's interesting indeed, but this task belong to a dedicated listening test.

I thought it was obvious why I and some others are interested about the faster options. We already have found out that LAME -V5 produces reasonably fine quality in the multiformat test. For many of us the LAME Info tag is very important and we would like to see its support extented to all possible HW & SW players. IMO, the HA members would need very strong reasons to use something else than LAME.

Personally, I have my ripped CDs archived in a lossless format. I have one copy of the complete archive in a high bitrate lossy format for home hifi usage. For portables I would like to encode smaller files on the need basis.

Normally I can do that with LAME, but there are times when I am for example about to start a holiday trip in about two hours and I would like to encode 30 hours of new music to take with me. (I know, I should have done that earlier, but sometimes things just don't go as planned.)

If the decoding speed from the lossless source is 50x and LAME encodes at 15x speed it would take about 2 hours 40 minutes. If a faster encoder encodes at 75x speed the process would take one hour.

The question is, if the quality of the used fast encoder is acceptable.

Edit: typos...

Alex B> I'm sometimes in the same situation (the need to encode quickly some lossless files). I also noticed that the effective speed of fast encoders may be drastically ruined by fragmentation and disk access (amplified by multi-core processors).

I'm also quite sure that the whole process lossless library -> MP3 isn't the most popular way to get MP3 files (most people are probably ripping CD to MP3 and in this case the encoding speed is usually limited by the extraction time). And as you said, speed is only needed in some specific situation (before holidays, etc... otherwise overnight encoding with best/slow encoders is fine). So is this marginal (I suppose) behaviour enough to justify a public listening test? Or shouldn't we use Sebastian's opportunity to evaluate for once the real potential of most popular/attractive LAME competitors (which seems to be all considerably faster than LAME)?

Perhaps the encoder/settings selection doesn't have to be based on the speed or on the quality. Let me explain.

In my speed test I included LAME -V5 -g5. It wasn't significantly faster. In my experience LAME -q5 at medium bitrates is the lowest usable q setting. So for LAME I think it is better to use the standard -V5 setting.

As we have seen iTunes doesn't have any fast VBR settings, so it is best to select the highest quality setting.

The choice between Helix and RealPlayer may not be difficult after all. We have no knowledge about the internal options used in RealPlayer and it doesn't seem to have any speed advantage. The RealPlayer UI is awkward and the converter feature is not available in the free version. A new RealPlayer version is on the beta stage, but I have no knowledge about it (other than it looks different). I would select the more versatile Helix CL encoder and use a suitable -V setting without any additional untested quality switches. However, it would be good to check if Helix has any newer source code available. Do we have any voluntary developers who could create a new compile if needed?

IMHO, the FhG settings should be quickly pre-tested. The difference between -q 0 and -q 1 can be anything. We don't even know if -q1 is better at all or for example if -q 0 is below all usability -- and we will never know that if don't try it before choosing the setting. I can contribute.

That leaves us GoGo. I could include it in my pretest. I hope some others would be able to do the same.

I would like to suggest the following: If GoGo is clearly worse than FhG -q 0 and -q 1 then we could drop it from the test and include both FhG settings (in case both settings seem to be promising). If GoGo seems to be able to provide useful or even better quality than FhG we could continue this discussion.

Also, I think 5+1 would be fine.

Very reasonable and practicable procedure IMO.

I fully agree with Alex B's suggestions.

Might I ask as to why you seem to have stowed away your idea of last spring to organize a (yet unexplored) mid-bitrate multi-codec test next, after my inquiry about a 96k test:
http://www.hydrogenaudio.org/forums/index....53134&st=78

I can't help but wondering if it isn't a bit of a waste of time and resources to explore 128k MP3 (once again), probably the most extensively tested/known/used codec setting today, while the sub-100k range almost remains terra incognita in the field of public listening tests.

Edit: not to mention, as was coined in this thread already, that the prospect of getting all-tied 4.5ish results, seems hardly challenging or even interesting to me.

Polar,

This MP3 test has been discussed a lot. We did a long pretest discssion about a year ago and we decided to postpone this test in favor of lower bitrate multiformat tests (48 and 64 kbps). The last about ~130 kbps Mp3 test is outdated and now it is a very good time to do this test for the discussed reasons. (Check also the the last year's 14-page discussion).

Sebastian has said that the next test in his list is a 80 kbps multi-format test and I hope we can continue testing codecs in a faster cycle so perhaps an about 100 kbps test could follow soon after that. Then we would have tested all important bitrate classes between 48 and 128 kbps.

I think after the MP3 test we will have an AAC test with both LC and HE-AAC at 80 kbps. The winner of that test will be included in the multiformat 80 kbps test.

As a spectator and (very grateful) user of these tests, can I explain why I should like to see the iTunes MP3 encoder tested?

I use a Mac, and have just got an iPod. To navigate on an iPod, tags are very important. It's my understanding that iTunes uses a better database than Max to get its tag information, so I should prefer to use iTunes. I also believe that the script that allows one to use Lame in iTunes uses a now somewhat out-of-date version.

So I'd like to get some kind of handle on how much quality I might lose if I switched from Max+Lame to iTunes with native encoder (assuming that Lame is superior, by some degree to be determined). Therefore I'd like to see iTunes tested at best quality settings.

I could, of course, test it myself, but I don't want to become artifact-aware, and a test like this is more systematic than anything I'd set up myself.

I'm very grateful to the people who organise and take part in these tests. One of their virtues to ordinary punters like me is in discovering those things that it is NOT worth testing for oneself.

I remember in a previous test iTunes did quite badly, and I don't think they would have upgraded the codec since then - they've abandoned it since they switched to using AAC. I think it's worth to note though, the default mp3 bitrate iTunes is 160kbps (where every other program defaults to 128) so i'm sure they knew it wasn't great.

edit: I would really like to see 128k AAC as a high anchor in this test. I know we've already discussed having a high anchor but I think it would be useful to see exactly how much of an edge AAC has over MP3.

As I promised, I tried to do a "quick pretest" with a few samples using foobar's ABX module, but that didn't work. The codecs' quality didn't have immediately obvious differences and the results would have been almost arbitary depending on the used samples. So I felt kind of obligated to do a full-scale personal test.

I used the following codec settings:
FhGc = FIIS 3.4 CBR 128 kbps
FhGv = FIIS 4.1 VBR -br 0 -m 4 -vbri -ofl
FhGv1 = FIIS 4.1 VBR -br 0 -m 4 -q 1 -vbri -ofl
GoG = GoGo 3.13 ABR -a -b 137
GoG2 = GoGo 3.13 ABR -a -b 137 -q 2

Edit: I forgot to mention that I used FhG Mp3sdecoder v. 1.3 for decoding the samples. (Looks like the -ofl option worked. ABC-HR Java didn't adjust the starting position of the FhG VBR samples)

After seeing the results of my previous speed and bitrate test I upped the GoGo ABR bitrate setting from 135 to 137 so that it would better match the codec average. FhG VBR produced still a bit higher average bitrate with the now tested new samples (134 kbps vs. 131 kbps).

I tested 30 varied samples. All samples are created by myself and most of them are brand new. About half of them are from Vangelis' Alexander soundtrack album, which has a lot of interesting music (it's a mixture of classical, electronic and ethnic genres) and I wanted to try if that material would be useful for testing codecs.

The reference samples and my ABC-HR result files are available in this link:
http://rapidshare.com/files/55027379/alexb_test.zip (about 76 MB)

Here are the test results:


Even though I did the test properly, I wouldn't trust the results too much.

Mostly I was able to differentiate the contenders without ABXing because of the lowpass. All encoders seem to use an about 15-16 kHz lowpass frequency at this bitrate/quality level. Beyond that the test was a tough call. The codecs were generally quite good and the slight lowpass didn't make the samples annoying (to me, it seems to be detectable only in a direct comparison with the reference).

I didn't want to spend several days with this test so I rated the samples rather quickly and it is possible that a new test with the same samples would produce slightly different results.

GoGo failed misarably on one killer sample, the sample number 29 (also FhG was bad with this sample, but better than GoGo). On the other hand FhG's VBR has obvious problems with some other samples.

I can make the following conclusions:
All encoders are tied and the higher quality options didn't make the encoders better.

Edit: Fixed a couple of typos and added the MP3 decoder info.

Thank you for your exhausting test.

At least it shows that

a) Gogo should be used without the -q2 switch
b) Gogo is competitive with FhG

Unfortunately it doesn't show much about how to use FhG.

I would ignore any difference that is smaller than 0.5 in my test. As you know, when you hear small artifacts, which may not be the same (i.e. when each encoder produces more or less different artifacts in different positions), it is quite difficult to put the contenders in an annoyance order. On each listening time you may rate the encoders differently. If you spend more time with each test sample the opinion may became more consistent, but as I said, I had to do the test rather quickly.

With some samples I found clear winners and losers, but also within this subgroup the winners and losers varied from sample to another.

So I think my test can prove only that I didn't find any of the contenders to be clearly worse or better in this specific test.

I wonder why the quality switches didn't work as one would like to expect? Is it possible that when the encoder does a more comprehensive analysis it can also go wrong in some cases and a more straightforward faster approach can actually produce better results?

However, I too would select the faster setting for GoGo, because I would like to get more proof of its usability in the faster default mode. *)

Regarding FhG, I think the newer 4.1 version & VBR mode should be selected, because my test didn't show any clear regression when compared with the 3.4 version. This would also make the test a VBR-only test and the average bitrates of the test contenders would be closer to each other. According to my test it would be safe to select the faster default -q value, but I really would like to see results from other forum members. Perhaps my test samples didn't reveal everything.


Edit: *) In addition, before selecting a specific -q value for GoGo we probably should pretest -q 0, -q 1, -q 3 etc too. Otherwise we would be just quessing the best value. It is better to use the default setting similarly like with Helix. FhG 4.1 VBR is a bit different case because it has only two possible -q values, which can be pretested easier.

Thanks for your test. From my understanding FHG 4.1 vbr is ABR at M4 / M5 ?

I wouldn't be worried about that. It looks like the encoders produce quite varied bitrates (including GoGo ABR):


Click to enlarge.

Does it really make sense to include GoGo? I think not:

1) Practically nobody uses it
2) Basically it's just an ancient version of Lame (3.92 I think) with some speed optimisations

So you didn't want to contribute this discussion by suggesting which encoders should be tested and especially which FhG settings (or version) should be selected. Just bashing GoGo does not really advance this discussion.

1) How do you know that? What encoder people usually select when they need a faster MP3 encoder than e.g. LAME? (Naturally I mean only those people who know about the options. It is obvious that most users on this planet do not change the used encoder. They just wait longer or if that is not possible they don't get everything encoded in time.)

2) It is based on LAME 3.88 and much of its code is completely rewritten in assemply language. In my test it was about 5x faster than LAME. This what the encoder displays about its version:

After my listening test I'm even more interested about GoGo. If it really is compentive against FhG it deserves to be tested in a public test. We are trying to find the best encoders that are available for encoding MP3 files just now. AFAIK, GoGo has no technical problems. The resulting MP3 files work fine. It doesn't matter for practical purposes if the development continues or not.

Well, GoGo was tested years ago in this test and came out clearly worse than the current Lame version (3.95) then. So anyone concerned about quality will avoid it. And that was years ago. How many people really need their MP3s so fast that they are ready to sacrifice quality today with all the superfast computers around? Sorry, I really don't see an audience for such a codec anymore. Just look at the almost complete lack of discussion here on HA about using GoGo for serious encoding work.

If it's really based on Lame 3.88 it's even more ridiculous to include it. Just think of the quality tuning work of 6 1/2 years that went into Lame since then, including Dibrom's Presets and Gabriel's complete reworking of the -V n modes.

I think it's a waste of time to test GoGo if there are more interesting and more widely used codecs around that can be included in the test. As the test will be very hard for most participants due to the hight quality of the codecs, I think we should limit the number of codecs as far as possible.

I was hoping 96 or 128 AAC would be tested as these are more commonly used. Are you going about this systematically from lowest bitrates to high?

I don't think any of the now tested other encoders can be competitive against LAME. In Roberto's test GoGo was tied with all other MP3 encoders except LAME:



I have a hunch that the situation has not changed much since then.

However, as you said that test is old and the encoders and suitable settings are going to be different now.

One of the reasons to do this test now was to check if the considerably faster encoders can produce usable quality when compared with LAME. The faster options seem to be FhG, Helix and GoGo. iTunes is not practically faster, but it should be included for other reasons as discussed in this thread.

The faster encoders are useful in many situations. One example is when you want to quickly create more or less temporary files for a small portable. Another application could be on-demand WAN or LAN distribution.


It would be nice to finally select the contenders after these 14- and 6-page discussions and start the test samples discussion. In Roberto's MP3 test and in my personal test the results varied clearly from sample to another. Actually, I think that a test like this could be easily manipulated to produce certain results by selecting certain test samples. The sample selection is going to be crucial.

@sebastian, can we also include lastest Coding Technologies AAC from Winamp. both LC an HE on 80kbps test?

Could we please discuss things about the AAC test when the right time has come?

Sorry for not showing up here lately, but I had a car accident and had to sort up some things. In the meanwhile I've got my car back from the workshop and hired a lawyer to deal with the insurance company of the guy who drove into the back of my car during a traffic jam.

So, where are we... We need to decide what encoders to feature and what settings to use. One (IMO) important question: how many contenders should we have?

Summary:


List of possible and pertinent (IMO) encoders:

• Fraunhofer surround
• GoGo
• Helix
• iTunes
• LAME


List of possible ambiguities:

• Fraunhofer surround : -q0 (fast & defaut) vs -q1 (slower but labelled as "high quality mode") // ABR vs VBR
• GoGo : -q2 or not
• Helix : no ambiguity (seems that nobody suggested ABR over VBR nor any magic commandline)
• iTunes : not discussed that much IIRC ; 128 VBR would maybe work (I mean bitrate-wise)
• LAME : no ambiguity (3.98b5 -V5)


List of possible goals of this test:

• best possible MP3
• best fastest MP3 encoder outside LAME
• best MP3 among "alive" encoders
• LAME vs major companies (iTunes, Fraunhofer, Real)


Correct me if I missed something.
___


I would personally go for a LAME + Fhg + iTunes + Helix match, set with VBR when possible and with settings we would trust to be the best ones (according to the community or the developers themselves).

So, since we have no high anchor, we could go for 5 contenders + 1 low anchor or 4 contenders + 1 low anchor. If 5 contenders are OK, we can test all 5 suggested MP3 encoders including Gogo. Otherwise, we have to decide which encoder we drop.

No please, 80 kbps is excellent! With Vorbis, it is *the* threshold bitrate that I use for my portable. The quality is good enough and not the least bit irritating (unlike 64 kbps). I think this is an essential bitrate to test and I'm glad Sebastian is testing it. Sorry for the OT, just want to show my support for the future planned tests.


As difficult as the last 128 kbps multi-format test proved to be, I think the fewer contenders the better, though this theoretically won't be as difficult since different implementations of the same (older) format are being tested here.

I agree. The encoders are good enough, that the high anchor can be dropped.

Are 5 contenders too many?

don't think so (this time i would try to ABX again )

I would prefer 4 but I'm not against this 5+1 configuration if it helps us to unlock the situation a bit

Let's move on to the settings...

Is --vb-new default in LAME 3.98? Should we choose the highest quality setting in iTunes? FhG -q0 or -q1? Gogo ABR or VBR, any -q setting? Helix CLI encoder or Real, what setting?