halb27, as far as I heard so far, FhG 4 seems to be a new encoder so maybe the VBR performance increased compared to the 3 series.

So should we do anything to avoid people to react strangely? IMO this is not a good target and we won't succeed anyway.

No, indeed - we can't avoid it. But there's one question we must ask to ourselve: would such reaction be strange? Why should someone believe that CBR performs better than VBR? Why should someone accept that ABR is globally a better choice when this superiority is only attested for specific problems? I personally don't see any reason to do that. I (can) accept some advice from developers themselves but from unknown people I simply can't do it. Or to be more precise: I could trust some people's advices for a personal usage of an unfamiliar encoder but I wouldn't engage the credibility of a test and the work of 20 listeners on such fragile knowledge.

We should do the best according to what we know - well conscious about the imperfections.

OK, but if someone (me for exemple) will provide samples revealing that FHG VBR performs better than CBR? What would be our knowledge?

The only sure thing is lame -V5 cause we know it well. Helix, Fhg should be tested @ 130k vbr too, not CBR.. Gogo shouldn't be tested as its too old and not really faster than the others. Why do I have a gut feeling that the results will be again near perfect not showing much difference ?

OK, I did my test with an earlier version so my experience may not be valid any more.

According to my bitrate table Fhg VBR -m3 mode looks as the perfect VBR setting for such listening test: 128 kbps on 150 full classical music tracks (16 hours of music). For comparison, LAME 3.98b5 -V5 reaches 129 kbps on the same corpus.
As usual advantage of VBR coding, bitrate fluctuates and can reach high values on critical samples. Here are four examples, coming from my usual gallery, showing that Fhg VBR (-m3 -q0) outperforms CBR/ABR encodings (tested -br 134):

http://rapidshare.com/files/54002892/fhgsamples.zip

You may notice it: individual bitrate is considerably higher than 130 kbps but that's exactly what we're expecting from VBR with such samples.

That would be a valuable piece in the puzzle of improving experience.

@halb27> I anticipated your wish (see previous message)

A small bitrate table (150 full tracks of music; genre = classical):
http://www.megaupload.com/?d=Z1VKHJ0U

In summary:




EDIT: commandline used (foobar2000):
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 1 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 2 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 3 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 4 -q 0
-if - -of %d -vbri -ofl -eof -sr 44100 -c 2 -br 0 -m 5 -q 0

-c2 = 2 channels
-br 0 = required to enable VBR
-m n = VBR mode
-q 0 = enables fast quality coding mode [damned! I used the wrong switch!!! I had to redo the whole table]

Thanks for testing guruboolez!

In a quick test with my AC/DC sample a "FhG IIS 4.1.0 @ -m 3" file was easy to ABX. A lot easier than a LAME -V5 encoding. The electric guitar in the beginning of the sample is totally ruined. The bitrate was 144 kbps.

I am going to gather a few of my favorite samples and test FhG 3.4 CBR, FhG 4.1.0 CBR & VBR and GoGo ABR. However that is not going to happen before I have run the bitrate and encoding time tests on Saturday or Sunday.

Test -m 3 with some of my audio tracks (Pink Floyd, Led Zeppelin, Alphaville, Dire Straits...) and the bitrate is around 150 kbps.

menno also said that -m 4 seems to produce ~130 kbps.

I tested on more than 100 samples gathered from listening tests over the years:

-m3 produces 140 kbps
-m4 produces 130 kbps

Lame -V5 produces 136 kbps on this set

BTW, what does the "Original File Length" actually do? Does it have anything to do with gapless playback?

This is the default mode and should be preferred for this test.

Yes, was just about to ask whether or not we should force encoders to prefer speed over quality, but I saw that -q 0 is default anyways.

Yes, it is for gapless playback. FhG decided to reinvent the wheel.

Edit:

It would be great if Nyaochi or some other developer could create a front end that would save a LAME info header to FhG files, as I wrote in this reply:

Why should speed be prefered over quality?

Testing the faster options vs. LAME has been the idea of this test since August 2006 (or even before).

Perhaps the new FhG version changes that, but could you first try if the encoder is any better at -q 1 than at -q 0.

I just tried with one “speed metal” album:
-m3 = 150 kbps
-m4 = 140 kbps
-m5 = 113 kbps (32 KHz)

I'd like to see a bitrate table including different musical genre (some members did it in the past) to see which -m mode would be the most adequate. -m4 sounds significantly worse than -m3 (crappy artifacts everywhere on metal).


Yes, I remember that some people requested a “fast MP3 encoder listening test”, right?
But is it still the purpose here? Or are we interested to see if the new and rather untested MP3 encoders are valid challengers to LAME?
If the goal of this test is to check which fast MP3 implementation offers the best quality, then I don't see the point of including LAME (excepted maybe as high anchor). But of the goal is to evaluate what degree of quality all "modern" MP3 encoders are offering, then I would use them at their “high quality mode” - not their “fast” ones.

-m 3 seems to encode at 50x with foobar2000 (two parallel encoders due to HT CPU).

The actual idea of the test was to see if LAME, which is really slower than the other encoders, offers a much higher quality that would justify the speed "issue". LAME could serve as high anchor, but -V 5 --vbr-new might not be that much better than the rest thus doing a bad job as high anchor.

On rock / metal i get around 152k (m3) 140 (m4)

I missed this idea. Sorry.
But shouldn't the first logical competitor to LAME be LAME itself, at -q7 or -q9 setting (which are significantly improving the encoding speed)? If the purpose is to evaluate the tradeoff between quality and speed shouldn't we question first the choice of LAME developers before testing alternative encoders?

Here are some results with -m 3:

Vivaldi's Four Seasons: 130
Chris Rea (3 albums): 150
Coldplay (2 albums): 145
Deep Purple's Made in Japan Remastered: 150
Dire Straits (3 albums, 1 live with 2 CDs, 2 studio, 1 remastered version of a studio album): 150

I can let fb2k encode and see what else comes out after encoding my large Pink Floyd collection, Jethro Tull, etc., but I think it will stay at around 150 kbps.

Eminem's Curtain Call - The Hits: 145



I was thinking about testing the recommended LAME settings for 128 kbps (= -V 5 --vbr-new) against other fast encoders with their default setting to see if there's a big quality difference. Another possibility would be asking LAME devs for the fastest LAME setting and then comparing that to the fastest setting of the other encoders OR trying to figure out what high quality settings to use for the contenders and then checking the results of the test - if the other contenders that were set to prefer quality over speed are still worse than LAME, then their fast setting would be for sure, too. If they are better and the speed is similar to LAME's, LAME no longer is the king of MP3 encoders. If the quality is equal, look at the encoding speed. If the speed was also similar to LAME's, then we still have the "problem" that we don't know how well the fast settings perform.

yeah, besides, gogo has already been tested.

These are only the samples. I thought you provided a listening test.

8/8 everywhere (I didn't mentionned it, sorry).
I can restart the test if you really want ABX log files.

No ABX log necessary, I just wanted to know the result.

Sample “A03” offers the most obvious difference IMHO.
I didn't try yet with VBR -m4 which seems much more suitable to a ~130 kbps listening test (with non-classical music samples).

What is fraunhofer mp3encs v1.4 20070711 please?

all4mp3's mp3sEncoder.exe V1.4 is dated 2007, July 5, and based on an encoder library V04.01.00 2007-05-18.

So this must be something else.

http://www.all4mp3.com/tools/sw_fhg_cl.html

check the filename

Thank you, but that's what my version originates from (the Win32 version). Guess you use another OS.

I checked all download links and all are providing “mp3sEnc(...)20070711.zip”

I see, you referred to the file name of the zip file, not the encoder itself.

exactly

1. I'm more interested in quality than speed, so, if there are other mp3 encoders being actively developed besides Lame (--I assume FhG is), and if unofficial but quick abx tests by known good listeners show they may plausibly be contenders, then I would like to see them included, or at least the one that is suspected of being the best rival. I also think mixing VBR & CBR can lead to some questions about the results (even in the 64kbps round), at least on tricky individual subtests where VBR produces a significantly bigger bitrate file than CBR, though I realize you keep it within reasonable bounds (10% or whatever). (Is any mp3 encoder not developed to take advantage of VBR likely to do as well as LAME in bitrates of 128kbps?)

2. Of course, if you are testing speed, then it would make sense to make sure all encoders are at the same level of speed (just as you would for bitrate)--produce what quality at the same bitrate in the same time. Else, you are comparing apples and oranges. What you are aiming for is quality per bit per second. Now you could correct for different times by normalizing the numbers after the test (ratios per time etc.), but that obviously is less meaningful (some encoders might do disproportionately bettter or worse at a different speed setting, so a linear ratio or normalization would be deceptive).

How about adding Lame 3.90.3 as well, just for kicks. And since I imagine there are quite a few of those mp3's laying around.

I'd say yes.


Does FhG have a set of recommendations in any open documentation? According to this:

That's not too terribly informative, but I don't see anything that leads me to believe that FhG advises against VBR, so I'd say VBR should be used.

Can't demand so much of the testers--I think they're trying to get the list down to four candidates, so 3.90.3 should not be included.

Yes, and besides, Lame 3.90.3 was most popularly used for VBR ~ 190-200 kbps (-alt-preset standard, equiv to -V2), not much at 128 kbps (for which no VBR mode was available, only CBR or ABR).

No thanks, definitely not an option.

Someone mentioned Radium/ProducerPro/l3codecp.acm v1.2.x before. I too would like to see this encoder included as it was so widely used. (heck, I still use it with virtualdub) I wan't to see how everyone else thinks it compares to the newer versions found in wmp10/11 (l3codecp.acm v3.4.x aka fastenc).

Personally I would drop GOGO since it is based on lame 3.88 (or something around there) and would much rather see lame 3.90.3/3.93.1 --alt-preset CBR 128 -h (only because you all don't like my crazy/custom settings ) to see how much lame really has/hasn't improved.

I think it would be better to have an FhG test, a Xing test, and a LAME test to see how much the encoders have changed over the years. Then compare the best with each other.

My $.02

I will definitely not include versions older than 3.97. 3.97 was an idea because it's the currently recommended encoder, but since Gabriel and others want to test 3.98, I prefer that over 3.97.

Any idea what was changed from Gogo 3.12 to 3.13?

I think before starting to discuss settings, we should decide what the test should be good for:

A. Compare recommended LAME quality settings to "defaults" of the competition (defaults in quotation marks because VBR will be probably enforced - what I mean is that no additional settings like optimization for speed or quality should be supplied)

B. Compare recommended LAME speed settings to fastest mode of the competition

C. Compare recommended LAME quality settings to best quality mode of the competition (but only if those settings have a note that they really increase quality, such as in FhG's case, so no funky Helix switches that nobody knows what they are actually good for)

I choose C. I'm interested in how competitors perform with quality-geared settings. Encoding speed should be mentioned but should also be a secondary consideration unless we can create a metric to gauge performance-per-time-unit as a fellow above suggested.

FhG MP3 Surround Encoder is an Next-Gen MP3 encoder & you must use the full version for any test...

C seems most preferable to me.

I don't understand what you mean with "full version".

I'm not sure to understand what Silver Wave means by 'full version' but if 'full' means 'full quality' I would agree with this point.



The first questions I would ask:

• now we have bi-, quad- and soonly octo-core processors working at several GHz, is there any point of testing encoders, which are already very fast, with their fastest settings?
• shouldn't we first respect these new competitors and check their full potential - especially if we oppose them to a reference called LAME which will be set to offer its best - before evaluating their quality on their fastest mode?


In my opinion the true challenge of this “MP3 listening test” is to see what potential offer the latest “2007” Fraunhofer encoder, the flexible HELIX one, and maybe the last Apple's implementation (totally different from Fhg & Xing). I would consider them as true competitors to LAME, and not the poor ones we could handicap even further with 'fast'/'lower quality' settings.
It's a listening test: we don't have to presuppose that LAME's competitors are worse in order to justify our choice to lower their quality a bit more. It implies the following rule: if one encoder is set to reach its full potential (LAME) all others should be set with the same principle. If our goal is to test ultra-fast MP3 implementations, then LAME 3.xx looks out of competition.

what are we interested for? To see if LAME's superiority is still valid? To see which x90 MP3 encoder offers the less distorted sound?


I know there's another question (already mentionned): is LAME relative “slowness” really justified by a quality boost? That's why some users are interested (for practical purpose) to "boost" speed to the max level. This question is perfectly valid and is truly interesting. But starting a listening test for the sole purpose to answer this question is problematic. If LAME appears as better, what would be our gain? We would only “discover” that a x60 encoder sound worse than a x15 one: what a lesson! And we would still ignore if Fraunhofer 4.xx in HQ mode is able to compete with LAME. Gain in this hypothesis = nada.

I absolutely second that. Speed really shouldn't be an issue as any encoder can be considered fast enough.
Sure it's remarkable in case it should come out that a superfast encoder has better quality than a sufficiently fast one, but that's all.

As far as I remember in the first thread for the 128 kbps listening test there was a rough motivation to see how Lame competes with other encoders which happen to be faster. The speed thematics was there, but it was not an overwhelming one.

Do you guys mind if Real is used for encoding the Helix samples?

I prefer the CLI encoder because I did use it and had a pretty good impression qualitywise.
I also bought Real when I cared about Helix a lot but wasn't impressed and continued to use the CLI encoder.
I must admit however I don't remember the details, and it may be that it was just the GUI that disappointed me.