Personal evaluation at ~130..135 kbps, 200 samples, AAC (iTunes, Nero) - MP3 - Vorbis aoTuV Options

[pepoluan]

Check the date of the original posts?

That might do it... however I think it is better for guru to edit his posting above for posterity...

[guruboolez]

Fantastic job, guru. Don't you have some kind of HA award yet?

The good surprise comes from LAME MP3, which get the best mark (3,95)

Am I missing something, or plot says that it's Vorbis who get 3,95 and LAME get 3,94?

Youp...
Gecko already noticed it... I finally changed the plot, but I can't currently upload it (my new ftp needs a provider access IP to access to the ftp). This week end

pepoluan> I publish tests few hours or days after I finish them. Evereux's advice is fine

[pepoluan]

About half have been integrated here.

Please check it up I'm sure there are mistakes. I have a terrible headache and can't really concentrate.

Will try to finish it up tomorrow.

[bond]

About half have been integrated here.

Please check it up I'm sure there are mistakes. I have a terrible headache and can't really concentrate.

Will try to finish it up tomorrow.

great thingie this page! thx a lot!

[pepoluan]

Dang. Those tables make me dizzy.

All guru's tests, I think, have been listed in this HA Wiki page.

I am darned sure there are mistakes. For instance, I'm not sure of the links. But I have no time today, so please check the page out and tell me what's the mistakes. Or fix it yourself if possible

Oh and please forgive the coloring. It's not yet finished. Still an "alpha version" page will "go beta" if you guys tell me where the bugs are...

Hmmm.... just one drawback here... nearly all tests are guru's... where are the others...

This post has been edited by pepoluan: Jan 26 2006, 13:48

[pepoluan]

Uhh, guru, I noticed your lossless test is no longer accessible...

So I haven't put that in the listening tests page.

[guruboolez]

Uhh, guru, I noticed your lossless test is no longer accessible...

So I haven't put that in the listening tests page.

http://guruboolez.free.fr/lossless/

[=trott=]

My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

...which just goes to show that statistics can prove anything. In sebastian's multiformat test mp3 and in fact all other contenders were statistically tied. (possibly making an exception to vorbis aotuv). This also holds true for this test, I believe. (excepting the one classical test...)

I'd rather say that itunes 130 comes super close to lame 196 while still not being all that much better than lame 140.

[guruboolez]

My interpretation is that iTunes @ ~130 Kb/s comes super close to LAME @ ~196 Kb/s (the "high anchor", LAME 3.97 beta 1 –V2 --vbr new) and is definitely better than LAME @ ~130 Kb/s.

In sebastian's multiformat test mp3 and in fact all other contenders were statistically tied. (possibly making an exception to vorbis aotuv). This also holds true for this test, I believe. (excepting the one classical test...)

Not exactly. There's a clear hierarchy for both tests:

NON-CLASSICAL (50 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3 and Nero Digital AAC

CLASSICAL (150 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3
5. Nero Digital AAC

Most tested contenders are not statistically tied

[=trott=]

Not exactly. There's a clear hierarchy for both tests:

NON-CLASSICAL (50 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3 and Nero Digital AAC

CLASSICAL (150 samples)
1. high anchor
2. iTunes AAC and Vorbis aoTuV
4. LAME MP3
5. Nero Digital AAC

Most tested contenders are not statistically tied

I must have misunderstood something about sebastian's test then...excepting the low anchor there of course, can they not be considered as tied? I see the (relatively small) difference in this test, but in sebastian's I...cannot

[pepoluan]

Uhh, guru, I noticed your lossless test is no longer accessible...

So I haven't put that in the listening tests page.

http://guruboolez.free.fr/lossless/

Ahh, merci beaucoup!

I've put that in the listening tests page, External Tests section.

BTW, the listening tests page has "gone beta"! Yay! Feel free to check it out.

Side note: FLAC lost by a wide margin?? Gee I really must check out LA...

[guruboolez]

I must have misunderstood something about sebastian's test then.

This also holds true for this [i.e. guruboolez's one] test, I believe. (excepting the one classical test...)

My comment was about the last sentence. All contenders are indeed tied for Sebastian's tests, but not for mine

[guruboolez]

Side note: FLAC lost by a wide margin?? Gee I really must check out LA...

Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included

This post has been edited by guruboolez: Jan 28 2006, 10:17

[pepoluan]

Side note: FLAC lost by a wide margin?? Gee I really must check out LA...

Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included

Uhh... stupid me... I just realized... it is a Listening Test... lossless tests surely do not belong here...

I've removed the link to guru's lossless test site.

By "lost", I mean that FLAC's compression is ... how I put it? not that good ...

[vinnie97]

Side note: FLAC lost by a wide margin?? Gee I really must check out LA...

Off-topic: there are no looser or winner with a lossless comparison. FLAC is also close to top for decoding speed.

EDIT: you should rather link the WIKI lossless page which offers several links for different lossless comparison. Mine is included

Uhh... stupid me... I just realized... it is a Listening Test... lossless tests surely do not belong here...

I've removed the link to guru's lossless test site.

By "lost", I mean that FLAC's compression is ... how I put it? not that good ...

Yes, but it has wider support than LA, encodes/decodes faster and has error correction.

[rjamorim]

I've put that in the listening tests page, External Tests section.

BTW, the listening tests page has "gone beta"! Yay! Feel free to check it out.

Jesus, dude, you didn't link to ff123's test. It's like, all heresies rolled into one - he is pretty much the man behind this whole mess, for starters

[pepoluan]

I've put that in the listening tests page, External Tests section.

BTW, the listening tests page has "gone beta"! Yay! Feel free to check it out.

Jesus, dude, you didn't link to ff123's test. It's like, all heresies rolled into one - he is pretty much the man behind this whole mess, for starters

Uh... URL please? I promise it will be put in the next revision...

*hit head with a sandbag* stupid me...

EDIT:

Uhh... /me = stupid^2 ... what's Google for but I see rjamorim has beaten me to it ...

Well I did add a link to ff123's artifact training page near the top... hope I can atone for my sin this way

This post has been edited by pepoluan: Jan 29 2006, 07:30

[Bodhi]

Few words to conclude the test…
It’s pretty clear that all encoders tested here correspond to a good or even a very good output quality. There are currently no winner between AAC (iTunes) and Vorbis. It’s funny to see that results are pretty close on the finish line when problems are so different. Encodings are not fully transparent, but quality is in my opinion excellent most often (but not always).
LAME offers to MP3 the chance to stay competitive against AAC and Vorbis. Not fully competitive, but the efficiency of this format forces the respect.
Nero Digital implementation of AAC is slightly disappointing, especially with classical music, which is still a weak point of this encoder. But the quality is far from disaster (it wasn’t the case two years ago), is on average really good, gets even better with “non-classical” music and should satisfy several users.
Last but not least, difference among all these encoders is really small (don't look too much on "zoomed" plots )

But the average mark is somewhat misleading. LAME quality is ~0.5 point lower to iTunes or Vorbis, but it doesn’t mean for example that quality of encoded albums are 0,5 lower. This lower ranking is rather the expression of higher fragility than lower quality. LAME, and Nero Digital, are more inclined to serious distortions than Vorbis or iTunes AAC at the same bitrate. The concept of quality may be replaced with such encoders by the concept of strength or robustness. To illustrate this I made the following histogram (sorry for poor quality, I’ll change it later):



Here, Vorbis and iTunes both get a mark comprise between 4.5 and 5.0 for 50% of the tested samples, whereas Nero only achieve this state (near-transparency or full transparency) for 20% of the same samples. With the classical group of samples, 30% of the them were ranked below 3.0 with Nero when iTunes or Vorbis got the same notation of less than 10% of the sample. The two winners are stronger, and could handle more situations than LAME and Nero Digital AAC.

Hi,

would you still conclude this test the same way with "today's codecs"?

I'm not asking for a new test but just your opinion>

Thank you.

[BaByB0y]

Very nice
Through the Test, iTunes AAC and Vorbis do good job :x
nXqd

[TechVsLife]

I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2. Is there a link to download those samples?
Thanks.

[halb27]

I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2. Is there a link to download those samples?
Thanks.

In tis thread: LAME problem samples you'll find harp40_1 which is the worst harpsichord sample to me.
It's not just a problem for Lame but a problem for many encoders (not just mp3), and usually it requires a higher quality setting than is usually necessary.

I don't know a sax problem but may be trumpet problems are similar.
There can be a tremolo issue with trumpets. There's a sample 'Trumpet: My Prince' (guess you'll find it when doing a HA Google search above) which has this tremolo issue with Lame 3.97 and 3.98 when using VBR, but also when using FhG CBR (I tried FhG which ships with current dbpowerAmp).
You can also find a trumpet problem in the above link (this was the very problem I started worrying about problem samples), but this problem is rather Lame specific at least when looking at mp3 (it's a problem to some other formats as well). 3.97final has improved on it, and with Lame 3.98b3 the problem is overcome (at least at a higher bitrate which I always use for having a safety margin).

I encourage you to use such a safety margin as well if you can afford the larger filesize. Looks like you're out for that.
Use for instance 3.98b3 -V1 or an ABR or CBR setting in the 200+ kbps range, for instance -b224 -h.
With such a setting music usually is transparent, and in those rare cases when it's not it's at least acceptable.

This post has been edited by halb27: May 29 2007, 22:59

[TechVsLife]

Thanks, I found it after googling "trumpet: My prince."

The sax solo problem is seen in the scores given by guruboolez at the beginning of this thread.

Nothing leapt out at me on an attentive but not painfully concentrated listening, so I'd have to cheat by training myself in order to catch artifacts. I'll keep a lossless archive in case I gradually acquire the Power (Curse?), or in case someone I know has It, but probably -V 5 is fine for me (I've only tested -V 2 on those samples). The typical scratches on LP records seem to have been a much greater "distortion" than the anomalies people are talking about here?

I'd hate to think there's a perfect correlation between technical ability to hear the slightest changes, and ability to understand the music, but there's obviously some correlation. On the other hand, there's prob. some inverse correlation between obsession with technical issues and musical understanding. (unless someone comes up with an encoder that recreates or betters the original music (-V negative 2?), or that has the ability to rank it, e.g. Bach over Britten.)

I'd like to test the harpsichord, sax solo, and any other classical music samples that are extremely distorted at v5 lame, at least to guruboolez (--I hope at least barely noticeable to me), to see if I need to go up to v3 or v2. Is there a link to download those samples?
Thanks.

In tis thread: LAME problem samples you'll find harp40_1 which is the worst harpsichord sample to me.
It's not just a problem for Lame but a problem for many encoders (not just mp3), and usually it requires a higher quality setting than is usually necessary.

I don't know a sax problem but may be trumpet problems are similar.
There can be a tremolo issue with trumpets. There's a sample 'Trumpet: My Prince' (guess you'll find it when doing a HA Google search above) which has this tremolo issue with Lame 3.97 and 3.98 when using VBR, but also when using FhG CBR (I tried FhG which ships with current dbpowerAmp).
You can also find a trumpet problem in the above link (this was the very problem I started worrying about problem samples), but this problem is rather Lame specific at least when looking at mp3 (it's a problem to some other formats as well). 3.97final has improved on it, and with Lame 3.98b3 the problem is overcome (at least at a higher bitrate which I always use for having a safety margin).

I encourage you to use such a safety margin as well if you can afford the larger filesize. Looks like you're out for that.
Use for instance 3.98b3 -V1 or an ABR or CBR setting in the 200+ kbps range, for instance -b224 -h.
With such a setting music usually is transparent, and in those rare cases when it's not it's at least acceptable.

[shadowking]

mp3 problems samples aren't slight changes at all. The worst cases make me physically ill the more I listen to them and there is no need to abx even on -v2.