IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
My personal 128K mutiformat test
SacRat
post Jan 7 2004, 16:59
Post #1





Group: Members
Posts: 222
Joined: 8-November 02
From: Sarov, Russia
Member No.: 3719



I've recently finished my latest multiformat test @128kbps, featuring iTunes AAC, LAME 3.93.1, WMA9 VBR and Vorbis 1.01.
Results were interesting enough...

On 7 of 12 "classical" samples I used (thanks to Roberto for sharing) results were more, than just interesting...
AAC: nearly perfect, but sometimes have little troubles with hi-freq (percussion) signals and noices.
Vorbis q4: very slightly better, than AAC, but its attempt to encode hi-freqs made it fail on one sample (too annoying difference). Generally the same, as AAC.
LAME -b128: Poor. Difference is mostly obvious. It either artifacts badly or has degraded sound.
WMA Q75: Complete crap. Mostly even worse, than LAME (sic!). Incredible amount of artifacts.

If anyone interested in details I could provide them...

What do you think about WMA? Should I retest it in CBR (no pro, 'cause test was oriented on codecs with HW support) or just leave this crap? Any suggestions?

Codecs chosen:
iTunes for AAC for high quality in previous tests and price wink.gif
LAME 3.93.1 - just it seems to be one of the most popular ones
WMA9 VBR - kinda the best (even though WMA8 CBR seems to be the most popular one)
Vorbis 1.01 for possible uality improvements over 1.0

So overall it's just a demonstration with no tweaking (this test was for newbies, not audiomaniacs).


--------------------
http://sacrat.blogspot.com
Go to the top of the page
+Quote Post
Yaztromo
post Jan 7 2004, 22:12
Post #2





Group: Members
Posts: 236
Joined: 28-July 03
From: England, UK
Member No.: 8031



Why did you not use alt-preset 128 for LAME?

Also, were these all ABX tests?
Go to the top of the page
+Quote Post
Continuum
post Jan 7 2004, 22:19
Post #3





Group: Members
Posts: 473
Joined: 7-June 02
Member No.: 2244



More details on the samples used would be nice. smile.gif
Go to the top of the page
+Quote Post
rjamorim
post Jan 7 2004, 22:27
Post #4


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (Continuum @ Jan 7 2004, 07:19 PM)
More details on the samples used would be nice. smile.gif

QUOTE
On 7 of 12 "classical" samples I used (thanks to Roberto for sharing)


I think he means the samples I used in my 64kbps test. (and will use in the MP3 test)

In case you want to check them out:
http://rarewares.hydrogenaudio.org/samples/samples.rar


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
SacRat
post Jan 7 2004, 23:45
Post #5





Group: Members
Posts: 222
Joined: 8-November 02
From: Sarov, Russia
Member No.: 3719



@Yaztromo: I expected this question biggrin.gif
Very simple. It was a test. oriented on newbies, so I decided to use default or popular settings, without tweaking anything. And -b128 is a way more popular, than ap128 wink.gif
And, surely, it was a blind test, using ABCHR. In many cases I used direct ABXing. Those cases, where I got reliable results with some efforts are given 4.5 (see below)

@rjamorim: Roberto, could I make a link to your sample set in my article?
btw, were there any "killer" samples? Myblood seem to be a good sample for getting a clear picture of how format's noise shaping works smile.gif

This article was made for one Russian magazine, so I can't publish it right now for at least two reasons (Russian and © stuff), but test results would be given below. btw, if anyone is interested in proofreading this text in Russian (don't even ask to translate, it's 70k symbols already biggrin.gif), just e-mail me.

A list of samples (all marks would be given in that order):
Mybloodrusts
Newyorkcity
Bigyellow
Gone
Dafunk
Enolagay

AAC results:
4
3.5
4.5
3.5
3.5
4.5

Vorbis results:
3.5
4
4.5
4.5
2.5
4

WMA results:
2
2
3
2
3.5
3

MP3 results
2.5
3
4
2
3
3

Reading results: they're only given to compare one format against another. Absolute values won't give you reliable results.

Another thing. I already noted, that this test is made for amateurs, who aren't interested much in command line tweaking. They only want to know quality related stuff. Next, each of these formats has support in portables, so 128K test might be interesting from this side.

I'm asking for help once again: had anyone experienced quality related problems with VBR mode on WMA9 files? In my previous test (http://www.computerra.ru/compunity/dos/25596/index.html) it gave me poor results. So, how do you think, would it be more honest to update test with WMA9 CBR?
Anyway, from my experience of work wih WMAs I seriously doubt, that resuts would change much: WMA8 at 96K is annoying even on my iRiver. I even get crazy on my old WMA8 160 kbps recordings...

One more funny thing: after this test I seem to become much less tolerable to artifacts. Now even some TV programs drive me crazy... I hear these damned encoding artifacts :\


--------------------
http://sacrat.blogspot.com
Go to the top of the page
+Quote Post
rjamorim
post Jan 7 2004, 23:56
Post #6


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (SacRat @ Jan 7 2004, 08:45 PM)
@rjamorim: Roberto, could I make a link to your sample set in my article?

Well, yes, as long as it doesn't become much popular. I don't want to abuse HA's bandwidth...

QUOTE
btw, were there any "killer" samples? Myblood seem to be a good sample for getting a clear picture of how format's noise shaping works smile.gif


The worst sample there is Waiting. There are no traditional problem samples there.


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
AtaqueEG
post Jan 8 2004, 01:14
Post #7





Group: Members (Donating)
Posts: 1336
Joined: 18-November 01
From: Celaya, Guanajuato
Member No.: 478



QUOTE (SacRat @ Jan 7 2004, 09:59 AM)
On 7 of 12 "classical" samples I used...

Actually it's "classic", not "classical".

"Classical" suggests music like Beethoven or Mozart, you know, hmm, "classical music"

smile.gif


--------------------
I'm the one in the picture, sitting on a giant cabbage in Mexico, circa 1978.
Reseņas de Rock en Espaņol: www.estadogeneral.com
Go to the top of the page
+Quote Post
kl33per
post Jan 8 2004, 04:17
Post #8


A/V Moderator


Group: Members
Posts: 841
Joined: 9-June 03
From: Brisbane, AUS
Member No.: 7078



I think that was why he put quotation marks around classical (i.e. "classical") meaning orchestrated music in general. I agree the word is overly (and incorrectly) used though. Anyway...

Interesting test although not entirely unsurprising. By averaging out your scores you'll see that AAC actually outperformed Vorbis (although I don't think thats statiscally valid and the difference is so insignificant that you couldn't draw any solid conclusions from it). They are as you say generally the same. WMA9 result is a little surprising, although considering WMA9Pro didn't outperform any codec (except LAME of course) in Roberto's second 128kbps test then you couldn't expect WMA9Std to do to well. The fact that it didn't perform closer to LAME though (considering LAME was using -b128 instead of --alt-preset 128) is a little disturbing.


--------------------
www.sessions.com.au - Sessions Entertainment
Go to the top of the page
+Quote Post
SacRat
post Jan 8 2004, 10:15
Post #9





Group: Members
Posts: 222
Joined: 8-November 02
From: Sarov, Russia
Member No.: 3719



@Roberto:
Thanks.

@AtaqueEG:
Please, don't blame me; I wrote this late at night smile.gif And thanks for correcting biggrin.gif

@kl33per:
You can see, that AAC score is very slightly higher in average, but on most samples Vorbis sounds slightly better. That was the most surprising thing for me as AAC is becoming an object of worship on HA, while Vorbis is often blamed for quality.
I tested WMApro some time ago and liked it a lot, but there's still hardware compatibility issue, so I didn't use it in my current test.

As I already said, my conclusions are simple: AAC and Vorbis are clear leaders, which often give you "near transparent" quality at that size, being a good replacement for mp3 on HW players. Using LAME MP3 at -b128 (you can't be sure, that ap128 would be better) is not a good idea, even though it sound is better, than WMAs. Sooth to say I expected WMA to be slightly better, than LAME ;\ Shame on MS for making such a crappy codec. It seems, that they don't care about quality at all. Using theoretically better codec plus VBR mode and performing worse, than MP3 is a shame. It has very noticeable artifacts, so even MP3's preecho is nothing compared to it.


--------------------
http://sacrat.blogspot.com
Go to the top of the page
+Quote Post
vinnie97
post Jan 8 2004, 22:44
Post #10





Group: Members
Posts: 472
Joined: 6-March 03
Member No.: 5360



No MPC? huh.gif
Go to the top of the page
+Quote Post
rjamorim
post Jan 8 2004, 22:50
Post #11


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (vinnie97 @ Jan 8 2004, 07:44 PM)
No MPC? huh.gif

QUOTE (SacRat)
'cause test was oriented on codecs with HW support


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
xmixahlx
post Jan 8 2004, 23:41
Post #12





Group: Members
Posts: 1394
Joined: 20-December 01
From: seattle
Member No.: 693



it works on my linux box smile.gif


--------------------
RareWares/Debian :: http://www.rarewares.org/debian.html
Go to the top of the page
+Quote Post
ScorLibran
post Jan 11 2004, 13:55
Post #13





Group: Banned
Posts: 769
Joined: 1-July 03
Member No.: 7495



QUOTE (xmixahlx @ Jan 8 2004, 05:41 PM)
it works on my linux box smile.gif

"HW support" generally means portable hardware support (iPod, iRiver, Rio, PocketPC, etc.) or support on proprietary systems (Phatbox, Kenwood Music Keg).

But you probably already knew this. wink.gif
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th April 2014 - 23:09