IPB

Welcome Guest ( Log In | Register )

7 Pages V   1 2 3 > »   
Reply to this topicStart new topic
128kbps Extension Test - FINISHED
rjamorim
post Aug 4 2003, 06:10
Post #1


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



Hello

I would like to announce that the 128kbps Extension listening test is now closed, and the results are available at my Listening Tests page:

http://audio.ciara.us/test/

Here are the overall ratings:


Please post comments at this thread.

Best regards;

Roberto.

This post has been edited by rjamorim: Aug 4 2003, 06:14


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
ff123
post Aug 4 2003, 08:27
Post #2


ABC/HR developer, ff123.net admin


Group: Developer (Donating)
Posts: 1396
Joined: 24-September 01
Member No.: 12



Nice test. The results clearly show how good the newer generation of codecs are. Looks like Blade served its purpose well. I wonder if there are still some Blade 128's out there on p2p? I wonder how Fraunhofer's FastEnc or the older "Radium" hack would have compared to Lame?

Death2.wav is really interesting. The bitrates for Ogg and MPC don't suggest anything out of the ordinary (about 115 kbit/s), but WMA9Pro really tanks. A lot of people comment on noisy transients, with something also wrong with the stereo during those parts.

Several people have very good high frequency hearing, and can hear the lowpassing of MPC and AAC (both around 16 kHz). One person (gecko) seems to find this to be a significant defect.

If new samples are chosen for the next test, I think Waiting.wav should be retained. It seems to be one of those killer samples which have a big effect on every codec.

ff123
Go to the top of the page
+Quote Post
treech
post Aug 4 2003, 08:29
Post #3





Group: Members
Posts: 208
Joined: 28-May 02
From: Helsingborg
Member No.: 2147



SWEET!

this will help me decide on what portable (player) to get.

looks like aac and ogg are very good indeed at this bitrate, really good news for portables, since those 2 are most likely to be the ones to get implemented first, (i'de love to get the iriver perl, ogg support)

and the ipod already had aac support iirc ... biggrin.gif
Go to the top of the page
+Quote Post
superdumprob
post Aug 4 2003, 10:26
Post #4





Group: Members
Posts: 212
Joined: 30-March 03
Member No.: 5752



VERY interesting that Musepack came out on top, albeit marginally. Thanks to Roberto, all the test takers and everyone who helped. These comparisons are very useful. smile.gif Keep up the good work!


--------------------
superdumprob
____________________________________________

"If we knew what it was we were doing, it would not be called research, would it?" - Albert Einstein
Go to the top of the page
+Quote Post
guruboolez
post Aug 4 2003, 10:31
Post #5





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



I did two different tests : one on a portable PC, with a (I suppose) crap audio chip (Conexant AC) but good headphone ; another with my main soundcard, Terratec DMX6Fire.

Results are here :

http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm
http://membres.lycos.fr/guruboolez/AUDIO/t...MF/TERRATEC.htm

Comments (log file), (in anglische) are available too :
http://membres.lycos.fr/guruboolez/AUDIO/test_MF/


I've two different winners :
WMA9PRO on portable, and Quicktime AAC on Terratec (never mpc).
The three best are close each others ; Vorbis was fourth, on two test, not as good as the three others, but more constant than WMA9PRo, and maybe MPC.
LAME is now old, and Blade looks more as a joke than as an encoder.

I'm a bit surprised to see mpc "wining" here. I hope that Frank Klemm is just in holydays : I didn't see him for some month on HA. The development of mpc can't stop !

EDIT : Roberto, thank you another time for publishing these results immediately after then end of the test smile.gif

This post has been edited by guruboolez: Aug 4 2003, 10:33
Go to the top of the page
+Quote Post
spoon
post Aug 4 2003, 10:42
Post #6


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2725
Joined: 24-March 02
Member No.: 1615



I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
Volcano
post Aug 4 2003, 11:02
Post #7





Group: Members (Donating)
Posts: 916
Joined: 30-September 01
From: Berlin, Germany
Member No.: 112



Well done, Roberto. B)

I too was surprised at how well the codecs performed generally, some tests were *extremely* hard. (However I have to say that in some cases where added background noise was the only problem, I had absolutely no chance of detecting it because my sh*tty equipment hisses like mad. Gotta get a better soundcard soon. blink.gif)

I'm beginning to lose hope for Vorbis. It gets beaten by MP4 in most cases, and generally performs worse than one would expect, given the fact that this type of bitrate range is considered Vorbis' speciality. Add to that the sluggish development (also with regards to hardware support), and its future doesn't look too bright... sad.gif


On a side note, @ff123: Have you considered perhaps adding XML support for the results file to ABC/HR? That would make parsing the results into a database a breeze, and they could be published on one HTML page much more easily.
Go to the top of the page
+Quote Post
spoon
post Aug 4 2003, 11:17
Post #8


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2725
Joined: 24-March 02
Member No.: 1615



I wouldn't loose hope smile.gif it just shows how competitive things are in that bitrate area, really there is not much between AAC, Ogg, WMA and MPC. Big bucks have gone into developing some of those codecs and it is good that ogg achieves the same without patent infringing (if WMA wanted to use some technology as AAC, they would just cross license patents, that was not an option for Ogg).

I think the 'winners' in years to come will be AAC and WMA, both of these will have large online commercial music shops and wide support on portable players.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
Lev
post Aug 4 2003, 11:18
Post #9





Group: Members
Posts: 524
Joined: 7-November 02
From: Gloucester, UK
Member No.: 3716



QUOTE
The website you have attempted to access is on a filter list maintained
for Unilever. In an effort to avoid potential embarrassment to you or
subsequent users of this PC, this material will not be displayed.

However the list of filter sites can sometimes be inaccurate or out of
date. If you believe this is the case please call the GIO-ES service desk
on +44 (0) 1244 50 4180 and access to this site will be restored for you
and all other users in Unilever.

The URL which you have attempted to access is listed below. You will be
asked for this information when you call the service desk.

GIO apologises for any inconvenience this may have caused.



--------------------------------------------------------------------------------

URL = http://membres.lycos.fr/guruboolez/AUDIO/t...t_MF/COMPAQ.htm 

heh smile.gif

It seems that you are sensitive to MPC artifacts, Guru. rjamorim wants your ears, I am very happy not to have them wink.gif


--------------------
http://www.megalev.co.uk
Go to the top of the page
+Quote Post
guruboolez
post Aug 4 2003, 11:33
Post #10





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Lev > Is this what appeared when you tried to reach the two results matrix I uploaded ? It works with me... Can someone confirm ?

P.S. I created the webpage with MS Word XP (I haven't anything else installed, and I'm bored by notepad wink.gif) : It's big, and maybe problematic.

This post has been edited by guruboolez: Aug 4 2003, 11:35
Go to the top of the page
+Quote Post
spoon
post Aug 4 2003, 11:36
Post #11


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2725
Joined: 24-March 02
Member No.: 1615



Works for me, the filtering is through his company proxy server (unilever).


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
XXX
post Aug 4 2003, 12:19
Post #12





Group: Banned
Posts: 32
Joined: 4-August 03
Member No.: 8154



For the record, if you have six players and five tie for first place, the sixth player is in sixth (or last) place, not second place. If you have six players, and two tie for second place, there is no third place; the next place is fourth place. Win. Place. Or Show. The rest don't really matter, you know.
Go to the top of the page
+Quote Post
Mac
post Aug 4 2003, 12:32
Post #13





Group: Members
Posts: 650
Joined: 28-July 02
From: B'ham UK
Member No.: 2828



Taking into account the deviations possible in the results, I see it fair to tie AAC, MPC, OGG & WMA in joint 1st.. (MPC could be the lowest of the four and OGG the highest if the errors fell in a parculiar way)

At first I thought it is surprising that 4 very different codecs can achieve 4 levels of quality that converge so closely.. but then I thought, does this just mean they are approaching the soft-limit of achievable compression? Until radical re-thinks in compression schemes (SBR @ 128k, wavelets) I wouldn't imagine a codec *could* progress much further than this ~4.5 limit.

Thankyou for organizing this test Roberto, these results provide the interesting food for thought that can only come about from a professionally conducted group test smile.gif


--------------------
< w o g o n e . c o m / l o l >
Go to the top of the page
+Quote Post
Lev
post Aug 4 2003, 13:26
Post #14





Group: Members
Posts: 524
Joined: 7-November 02
From: Gloucester, UK
Member No.: 3716



Yep - its a Unilever filter... you didnt include 'bondage' or 'granny fisting' in the title bar, did you? biggrin.gif

I agree with Mac - its almost like a plateau has been reached, and asking people to judge encoders is almost like asking people if they like celery. Guruboolez seems sensitive to MPC artifacts, whereas many of us aren't, hence it slightly won in the test. Its almost like a genetic thing, rather than a matter of training or practice.. I have a self proclaimed talent for picking out Oggm for example.

But yep, I love test results like this. I love figures as opposed to words. Thanks smile.gif


--------------------
http://www.megalev.co.uk
Go to the top of the page
+Quote Post
phong
post Aug 4 2003, 13:51
Post #15





Group: Members
Posts: 346
Joined: 7-July 03
From: 15 & Ryan
Member No.: 7619



I think one certain conclusion we can make is that anyone selling 128k anything and calling it "CD quality" needs to be thrown in jail. smile.gif

A couple notes (based on my listening):

For me, each codec had at least one sample with a pretty catastrophic falure that I would probably notice in casual listening. MPC came the closest to being "acceptable" to me for all samples, but had a quite noticeable stereo separation issue with TheSource. Nobody else seemed to mention that problem though...

Blade did serve its purpose, however, I had one sample where lame lost to blade. I one other person ranked them in that order for that sample (Waiting), and another had them tied, so I might not be completely crazy. Also on death2 if it weren't for blade for comparision, WMApro would have gotten < 2 and AAC would have been scored quite a bit lower too.

I was able to ABX a lot more of these than I initially expected to. My ears are not all that trained (well I suppose they're getting there).

I would have ranked them (based on my results before I saw everyone elses):
1. MPC
2. WMApro
3. AAC and vorbis tied (very close, throw out one test and they'd flip-flop)
5. lame (fairly far back)
6 . blade (way way back).

For my results, MPC was the most consistant, almost always either getting first, or very close (with the one exception.) WMApro was ususally near the top, with one pretty big failure, and somes falling back to the middle. AAC consistantly came in around third, close to MPC and WMApro, while vorbis tended to jump around more. Lame usually trailed but occasionally came out near the top, and blade was always way back. For me MPC was definately "least likely to have serious problems" which is pretty consistant with the consensus of these forums.

My ears got trained as I went along... Problems that were hardly noticeable initially became really easy (specifically the "brightness" of vorbis on some samples, and the absolute crappiness of blade on all samples). Other things got harder from having listened to them too much. smile.gif I think I would have fairly different results if I were to do another test (WMApro, AAC and esp. vorbis would probably get lower scores now that I am more familiar with their problems).

I thought it was interesting how much people varied in their sensitivity to different problems. Even though I knew harpsichords were supposed to cause trouble, I really couldn't identify too many with Bachpsichord (and I did try very hard). OTOH, I'm the only one (I think) that mentioned the stereo problems with TheSource and MPC.

I'd have to say I was "heartened" by the the final results. While I had MPC as a winner, I don't ever see it having success in the portable market or widespread support. After looking at my results, my sentimental favorite (vorbis) didn't look like it could stand up against against WMApro, nor could AAC (which I reguard as "less evil" than WMApro). To see that there's pretty much a four way tie for first is a bit surprising. If vorbis gets further tuning, I think it could stay competitive.

Who gets the golden ear award? Guruboolez? Gecko? I suppose handing out awards isn't a good idea. It might encourage people to fake results. Plus from my comments, I think I'd get the "excessive verbosity" award.

I want to say thanks one more time to rajamorim (espescially for the speedy tabulation) and to all the other participants.

This post has been edited by phong: Aug 4 2003, 14:01


--------------------
I am *expanding!* It is so much *squishy* to *smell* you! *Campers* are the best! I have *anticipation* and then what? Better parties in *the middle* for sure.
http://www.phong.org/
Go to the top of the page
+Quote Post
guruboolez
post Aug 4 2003, 14:33
Post #16





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



Here is the ranking table for this test (based on overall results) :

http://membres.lycos.fr/guruboolez/AUDIO/t.../test_MF/HA.htm

Mpc never goes below the third place (only codec to perform that). Must be the most constant encoder (thanks to VBR I suppose) of the five challengers (with lame, always at the two last places).

Vorbis can't seriously claim one of the first three place.

Lame is outdated (Lev, you could call that 'granny fisting' wink.gif)

This post has been edited by guruboolez: Aug 4 2003, 14:44
Go to the top of the page
+Quote Post
S_O
post Aug 4 2003, 15:14
Post #17





Group: Members
Posts: 296
Joined: 27-July 02
From: Germany
Member No.: 2821



Wasn´t the test open to 5th august? I´ve done test 1 - 10 already, with comments etc., just hadn´t enough time to do the last both test.
Now I´ve done the ranking of myself. also the last both samples are not included my result is a bit different:

1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0

I always rated the blade sample to 1, because it´s clearly the worst, also on some samples Blade performs better. The results are not manipulated and didn´t thought that would come out (I thought AAC clearly beats Vorbis and Musepack as higher-bitrate codec is the worst after blade). here are results for sample 1-10:
CODE
Vorbis    5   - 5   - 4,5 - 4,5 - 5   - 5   - 4   - 4,5 - 4,5 - 5 -- 4,7
Lame      4,5 - 4,8 - 4   - 5   - 2,5 - 3,8 - 4,2 - 3,8 - 2,5 - 5 -- 4,01
MusePack  4,6 - 4,9 - 4,2 - 3,8 - 5   - 5   - 3,8 - 4,5 - 4   - 4 -- 4,38
WMA Pro   5   - 3,2 - 5   - 4,2 - 1,5 - 3,5 - 4   - 4,5 - 4,5 - 3 -- 3,84
QT AAC    4,4 - 4,5 - 5   - 5   - 4,8 - 4,2 - 4,4 - 4,8 - 4,5 - 5 -- 4,66

If you like I can also post all my comments to the samples.
Go to the top of the page
+Quote Post
smok3
post Aug 4 2003, 15:27
Post #18


A/V Moderator


Group: Moderator
Posts: 1709
Joined: 30-April 02
From: Slovenia
Member No.: 1922



QUOTE (spoon @ Aug 4 2003, 11:42 AM)
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

that would be interesting to see.


--------------------
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung
Go to the top of the page
+Quote Post
fragtal
post Aug 4 2003, 15:47
Post #19





Group: Members
Posts: 274
Joined: 1-January 03
From: Old Europe
Member No.: 4353



QUOTE (S_O @ Aug 4 2003, 04:14 PM)
1. Vorbis: 4,7
2. QT AAC: 4,66
3. Musepack: 4,38
4. Lame: 4,01
5. WMA Pro: 3,84
6. Blade: 1,0

That's the ranking I've expected. sad.gif

I hope this test will encourage Xiph.org to do some further Vorbis tuning.


--------------------
I love the moderators.
Go to the top of the page
+Quote Post
Jojo
post Aug 4 2003, 15:53
Post #20





Group: Members
Posts: 1361
Joined: 25-November 02
Member No.: 3873



I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it? unsure.gif

This post has been edited by Jojo: Aug 4 2003, 15:54


--------------------
--alt-presets are there for a reason! These other switches DO NOT work better than it, trust me on this.
LAME + Joint Stereo doesn't destroy 'Stereo'
Go to the top of the page
+Quote Post
guruboolez
post Aug 4 2003, 15:58
Post #21





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Jojo @ Aug 4 2003, 03:53 PM)
I would like to know which options have been used for the wma 9 pro samples. I mean whether vbr or cbr.
Also, where can I download wma 9 Pro? Or do I have to pay for it? unsure.gif

Just read Roberto's presentation :

wma9pro VBR 128 two pass (it's more ABR, and very accurate : 128 kbps)

WMA9PRO is available on Microsoft website. Just DL for free Windows Media Encoder 9. If you're running on 98SE OS, take a look at Spoon's dBPowerAMP. It's an easy way to encode with WMA9 (pro, lossless, voice...)
Go to the top of the page
+Quote Post
rjamorim
post Aug 4 2003, 18:09
Post #22


Rarewares admin


Group: Members
Posts: 7515
Joined: 30-September 01
From: Brazil
Member No.: 81



QUOTE (spoon @ Aug 4 2003, 06:42 AM)
I would be interested in doing a little statistics based on bitrate, if someone who has access to the samples please post either the total file sizes for each encoder, or the average bitrate (ie 134kbps + 150Kbps...) for each of the codecs.

I will add a table with all the bitrates to the results page later today.

QUOTE
Wasn´t the test open to 5th august?


Oh, sorry, S_O, but quoting the readme:
"7. After you finish the test, save the results and mail them to
rjamorim@<blanked>. The test ends on August 3rd, 2003. No
results will be accepted after that date."

sad.gif



Some info for you guys:

Next test will be at 64kbps. I'll compare Vorbis, HE AAC, WMAv9, MP3Pro and Real Audio Cook. But that's discusseable, and I don't want to discuss it right now. I gotta take a break... wink.gif

Hopefully, people won't have to struggle that much to find artifacts in this test.

I plan to start this test by the beginning of September, probably the 3rd.

People are too fed up with the current samples, so I think I'll only keep Waiting (per ff123's suggestion) and do a call for samples next week.

No decision has been taken yet though, and I sense there'll be a very heaten discussion about whether choosing VBR whenever possible or only CBR. I'll start a pre-test thread as the test gets closer.

Thank-you very much.

Best regards;

Roberto.

This post has been edited by rjamorim: Aug 4 2003, 18:10


--------------------
Get up-to-date binaries of Lame, AAC, Vorbis and much more at RareWares:
http://www.rarewares.org
Go to the top of the page
+Quote Post
music_man_mpc
post Aug 4 2003, 18:19
Post #23





Group: Members (Donating)
Posts: 707
Joined: 20-July 03
From: Canada
Member No.: 7895



These results are very supprizing, even on MusePack.net it says that Ogg is probably better for bitrates <140Kbit.

QUOTE
Sound tests with other Audioformats show that MPC is the MPC for high quality with bitrates above 140kbits. Below OGG Vorbis is a bit better.


This post has been edited by music_man_mpc: Aug 4 2003, 18:19


--------------------
gentoo ~amd64 + layman | ncmpcpp/mpd | wavpack + vorbis + lame
Go to the top of the page
+Quote Post
Gecko
post Aug 4 2003, 18:29
Post #24





Group: Members
Posts: 934
Joined: 15-December 01
From: Germany
Member No.: 662



I find it very interesting that aac being cbr and wma pro being two passed reach such a high quality. I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).

Looking at the results, I am also glad mpc was included in the test since it is a serious contender.

I noticed stereo collapse on FloorEssence with mpc (but not on TheSource).

Thanks Roberto, for performing the test and such a quick evaluation!
Go to the top of the page
+Quote Post
guruboolez
post Aug 4 2003, 18:39
Post #25





Group: Members (Donating)
Posts: 3474
Joined: 7-November 01
From: Strasbourg (France)
Member No.: 420



QUOTE (Gecko @ Aug 4 2003, 06:29 PM)
I've been especially surprised by WMA Pro - it's predecessors being so crappy (allthough apparently it could use some more tuning if we look at the death2 slipup).

I'm surprised too. The WM9pro encoder used for this test is the first never released. I never heard a baby singing so good for its birth.

Quality is amazing for half samples of this test, which is simply remarkable (first place, with sometimes great notation !). More tuning is needed - death5 is totally crap - the biggest flaw in my opinion on 12 samples, even when including lame.

Last but not least, wma9pro is gapless... AAC/MP4 isn't for the moment.
Go to the top of the page
+Quote Post

7 Pages V   1 2 3 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 23rd April 2014 - 10:53