Multiformat listening test @ ~64kbps: Results, Results and post-test discussion |
Multiformat listening test @ ~64kbps: Results, Results and post-test discussion |
 Apr 12 2011, 00:40
Post
#1
|
|
|
Group: Members Posts: 1315 Joined: 3-January 05 From: Argentina, Bs As Member No.: 18803  |
The test is finished, results are available here:
http://listening-tests.hydrogenaudio.org/igorc/results.html Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC. |
 |
 
|
 |
|
Apr 13 2011, 04:42
Post
#2
|
|
 Group: Developer Posts: 191 Joined: 8-July 03 Member No.: 7653 |
QUOTE (IgorC @ Apr 11 2011, 16:40)  The test is finished, results are available here: http://listening-tests.hydrogenaudio.org/igorc/results.html Summary: CELT/Opus won, Apple HE-AAC is better than Nero HE-AAC, and Vorbis has caught up with Nero HE-AAC. Hey all, you may be interested in a presentation of the results that I did which summarizes some of the results I found while looking at the data. http://people.xiph.org/~greg/opus/ha2011/ I'll probably be adding a bit to it over the next few days as I analyze some more things. I'm posting the link here first before linking it up elsewhere in the hopes that if I've done something obviously dumb that the folks here will cluestick me. Thanks! |
|
|
|
|
Apr 13 2011, 22:46
Post
#3
|
|
 Group: Developer Posts: 618 Joined: 6-December 08 From: Erlangen Germany Member No.: 64012 |
Thanks, Garf, for the plot! And thanks, Christoph, for clearing up your (very unfortunate) headphone jack issue.
QUOTE (NullC @ Apr 13 2011, 05:42) Hey all, you may be interested in a presentation of the results that I did which summarizes some of the results I found while looking at the data. http://people.xiph.org/~greg/opus/ha2011/ I'll probably be adding a bit to it over the next few days as I analyze some more things. I'm posting the link here first before linking it up elsewhere in the hopes that if I've done something obviously dumb that the folks here will cluestick me. Thanks! *raising my hand* Please don't write "HE-AAC" but something like "two popular HE-AAC encoders" in QUOTE Now, these results demonstrate that Opus's performance against HE-AAC, one of the strongest (but highest-latency) codecs at this bitrate, are very strong, besting its quality on the majority of individual audio samples and receiving a higher average score overall. Sorry, but being an AAC developer I have to stress this. The nero and Apple encoders have never been proven to be the best encoders around. Chris -------------------- If I don't reply to your reply, it means I agree with you.
|
|
|
|
|
Apr 14 2011, 00:58
Post
#4
|
|
|
Xiph.org Speex developer Group: Developer Posts: 430 Joined: 21-August 02 Member No.: 3134 |
QUOTE (C.R.Helmrich @ Apr 13 2011, 17:46) Sorry, but being an AAC developer I have to stress this. The nero and Apple encoders have never been proven to be the best encoders around. I don't know much about all the HE-AAC implementations around, but keep in mind that we're talking about two implementations that have matured over several years compared to the Opus *reference* implementation that is still very immature in terms of tuning. So if anything, I think the comparison would be biased against Opus. Not to mention that Opus makes quite a few quality-affecting compromises to achieve its low delay. To be fair, I'm probably just as surprised as you are with these results. |
|
|
|
IgorC Multiformat listening test @ ~64kbps: Results Apr 12 2011, 00:40
Garf If someone can assist with a bitrate table or per-... Apr 12 2011, 01:02 Garf Oh, and given that Opus is open sourced, if one of... Apr 12 2011, 01:06 AllanP I just wonder one thing, when the Vorbis encoder w... Apr 12 2011, 01:14
Garf QUOTE (AllanP @ Apr 12 2011, 02:14) I jus... Apr 12 2011, 01:15 AllanP QUOTE (Garf @ Apr 12 2011, 02:15) You can... Apr 12 2011, 01:22 romor Congratulation to CELT/Opus!
I wanted to com... Apr 12 2011, 03:08 IgorC I think the results of lessthanjoey and AlexB are ... Apr 12 2011, 03:50 googlebot I'm stunned by the CELT/Opus results! I wo... Apr 12 2011, 08:06 NullC QUOTE (googlebot @ Apr 11 2011, 23:06) I... Apr 14 2011, 05:00 saratoga QUOTE (NullC @ Apr 14 2011, 00:00) We als... Apr 14 2011, 06:29 NullC QUOTE (saratoga @ Apr 13 2011, 22:29) Is ... Apr 14 2011, 08:30 Garf QUOTE (NullC @ Apr 14 2011, 06:00) Low-la... Apr 14 2011, 12:19 jmvalin QUOTE (Garf @ Apr 14 2011, 07:19) Is the ... Apr 14 2011, 14:04 NullC QUOTE (Garf @ Apr 14 2011, 03:19) QUOTE (... Apr 14 2011, 17:47 C.R.Helmrich QUOTE (NullC @ Apr 14 2011, 18:47) The sw... Apr 14 2011, 23:39 jmvalin QUOTE (C.R.Helmrich @ Apr 14 2011, 18:39)... Apr 15 2011, 05:49 Alex B Thanks guys! Interesting results.
One note t... Apr 12 2011, 12:59 Garf QUOTE (Alex B @ Apr 12 2011, 13:59) I got... Apr 12 2011, 13:53 NullC QUOTE For processing the result .txt files with ch... Apr 12 2011, 14:48 Garf QUOTE (NullC @ Apr 12 2011, 15:48) Sounds... Apr 12 2011, 14:59 Alex B QUOTE (Garf @ Apr 12 2011, 16:59) But the... Apr 12 2011, 15:09 Alex B QUOTE (NullC @ Apr 12 2011, 16:48) Sounds... Apr 12 2011, 15:02 NullC QUOTE (Alex B @ Apr 12 2011, 07:02) QUOTE... Apr 12 2011, 15:17 motion_blur QUOTE (Alex B @ Apr 12 2011, 16:02) QUOTE... Apr 12 2011, 16:15 NullC QUOTE (motion_blur @ Apr 12 2011, 08:15) ... Apr 12 2011, 17:54 motion_blur QUOTE (NullC @ Apr 12 2011, 18:54) QUOTE ... Apr 12 2011, 19:42 Alex B For comparison I uploaded a rar package of my ... Apr 12 2011, 14:14 Garf QUOTE (Alex B @ Apr 12 2011, 15:14) For c... Apr 12 2011, 14:49 Alex B QUOTE (Garf @ Apr 12 2011, 16:59) But the... Apr 12 2011, 15:35 Garf QUOTE (Alex B @ Apr 12 2011, 16:35) QUOTE... Apr 12 2011, 15:42 Alex B Regarding the bitrate table,
I guess that CELT/Op... Apr 12 2011, 16:14 NullC QUOTE (Alex B @ Apr 12 2011, 08:14) Regar... Apr 12 2011, 18:10 IgorC Yes, I was too strict. Sorry about it.
Some of th... Apr 12 2011, 18:13 motion_blur QUOTE (IgorC @ Apr 12 2011, 19:13) Yes, I... Apr 12 2011, 20:09 NullC QUOTE (motion_blur @ Apr 12 2011, 12:09) ... Apr 13 2011, 00:59 motion_blur QUOTE (NullC @ Apr 13 2011, 01:59) QUOTE ... Apr 13 2011, 10:06 markanini I figured ratings would vary between testers depen... Apr 12 2011, 18:52 NullC QUOTE (markanini @ Apr 12 2011, 09:52) I ... Apr 12 2011, 20:57 lessthanjoey I've done some more testing with headphones af... Apr 12 2011, 19:51 C.R.Helmrich Thanks for organizing the tests, guys! Sorry f... Apr 12 2011, 20:11 motion_blur QUOTE (C.R.Helmrich @ Apr 12 2011, 21:11)... Apr 12 2011, 20:41 Garf QUOTE Please provide the number of valid results (... Apr 12 2011, 21:44 C.R.Helmrich Sorry, Christoph, can't reproduce it. What you... Apr 12 2011, 22:12 Garf QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)... Apr 12 2011, 23:37 motion_blur QUOTE (C.R.Helmrich @ Apr 12 2011, 23:12)... Apr 13 2011, 00:41 NullC QUOTE (C.R.Helmrich @ Apr 12 2011, 11:11)... Apr 12 2011, 21:47 IgorC motion_blur,
You can download the results of all... Apr 12 2011, 20:21 _mē_ Some presentation suggestions:
1. Codec versions a... Apr 12 2011, 20:43 IgorC QUOTE (_mē_ @ Apr 12 2011, 16:43) Some pr... Apr 12 2011, 20:46 Alex B QUOTE (_mē_ @ Apr 12 2011, 22:43) 2. Link... Apr 12 2011, 21:06 Alex B QUOTE (Alex B @ Apr 12 2011, 23:06) QUOTE... Apr 13 2011, 11:38 NullC QUOTE (Alex B @ Apr 13 2011, 02:38) The b... Apr 13 2011, 12:41 Garf QUOTE (NullC @ Apr 13 2011, 13:41) QUOTE ... Apr 13 2011, 12:54 Alex B QUOTE (NullC @ Apr 13 2011, 14:41) Any id... Apr 13 2011, 14:46 NullC QUOTE (Alex B @ Apr 13 2011, 05:46) QUOTE... Apr 13 2011, 22:48 Garf QUOTE (_mē_ @ Apr 12 2011, 21:43) Some pr... Apr 12 2011, 22:24 Alex B Here is the raw data for a bitrate table. The bitr... Apr 12 2011, 20:30 IgorC Thank you for your help, AlexB. If you can do the ... Apr 12 2011, 20:44 C.R.Helmrich Christoph, do you mean the slightly washed out bas... Apr 12 2011, 20:52 motion_blur QUOTE (C.R.Helmrich @ Apr 12 2011, 21:52)... Apr 12 2011, 21:11 C.R.Helmrich QUOTE (motion_blur @ Apr 12 2011, 22:11) ... Apr 12 2011, 21:38 motion_blur QUOTE (C.R.Helmrich @ Apr 12 2011, 22:38)... Apr 12 2011, 21:56 IgorC I've checked. The decoder on Christoph's s... Apr 12 2011, 20:54 Alex B Here's the bitrate table:
In Excel format:
... Apr 12 2011, 22:20 saintdev QUOTE (Alex B @ Apr 12 2011, 14:20) Here... Apr 12 2011, 23:21 Garf QUOTE (NullC @ Apr 13 2011, 05:42) Hey al... Apr 13 2011, 08:27 IgorC QUOTE (Garf @ Apr 13 2011, 04:27) One thi... Apr 14 2011, 05:57 Garf QUOTE (C.R.Helmrich @ Apr 13 2011, 23:46)... Apr 13 2011, 22:57 C.R.Helmrich QUOTE (Garf @ Apr 13 2011, 23:57) From th... Apr 13 2011, 23:06 Garf QUOTE (jmvalin @ Apr 14 2011, 01:58) I do... Apr 14 2011, 09:13 jmvalin QUOTE (Garf @ Apr 14 2011, 04:13) You are... Apr 14 2011, 11:41 Garf The result page is now updated with per-sample gra... Apr 13 2011, 12:41 mixminus1 Thanks much to all for their work in both setting ... Apr 13 2011, 14:55 Garf QUOTE (mixminus1 @ Apr 13 2011, 15:55) Th... Apr 13 2011, 15:04 mixminus1 :facepalm: Good God...
Thanks, Garf, I was scour... Apr 13 2011, 15:15 romor @Garf: can you please reupload results you posted ... Apr 13 2011, 16:35 Garf QUOTE (romor @ Apr 13 2011, 17:35) @Garf:... Apr 13 2011, 18:59 IgorC AlexB, thank you for bitrate verification.
I real... Apr 13 2011, 17:12 pdq Whether or not classical should be considered to b... Apr 13 2011, 17:51 IgorC QUOTE (pdq @ Apr 13 2011, 13:51) Whether ... Apr 13 2011, 18:01 pdq QUOTE (IgorC @ Apr 13 2011, 13:01) QUOTE ... Apr 13 2011, 18:53 romor file: http://people.xiph.org/~greg/opus/ha2011/2..... Apr 13 2011, 19:28 NullC QUOTE (romor @ Apr 13 2011, 10:28) file: ... Apr 13 2011, 19:53 IgorC Bitrate verification on my set of albums:
http:/... Apr 13 2011, 21:45 NullC QUOTE (IgorC @ Apr 13 2011, 13:45) Bitrat... Apr 21 2011, 19:15 jmvalin QUOTE (NullC @ Apr 21 2011, 14:15) QUOTE ... Apr 21 2011, 19:53 Garf QUOTE Has anyone ever seriously blind-tested e.g. ... Apr 14 2011, 00:10 C.R.Helmrich QUOTE (Garf @ Apr 14 2011, 01:10) I'm... Apr 14 2011, 10:46 .alexander. The second graph seems to be consistent with ... Apr 15 2011, 12:40 jmvalin QUOTE (.alexander. @ Apr 15 2011, 07:40) ... Apr 15 2011, 15:17 Xanikseo QUOTE (IgorC @ Apr 13 2011, 21:45) Bitrat... Apr 20 2011, 16:34 Garf QUOTE (Xanikseo @ Apr 20 2011, 17:34) Igo... Apr 20 2011, 18:02 IgorC QUOTE (Xanikseo @ Apr 20 2011, 12:34) Igo... Apr 20 2011, 18:19 Zarggg QUOTE (Xanikseo @ Apr 20 2011, 11:34) EDI... Apr 20 2011, 19:09 Xanikseo QUOTE (Zarggg @ Apr 20 2011, 19:09) QUOTE... Apr 20 2011, 21:20 IgorC NullC,
h*tp://www.mediafire.com/?s7i9usu2qr27pcg
... Apr 21 2011, 23:52 |
|
Lo-Fi Version | Time is now: 24th May 2013 - 06:42 |
