64kbps public listening test, FINISHED Options

[rjamorim]

Is it possible to re-assignate a different place for the different plot ? I mean : from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.

Yes, but that will take longer, since I'll have to redo all the excel plots.

I'll do it later tonight, since I must now leave to Uni.

[smok3]

@rjamorim:

tnx for organizing and conducting the test.

flame verloren:
can't open http://audio.ciara.us/test/64test/comments...s/comments.html

B)

[AstralStorm]

Thank you rjamorim for making my results 'anon01'.

[rjamorim]

flame verloren:
can't open http://audio.ciara.us/test/64test/comments...s/comments.html

OK, I changed comments.html to index.html.

So please use this link now:
http://audio.ciara.us/test/64test/comments/

Hopefully it'll work :B

[rjamorim]

Thank you rjamorim for making my results 'anon01'.

Well, dude, you didn't specify if you wanted me to associate your results with your name...

Now you're "just another number"

[bond]

good idea calling it low/high anchor (perhaps you should explain than on top of the page what these are, which encoder, settings and of course bitrate

from left to right, the winner (HE-AAC) to loser (Real?). This may be useful, to see on which sample winner(s) fail(ed). Random position aren't useful in my opinion.

also nice
but i think it would be good if the order is the same in every plot, so perhaps the codecs should be ordered as they were judged in the overall results (so it is also very easy to see when a codec performed worse and when better)...

and one last optical thing:
if you do the plots again, plz write the codec names horizontally, perhaps use a smaller type size too because now the names are really too big imho (like half the size of the spreadsheet itself)

[guruboolez]

Glad to know anon1's name. It's one of the rare listener that spoke about reverberation of vorbis encodings.

[2Bdecided]

I was sad to be anon05 - I'll have to demand credit for my hard work next time - though I didn't work half as hard as you guys who did them all!

What about processed by rank rather than grade? Is that too much hard work?

Cheers,
David.

[ff123]

I was sad to be anon05 - I'll have to demand credit for my hard work next time - though I didn't work half as hard as you guys who did them all!

What about processed by rank rather than grade? Is that too much hard work?

Cheers,
David.

It should be easy to process by rank -- just run the raw data through the statistical tool again, this time using the friedman analysis.

ff123

Edit:
Both versions of abchr need to have a field for nickname (default nick is anonymous) and a checkbox to indicate whether the listener wants his comments associated with his nickname (just to be doubly sure about privacy); this would be written to a line in the results file.

This post has been edited by ff123: Sep 22 2003, 16:35

[c_haese]

OK, I changed comments.html to index.html.

So please use this link now:
http://audio.ciara.us/test/64test/comments/

Hopefully it'll work :B

It's not working for me. The page comes up, but none of the links on that page seem to work. Oh, and the title of that page says "AAC@128kbps test comments."

-Carsten

[Niknak]

Can you give us the actual bitrates or filesizes with each codec and for each of the samples? Are there any significant differences?

[de Mon]

I can't understand why didn't the best version of Vorbis used - GT3??? Why did realy hissy and not tuned version compared with quiet old and tuned products when there exists GT3?

[Continuum]

GT3 is only real tuned for high bitrates (>160 kbit). It would be no better at 64 kbit than the standard version.

[sony666]

the mp3PRO result is interesting.. too bad the format never took off due to the licencing nightmare I see the same fate for AAC coming.. unless there will be a free and superior "LAME aac" encoder, mp3 is gonna stay (yes, except the divx ripping ppl)

thanks roberto e.a. for organizing this

This post has been edited by sony666: Sep 22 2003, 19:00

[JohnV]

I can't understand why didn't the best version of Vorbis used - GT3??? Why did  realy hissy and not tuned version compared with quiet old and tuned products when there exists GT3?

The post 1.0 cvs version used includes actually some new tweakings for low bitrates, and as continuum said, it's gt3 which has nothing new for -q0.

[JohnV]

the mp3PRO result is interesting.. too bad the format never took off due to the licencing  nightmare I see the same fate for AAC coming.. unless there will be a free and superior "LAME aac" encoder, mp3 is gonna stay (yes, except the divx ripping ppl)

1. mp3Pro is no standard -> AAC/AAC-HE is
2. mp3Pro is completely proprietary -> there are several different AAC implementations
3. FAAC is getting better and it's free. It might not be state of the art, but it's not that bad it used to be. I think it's very soon and partly already ready to challenge at least Lame MP3.

There's still no doubt that mp3 will be around for a looong time..

[AstralStorm]

I can't read the comments from the site.

If I click any link on that page
after trying 20 times to connect and then getting always redirected
Mozilla says the site exceeds redirection limit.

This post has been edited by AstralStorm: Sep 22 2003, 20:16

[de Mon]

Thanks you for your replies, Continuum and JohnV. I had not known about that info.

[Dologan]

Heh, I find it weird that I usually scored MP3Pro higher than HE-AAC. Those two (apart from Lame) were generally the only ones that needed careful listening to single out, but once I got hold of the artifacts I usually found HE-AAC's to be a little bit more annoying than those of MP3Pro, although there were times where HE-AAC was amazingly difficult to ABX for me, whereas that didn't happen with MP3Pro
Another deviation of mine from the average is that often WMA was usually almost at the level of FhG 64 in terms of annoyance, separated from QT and Real...

My overall: LAME > MP3Pro, HE-AAC > Ogg > Real = QT > WMA > FhG

~Dologan

[ff123]

Results have been slashdotted:

http://slashdot.org/article.pl?sid=03/09/2...tid=141&tid=188

Get your comments and rebuttals in while the thread's still hot.

ff123

[AstralStorm]

My results:

CODE

LAME     MP3Pro   HE-AAC   Vorbis   LC-AAC   Real     WMA      FHG
 4.07     3.39     3.13     2.67     2.22     2.05     1.77     1.55

LAME is better than MP3Pro, HE-AAC, Vorbis, LC-AAC, Real, WMA, FHG
MP3Pro is better than Vorbis, LC-AAC, Real, WMA, FHG
HE-AAC is better than LC-AAC, Real, WMA, FHG
Vorbis is better than WMA, FHG
LC-AAC is better than FHG

[pseudoacoustic]

Great test Roberto. However, I also cannot access the comments page. Would it be too much to ask you to email them to me (the same email I sent you the results with)? Thanks.

[rjamorim]

Results have been slashdotted:

http://slashdot.org/article.pl?sid=03/09/2...tid=141&tid=188

Get your comments and rebuttals in while the thread's still hot.

ff123

"Hydrogenaudio has just wrapped up a listening test of various audio codecs at 64kbps."

That pisses me off. A lot.

Sure, it was "Hydrogenaudio" that stayed up till 5:30 AM sorting results, it was "Hydrogenaudio" that took shit on every issue that surfaced, it was "Hydrogenaudio" that screened 500+ results ditching the ones ranking the reference...

I'm not an asshole, but I think credit should be given where credit is due. Credits weren't given to ff123's test either, but at least it wasn't misrepresented as being conduced by HA.

sigh...

OK, rant off.


@pseudoacoustic: Check the 4th post in this same thread
http://rarewares.hydrogenaudio.org/rja/comments.zip

[ff123]

I always find stuff to reply to in the /. threads way too late. In the current thread there were a couple that got my goat (having to do with the legitimacy of subjective testing and the drawing of the confidence interval bars).

ff123

[Dologan]

That pisses me off. A lot.

Sure, it was "Hydrogenaudio" that stayed up till 5:30 AM sorting results, it was "Hydrogenaudio" that took shit on every issue that surfaced, it was "Hydrogenaudio" that screened 500+ results ditching the ones ranking the reference...

I'm not an asshole, but I think credit should be given where credit is due. Credits weren't given to ff123's test either, but at least it wasn't misrepresented as being conduced by HA.

sigh...

OK, rant off.

Don't be sad, Roberto. Who needs Slashdot appreciation, anyway? We all love you here at HA.org, we all know you're da man.

This post has been edited by dologan: Sep 23 2003, 04:21