I'm new to this and have a quick procedural question. I successfully ABX'd one of the test samples, and I was wondering if this is all that is required when submitting results. I've noticed that some people don't include information from the Left/Right sidebar test - can I just submit my ABX results or do you also need the Left/Right sidebar test info as well?

Sure, the sidebars are very important. They are there so that you can rank codecs and define how they sound compared to each other.

This test is not about finding out if the samples sound different from the original (ABX). It's about finding out how bad each one sounds compared to the original.

Thanks for the reply (I feel dumb now )

Well at least I know I can ABX one of the samples!

Whoaa... It takes so long time to do all... I think I spend too much time on abxing or something... Is it ok to submit results without abx?

(Uhm.. correction: I hope it's ok... without "?")

Sure. If you are confident you know which sample is the reference and which is encoded, there's no need to ABX them.

Rjamorim, why I get to know about the test now and not when it did start?
You need to create some automailer or mailing list...

I just got the big rar through BitTorrent. It seems to contain only wav files. No flac, no mp3.

Good idea!

Maybe Dibrom can set up Mailman for us



No idea, I didn't create it. I'll check it out, but I guess you can just decompress it to the same dir you decompressed ABC-HR_bin.zip into and try to load the config files.

I didn't create this file either, for the record. I found it in the rarewares directory along with the other samples, so I assumed rjamorim created it.

hehe. These are the 12 samples uncompressed. I keep them there because often someone asks for them, and also sometimes I have to ask people to encode the samples for me.

I think that torrent can be safely deleted

Perhaps it should be replaced with the correct files. It's more convenient to work with one URL from my point of view.

I have a small question:

How do you consider an abc/hr result without all contenders ranked?
As an example, I have been unable to hear differences between some samples and originals on the 128k test, so I left the slider on its original position (5). However, the resulting saved file does not mention those ones.

All 5's would be inferred by the processing program. I agree it can be disconcerting to see a blank result, though.

As an interesting trivium, the general comments button was added to satisfy what seems to be a basic human urge for people to have something to show for their efforts -- quite a few people in earlier listening tests would pull down sliders, to 4.9 for example, just so that they could comment about how they couldn't hear differences.

ff123

There should be a button 'I've tested this' somewhere.
This would allow for incomplete results.

Appearantly not. After fooling around with some firmware variables, my mac is in an unbootable state. Sorry about that

Gogo posts moved here: http://www.hydrogenaudio.org/show.php/showtopic/17892

I use the "General Comments" button for exactly that. If any test groups are perceptually transparent to me, then I point out which ones in the general comments, to explain any "5.0"s in my test results (though it's probably not necessary).

---------

Well, I'm moving this weekend and starting a new job next week, so with everything I've had to do in preparation, I was only able to complete testing of three samples.

My results are submitted.

Hello. As I start writing this post, there are still 25 hours until the test closure.

And I bring bad news.

So far, this is the amount of results I received:

Sample 01: 08
Sample 02: 07
Sample 03: 07
Sample 04: 06
Sample 05: 08
Sample 06: 05
Sample 07: 07
Sample 08: 05
Sample 09: 07
Sample 10: 05
Sample 11: 07
Sample 12: 12

Therefore, Waiting is so far the only sample with enough results to generate statistically valid rankings. And I would consider that the bare minimum, I would be more comfortable working with at least 15 results.

Then, we have some options:

1 - Pray to God that my mailbox will be flooded with results tomorrow, so that the results can be published on schedule
2 - Delay the test closure in one week (and NO MORE than that)
3 - Give up and call it quits, and hope there will be more interest in the AAC test.

So, what do you think?

Best regards;

Roberto.

I will increase each of those counts by 1 at the last minute, but I can't do more than that .

The test was very hard on most samples, its tedious when you try for hours but cant ABX anything

I sent results for 2,7 and 8, all others (even this #12 "waiting" stuff") were transparent to me.
Sorry to hear about the lack of participation after all your work roberto :/

5 more test result files sent. I just can't listen to stravinskij though. Drives me mad...

I opt for nr 2. You can do nr 1 as well, but I doubt it helps

Hopefully your post will bring some more people who get bad conscience for not participating (or only submitting partial results).

Yeah. I agree, 2 has the highest potential of leaving everyone satisfied at the end.



I hope so too

I would contribute (in fact I downloaded sample 9 and the ABC/HR program), but my effort is futile without at least a medium-end pair of headphones.
Unless my ears are really that bad.

Well, you sure picked the most difficult sample to start with. Try nr 12 or 7 - they should be easier... Or if you need to borrow a pair of Senns (hd-600), I guess that could be arranged too.

I'll try to work on them. Thanks for the suggestion.

I would vote for a 1 week extension if needed.

sample 1 kicked my ass i was about to give up, but i tried 11 because it had a cool name and FINALLY felt better because i could at least ABX between 128 and CDDA sample...

Well, that's bad news indeed. I really hope some more people will send results, because judging from my own results this test has the potential for some surprises.

I think the difficulty level of this test is generally a bit overestimated. After taking this test, I'm almost convinced that in this bitrate range, you will find a glitch in every piece of encoded music, if you just search hard enough. As I mentioned in an earlier post, training makes a huge difference. If you don't hear differences in some samples at first, try some of the easier ones (personally I found 2,7, and 8 quite easy), and you will most likely be able to identify some of the harder samples later on.
I'm certainly no golden-ear, but in the end I was nonetheless able to identify every single encoded sample in this test (and I don't even own any headphones, so that doesn't count as an excuse either ).

yeah one more week is a great idea!

perhaps it makes sense if people post if they are planning to send results: i will do!

maybe its also a good idea to post on slashdot!

I don't find trying to identify the encoded sample to be all that difficult (although, yes some are harder than others). The thing I find the most difficult and time consuming is subjectively rating (as objectively as possible, ofcourse) one encoded sample above another. To do this as objectively as possible I believe you need to ABX every encoded sample against another to again prove that you can hear any difference in them. I'll concentrate on one time range compare them all, only to discover that the codecs behave differently in another part of the sample which makes the work I've done semi-redundant.

for everyone who has troubles finding differences between the samples:

nothing wrong with that, if you dont hear differences the files are transparent for you!
simply send your 5 of 5 points voted result files


but its important not to give up too fast, even i, with my bad hearing, can spot some differences after listening very carefull to the samples

Yes, you're certainly right. Though, I don't think you really need to do ABX tests of each sample against each other (just as you don't need to provide an ABX test of a sample that's just plain horrible against the original). For example, I think it's unnecessary to ABX some 1.0 sample against a 4.0 sample (as an extreme example), but it is certainly a good idea to do ABX tests of samples you're planning to rank similar. There were a few instances where I wanted to rate some sample a bit lower than another and then discovered in ABX that I can't really hear the difference reliably.
Still, as you suggested, only ABXing one encoded sample against another isn't even the really difficult part, since often the artifacts are different enough to ABX them reliably. What I find really difficult is to decide which artifact is worse. For example, I'm quite sensitive about high frequencies, so if some encoder lowpasses excessively, I can usually pick it out immediately. Still, I don't think (moderate) lowpassing is such a horrible artifact, compared to some of the stranger artifacts, which might not be as obvious at first listen, but are really disturbing once you hear them.

I hope that 1 week delay would be right thing to do. I have trouble to find time to this test, but i could send my results next week.

I agree with evereux and schnofler. Until now I only found time for the first sample and it wasn't very hard to hear the difference between reference and encode... but ranking them is really difficult.

If the test would be extended by another week, I hope to contribute some more results, but I don't think I will do the complete package.

OK, since everybody seems to agree extending the test is better, I'll make it run until next weekend.

I'll update the test pages and posts to reflect this.

Thanks for your opinions.

Regards;

Roberto.

Yes, you're right there.

I'm really glad you're giving it another week, I REALLY want to see the results, I find this fascinating. I started but found it too time consuming, but since you need more input, I'll try to give it a shot over the course of the week a little at a time. Best chances for me are if you run it through next weekend at least. I've got some work related stuff going on and a wife and two kids at home, time is precious!

One thing this seems to indicate is that 128kb MP3s have really improved over time.

Thanks for the extension, I've just found the time to finish the test.
By the way, you should enable e-mail notifications, because now I don't know
whether my present results have reached you.

Well, Yahoo is fairly reliable, in my experience.

By notification, you mean doing auto-reply?


Also, something you should know:

This extension has messed my schedule a little. I have a trip planned to start tomorrow morning (Stanley Jordan will play tomorrow in São Paulo! ), from there I'll go to Rio de Janeiro to visit my brother, and will stay there until next week. So, you should be warned that the test WILL END on Sunday - no results will be accepted after that date, we need some seriousness here - but the results will only be published when I return home, probably tuesday or wednesday. Sorry :/

Regards;

Roberto.

Maybe Yahoo indeed is stable, but I recall bad things happening to my results of 128k extension test.

BUG REPORT:

Java ABC/HR crashes with both latest Sun's Java 1.4.1 and 1.4.2,
to reproduce play the sample to nearly the end (less than buffer size) and then switch to another file (with fast switching on and loop off). It starts to eat 100% CPU then.

Hrm?

Sorry, but I honestly don't remember what went wrong.

In the 64kbps test your results were processed without any problem. Maybe there was a problem in the 128kbps extension test because you copy-pasted the results to the e-mail body instead of attaching them, but I don't think that lead to any problem, really. And you didn't participate in the AAC test...

ok guys, today is the last day to send in your results

everyone who till now didnt start to download and listen to the samples, get your ass up and do it

we have more than 10000 members here, note that i will ban each one of them who didnt join the listening test!

so do it, you lazy bastards!

Argh, wish I'd seen this sooner. I'm very interested in the results of this test for streaming purposes.

I'd do it myself now, except I don't have a sound card to do it on . So yeah, I hope there are enough results to make it a decent test.

Hello.

@Bond: Sorry, but I was mistaken in my schedule. My tests always end on a Sunday, so actually, tomorow is the last day for result submission. I'll receive results until midnight in Brazilian Standard Time (Brasilia), and then close the test.


Also, please keep in mind results will only be published after I return from Hell (Rio de Janeiro), so expect them to appear around Wed-Thu next week.


I'm fairly sure we have enough results now, but given the public response has been less than enthusiastic even after the extension (yes, you read that right, I got few results this week), I would like to urge anyone that tested even one or two samples to send whatever you got. I need all help available ATM.


Thank-you very much.

In the hopes that the AAC test will draw more interest...

Roberto Amorim.

ic


np enjoy your trip to hell and back again


i think so,
imo the mp3 test pretty much shows the profile of the ha community
mp3 is still the most widely used audio codec but with the exception of lame we dont really care
judging from this i think aac will draw more interest

Hmm, I really hope this is not the case. After all, the HA community also prides itself on its objectivity. Refusing to test any other mp3 codec in the firm believe that nothing will ever beat LAME hardly fits into that profile.

yeah exactly
i hope that today some more people join, its still enough time to do it cause you dont need really a lot of time to test the samples yourself!

its a good thing to help bringing transparency into the codec jungle

so everyone who is not going skiing or is lying in the sun on the capacabana start listening

I hope that some of the samples aren't in AAC test. I couldn't even test all those samples in MP3 test.

yep its not easy to spot differences when you dont have practice do to it
but they are there, and hearable also for people not having golden ears, like me

a good tactic to do it is to listen only to a small part of the samples (some seconds) where you think you already heard a difference maybe
than you maybe can listen only to specific instruments or tones whether they are the same in every sample, for example listening to the hi hat sounds its pretty easy to detect whether there are differences or not
also you have to listen to one sample (meaning one codec vs the source) various times, its not enough to listen to it only once or so

as i said it has a lot to do with getting a feeling for what you have to listen to to spot errors
but once you have little practice its very easy to hear them