Public Multiformat Listening Test @ ~64 kbps [March/April 2011], Nero & Apple HE-AAC, aoTuV Vorbis, CELT Options

[C.R.Helmrich]

I want to ask everyone. Before the test finish, which codec will win? Of course you may have to take a guess/bet or based on your experience/personal preference.

I have certain vague expectations about which codec will win, but I won't tell you (and I ask everyone else with knowledge about the codecs under test not to do so either) since this could influence the test.

A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris

This post has been edited by C.R.Helmrich: Mar 23 2011, 21:29

[IgorC]

Hello, Chris.

A (IMHO) more important question: Igor, are all session results excluded for which some hidden reference is graded with less than 5.0? Maybe you should mention the post-screening procedure so that listeners behave carefully enough.

You're right.
Here are the rules:

CODE

If at least one of the following errors takes place then the result will be INVALID.
1. graded the reference lower than 5.0
2. didn't grade the low anchor.

If one listener will submit INVALID result then he will be informed and will have one more and unique posibility to submit result for one particular sample.
Now if the listener will submit 3 or more invalid results then only ABX results will be accepted from him/her or directly rejected in abussive cases.

And about the resampling: Except for CELT, was the input or the output of the codecs resampled to 48 kHz?

Chris

The output of decoded (vorbis, aac) files was resampled.

This post has been edited by IgorC: Mar 23 2011, 21:45

[Fool_on_the_hill]

Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994?
The same result. Strange...

[LithosZA]

Tried to abchr.jar (+jdom.jar, looks-1.3.1.jar) from here http://www.hydrogenaudio.org/forums/index....st&p=701994?
The same result. Strange...

You could try uninstalling the 64bit version of JRE/JDK and try using the 32bit version of the java runtime instead.
Don't think it will make a difference, but it is worth a try.

[Sebastian Mares]

Your JRE is probably too new. Had this problem in my tests, too.

[IgorC]

Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924

Fool_on_the_hill, you can try it

CODE

The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.

This post has been edited by IgorC: Mar 24 2011, 16:56

[C.R.Helmrich]

Some relevant knowledge collected for the 128-kb test that didn't take place (yet...):

Instruction sheet for audio codec listening tests (high quality)

(German: eine deutsche Version der Anleitung liegt hier).

The best codecs in this 64-kb test can still be considered high-quality, so the description is applicable.

By the way, two posts above mine in that thread is a Mac binary of faad. In case any Mac user needs it. [Edit: unfortunately, not any more ]

Chris

This post has been edited by C.R.Helmrich: Mar 24 2011, 22:46

[Alex B]

Thanks for organizing this test.


To help the testers who do not have access to a Windows computer I created FLAC versions of the decoded samples and put them online on Dropbox.

I decided to pack the sample folders into six zip files (a compromise between storing them individually or in a single package). Each zip file contains five sample folders:

http://dl.dropbox.com/u/7690288/samples01-05.zip (48.6 MB)
http://dl.dropbox.com/u/7690288/samples06-10.zip (67.3 MB)
http://dl.dropbox.com/u/7690288/samples11-15.zip (67.6 MB)
http://dl.dropbox.com/u/7690288/samples16-20.zip (52.7 MB)
http://dl.dropbox.com/u/7690288/samples21-25.zip (61.6 MB)
http://dl.dropbox.com/u/7690288/samples25-30.zip (67.7 MB)

I compressed the original, decoded 48 kHz WAVE test samples with FLAC 1.2.1 (-8) and verified the resulting FLAC files with foobar2000's bit compare tool.

The sample folders are included in the zip files. You must unzip the folders and decompress the FLAC files back to WAVE before they can be used in the test. The new filename extension must be .wav instead of .flac, otherwise the filenames must be unaltered. The folders must be placed in the ABC-HR folder:

ABC-HR > Sample01 > the six uncompressed .wav sample files
ABC-HR > Sample02 > the six uncompressed .wav sample files
ABC-HR > Sample03 > the six uncompressed .wav sample files
etc.

The 30 sample folders (= 180 WAVE files) will need about 712 MB of disk space.

You can have the FLAC files in the same folders if you don't want to delete them. They will just occupy some additional disk space (about 366 MB for all 30x6 samples).


Please don't abuse the bandwidth that is available for this Dropbox account. It makes no sense to download these versions if you can use the smaller original package (or packages) and let the bat file (or files) automatically process the test samples.

EDIT

If anyone else wants to host these FLAC versions I can provide also the individual sample packages and all samples in a single package (PM me).

This post has been edited by Alex B: Mar 25 2011, 10:46

[IgorC]

Thank you, AlexB
That is exactly what people asking with Linux and OS X.

[IgorC]

Here is more clear interpretation of rules:

Participants who don't want to worry too much about the grading rules
can simply ignore them. Listeners should do their best to rank the samples and
be careful to identify the hidden references. Listeners should ABX tests
they are at all unsure.

1) If the low anchor is not graded, or if any hidden reference is graded
below 4.5-5^(*1a) (see App.) the result is INVALID.

^*1a It will be discussed with people who have experience of conducting public tests when final results will be ready.

2) For each sample with a ranked reference or an ungraded low anchor the
listener will have a single chance to submit a replacement test run for
that sample. The replacement test must cover all codecs, not just the
codecs with the ranked reference. (This also covers cases where the
reference is ranked but still at or above 4.5)

3) If a listener submits more than 2/15 (4 for 30 samples submitted) INVALID
results then only ABX results will be accepted, or the listener will be excluded
completely in cases of apparently abusive behavior.


App. These rules aren't extremely strict in order to allow for simple human
error while still excluding careless participants.

A stricter procedure to exclude all ranked references risks a systemic
bias against any codec which are very good on a few samples and thus
subject to more reference confusion by causing those samples to be excluded
and weighing the test towards other samples.

[Alex B]

I just noticed that this test is not in the forum news.

[IgorC]

I just noticed that this test is not in the forum news.

Oh, it will be great to have it there

This post has been edited by IgorC: Mar 26 2011, 02:26

[Canar]

Create a post in the News Submissions forum and I'll see what I can do.

[Alex B]

Actually the first post in this thread is the news announcement.

During the past tests we usually had 1) a pretest discussion in "Listening Tests", 2) a news thread in which the open test was announced and commented and finally 3) a news thread in which the results were published and commented.

For example, the previous Multiformat 64 kbps test:

1) pretest discussion: http://www.hydrogenaudio.org/forums/index....showtopic=53134
2) announcement: http://www.hydrogenaudio.org/forums/index....showtopic=56397
3) results: http://www.hydrogenaudio.org/forums/index....showtopic=56851

This time we did not have a pretest discussion (at least not in a single dedicated thread). This thread is already the news & comments thread.


To: Admins/Mods

- Could you simply create a short (perhaps slightly edited) news article from the first post that would point to this thread?
or
- Could you move this thread to "Validated News", create a short (perhaps slightly edited) news article from the first post that would point to this thread and perhaps leave here, in "Listening Tests", a short note about the moved thread and a link to the new location?

[IgorC]

AlexB has a point. We need the announcement of this test on News Section. This way more members will participate.

Some members have already sent their results.
it's on approximately half way to get enough results.
Hopefully people will have time to conduct test during this weekend.

Hydrogenaudio community, please, participate in test!

This post has been edited by IgorC: Mar 26 2011, 16:34

[IgorC]

BTW, If you don't want to download separate packages there is All-In-One package (ABC-HR and all samples)
It should be more easy.

ABC-HR_bin_and_samples (All-In-One)

This post has been edited by IgorC: Mar 27 2011, 17:21

[LithosZA]

The test was really fun. I was supprised how nearly transparent some of these samples were at 64Kbit/s.
I would encourage that more people take this test.

[Fool_on_the_hill]

Thank you, Sebastian.

There are two versions of ABC-HR for Java 5 and 6. http://www.hydrogenaudio.org/forums/index....st&p=683924

Fool_on_the_hill, you can try it

CODE

The Java 5 build should also run fine on Java 6 machines. In both cases just double-click the jar to start.

still the same error

[IgorC]

still the same error

It's a sort of particular incompatibility. I have the same Windows 7 x64 and Java 5/6 packages work fine on it. Also have checked on Window XP machine. No problem.

This post has been edited by IgorC: Mar 28 2011, 19:01

[IgorC]

Some people wait for the last day to send their results. That's ok.
But it might be the case that someone can forget to send the results until 30th of March.
So if You have already even a few results send them soon.

Of course, the last moment results will be accepted during 30th and 31st (~01.00 of GMT -3.00) of March.

As far as I can see all participants have received the answer. Thank You to All who participate.



This post has been edited by IgorC: Mar 29 2011, 20:23

[Sebastian Mares]

I usually got a decent number of results on the last day from people who wanted to test more but didn't make it in time.

[markanini]

I've had fun doing the tests but please make the samples shorter next time so that the participant can more easily establish points in the audio to focus on and avoid fatigue. For example sample 15 starts of with a female voice and then switches to a male voice, forces you to acquire a split personality

[lessthanjoey]

I'm extremely impressed by all four (non low anchor) codecs. Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case

[romor]

Two in particular appear essentially transparent to my ear across the entire set of samples. I can only hope that those two are consistent and it's not a cycling of which is best in each case

Notices the same - I downloaded bundle out of curiosity and 2 codecs perform superbly on couple of samples I tested

2m to midnight - is there minimum amount of samples that can be reported? Like 1/3 of all or 10 for reckless ones?

edit: I realised now (readme.txt) that it's not necessary to test all samples, sorry for wasting space

This post has been edited by romor: Mar 30 2011, 06:52

[IgorC]

The test is extended until the 10th of April.
We still need more results for some particular samples.
I will post the numbers of those samples in a few minutes.

This post has been edited by IgorC: Mar 30 2011, 19:40