Public, Multiformat Listening Test @ 64 kbps - OPEN Options

[Whelkman]

I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)

I forgot about needing to manually select sound source; that was the first workaround I instituted. I haven't experienced clicks, but the audio doesn't always start/stop in exactly the same place. The latter is known; I saw it mentioned somewhere.

Speaking of bugs, your handle breaks quoting.

[Sebastian Mares]

Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.

[elmar3rd]

and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard),

I can confirm this (JRE Version 1.6.0 (Build 1.6.0_02-b05 / Win XP Pro SP2). But I can't confirm any other problems with ABC-HR neither in this test nor in the previous three or four tests.

This post has been edited by elmar3rd: Aug 13 2007, 17:44

[ff123]

The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).

N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec...

Guru plays hard too. I can't imagine ABXing all files. In fact, I didn't ABX any of them. If I think I might not be hearing a true difference I just give it the benefit of the doubt and don't pull the slider down.

ff123

[Alex B]

Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.

I have noticed this too. In my case it is enough to close the cropped "New ABC/HR Test..." window and reopen it. On the second time I get the complete window. I don't need to restart the program.

(XP SP2, jre1.6.0_02)

This post has been edited by Alex B: Aug 13 2007, 17:56

[Sebastian Mares]

Maybe a mod can split this and create a new thread called "ABC/HR Bitching" or something.

[muaddib]

Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.

I have noticed this too. In my case it is enough to close the cropped "New ABC/HR Test..." window and reopen it. On the second time I get the complete window. I don't need to restart the program.

Really?! I will have to try your solution. I was up to now always restarting application like Mares.

[Sebastian Mares]

BTW, you can take your time on Wednesday because the test will end around 20:00 since I am in the new apartment fixing things and talking to various mechanics (or what is the correct translation of the German "Handwerker"?).

[StillIll]

BTW, you can take your time on Wednesday because the test will end around 20:00 since I am in the new apartment fixing things and talking to various mechanics (or what is the correct translation of the German "Handwerker"?).

Handyman?

I'm looking forward to seeing the results of this test

This post has been edited by StillIll: Aug 13 2007, 22:22

[robert]

or what is the correct translation of the German "Handwerker"?

craftsman / craftswoman

[guruboolez]

I hope you will not get bored until you get to samples 7.-18.

Samples 1 to 9 are OK. I did 50% of the job and I have 44 hours to finish the 9 others.

Guru plays hard too. I can't imagine ABXing all files. In fact, I didn't ABX any of them. If I think I might not be hearing a true difference I just give it the benefit of the doubt and don't pull the slider down.

At this bitrate I've no problem to ABX these encodings. I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds - I will see it on log files in two days). It's not very stressing and I can easily afford this for all samples. It would be very different at 128 kbps.
I only failed for one sample (bibilolo, the 7th IIRC).

__
I noticed that for most samples I already tested two are sharing the same kind of artefact: a grainy texture I only heard in the past with SBR (HE-AAC & MP3Pro). So I guess that the second one is WMAPro and that it uses a technology which seems very close to Coding Technologies' one.

[muaddib]

At this bitrate I've no problem to ABX these encodings. I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds - I will see it on log files in two days). It's not very stressing and I can easily afford this for all samples. It would be very different at 128 kbps.
I only failed for one sample (bibilolo, the 7th IIRC).

Sample7 was the easiest one for me. I could even ABX (8/8) high anchor, which on most other samples I could not (at least I think it was the high anchor that I can not ABX).

[halb27]

... I can ABX most of them in less than a minute (some of them were probably done in ~20 seconds ...

Guess this means even you wouldn't easily hear problems in normal listening situations outside of abxing. Not talking about most of us with our common ears.
Sounds very good for these codecs at such a low bitrate.

[guruboolez]

Guess this means even you wouldn't easily hear problems in normal listening situations outside of abxing.

I was maybe not very clear. I can ABX (8/8...12/12) most samples in a few seconds. It's only possible if obvious artefacts are audible (i.e. when quality sounds usually poor compared to the reference). For me 64 kbps are still unpleasant even on "normal" listening conditions ("normal" has of course to be defined... as well as "unpleasant": the ears adapt themselves very fast to poor listening conditions such as AM radio, etc...).
I would say (from my own experience) that a huge quality gap exists between this quality and the one obtained by the same competitors at 128 kbps. But without a solid experience in artefacts hunting I'm pretty sure that this quality would be amazing to my ears.

This post has been edited by guruboolez: Aug 14 2007, 12:31

[muaddib]

I would say (from my own experience) that a huge quality gap exists between this quality and the one obtained by the same competitors at 128 kbps. But without a solid experience in artefacts hunting I'm pretty sure that this quality would be amazing to my ears.

It is great that you are helping us with this test Big thanks from my side!
Also have in mind that results from this test will also help in improving Nero AAC at high bitrates.

[guruboolez]

Sebastian has now one new result for each sample. All must be valid (successful ABX or 5.0). I think wasn't particularly soft with the competitors, especially on the second half of the test. Now I'm waiting to see the final results and to discover mine.

[Ron Jones]

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test

I concur. It's not that the other planned tests would be invaluable (because they certainly will be), but it seems that many are very interested in a multi-format test at 96 kbps, myself included. I also think an aoTuV B5 Vorbis-only test at various bit rates, between Q1 and Q4, with non-music samples (perhaps both effects and voice) would be interesting, seeing as how many game developers are using Vorbis as a staple for audio assets, and the results of such a test may be particularly valuable for them.

[Sebastian Mares]

I also think an aoTuV B5 Vorbis-only test at various bit rates, between Q1 and Q4, with non-music samples (perhaps both effects and voice) would be interesting, seeing as how many game developers are using Vorbis as a staple for audio assets, and the results of such a test may be particularly valuable for them.

With all due respect, these tests are made for users who want to listen to music either at home (>= 128 kbps) or outside the house (< 128 kbps). If game developers want to encode their car engine sounds or whatever to Vorbis, they should conduct their own tests - I doubt EA or any other company is going to come and pay me for conducting tests or pay you folks for investing your time in testing.

[rbrito]

Well, speaking of bugs, creating the ECF files was a pain since I had to close and restart ABC/HR every time because otherwise, if I opened the Create New Test window again, it would have only half of the normal size.

Sorry to ask something which may be highly obvious, but where are the sources of the ABC/HR program in Java?

I see that there's a directory with licenses and it contains some mix of BSD-like (both 3 and 4 clause) and LGPL 2.1. I would like to see the sources at least of the LGPL program and, quite possibly, package it for Debian (and hence, Ubuntu), so that people can use things like:

aptitude install wavpack abchr vorbis-tools faad

and use the corresponding programs to decode the files (with all the dependencies already pulled in).

This would simplify the process of distributing the binaries for at least one platform. I can also create a meta-package that pulls other pieces (or I can include scripts as examples) that helps in listening tests conducted by Hydrogen audio, with all the packing and unpacking going on under a Free platform.

BTW, such script in that package would help with users on MacOS X and even those on *BSD, as I always try to write programs in a a POSIX compliant shell like ash.

In summary, I think that we could indeed make this easier for at least a portion of the potential test listeners and I volunteer to make and maintain such a package in Debian proper so that it would be there even if I get hit by a bus.


Regards, Rogério.

[guruboolez]

Sources are hosted on rarewares: http://www.rarewares.org/others.php

[IgorC]

I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.

[rbrito]

Sources are hosted on rarewares: http://www.rarewares.org/others.php

Thank you very much. I see that the other .jar files are already in Debian (which probably mean that they are already in Ubuntu Linux) and now, I only have to modify the build process (with ant) to get a built package.

Nice.


Thanks, Rogério.

[mgabi]

I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.

YES. Me too!!!

[kdo]

I've done 17 samples of 18. Only White America sample sounds strange and confuses me. I hear difference at the first try but then can't abxed with 5 tries. Original already sound artificially however it's lossless.

YES. Me too!!!

+1

It was the only sample where I could distinguish only low anchor.

[muaddib]

Are you people inpatient to see the results?