To illustrate the debate about noise and clicks I uploaded a new harpsichord sample (recorded in the 60') with much more mechanical clickety-clack :
http://www.megaupload.com/?d=WEJXUM18

I also have a fresh and expensive opera live recording with several stage noise (feet or object on the parquet floor) which were so close to ripping artefacts that I assimilated them first to mastering issues. I also have an excellent opera in Hybrid-SACD with weird pops on one track I can't explain. And to finish, a new CD re-issue (licensed by a label to a second one) with a ripping click noise: the CD is new, pressed but with a flaw typical of amateur ripping...


I don't think we should include samples that weren't correctly ripped unless with explicitely want to test how will react different encoders to a common issue. Better use a good rip of interesting samples rather than poor ones. But those clicks are far to be unsignificant: millions peoples have the same ripping artifacts on their library and the exemples I listed above are showing that even EAC-maniac can have the same or similar ones.

12.8. is near... How is the test going? How many results are expected?

Will it be possible to get the key file for the result files when the test has finished? It would be interesting to compare your own results with the averages.

Normally it has been published after the test has finished. Also, the complete test data must be publicly available for anyone who wants to verify the results or further analyze the data.

BTW, I mailed my result files today.

Absolutely, yes!

I received both results, BTW. So far, I haven't been able to sort the results, but I hope to do so tomorrow. After that, I can also tell you which samples have a higher priority. What I can tell is that I have 22 mails in my inbox.

It will be very interesting to see the results. I felt I was finding more serious artifacts in the lower-numbered samples, ~1-5. Will be interesting to see if other people were doing the same.

Sample 8 is the only one with less than 10 results. For the other samples, here is the number of (valid) results so far:

Sample 01: 18
Sample 02: 18
Sample 03: 16
Sample 04: 17
Sample 05: 15
Sample 06: 14
Sample 07: 11
Sample 08: 9
Sample 09: 12
Sample 10: 10
Sample 11: 11
Sample 12: 11
Sample 13: 13
Sample 14: 11
Sample 15: 10
Sample 16: 10
Sample 17: 12
Sample 18: 11

Unfortunately, there were several invalid results with ranked references which I am going to ignore.

Oops. Hope none where mine

Can you describe invalidities? I read ABC/HR's documentation, but it is out of date. I got through five rounds of samples or so before I understood the behavior of the ABX "Testing Mode" but went back and corrected or reran previous trials (good thing I wrote notes for each sample). I should be okay now, but one never knows.

EDIT: I see that I've downloaded the Java version yet documentation points to the Win32 version, so it's not out of date as much as mildly incompatible.

As I said, results with ranked references.

Whenever you are not sure whether you can hear differences or not, you should do ABX in testing mode. When you do enough correct trials (5 out of 5, or 7 out of 8) then reference is grayed out and you can only move slider for coded sample (you can only rank coded sample). Before doing ABX in testing mode it is advisable first to do it in training mode until you are sure that you can hear differences.

IMO whenever you can not successfully do ABX in testing mode, you should give 5 to an encoder.

And one more thing. Come on people please do a bit more on samples 7-18! At least sample 7 is very easy even for high anchor.

Great summary. This information would be helpful to newcomers were it included somewhere in the test summary or linked elsewhere.

That is precisely what I do. I have no problem in giving 5 to an encode. If I cannot spot any obvious difference after listening to the sample two or three times it gets a 5 rating.

I try a bit harder than that, because I like to know that I have all the detail in the music that I can possibly hear when I dish out a big 5.0.

Editorial comment:
Your testing procedure is just too complicated for a senior citizen old codger like me. Too many installs, too many directories, to many apps... Too much.... If anyone comes up with a simple web based user interface count me in....


Note to moderators: If my comment seems overly harsh just delete it.

Woody

I didn’t realize there were 18 samples … At first I only found sample 18 from the torrent mentioned in post #6, later I found the other samples but thought they were from earlier listening tests

Well, all you need to do is simply download the ABC/HR ZIP file provided on the test page, unpack it to a folder of choice, then download the samples you want to test, unpack them in the ABC/HR folder, run the corresponding DecodeXX.bat files from the "bin" directory and that's it. All you need to do then is to start ABC/HR, load the corresponding "*.ecf" file, test, save as "*.erf" and send me the file. That's not too complicated.



Well, both the TXT and HTML readmes list all 18 samples - both as torrent and HTTP downloads.

I was a bit confused … the readme says "Place the sample package ("SampleXX.zip") in the same folder as ABC/HR and uncompress it." I thought #18 was the "latest version"

It still takes a bit of work to get started on these tests, although it's not difficult. I can think of other ways to do this. For example: have the java app start from a click on a webpage. The application could have another dialog box specifically made for such tests which lists the samples the test administrator has selected, has buttons for downloading the compressed files for each sample (either by torrent or http), and has another button for starting the test once the download has completed (the app would automatically take care of decompressing/decoding everything). The only thing you would need to specify is where to download the compressed file to, and where to load it from when you want to start/continue the test. Instead of emailing the .erf file, you could choose to submit it directly by pressing a button (perhaps with IP logging to identify possible multiple submissions from the same person).

The only downside to this is people may not like the abchr application to connect to the internet, because then there are always questions about what it does when it connects.

The people who care about their the nature of their Internet traffic probably aren't the same who want the most convenient experience possible. I assume the application would still work manually for those inclined to run it thusly.

why can't an installer with ALL ready be made? (not that i care, but for others like the prev poster)

Someone would have to write an installer in Java or something that runs on Windows, Linux and Mac.

I've got two comments about the current abc/hr java version:

1) in ABX mode there are keyboard shortcuts Q,W,E,D to play/stop samples -- this is VERY helpful.
However, in the ABC/HR mode, the same shortcuts are available but which sample do they refer to? That is, when I press W or E - which sample of the 5 it is going to play? Had to click the buttons with the mouse to make sure it's playing the ones I wanted.

2) on my laptop the playback via the java sound engine skips every so often. Setting high priority in the task manager (windows2000) doesn't help.

Well, I cannot help in those cases since I didn't write the program, sorry. Also, there is no better thing we can use that is also available for multiple platforms.

Anyways, today is the last day you can test and submit results. I will close the test tomorrow at around 12 PM German / Central European Time (GMT +1 with daylight saving time). Once the test is closed, no results will be accepted, so please don't wait until the very last minute.

Important! If you decide to start testing now, please focus on samples 7+ since they are the ones with the least results. This only applies to people who start testing now - if you already tested samples 1 to 6 and your results are pending, submit them, of course. Again, if you start testing now, please do so for samples 7 and higher - do not start testing now while choosing samples 1 to 6!

That of course makes sense. Maybe I am just not patient enough.

Sebastian, can you tell me if I rated any references? Results should've arrived from vince dot hart at gmail. I fear the answer.

The key will be available after the test is over so you can check yourself.

People, about two hours left!

Hmm... I could use a bit more results for the last 10 samples. I am extending the test one last time until 15th since I expected a bit more results to arrive shortly before the test ends. Maybe Guru can test, too (BTW, received two mails from you - thanks!).

Come on people! If you feel geeky doing the tests, remember it's all in the name of science!

Also, even if you are not personally interested about this bitrate range this a good opportunity to practice with relatively easy samples. A future higher bitrate test will be much more difficult for the testers. We have recently tested 48 kbps and now 64 kbps. Possibly the next multiformat test will be 96 kbps. I also look forward to see the discussed 128 kbps MP3 test happening sometime soon.

This is why I joined. 48 kbps is unrealistically low and thus uninteresting to me right now, but I'm having problems as early as 96 kbps. A new 128 kbps test may be more challenging than many are expecting. 64 kbps is just the right spot where you might be fooled some of the time, making it interesting for testing purposes.

Well, in case anyone is interested, the next tests will be:

MP3 at 128 kbps
AAC at 80 kbps (featuring both LC and HE-AAC)
Multiformat at 80 kbps with winner of the AAC test vs. other codecs
And finally maybe a 32 kbps and a 96 kbps test, but these have very low priority

It is good to see that you have plans for several tests since no one else has been interested about conducting listening tests.

The 128 kbps MP3 test will be interesting because LAME 3.98 should be soon ready and FhG has a new version too.

Personally, I think 80 kbps is quite near of 64 kbps and 96 kbps would be a better next step for a multiformat test, but I suppose you have carefully thought about the options and followed the ongoing HA discussions more keenly than I.

It would be good to keep the tests flowing in a steady stream and not allow as long breaks as we had now. I think more testers would be willing to participate if the test discussions are constantly kept alive. For example, if a test is not possible in the scheduled time frame because of codec development reasons you could start the next test from the list instead.

Also, I wonder if the test presentations, instructions/help/faq and the results presentations could be developed further. It would be nice to have an attractive and carefully organized web site where you could browse info about the ongoing test, the test schedule and the results easily. Perhaps naylor83 would be interested in helping with this. He did excellent work with the rarewares site.

I too would be more interested in a multi-format at 96 with the AAC candidates being Nero and Itunes VBR.

This bitrate would be of more interest for those using portable devices I think.

And science should be in the name of better life

I disagree. 48 --> 64 --> 80 = 16 kbps between each test iteration. The difference between Vorbis at 64 and 80 kbps has me very interested in this test. 80 kbps is the portable sweet spot for me when it comes to Vorby...I'd like to see if it's still #1 or if there's a similar transparency being reached across all formats.

Sebastian, you're killing me....I need to know how many references I ranked.

I'm also with vinnie97 wholeheartedly as regards 80 kbps tests - I'd also be interested in the results & suspect quite a few participants would be pleasantly surprised with the fidelity of the sounds; I for one was a little surprised in this test at the fidelity in some of the non-problem/general-music samples e.g. "Big Yellow" & "Les Voleurs" come to mind as being mostly quite listenable to my ears/head/brain IIRC.
Can't wait for the overall results!

Edit P.S.: I also dread my contributions being thrown out for whatever reason, so I tend to be a bit conservative and listen/rank like beto wrote above.
P.P.S.: Blimey, the next 128 mp3 test will be ruddy hard, I reckon!

It would be really great if people with golden ears would help us with this test. I guess for them it would be very easy and they could do it in 2-3 hours.

But the golden ears would have to rate all these samples between annoying and very annoying.

Whereas people with average hearing can use the full scale for the ratings. Like I did.




I have a question: When somebody ranks the reference, do you disqualify the whole test for this sample or only the result for this particular encoder within the test?

Would it make any sense for those of us who ranked the references to repeat the discarded tests more carefully and re-submit?

Giving low grades wouldn't make any problems.
And because someone can hear very well and find those small differences it doesn't mean that he/she will grade those differences as annoying.

I'll do it today or maybe tomorrow for all 18 samples - Sebastian will therefore have one more result for each one.

I think the amount of ranked references in the future tests could be lessened with better instructions. ff123's practise page is very useful, but an updated page (with the JAVA version screenshots) inside the test site would be better.


The following things should be said very clearly:

- If the difference is not obvious, you should do an ABX test.
- If you can't succesfully ABX you should not try to guess. Don't pull either of the rating sliders down.
- If you didn't ABX test an obvious sample it would be good to listen to it once again and verify that you moved the correct slider.


Perhaps the instructions could have some step by step examples with screenshots. For example:

[Screenshot 1, the initial view] & a speech bubble:
"Hmmm... the sample on the left has less overall clarity and the symbals are a bit diminished, could it be the lossy file? Though, I am not sure, better go to ABX."

[Screenshot 2, ABX view, configuring the test]

[Screenshot 3, ABX view after a succesful ABX test] & a speech bubble:
"Oh dear, it was the other way around. This encoder has increased the volume of the higher frequencies slightly and at first sight it sounded better because the reference is not very bright."

and so on...

"... Ah, that must mean this is the WMA sample! I'll bring it down an extra notch just for spite."



Edit: Seriously though... Good suggestions!

All participants in this test that have saved their sessions, could use Alex B instructions to verify that no references were ranked.

I am willing to help in building instructions for future listening tests (for this one it is to late I am afraid). Maybe someone could start this in Hydrogenaudio Wiki (if that is a good place).

That's a fantastic idea. I downloaded the program and a few samples but got bogged down in the mechanics of things. Wasn't really sure what I was doing.

A single clear " how to " manual might have helped.

In my case it would mean practically re-doing the whole test from scratcg, because most of the time I didn't do abx.
I did go through abx a few times when I was in doubt, and I did manage to abx successfully every time (or almost every, cannot remember). But there is still a chance I could have ranked a reference somtimes.

It would be nice if Sebastian could notify us by e-mail which of our tests are discarded and which we should re-do and re-submit. (Of course, if such thing is allowed by the test protocol)

The four first samples are tested on my side.
In order to avoid some mistakes I begin the procedure by ABXing all files and I start ranking them after this ABX procedure. Advantage: it's impossible to accidentaly mark the reference (only one slider can be lowered after a successful ABX test).

N.B. It seems that one competitor tends to make higher frequencies brighter (more noise/energy) which wouldn't be unpleasant if there wasn't additional artefacts. After lossy encoders, maybe the first "glossy" codec...

I hope you will not get bored until you get to samples 7.-18.

I don't understand why so little results for sample 8. It is easy after short 'getting used to it'. However I will not send my results because we at Nero don't want to have influence on the results.

Add to the list explanation of ABX testing behavior. It was far from obvious to me, and I had to redo several samples after experimentation (at the time I thought using the testing mode to eliminate the reference was required in all cases, but apparently this isn't true--this should be covered, too).

Not obvious is that the listener gets only one shot at testing mode unless he wishes to redo all samples within the test or load from a good session. The program also has some strange ideas of what qualifies as a commit; if I begin testing, reset my trials, then decide to come back later, I'll find the test was marked as a failed commit and will not be able to try again without starting over or loading from a successful session save. At least twice I successfully ABXed only to have my results discarded and replaced with a failed commit.

The listener should also know that he needs to run at least eight trials in testing mode. Early on I tried finalizing when ABC/HR first displayed a number (6/6, I believe), but results came up in red and the reference was not discarded, which is not helpful, especially given the retesting issue.

I experienced numerous bugs while using ABC/HR Java including crashes, disappearing menu options, nonfunctional sliders, failed saves, and phantom ABX commits. The program becomes especially unstable after a session save. Are these issues known?

Had I not kept notes on each sample, I probably would have quit due to the above frustrations.

You play hard, it seems...

I've done three listening tests so far, and the only bugs i found were the non working audio (that gets fixed manually selecting the output audiocard), and the clicks when pressing play (which doesn't happen to me if i use the soundcard directly instead of java sound engine)

But if you say it happens *After* saving session... i've never saved session. (don't use to get interrupted in the middle of a test)