Hello, people.

These last few months I have been working on some sort of guide to help newcomers get their ways around listening test conduction. Hopefully it'll help spark interest in people that were still just wondering whether to conduct their own tests or not.

http://www.rarewares.org/rja/ListeningTest.pdf

It's still not "officially released". So, I'd like to ask you guys for suggestions on improvements and corrections, or just general comments on how do you like it.

Thank-you, and I hope you enjoy reading it.

Best regards;

Roberto.

Oops. I'm sorry, the version that was available there is outdated.

If you downloaded it already, please redownload. The current version is the correct one. Thanks.

I once got reported to my ISP when I announced a new version of abc/hr at rec.audio.opinion, for violating the group's charter.

Announce listening tests there at your peril.

Other random thoughts:

Listener Training
It would be nice to have a section on listener training. In one of my tests, I had prospective listeners download a small training package before the main test started. Perhaps the instructions to acquiring the main test could be embedded in the training package.

sample01
A note on listener psychology: they will tend to download and listen to sample01 first, and then decide whether they want to continue based on their experience on that first sample. I know that's what I do ;-) Ideally, there would be some sort of randomizer which assigns different music to each of the samples dynamically, but that would require some way to sort things out in the end. Barring that, I would try to make sample01 as friendly as possible.

Keep the ball rolling
Try to keep the discussion thread going during the test to keep interest up.

sample durations
The samples should be about the same duration. The idea is that the average bitrate of the sample set depends on the individual sample durations as well as their difficulty -- the longer the sample, the more it affects the overall bitrate.

sample bitrate distribution
For a vbr codec, I think the distribution of bitrates in the small sample set (eg., 20 samples), supposing you draw a histogram of it, should resemble the distribution of a large sample set chosen from a wide variety of music.

ff123

Eliminate 1st second of encoder output
I seem to remember some codecs having problems during the 1st second or so of the output. The config file should start the sample such that the 1st second is not included in the listening test.

Well, isn't that a flaw in the encoder which should be allowed to affect the result negatively? A more extreme analogue: Some codecs have problems during sharp attacks. One should take care to eliminate such attacks from all test samples.

Very big thanks for the usual awesome help, ff123

Except that 99.5% of the time, you won't be listening to that 1st second. It's unfair to feature such a fault in every sample.

sample content
The sample should be as homogenous as reasonably possible, otherwise the listener may have difficulty rating a codec (eg., the first part codec A was better, but in the last part, codec B was better).

And now that I read through the whole document, I can only say: well done. I hope it will help to bring forward a successor that will pick up the testing business again.

Just one question: what does the title you write after your name on the front page, PITA, mean? "Pain in the ass" is the first thing that comes to my mind...

That's precisely it!

I looked at these articles you find around in the web, and most of them have the name followed by fancy acronyms, like "John Doe, PhD". Since I'm no PhD or anything like that, PITA is what comes closest

Obviously, I will remove it from the final version...

Not PITA for those that know...

1st second removal:
This is not because of problems in codecs, but to allow a realistic behaviour. With real encoding, encoders might be adapting themselves to content. In real tracks, this adaptation can be progressively done as the track is starting, thus the beginning of a extract from a track is not representative of the encoding of this part inside the full length sample.

Completelly off topic, but i must do it :



This, indeed, is SYSTEMIC ! ;)

For the rest, it's very very good !

Things like the High Bitrate question always seems to me like a paradox...i always thought you have to slip from "how good this sounds" to "how big the file size is"...cause as you said, most people won't notice a difference...

But then it's not listening tests....

MaB_fr

Here are some random thoughts and nitpicking - hopefully some of them are actually useful...

• Mention bias perhaps - not just placebo.
• QUOTE
  To counter the claims of the subjectivists, the objectivists created a method to reliably compare two audio signals called ABX.
  Wouldn't this sentence strictly be saying that the audio signals are called ABX and not the method?
  
• QUOTE
  Still on the samples subject: avoid the obvious choice of problem
  samples (samples that trip codecs producing very nasty artifacts) like Kalifornia,
  Castanets and IDM stuff because their artifacts are easily detectable, and
  therefore less fatiguing for your listeners.
  Maybe you don't want to mention sample names that only people that have been around for years will know... dunno. Perhaps it just adds to the confusion.
  
• QUOTE
  <nostalgia>
  Stuff like this makes it seem unserious IMO. I think the anecdotes should be left out if I understand what you want with this paper...
• Add an Index perhaps
• I think a more comprehensive explanation of what you actually do in the ABX would be good.. what buttons do what. How you ABX and rate. That part was very unclear to me.
• More excel guide! You can't just use the normal x,y-graphs so if you want to make it easy for people you should add steps for the graph creation. I think this is a bigger issue than you suggest in the paper.
• Where to get programs
• Link to ITU doc

Since the time you ran your early tests, I made the parametric Tukey's HSD option available on the web-based tool and made it the default:

http://ff123.net/friedman/stats.html

The document could still use a bit of proofreading, e.g., I see 'conduce' used where I believe you mean 'conduct'. I'll help you out with that if you like.

Compliments on a good document!

Suggestion: Place links and references at the end of the document. To doom9, HA, programs' homepages, etc...

Hello.

I would like to apologize for not producing an updated version of this document yet. I am now facing finals and papers, besides working 6 hours per day at Siemens. If that wasn't enough, I am helping Sebastian with his listening test and creating a new site design for LAME.

I guarantee you all your comments are being taken into account, and I hope to be able to release a new Work In Progress soon.

Thank-you very much.

Best regards;

Roberto.

Hey Roberto, any update?

Care to put some in here:

http://wiki.hydrogenaudio.org/index.php?ti...listening_tests

OMG! A new version at least!




Added a warning there.



Added all of these. Thank-you very much!



Added it. thanks!



Done



Good point! I removed the ambiguity.


Done


Bummer

OK, removed it


Done



Well, I think that part belongs more in the listener training part. Remember, that document is for test conductors, not test participants.


Augh. Maybe later. Guiding step-by-step in Excel is quite the pain :B


Done (most of it)


Done

Thank-you very much for all your suggestions, Jan!

Moving on...



Yes, please! All feedback related to grammar (and everything else, really) is welcome!



Added them as footer notes.



To be quite honest, I'm not too fond of the idea of wikifying it. I want to have responsability and authorship on this document, so that people can easily come to me if they need help. If wikifyed, both responsability and authorship get diluted...


Anyway, the new version is already uploaded, at the same location. Please download, read and send in your comments!

I promise I'll try to respond to the comments faster this time

New version up, fixed several small errors spotted by Sebastian Mares.

Perhaps I should document the GnuPlot way to produce graphs (way easier than with Excel or OOorg...

I would be very grateful

About running friedman.exe

"friedman -tp" which selects Tukey's parametric analysis is statistically more "proper" than "friedman -a" which selects the Anova analysis with a Fischer LSD. The former corrects for multiple codec comparisons, while the latter does not.

I've also made the Tukey's HSD the default analysis on my web page.

ff123

Ah, thanks. Fixed that on the document.

I already read parts of the comments on friedman.c to try to figure out what's the difference between all those modes, but because of the considerable amount of statistical terms (and I don't know much about statistics), I couldn't understand everything.