anyone knowing marx' dialectic?
here is my approach using this method to find a synthesis between garfs and schnoflers theses

imo in the case of a ranked reference there a two possibilities why the user voted this way:
he thought to hear a difference which
1) simply wasnt there
2) was there and he liked the enode quality better than the source (for whatever reason)

ad 1)
this could be caused by a mistake (as i understand schnofler's thesis)
but
i doubt that anyone does the final ranking as 1 more abxing without double checking that his final vote is correct (at least i wouldnt act this way)

so to say it can be divided into serious testers and not serious testers:
for people who serioulsy attend this test such "failures" shouldnt happen normally (also considering point 2 i wouldnt discard the results from these)
results of people who didnt seriously attend the test and voted in a hurry could be discarded (for example if there are too many (over the average) ranked sources in the results aso...)

ad 2) well thats how garfs thesis can be understood
in that way the voting would look that way:
source: 5
encode: a score higher than 5
as the later isnt possible, a vote of 5 is ok for the encode, when the source was voted worse


to sum it up/the synthesis:
ranked sources from not serious testers (which look like the person voted anything, too many (over the average) ranked sources aso...) can be discarded
all others should be used and used with score 5.0

You forgot the possibility:

a) User could hear a difference, ABX it, but it was rather small and he missed the right slider

thats point 1) voting by mistake
shouldnt happen (even without abxing), in fact abx helps to avoid these mistakes,
i mean if someone handles to really abx the sample with a high propability i doubt that on the final vote he will suddenly make a mistake

No, you are completely wrong, see my earlier example.

You can ABX 110/200, which is significant, but your chance of pulling the correct slider is only 55%.

I think a kind of matrix can be constructed, with some of the options and reactions:

Options
1a. Don't allow any ranked references
1b. Allow only 1 ranked reference
1c. Allow multiple ranked references
2a. Ranked reference must be accompanied by ABX to 95% confidence
2b. Ranked reference does not need to be accompanied by ABX results
3a. Score of ranked reference is not lower than another properly-ranked codec
3b. Score of ranked reference is allowed to be lower than another properly-ranked codec


Reactions
A. Entire file is thrown out
B. Score of codec with ranked reference is changed to 5.0
C. Score of codec with ranked reference is given the listener rating

Obviously, the most conservative approach is 1a + A

Here's how I might order the choices, from most conservative to less so:



ff123

I'll stop accepting results tonight, at midnight brazilian time.

I probably have enough results, byt maybe not enough if I dump the ranked references.

Check my proposal, it should eliminate those 'tiny differences'.
If you can ABX it well and you make a mistake, it shouldn't be counted.
If you DIDN'T ABX it (or barely ABXed it), it should be treated as 5.0.

-
Midnight brazillian? So it is already closed?

-
What does A option mean? WHOLE results file?

I'd consider it if there are >2 ranked references.
I'd throw out all results w/o passed ABX in the file then and of course all ranked references.

the results of abx'es should in no way influence the decision whether to use the ranked sources results or not!!!
noone is forced to do abx, you cant rely on whether someone did abx or not

in fact someone can do the whole test without abxing and without being unserious or having anything bad in mind


as i proposed unserious testers should be sorted out via the way if there are far over the average ranked sources in the results


well your example is not usable in this case as its unrealistic/only theoretical
noone will do 200 abx'es

Nope, there are still more than 6 hours to go:
http://www.timeanddate.com/worldclock/

That makes no sense. Just because a listener is serious doesn't mean he'll come close to the average results.



Garf did almost that once, in the MAD challenge :B

In my matrix, this is an option, so it is a matter of deciding (debating) how to order the list from most conservative to least conservative. Another way of stating your proposal would be to come up with a numerical score which says how bad ranking the reference is with respect to the other scores. For example, let's say somebody scores:

A = 4.9 ranked reference
B = 3.0
C = 2.0

A figure of merit score might be the ratio of the ranked reference to the average of the other scores. First, transform into difference scores:

A = -0.1 ranked reference
B = -2.0
C = -3.0

then, F.O.M = A / average(B, C)

and if this ratio is under some acceptable value, you would either accept the results file as is, or change the rating of codec A to 5.0

This is a mess, isn't it? Much easier just to discard files with ranked references.

ff123

sure it makes sense, if you see it from that point that you dont want to sort out the serious ones, but the unserious ones (which will surely be far over the average)

Jesus Christ, it is!

first of all i assume that we dont have that much results, we can discard many, only because there are some ranked references

second your calculations show you are a developer (this isnt meant in a bad way )
my proposal:
i would say the average user has 1 ranked reference per sample
if someone has an average (over all samples) of 2,5 he is out, all others are in with ranked sources scored as 5.0 (of course the reality can be different, but rjamorim will soon find this out)
easy and clean solution with no mess

third everyone who does this hard test and than gets discarded because he did ranked sources will feel pissed off

First, it was never in the plans to drop all of a listener's results because he ranked part of them. Unless something very creepy is going on (check the results package I linked earlier), even if a guy got 11 ranked references, the clean result will stay.

Also, what's with the 2,5?

A means throw out the whole file, which in the past was done if any reference was
ranked. If you don't throw out the whole file, then you have the option of changing
the scores of the ranked references to 5.0 or keeping the score (but assigning it to the codec instead of to the reference, of course).

It isn't possible (at least with my statistics software) to only throw out part of a file.

ff123

2.5 ranked references per sample (or a similar value) can be used as indication as unserious testing, meaning the whole tester is out
all other ranked references are considered as from serious testers and will not be dropped and given a 5.0 score as garf proposed

thats my proposal, but it doesnt seem to have much friends anyways so do as you like

Another problem introduced by this is: where to draw the line?

calculate the average (i guessed its 1 ranked reference per sample) and add 1.5

also someone can say that a user has a 50% chance to vote the reference, which is equal to 2.5 votes per sample, over this 50% is too much bad luck for a serious tester

Maybe those, who rated the reference below 4.5 or 4.0?

bond and Roberto were referring to how many ranked references should be acceptable. bond's proposal is to actually look at the data and find out how many references are ranked on average in those results files where it occurs. Then add 1.5 to that number to determine where to draw the line of how many ranked references are acceptable.

Over that line and the entire file is thrown out. Under that line, and the scores of the ranked reference codecs are changed to 5.0. This is a reasonable and conservative proposal.

ff123

Not a bad thing to do.
I'm for this proposal, it should filter out people trying to make results a white noise.

I still think that the existence of a successful ABX test is a much better indication of seriousness. Of course, as bond noted, a listener can be serious without doing any ABX tests. But if he has done some, then we can be *certain* that he was indeed serious about it.
Just to clarify, my thoughts, which Roberto posted above, were only about ranked references with successful ABX tests. If there is no ABX test or a failed ABX, I think the file should be discarded, since for all we know the listener just played around with the sliders (see the results package Roberto posted earlier).
If there is a successful ABX result, I personally don't find it necesary to discard the results, especially if there are not enough results for such luxury. It's unfortunate that we can't just throw out the ranked reference and still consider the other rankings. On the other hand, it's obvious that it might cast a shadow on the professionalism of the test if we consider the rating of the reference for the encoded sample. In my personal opinion, I wouldn't mind using this method if the listener obviously made an effort, but I can see that it might not be in the best interest of the test as a serious reference later on. That considered, it might be best to count a ranked reference with a valid ABX result as 5.0 (contrary to my earlier thoughts).

Furthermore, I guess Roberto will as usual publish the results files, so if anyone is interested in what the results would have looked like when calculated with a laxer method, we could still do this after the test ("inofficially" so to say).

i think our different opinions on that maybe simply depend on the personal way of doing the test? (as i said before i dont think that the relationship pointed out by schnofler is that clear at all)

anyways i think i made my point clear on how to detect unserious testers on a, imho, clearer way than via abx results

It is indeed the case that I never rank a sample without an ABX unless the difference is extremely obvious to me. However, I prefer to see it the other way round: my preference on how to do the test stems from my opinion on what should be considered serious, not vice versa.

Anyway, please don't take this personal. As I said above I personally don't have any problems considering results as valid if it is somehow clear to me that the listener was serious (and yes, simply because you participate in this discussion, I trust you on your seriousness). The problem is how to make this reasonable for others. We have to choose a definite, easily justifiable way of doing this, so the results of this test can be used as a serious reference. A successful ABX test is, in my opinion, a very strong sign that the listener did indeed put a considerable effort into this. The proposal you support just doesn't seem as strong to me.

Just to clear things up, I think there is a bit of misunderstanding about this proposal. If I understood you correctly (please correct me if I'm wrong), you want to look at all the results files from a certain listener, count the ranked references, and calculate the average of ranked references per file. If it's above 2.5 throw out the whole set of files from this listener.

But from this

I understood that ff123 wants to decide this on a file-by-file-basis (again, please correct me if it's a misunderstanding).

In any case, if you move sliders randomly you still have a 50% chance that your results (single files or the whole set) will be accepted, which is far too high in my opinion. If it is done that way, I would draw the line much lower (e.g. 1 ranked reference on average).

I still like using ABX results better, because they actually give you evidence of the listener's efforts, while the other proposal just aims at making an educated guess.

I stand by my proposal. Untamperable ABX scores is IMO the most reliable tool to know whether the person could hear the difference or not. Since we are in a position where every valid result is valuable, dumping ranked references which have been ABXed successfully seems a waste of data we usually cannot afford. Moreover, ABX test provide data not only from the p-value but also, to a certain degree, from the number of trials the person needed to get the significant result (esp. if we have access to the ABX trial sequence for that sample). If a person needed 40 trials to get a statistically valid result you can tell at least that the difference wasn't all that obvious on the part he ABXed (and much less with 200 trials). However, a respectable result of say, 10/11, 14/16 or 20/24 speaks for a consistently picked up difference. In such a case, counting ranked references as 5.0 also removes potentially relevant information about a codecs quality, since we are disregarding the evidence that the sample indeed is not transparent.
Ranking a sample >4.0 acknowledges that the sample doesn't really sound worse than the original to the listener, only "different"; and therefore, with good ABX results backing it, it is reasonable to assume that due to fatigue or a lapse of concentration, the reference could be mistakenly ranked, without making the result less valid.
The course of action I would take is:
1) See if we have enough kosher results to get statistically valid results.
2) If not, or if the non-kosher, marginally valid results are too voluminous to be discarded with a clear conscience; then take the rank value of the ranked references with successful, non-extreme ABXs above 4.0.

Sorry I didn't help for this test

For the discussion about ranked references, I would take them as 5.0, unless there is evidence of a mistake at the time of ranking (ABX results with small p-value) in which case I would use the specified rank.

I would also contact the authors to inform them of what you do with their ranked references.

Bogus data (someone playing with the sliders) can be recognised in that it doesn't fit with the other results and makes the error margin excessive.

Just realised that ranked reference with a small ABX value does not necessarily indicates a mistake. As commented before, the tester could like more the encoded sample. Make them all 5.0 I think.

From my own experience:

it happens that I've rated the reference, but ABX the file without any difficulty. How? Often, the mistake occurs with the first pair of the test. I'm not careful, and I hear weird sound which is a property of the reference (some samples are weird for some users). Then, I'm rating (correctly) all other files, but I forget to reevaluate the first file.
This mistake is rare (occuring when I'm performing the test too quickly), but it happens to me some times...
In my opinion, if ABX tests are correct, the wrong notation must be invert, and not be cancelled.

Fortunately, Phong's wonderful chunky result parser can take into consideration ABX pvals to accept or discard ranked references (you may now bow to Phong), so it would be quite easy and fast for me to calculate results without ranked references and with ABXd ranked references, and then decide if the ranked references will make a difference in final scores or not.

Considering I/we decide to go with ABX results to decide about using a ranked references or not, what would be a good pval cutoff line? p < 0.05? p < 0.01?


BTW: It can also parse results taking into consideration number of ABX trials (Phong rocks). Do you think this should be used to select results too?

Just a comment (to ABC/HR software development).

I'm somtimes tempted to perfom multiple ABX tests on a difficult part of a sample. There are some points where artifact is obvious, but I'd like to test something more difficult. I never do it, simply because I fear to fail on this test. It wouldn't be serious to rank a file 2.0/5, with bad ABX scores.
Therefore, comment should be added ofor ABX sessions (and why not, the possibility of doing multiple ABX tests for one single sample). Simply to inform the reader of the final log about the real conditions of the ABX tests (and score)

So? Opinions on pvalues? Pretty please? :/

By "using a ranked reference", do you mean using the score for the encoded sample or counting it as a 5.0?
As for pval cutoff, that's not an easy question. I usually try to get it down to 1% if the sample is really hard. But if I can hear a somewhat clear difference, I usually don't bother going on after achieving 7/8 (pval=3.5%) or something similar. So if you cut off at 1% you might discard some instances where the listener heard a pretty good difference but just, well, missed the slider. I don't know how others handle this. Maybe you could try it and see what p-val most people go for, and if too much would be discarded at 1%, use 5% as cutoff. Or you could count the listener's rating if pval is <1% and count it as 5.0 if pval <5%.


No. A big number of trials is not necessarily a sign that the listener didn't hear a clear difference. He might have switched the playback range halfway through or needed a bit of practice (I remember one test where I got 1 correct during the first 8 trials, and 23 correct in the following 24 trials). The pval is the measurement of significance, after all, and it should be used as such.


Yeah, I know that problem. You could try a different part and if it doesn't work out, add a few more trials from a part which is easy for you to get the pval back down. But I admit this is not the most elegant of solutions.


How would you want such a comment feature to work? (I mean, how would it be different from the sample comments?). Multiple tests might be a solution, but it also might be hard to handle. Or the program could just write a more detailed ABX log into the results file, maybe something like WinABX. But then the ABX tests would probably make up the largest part of the results files. Anyway, ideas are welcome.

edit: Woohoo! 100th post.

One other option:

Recalculate the ABX p-value, including the trial where the listener ranks the reference. So if the ABX reads 14/16 in the abx section and then he ranks the reference in the ABC/HR section, the total ABX would be 14/17. That's what I did for the first 64 kbit/s test. It's a lot of manual labor, though

But to answer your question, I would use p = 0.05

I'd say p < 0.05, since it's usually the scientifically accepted p-value of significance and I don't think we want to be too strict with the "ranked reference consideration threshold". I do, however, think we should take into consideration the number of ABX trials, since a large amount of trials does speak for a hard to distinguish difference which might as well be considered as 5.0 (for the sake of Garf's rationale). (*) How large should this amount be, is arguable. I would set the cutoff whenever the number of successes to get a p <0.05 for the given number of trials is less than 75%, this could be seen from some table. (As a rough estimate I'd say the treshold should lie above 30, which is, from my experience, the number of trials I need for nearly-transparent samples).

BTW, does Phong's chunky enable you to set a rank limit (such as my proposed >4.0) for consideration as well as how to consider the ranked references (as a 5.0 or the given value)? If so, I would also enforce the minimum rank of 4.0 for taking the given score and set a 5.0 for below-threshold scores. How credible is a test that states the original is annoying with respect to the encoding?

(*) EDIT: After reading Schnoflers reply, I acknowledge he has a point regarding the "warming up" or range selection during ABXing. This kind of issues could be easily solved if the program tracks the ABX trials, instead of the final result only. A long run of successes after several failures suggests the warm-up kicking in or a change of range to an easier one. However, interspersed successes and failures over many trials DOES almost unequivocally speak for a hard to distinguish difference.

Yes, it seems to freeze when it plays past the end point (which is shouldn't be doing)

I just saved my sessions after any significant change... unfortunately I did lose hard ABX results sometimes

Unfortunately, I don't think it does.

Anyway, I am fairly sure I got enough "clean" results. And preliminary testing shows that there won't be much of a difference if I count ranked results or not. So I will publish "clean" results at the results page, but will also make the ranked individual results available for download so that people can use them to play around with and see if they come with different rankings somehow.

Just to clarify: If there are enough clean results (and I'm fairly sure there will be), the clean results will be used to create the "official" rankings that will be posted at rjamorim.com/test.
I will discard any files where the reference was ranked, even if it was ABXd.

Edit: by "clean", I mean result files where the test participant didn't rank the reference files.

And enlightening comment about the fun of ABX, courtesy of music_man_mpc:





@bond: People do take more than 100 trials, it seems

ABX results are part of the test dude

I know someone who abxed every single sample to >0.001

And yes... some people do abx 200+ times... although the most I ever went was about 50 I think

Luckily I don't claim to be a golden ear

One hour to go!

After that, I will stop receiving results and will start calculating the plots.

If you didn't send me your results yet, do so now!

So, is it normal for a test to have this many ranked references?

Or is it a sign of how aac@128kbps is fairly transparent for a large segment of the test population?

Second answer

I guess several people are finding out their transparency thresold is around 128kbps, and not around 160-192 as it has been believed for years (although the latter figures probably still apply to MP3)

Yes. It's also a sign that people are not always careful, that placebo exists and that they dislike not being able to hear a difference and put a 5.0.

Whatever it is when ABC/HR.jar goes green

I don't think non abx'd non-ranked-reference results should be discarded , after all abxing is NOT compulsory...

Yes, for instance, you want to see if you can ABX one artifact, but know you can ABX another artifact...

Anywho, you could do this by opening up a new session and doing your 'hard' abx in that session...

Mmhhhh...
It's possible if the whole test include one file against the reference. But if you have 5 files (and therefore 10, because of blind reference for each encoded sample), you have first to find again the specific sample where you believe you heard something wrong. Not easy... and waste of time. And with encrypted log files, it's even harder to be sure that you're ABXing the good file

Can someone enlighten me on the origins of Velvet?
http://lame.sourceforge.net/download/samples/velvet.wav

All I know is that it was submitted by Roel (r3mix).

Does anybody know artist (Velvet Underground?), title and album of this song? Also, what would be the style (no way to figure out from just the introduction)

Thank-you.

Listening test is now CLOSED

Expect results to be posted real soon...

Regards;

Roberto.

rjamorim
sorry for bugging again, but what way on how to treating ranked refernces did you use now? i couldnt see that from your posts

btw also you seemed to have missed out my sample 09 results!? maybe it was because they were all transparent rated?


no its not

I simply ignored ranked references. Since I got enough "clean" results, it was OK.



Nope. I couldn't decrypt your sample 09 results. It's the only result file that gave me problems in the entire test. I sent it to schnofler so that he can investigate. Sorry about that.