Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.
Topic: Friedman analysis question (Read 6229 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Friedman analysis question

I have a small question on Friedman analysis (Analysis of Variance by Ranks). Say, I got a matrix with N rows and M columns and I perform a Friedman analysis on it, working with a tolerance level of tau = 0.05.

Assume that the outcome is equal to pval = 0.1, so for the total matrix the difference between the columns is not statistically relevant.

Now, if I perform the analysis again on only two columns out of the M total columns, the result is equal to pval = 0.01, for example. Is it allowed to say that this difference is statistically relevant, or is this value for pval overuled by the value for the total matrix?

Friedman analysis question

Reply #1 –
No, it is not allowed.
It would allow for people to bias their analysis by choosing on purpose columns with significant differences, and throw away columns with no significant differences.
This way, you would just have to feed enough random columns in your test so that you are certain to find some significant results by chance among them.

I think that the total result for two columns is equal to the result for these two columns alone, multiplied by the number of ways to choose two columns in the matrix. Something like this. In short, the global analysis takes into account the probability to get some significant pairs of columns by chance inside a random matrix.

Friedman analysis question

Reply #2 –
Quote
No, it is not allowed.
It would allow for people to bias their analysis by choosing on purpose columns with significant differences, and throw away columns with no significant differences.
This way, you would just have to feed enough random columns in your test so that you are certain to find some significant results by chance among them.

I think that the total result for two columns is equal to the result for these two columns alone, multiplied by the number of ways to choose two columns in the matrix. Something like this. In short, the global analysis takes into account the probability to get some significant pairs of columns by chance inside a random matrix.
[a href="index.php?act=findpost&pid=324459"][{POST_SNAPBACK}][/a]

Aren't you talking about rows? Sorry if I do not understand it correctly.

What I mean is (for example), I want to test the difference between conditions A, B and C. The pval for the total matrix is 0.1, but the pval for the matrix with only columns A and B (I include all rows) is 0.01. I do not want to make the total result significant by using tricks, I just want to test the significance between two conditions.

Normally, you can also make a "pval matrix" for all combinations of two columns to test individual conditions against each other (the new ranks are then calculated for those two columns only). My question is: is that allowed if the pval of your total matrix is already above your tolerance level? Because I know it is allowed when the total pval value is lower.

Friedman analysis question

Reply #3 –
You cannot make a pval matrix like you describe, (and certainly not draw conclusions from it), because of what Pio2001 already explained.

Friedman analysis question

Reply #4 –
Quote
You cannot make a pval matrix like you describe, (and certainly not draw conclusions from it), because of what Pio2001 already explained.
[a href="index.php?act=findpost&pid=324463"][{POST_SNAPBACK}][/a]

Ok, thanks (and also thanks to pio2001).

One last question: the pval matrix *is* valid when the total pval is lower than the tolerance level, right?

Friedman analysis question

Reply #5 –
It all depends on what you want to test. Testing A vs B is not the same thing as testing A vs B vs C.

This illustrates a side effect of blind tests.

Say that three independant teams want to test A vs B, B vs C, and A vs C respectively. They will give the result of the comparison of their two columns alone, for example 0.01.
Now, imagine that these experiments were run by one team only, and that they performed a global analysis on the ABC matrix. The experiments were exactly the same, the answers were exactlty the same, bu tthey come to a different conclusion : p < 0.1 ! (in reality, combining three such columns into a matrix would rather lead to p < 0.03, but this is just an example).

How can the same experiment lead to two exclusive, but both valid conclusions ? I guess that this is the paradox that annoys you !

The problem comes from the fact that, the more experiment you run, the more likely it becomes to get a false success (Type I error).

By definition, if an infinity of teams perform the same experiment again and again, with a requirement of p<0.01, then one team out of 100 in avergae will get a false success. That's the definition itself of p !

Therefore when you consider the three independant results above (p = 0.01), you must take this into account, and compute a new p for your meta-experiment, that consists in combining the original experiments together. This combination affects the probability of success. Experimenting once is like throwing a dice with one hundred sides. Experimenting twice is like throwing the dice twice, etc.

The extreme case would be to consider the only scientific publication in the world that would suit you and tell "this proves my theory with a relevance of 99 %", disregarding all the other negative results published elsewhere.

Ok, A vs B and B vs C are not the same. So the failure of one combination shouldn't affect the success of the other.

The trick here is that you are forced to choose the only pair that you want to analyse before you know the test result. For example A vs C. Then, once the experiment is done, you have the right to consider the A vs C analysis alone, and disregard the influence of B.
However, once you know the results, you are not allowed anymore to compare A and C alone. The purpose of the experiment must be chosen beforehand : comparing A, B, and C, or comparing A and C alone. It can't be changed once the results are known.

Friedman analysis question

Reply #6 –
Quote
[...]
[{POST_SNAPBACK}][/a]

Thank you very much, Pio2001. I think I understand what you're saying.

I'm thinking what it means in my case. I did a listening experiment to test 3 conditions A, B and C and asked a number of subjects their preference (ranking).

For me, the goal was to make statements on the preference of the group of listeners (a "final" ranking). But how can I say if A > C > B (for example) if I'm not allowed to test two columns individually (I planned this before I knew the test results, by the way).

What's confusing me, is the fact that the [a href="http://www.ff123.net/friedman/stats.html]online Friedman test[/url] from ff123.net does exactly that. For example, if I enter the following (artificial) data:

Code: [Select]
A B C
1 2 3
1 2 3
2 1 3
1 2 3
1 2 3
3 2 1
1 2 3


Then the answer from ff123.net is:

Code: [Select]
FRIEDMAN version 1.24 (Jan 17, 2002) http://ff123.net/
Friedman Analysis

Number of listeners: 7
Critical significance:  0.05
Significance of data: 4.98E-02 (significant)
Fisher's protected LSD for rank sums:   7.334

Ranksums:

C        B        A        
 19.00    13.00    10.00   

---------------------------- p-value Matrix ---------------------------

         B        A        
C        0.109    0.016*   
B                 0.423    
-----------------------------------------------------------------------

C is better than A


So, this tester does make a pval matrix.. and its final conclusion is based on that matrix alone.

Friedman analysis question

Reply #7 –
If you want to adjust the p-values for the fact that you are performing multiple comparisons, then you should use the "non-parametric Tukey's Honestly Significant Difference" instead of the "Friedman / Non-parametric Fisher's Least Significant Difference."

Non-parametric means that you use a ranking system (1st, 2nd, 3rd, etc.) instead of a rating system (4.9, 3.6, 2.7, etc.).

Similarly for parametric analyses, use the Tukey's HSD instead of the Fisher's LSD.

ff123

Friedman analysis question

Reply #8 –
Quote
If you want to adjust the p-values for the fact that you are performing multiple comparisons, then you should use the "non-parametric Tukey's Honestly Significant Difference" instead of the "Friedman / Non-parametric Fisher's Least Significant Difference."

Non-parametric means that you use a ranking system (1st, 2nd, 3rd, etc.) instead of a rating system (4.9, 3.6, 2.7, etc.).

Similarly for parametric analyses, use the Tukey's HSD instead of the Fisher's LSD.

ff123
[a href="index.php?act=findpost&pid=324575"][{POST_SNAPBACK}][/a]

Thank you, that is exactly what I need. 

To summarize: I can use the Friedman analysis to check the significance of the total data, and if the data is significant, I can use the Tukey HSD to perform a post hoc analysis between columns?

Friedman analysis question

Reply #9 –
Quote
To summarize: I can use the Friedman analysis to check the significance of the total data, and if the data is significant, I can use the Tukey HSD to perform a post hoc analysis between columns?


Just use the "non-parametric Tukey's HSD".

For the reasons PIO and Garf gave above, using the Fisher's LSD is not strictly correct for more than just a single comparison.  But for a small number of competitors (say 3 or 4), you'll probably get similar results.

ff123

Friedman analysis question

Reply #10 –
Quote
But how can I say if A > C > B (for example) if I'm not allowed to test two columns individually [a href="index.php?act=findpost&pid=324566"][{POST_SNAPBACK}][/a]


The application tells it to you.

Here, if we put aside the choice of the analysis, it says that C is better than A. It means that you can't tell that C is better than B, nor that B is better than A. Maybe they are tied. Otherwise, it would have been written too.

The p value matrix tells you all :

C is better than B with a probability of error equal to 0.109
C is better than A with a probability of error equal to 0.016
B is better than A with a probability of error equal to 0.432

The star near the C over A estimation means that the p value is below the threshold that you chose. Below, the text summarizes the strared comparisons. In this case, C is better than A only.

However, my understandings of these probabilities is not very good yet. I don't understand why C is better than A for both Tukey and Fisher non-parametric analyses, while if C and A are considered alone, C is not better than A.

Friedman analysis question

Reply #11 –
Quote
Quote
To summarize: I can use the Friedman analysis to check the significance of the total data, and if the data is significant, I can use the Tukey HSD to perform a post hoc analysis between columns?


Just use the "non-parametric Tukey's HSD".

For the reasons PIO and Garf gave above, using the Fisher's LSD is not strictly correct for more than just a single comparison.  But for a small number of competitors (say 3 or 4), you'll probably get similar results.

ff123
[a href="index.php?act=findpost&pid=324597"][{POST_SNAPBACK}][/a]

Ok, thanks. I will use the Tukey's HSD only, then.

Quote
Quote
But how can I say if A > C > B (for example) if I'm not allowed to test two columns individually [a href="index.php?act=findpost&pid=324566"][{POST_SNAPBACK}][/a]


The application tells it to you.

Yes I know. But the application does that by comparing two columns individually using a "new" Friedman analysis (at least, when I calculate it by hand, those are the exact values I get). That's why I was confused. I followed the same algorithm as ff123 but was not sure whether or not is was actually valid in my case.

Friedman analysis question

Reply #12 –
It's not the case for me. If I use your data with a Friedmann analysis, I get

A vs B alone : p = 0.257
A vs C alone : p = 0.0588
B vs C alone : p = 0.0588

While feeding the whole matrix I get

A vs B : p = 0.423
A vs C : p = 0.016
B vs C : p = 0.109

Friedman analysis question

Reply #13 –
Quote
It's not the case for me.
[a href="index.php?act=findpost&pid=324749"][{POST_SNAPBACK}][/a]

For me too. 

I must have made a mistake somewhere.