Part V: Analysis of Variance (ANOVA)
Planned Comparisons
| Why Analytic Comparisons? | Meaning of Omnibus F | Planned Comparisons |
| Contrast Formulas | An Example | More Demonstrations |
 
 
Analytic comparisons are used to determine which levels of an IV are significantly different. The logic behind analytic comparisons rests on the composite nature of the SSA component. Recall that:

Picture (422x85, 2.7Kb)

suggesting that the average of observed differences between all unique possible pairs of means or of conditions is contained within the SSA:

Picture (419x137, 3Kb)

So, the overall SSA may be better understood by examining the contribution of each of its parts. To do this we use analytic comparisons among means.
 
 
Whenever the omnibus F (overall F test from SRD ANOVA) is significant, we know that the treatment or variable of interest is effective (a difference between the groups is highly probable). At least two of the groups are significantly different (usually the two groups that are the most different), but we don't know for sure which pair(s) are.

If the F test is non-significant, we conclude that the treatment was not significant and we have insufficient evidence to rule out chance for the differences between the group means. Although we are willing to assert that there is no real difference between the means, we may be in error since the omnibus F test is testing the average difference between the means.

For example: if we have a single IV with 4 levels:

Picture (307x224, 3.3Kb)

However, if we had a hypothesis that the group 1 mean is less than the group 4 mean and used a single planned comparison we would likely have come up with a difference of -10. This difference of -10 is more likely to be significant than the omnibus F which yielded an average difference of -5. So, it is possible to find a non-significant F when some components would have been significant if you had specifically tested for them.
 
 
Planned comparisons are driven by theory or past data. Most of the time when you conduct a study you are not just interested in the overall significance of the independent variable. Usually the experimenter has some hypothesis about specific comparisons among levels of an IV or an hypothesis about which comparisons are more theoretically important or relevant. So you focus attention on certain comparisons because of their theoretical importance.

If the comparisons are planned in advance of collecting data, we just make the planned comparisons instead of performing the omnibus F. However, you still run a risk that comparisons which are experimentally relevant may not be significant. Also, comparisons which are not tested may be significant.

To make planned comparisons, first we must set an alpha level for each comparison. Typically we limit the total number of planned comparisons to the number of degrees of freedom for a particular IV and just use alpha = .05 as our alpha level for each comparison. If there are more meaningful comparisons we feel we should make than there are degrees of freedom, we must pay an "error rate penalty", since as we make more and more comparisons, we are more and more likely to find one of those comparisons significant just due to chance. One method of controlling for this problem of compounding error rate is called the modified Bonferroni technique. All it amounts to is adjusting the "per comparison a level" so that the overall "familywise or experimentwise a level" is .05 (or .01 if desired). Simply divide the desired a by the number of comparisons that are going to be made. Then carry out the F test for each comparison using the new more stringent a level for each comparison. You only do this adjustment if you wish to make more comparisons than you have degrees of freedom for a particular IV.

Once we have the correct alpha level, we then set up contrasts which represent our comparisons of interest. A contrast is nothing more than the sum of weighted group means or totals. They are also a comparison of just two things: means which have a positive weight versus means which have a negative weight. As a general formula, any contrast may be expressed as follows:

Picture (526x77, 3Kb)
 
 
To set up a contrast:
  1. Attach a negative sign to the group(s) you want on one side of the comparison.
  2. Attach a positive sign to the group(s) you want on the other side of the comparison.
  3. Attach a 0 to groups you don't want to be in the comparison at all.
  4. Weight each group on the positive side by the number of groups with a negative sign.
  5. Weight each group on the negative side by the number of groups with a positive sign.
  6. The newly created positive and negative weights should sum to zero, taken by themselves.
  7. Calculate the SScontrast according to the formulas below. Divide this by the mean square error term from the omnibus F test to get the Fcontrast with 1 df. You can now determine whether the comparison is significant.
When using Means: Picture (316x72, 2Kb)
When using Totals: Picture (289x64, 1.9Kb)
 
 
Take a 5 group study with the following group means: 10, 20, 30, 40, and 50 respectively. And say I wish to make the following comparisons:
  +   -
C1: 1+2+3 vs. 4+5
C2: 1+2 vs. 5
C3: 4 vs. 5

I therefore use the following weighted means:

C1: +10 +20 +30 vs. -40 -50
  +2(10) +2(20) +2(30) vs. -3(40) -3(50)

Picture (317x114, 2.6Kb)

C2: +10 +20 +0(30) vs. 0(40) -50
  +1(10) +1(20) +0(30) vs. 0(40) -2(50)

Picture (222x114, 2Kb)

C3: +0(10) +0(20) +0(30) +40 vs. -50
  +0(10) +0(20) +0(30) +1(40) vs. -1(50)

Picture (181x96, 1.7Kb)

Notice that I could have made one more comparison. I don't have to use up all my available comparisons. I just need to limit myself to the number of df's for the IV. In this case 4. Also notice that these comparisons are not orthogonal. They don't have to all be orthogonal. As long as they are of substantial meaning to the researcher, any set of comparisons may be made as long as they are limited in number to less than or equal to the number of df's for the IV. However, orthogonality of comparisons is usually desirable. To tell if the contrast weights are orthogonal, display them in a comparison by group matrix:

Group

  1 2 3 4 5
C1: +2 +2 +2 -3 -3
C2: +1 +1 0 0 -2
C3: 0 0 0 +1 -1

If comparisons are orthogonal, the sum of the cross products between each and all comparisons should be zero. Take C1 and C2 and sum the cross products:

grp1 grp2 grp3 grp4 grp5
C1C2 C1C2 C1C2 C1C2 C1C2
(2)(1) + (2)(1) + (2)(0) + (-3)(0) (-3)(-2) = -2

Since this is not equal to zero, these two comprisons are not orthognal. Let's look at C1 vs. C3:

grp1 grp2 grp3 grp4 grp5
C1C2 C1C2 C1C2 C1C2 C1C2
(2)(0) + (2)(0) + (2)(0) + (-3)(1) (-3)(-1) = 0

These two contrasts are orthogonal. For the sake of instruction, all of the following contrasts in the matrix below are orthogonal to each other:

Group

  1 2 3 4 5
C1: +2 +2 +2 -3 -3
C2: +1 -1 0 0 -2
C3: 0 0 0 +1 -1
 
 
The following short examples will give you a few more ideas about the properties associated with planned comparisons.
  • Signs used for coefficients are arbitrary.
  Control Time 1 Time 2 Time 3
Data: 10.4 12.4 14.2 18.3
Weights: -3 1 1 1

Picture (491x173, 3.7Kb)

  • When there is no treatment effect the calculated sums of squares will not significantly differ from 0.
  Control Time 1 Time 2 Time 3
Data: 21 21 21 21
Weights: 3 -1 -1 -1

Picture (352x52, 2Kb)

  • Planned Comparisons are insensitive to the direction of the effect. As the analyst you must decide if the observed result is in the predicted direction if the test was one tailed.

Time 3 > Time 2 > Time 1 >Control

  Control Time 1 Time 2 Time 3
Data: 10.4 12.4 14.2 18.3
Weights: -3 1 1 1

Picture (413x45, 2Kb)

Control > Time 1 > Time 2 > Time 3

  Control Time 1 Time 2 Time 3
Data: 10.4 12.4 14.2 18.3
Weights: -3 1 1 1

Picture (415x48, 2.1Kb)