Tukey or Bonferroni: which post hoc test after ANOVA?

Direct answer: A Tukey test compares every possible pair of group means after an ANOVA has told you that at least one mean differs. It controls the chance of a false positive across the whole set of pairs, so the overall error rate stays at the level you chose, usually .05.

The NIST/SEMATECH (n.d.-a) handbook describes Tukey's method as a way to test all pairwise differences simultaneously, with confidence exactly 1 − α when the groups are the same size. Lee and Lee (2018) add that the test uses the studentized range distribution and was originally called the "Honestly significant difference" test, which is why SPSS and SciPy label it Tukey HSD. Tukey (1949) is the early source. The handbook's worked example has four groups with 20 observations in all, so six pairwise comparisons, and it finds that the differences between groups 1 and 4 and between groups 2 and 3 are not significant while the other four pairs are. That is the typical shape of a Tukey result: a list of pairs, each marked as different or not, rather than a single yes or no for the whole experiment.

Direct answer: ANOVA asks one overall question: are all the group means equal? A Tukey test answers the follow-up question: which specific pairs differ? You run ANOVA first, and the post hoc test only matters when the omnibus F is significant, because only then is there a difference to locate.

Kent State University Libraries (n.d.) explain that the omnibus F in a one-way ANOVA does not tell you which mean differs, so you need either planned contrasts or a post hoc test, and that contrasts are decided a priori, before you see the data. Our guide to one-way ANOVA covers the F test, its degrees of freedom and the SPSS dialog in detail, and this post picks up where that one stops. The division of labour is simple. ANOVA protects you from declaring a difference when the data are consistent with all groups sharing one mean. The post hoc test then protects you again at the pairwise level, because comparing many pairs gives many chances for a chance difference to look real. Skipping the post hoc step leaves you with "something differs", which rarely answers a research question on its own.

Direct answer: Every extra test adds another chance of a false positive. With several independent t-tests at α = .05, the probability of at least one false positive across the family climbs quickly, which is why post hoc procedures adjust the criterion for each comparison.

Lee and Lee (2018, p. 354) give the formula for the inflated family-wise error rate, 1 − (1 − α)^N for N comparisons, and show that three pairs already push it to 0.143. The table below reproduces their figures for one to six comparisons at α = .05. Four groups need six pairwise tests, so the chance of at least one spurious "significant" pair is about 26 percent, more than five times what you intended. The Bonferroni correction, as Lee and Lee (2018, p. 358) describe it, divides α by the number of tests k, so that 50 t-tests would each be judged at .001. Tukey's test reaches the same goal differently, by using a distribution built for the largest difference among the means rather than for a single pair.

Direct answer: Neither wins every time. If you want all pairwise comparisons, Tukey has the edge. If you chose a small subset of comparisons before collecting data, Bonferroni may be better. The right choice is the method that gives the narrowest intervals for the comparisons you actually need.

The NIST/SEMATECH (n.d.-b) handbook states the rule in two sentences: "If all pairwise comparisons are of interest, Tukey has the edge. If only a subset of pairwise comparisons are required, Bonferroni may sometimes be better." It also says that "No single method of multiple comparisons is uniformly best" and advises choosing the method that gives the smallest confidence band. Lee and Lee place the two methods on a spectrum of strictness: "the Bonferroni method is more rigorous than the Tukey test, which tolerates type I errors, and more generous than the very conservative Scheffé's method" (Lee & Lee, 2018, p. 357). The same paper warns that Bonferroni can be unnecessarily conservative, with weak power, so it often fails to detect real differences, and that Holm and Hochberg procedures are less conservative alternatives. The table summarises the contrast using only what these sources report.

Direct answer: Use the Tukey-Kramer version for unequal group sizes and a variance-robust procedure such as Games-Howell or Dunnett's C when the variances differ. Check the variance assumption first, then choose the post hoc test that matches what you found in your own data.

Lee and Lee (2018, p. 355) explain that Tukey's studentized range method assumes equal sample counts, and that Kramer modified it for unbalanced designs by using the harmonic mean of the cell sizes, giving the Tukey-Kramer test. The NIST/SEMATECH (n.d.-a) handbook adds that with unequal sizes the confidence level is greater than 1 − α, which means the procedure is conservative. For unequal variances, Lee and Lee (2018, p. 360) list Tamhane T2, Dunnett T3, Games-Howell and Dunnett C as procedures that do not assume equal variances. Games and Howell (1976) is the early source for the procedure, and Lee and Lee (2018, p. 358) describe it as an improvement on Tukey-Kramer that uses a t-test with Welch degrees of freedom, applicable when each group has six or more cases. Kent State University Libraries (n.d.) point SPSS users to Dunnett's C in the "Equal Variances Not Assumed" panel. Our post on Levene's test shows how to check the variance assumption.

Direct answer: In SPSS, click Post Hoc in the One-Way ANOVA dialog and tick Tukey under Equal Variances Assumed, or a test such as Dunnett's C under Equal Variances Not Assumed. In Python, call scipy.stats.tukey_hsd with one array per group; set equal_var=False for Games-Howell.

Kent State University Libraries (n.d.) show that the SPSS Post Hoc dialog has two panels, Equal Variances Assumed and Equal Variances Not Assumed, and that you pick from the panel that matches your Levene result. The SciPy Developers (n.d.) documentation states that tukey_hsd with the default equal_var=True performs the Tukey HSD test for equal sample sizes and the Tukey-Kramer test for unequal ones, while equal_var=False performs the Games-Howell test. Their worked example compares headache relief for three brands, five people each. Brands 0 and 1 differ by −4.600 with p = .014, brands 0 and 2 by −0.260 with p = .980, and brands 1 and 2 by 4.340 with p = .020. Reading across, brand 1 stands apart from both others, while brands 0 and 2 are indistinguishable. These sources do not cover Excel, so use SPSS or Python when you need a documented, repeatable Tukey procedure.

Direct answer: Name the post hoc test and your reason for choosing it before you show any output, then report each pair's mean difference, confidence interval and adjusted p-value. State the correction explicitly so an examiner can see you did not shop for significant results.

Lee and Lee (2018, p. 360) reviewed 142 papers in three journals over ten years and found that 33 percent (47 of 142) applied no correction, 61 percent (86 of 142) used a correction without giving a rationale, and only 6.3 percent (9 of 142) used suitable methods; Bonferroni appeared in 35.9 percent. Those numbers show what examiners meet most often, and the way to stand out is to give the rationale. A sentence such as "Tukey HSD was used because all pairwise comparisons were of interest and group sizes were similar" is complete and checkable. If your variances differ, say that you used Games-Howell for that reason. Decide the test before you look at the output, because picking the procedure that yields the most significant pairs is a form of selective reporting. If you want a second pair of eyes on your tables, MAAS data analysis support lets a mentor work through your output with you, while you run and write up the analysis yourself.