← Chapter 13
Statistics · Chapter 13

Which Groups Actually Differ?

The ANOVA has come back significant. That is a real result — the arms of this trial are not all alike — but read it carefully, because it says less than it seems to. F is a single number built from every group at once, and a single number cannot point at a pair. It tells you something is going on among the means; it is silent on what. So the temptation is to glance at the chart, find the highest arm and the lowest, and announce that those two differ. That glance is exactly the trap. The most extreme pair in any sample is the one most flattered by chance, and testing every pair of arms at the ordinary 5% each quietly stacks the odds of a false alarm far above 5%. Below you can run a study whose F is genuinely significant, then work through the follow‑up comparisons two ways — one test at a time, and with a correction that keeps the whole family of tests honest — and simulate a thousand studies where nothing differs at all to watch the two approaches part ways.

The arms, with their means

Each dot is one person's outcome (a symptom score, lower is better); the heavy tick is the arm's sample mean and the thin whisker is a 95% interval for that mean built from the pooled within‑arm scatter. The dashed line is the grand mean of everyone. The ANOVA verdict for this sample is stamped on the banner to the right — but notice that the picture alone will not tell you, reliably, which specific arms are truly apart. Press Draw a new sample to feel how much the ordering of the middle arms jitters run to run even when the truth is fixed.
arm mean 95% interval for the mean grand mean

Every pair, compared

Now the follow‑up. Each row below is one pair of arms; the dot is the observed difference between their means and the bar is a confidence interval for that difference, using the pooled within‑arm variance from the ANOVA as the yardstick. When a bar clears the dashed zero line, that pair is declared different. The switch decides how wide each bar is drawn.
interval clears zero — pair declared different interval straddles zero — not shown to differ

Run a thousand studies where nothing differs

Here is the argument that settles it. Set every arm to the same true mean — so any pair we flag is a mistake — and draw study after study. For each one, first check the omnibus F, then run all m = k(k−1)/2 pairwise tests both ways. A study is a family‑wise error if it flags even one pair. Testing one at a time, that rate climbs well past 5% and keeps climbing as arms are added. The Bonferroni bars hold near 5% no matter how many arms there are — that is the whole point of dividing α by m.
Studies
run0
Comparisons
each study, m—
Omnibus F
false alarms—
A trial with several armsan ANOVA that has already come back significant

The study you are running

A trial with k arms and n people in each. Set the true separation and scatter, or jump to a situation worth seeing. Everything on the left recomputes from the sample you draw.
The design
4
12
The truth behind the arms
13.0
8.0
Jump to a situation worth seeing

The omnibus test, computed live

People
total N— 
F
statistic—MSb ÷ MSw
ANOVA
p‑value— 
Pooled SD
√MSwithin—the common scatter
Pairs to
compare, m—k(k−1) ÷ 2
Per‑test α
Bonferroni—.05 ÷ m
Pairs flagged
one at a time—of m
Pairs flagged
corrected—of m
The pairwise comparisons in full
PairDifferenceSEtp (raw)Verdict

Say it right

Nine things people say once the ANOVA turns up significant. Four of them are safe to write down. Tap each one.
Pick a statementEach one is either a fair reading of a significant F or a habit worth breaking. The four correct ones are the sentences you can defend.

What to carry out of this section

ANOVA is an omnibus test: one question, one answer. Its hypothesis is that every population mean is equal, and rejecting it means only that they are not all equal. Which ones, and by how much, is a separate question the F statistic never addresses. Treating a significant F as if it named a winning arm is reading a headline as though it were the whole article.
The follow‑up is where "which" gets answered — and it must pay for asking. Post‑hoc procedures (Bonferroni, Tukey's HSD, Scheffe) compare the pairs while widening every interval to account for how many pairs are being inspected. Bonferroni is the plainest: split the 5% budget evenly, testing each of the m pairs at .05 ÷ m. The intervals get wider, fewer pairs clear zero, and the price of that caution is missing some real but small differences — a trade every correction makes.
Testing every pair at .05 apiece is not a shortcut — it is a leak. Each test carries its own 5% chance of a false alarm, and with several arms there are many tests: three arms give three pairs, six arms give fifteen. Under a true null the chance that at least one pair looks significant climbs far above 5%, as the simulation on the left shows directly. The single omnibus F, and the corrected follow‑ups after it, exist precisely to cap that overall rate.
The p‑value is not the probability the arms are equal, and a significant F is not proof of cause. The ANOVA p is computed assuming the means are equal, so it cannot also be the probability of that assumption. And F compares variances; it knows nothing about how people landed in their arms. A causal reading is earned by random assignment at the design stage, never by the size of F.
A closing exercise. Load All arms equal and run five thousand studies. The Bonferroni bar sits near 5% the way it was built to; the one‑at‑a‑time bar sits far higher, and climbs further every time you add an arm. Both are drawing from a world with no difference in it whatsoever. That gap between the bars is the entire reason "which groups differ?" is a question you answer with a correction, not with your eyes.