Almost every real question is a comparison. Does this route get you to work faster than that one? The honest answer never comes from a single number, because you cannot measure a route — you can only sample it, a few mornings at a time, and every sample lands somewhere slightly different. So the quantity you actually care about, x̄A − x̄B, is not a fixed fact about the world. It is a statistic, and like every statistic it has a distribution. Below you control two populations of commute times and then draw real samples from them. Watch the observed gap jump around from draw to draw — sometimes overshooting the truth, sometimes landing on the wrong side of zero entirely. Then watch where its variability comes from: the two sample means each wobble, and their wobbles combine by adding variances, not standard errors. That one rule is the engine of this whole chapter.
One morning’s worth of data from each route
Each dot is one commute, timed in minutes. The pale hills are the two populations you set on the right — the truth, which in real life you never get to see. The solid vertical line in each strip is that sample’s mean; the bar beneath it is a 95% confidence interval for that route alone. Press Draw a new pair of samples repeatedly and watch how much the two means move even though the populations underneath never change.
Collapse it to the one number you care about
Two means are two facts; the difference is the answer. The curve below is what x̄A − x̄B would look like across endless repeats if the two routes truly took the same time on average — centred at zero, with width equal to the standard error you computed from these samples. The dark marker is the difference you actually got. The shaded tails are every outcome at least that far from zero in either direction; their combined area is the p‑value. Underneath, on the same scale, is the confidence interval for the difference.
if the two means were equalas far out as observed, or fartherthe difference you got
Where the wobble comes from: variances add
Route A’s mean has its own standard error. So does route B’s. Subtracting one from the other does not cancel the noise — it collects it, because two independent sources of error never conspire to line up. But they do not simply add either. They combine the way the sides of a right triangle do: square them, add, take the root. Drag the sample sizes and watch the triangle change shape.
Do it a thousand times
One pair of samples proves nothing. So repeat the whole study: each run below draws fresh samples from both populations and records only the difference between their means. The bars are those simulated differences. The dark line is the theoretical curve — centred on the true difference μA − μB, with spread √(σA²/nA + σB²/nB). Nothing was fitted; it is drawn straight from the formula. The bars landing on it is the claim of this section, verified.
simulated differencestheory, not fittedzero — no difference
Studies run0
Mean of the differences—
SD of the differences—
Share calling a difference—
Two routes to workset the truth, then go sampling
The truth you are sampling from
In a real study none of these six numbers is knowable — that is the entire problem. Here you get to set them, so you can check the machinery against an answer you already know.
Route A — the highway
34
7
25
Route B — the back roads
30
7
25
Jump to a situation worth seeing
How to combine the two spreads
This study, computed live
Route A mean x̄A—
Route B mean x̄B—
Observed difference—
True difference—you set this
SE of x̄A—sA ÷ √nA
SE of x̄B—sB ÷ √nB
SE of the difference—
Degrees of freedom—
t statistic—difference ÷ its SE
p‑value two‑sided—
95% confidence interval for μA − μB—
The error‑bar trap, checked against your own numbers
Look back at the two confidence intervals in the first panel — one per route. It is enormously tempting to eyeball them and conclude that overlapping bars mean no real difference. That shortcut is wrong, and it is wrong in a specific, predictable way. Here is what your current numbers say.
The reason is arithmetic, not opinion. Each individual bar reaches out by t* × SE for its own group, so judging “do they touch?” effectively tests the gap against SEA + SEB. But the difference is judged against √(SEA² + SEB²), which is always the smaller of the two. The eyeball test is therefore too conservative: bars can overlap while the difference is comfortably significant. Compare the two bars in the triangle panel above to see the size of the gap you are giving away.
Say it right
Nine things people say after comparing two means. Four of them are correct. Tap each one.
The section in one lineThe difference between two sample means is a statistic with its own sampling distribution: centred on the true difference, and spread out by √(SEA² + SEB²) — the two wobbles combined the way perpendicular sides combine, never by simple addition.
What to carry out of this section
Variances add; standard errors do not. If you remember one line, make it this one. For two independent samples, Var(x̄A − x̄B) = Var(x̄A) + Var(x̄B) — note the plus sign even though the statistic has a minus sign. Independent noise cannot cancel; it can only accumulate. Taking the square root at the end is what turns that sum of variances back into a standard error, and it is why the hypotenuse in the triangle above is shorter than the two legs laid end to end.
The word independent is doing real work. Everything on this page assumes the two samples are separate sets of observations — different mornings, different drivers, different subjects. If instead each observation in A is paired with a specific observation in B (the same driver on both routes, the same patient before and after), the two means are correlated, the variances no longer simply add, and this machinery gives the wrong answer. The fix is to take the difference within each pair first and analyse that single column of differences with one‑sample methods. Deciding which design you have is a question about how the data were collected, and no amount of computation will answer it for you.
What a small p‑value is not. It is not the probability that the two population means are equal, and it is not the probability that you are wrong. It is computed assuming the means are equal, and it reports how often a difference at least this extreme would show up under that assumption. Turning that around — treating it as the probability of the assumption — is the single most common error in applied statistics, and no sample size protects you from it.
Significant is not the same as important. Load the Tiny but certain preset. A half‑minute difference in commute time is worth nothing to anybody, but with 200 tightly clustered drives on each route the standard error falls to about a fifth of a minute — easily small enough to flag a gap that size, and the interval comes back at roughly a quarter to one minute. Statistical significance answers “can we tell this gap apart from zero?” — a question about precision. Whether the gap matters is a question about minutes, dollars, or symptoms, and the confidence interval, reported in the original units, is the tool that answers it. Always read the interval, not just the verdict.
A difference between groups is not automatically caused by the grouping. Every number on this page would be identical whether route A was assigned at random each morning or chosen deliberately on days the weather was bad. The statistics measure the size and precision of a gap; they say nothing about what produced it. Only the design — random assignment versus observed groups — licenses a causal reading, and a bigger t statistic never upgrades an observational comparison into a causal one.
One last thing worth noticing. Run the simulation with the No gap at all preset, where the true difference really is zero. The histogram centres on zero, as it must — but individual studies land as far as two or three minutes out on either side, and roughly one run in twenty declares a difference anyway. Those runs are not mistakes in arithmetic. They are the guaranteed cost of the 5% rule, showing up exactly as often as advertised.