A test hands back a verdict. An interval hands back a list of the differences the data can live with — which is nearly always the more useful object. Build one here from two live samples, stretch it and shrink it, and then face the question the whole section turns on: what does it mean when the interval happens to cover zero? A good deal less than most people assume, and occasionally a good deal more.
One study, one interval
Two groups get two treatments; the outcome is the drop in systolic pressure, in mmHg. Every dot below is one patient, freshly sampled from the truth you set on the right. The heavy tick in each band is that group’s sample mean. Subtract the two and you get a single number — and underneath, the interval built around it. Draw again and again: the dots move, the means move, and the interval slides along with them. The truth never moves.
treatment Atreatment Bthe intervalzero — no differencethe true difference
How the interval is built
Three ingredients and nothing else. The point estimate is the difference in sample means — that is where the interval is centred. The standard error says how far that estimate typically wanders from the truth. The critical value t* converts “typically” into the confidence you asked for. Multiply the last two, step that far in each direction, and you are done.
Notice the two levers. The centre is set by the data you happened to get; the arms are set by how much data you got and how confident you insisted on being. Sampling error moves the whole thing sideways. Sample size and confidence level change only the arms.
The same study at six confidence levels
Same patients, same means, same standard error — only t* changes. Ask for more confidence and the interval must widen to deliver it, because the only way to be surer of catching the truth is to cast a wider net. Watch the bars turn colour: on many settings there is a level at which the interval stops missing zero and starts covering it. That is worth sitting with. Whether a difference is “significant” is not a fact about the data alone; it is a fact about the data and the level someone chose before looking.
misses zerocovers zerozero
One hundred studies, one hundred intervals
The confidence level is a promise about the procedure, not about any one interval, and the only way to see a promise about a procedure is to run the procedure many times. Each thin bar below is one complete study — new patients in both groups, new means, new interval. Bars that catch the true difference are dark; the ones that miss it are red. You cannot tell which is which from inside a single study. That is precisely the difficulty the word “confidence” is built to describe.
caught the truthmissed itthe true differencezero
Studies run0
Caught the true difference—
Missed zero—
Average width—
Two treatmentsset the truth, then draw a study
The truth you are sampling from
In a real study not one of these numbers is knowable — that is the whole reason intervals exist. Here you set them, so you can grade the interval against an answer you already hold.
Treatment A
12.0
8.0
40
Treatment B
6.0
8.0
40
Confidence level
95%
Jump to a situation worth seeing
This study, computed live
Mean A x̄A—
Mean B x̄B—
Point estimate—x̄A − x̄B
True difference—you set this
Sample SD sA—
Sample SD sB—
Standard error—√(sA²/nA + sB²/nB)
Degrees of freedom—Welch
Critical value t*—
Margin of error—t* × SE
Interval width—2 × margin
Caught the truth?—
95% confidence interval for μA − μB—
The same three numbers also run the test. Divide the point estimate by the standard error and you have t; the two‑sided p‑value follows from the same t distribution with the same degrees of freedom. Because the interval and the test are built from identical parts, they can never contradict each other: the interval misses zero exactly when p < α. Two reports of one calculation.
t statistic—estimate ÷ SE
p‑value two‑sided—
Do they agree?—
Does the difference matter?
“Covers zero” and “misses zero” are only two of the five things an interval can tell you, and on their own they are the least interesting two. Decide first — before looking — how big a difference would actually change what you do. Everything smaller than that is noise you would ignore even if it were real. Set that threshold below and the interval sorts itself into one of five verdicts.
3.0
The pair that gets confused most often is the bottom two. Both intervals cover zero; they mean opposite things. A narrow interval hugging zero has ruled out every difference big enough to care about — that is a real, publishable, useful finding of no meaningful effect. A wide interval covering zero has ruled out almost nothing; it is consistent with no effect and with a large effect at the same time. Reporting the second as “no difference” is one of the most common errors in applied statistics.
Say it right
Ten things people say after building an interval for a difference. Four of them are sound. Tap each one.
0 of 10 tried
Start anywhereTap a statement to see whether it survives scrutiny. Several of these are things you will hear said out loud with total confidence.