Two-Sample Tests

Almost every interesting question in statistics is a comparison. Does this treatment beat that one? Do these two groups of customers behave differently? Did the change we made actually change anything? A one-sample test asks whether a single group differs from a fixed number; a two-sample test asks whether two groups differ from each other — and that is the question the world usually cares about. This chapter builds the four workhorse procedures (two independent means, paired means, two proportions, and the confidence intervals that go with them), shows you exactly how to decide which one a given study calls for, and works the arithmetic all the way through so you can see there is no magic in it.

Section 1Two independent means

Two groups, one question

You have measured a numeric outcome — a test score, a reaction time, a blood pressure, a delivery time — on two separate groups of subjects, and the group averages came out different. They always do. Two samples drawn from the very same population will differ by chance alone. The entire job of a two-sample test is to answer one question: is the gap we observed bigger than chance would comfortably produce?

The samples here are independent, which has a precise meaning: knowing any value in group 1 tells you nothing about any value in group 2, and no subject appears in both groups. Two randomly assigned treatment arms are independent. Men and women in one survey are independent. The same forty people measured in January and again in June are not — that is Section 2's problem.

Write the population means as μ₁ and μ₂. The hypotheses are about the difference:

H₀: μ₁ − μ₂ = 0     H_a: μ₁ − μ₂ ≠ 0

Notice what the null says: the two populations have the same mean. It does not say the two samples had the same mean — they plainly did not, or you would not be testing anything.

The test statistic, and where it comes from

Every test statistic in this book has the same shape: (what we saw) minus (what the null predicted), divided by how much that quantity naturally wobbles. Our estimate of μ₁ − μ₂ is x̄₁ − x̄₂. Its wobble is measured by the standard error of the difference. Because the two samples are independent, their variances add (never their standard deviations):

SE = √( s₁²/n₁ + s₂²/n₂ )

and the test statistic is

t = ( x̄₁ − x̄₂ − 0 ) / SE

Since we estimated both standard deviations from data, the reference curve is a t-distribution rather than the normal. The degrees of freedom are the awkward part. Welch's approximation — what statistical software gives you by default and what you should use by default — is

df = ( s₁²/n₁ + s₂²/n₂ )² ÷ [ (s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1) ]

This is rarely a whole number, and that is fine. Working by hand without software, you may round the df down to the nearest integer; that is deliberately conservative, giving a slightly wider interval and a slightly larger p-value, so you never overstate your evidence.

A worked example, all the way through

A trainer compares two coaching formats. Format A: n₁ = 12, x̄₁ = 78.4, s₁ = 8.2. Format B: n₂ = 15, x̄₂ = 71.9, s₂ = 9.6. Participants were randomly assigned, so the groups are independent. Test at α = 0.05, two-sided.

Step 1 — the observed difference. 78.4 − 71.9 = 6.5 points.

Step 2 — the standard error. First the two pieces:
s₁²/n₁ = 8.2²/12 = 67.24/12 = 5.6033
s₂²/n₂ = 9.6²/15 = 92.16/15 = 6.1440
Sum = 11.7473, so SE = √11.7473 = 3.4274.

Step 3 — the test statistic. t = 6.5 / 3.4274 = 1.896. The observed gap is about 1.9 standard errors from zero.

Step 4 — degrees of freedom.
Numerator: 11.7473² = 138.00
Denominator: 5.6033²/11 + 6.1440²/14 = 31.397/11 + 37.749/14 = 2.8543 + 2.6963 = 5.5506
df = 138.00 / 5.5506 = 24.9.

Step 5 — the p-value and the decision. With about 25 degrees of freedom, the two-sided cutoff at α = 0.05 is t* ≈ 2.06. Our 1.896 does not reach it; the two-sided p-value is roughly 0.07. Since 0.07 > 0.05, we fail to reject H₀.

Step 6 — say it in English. "A 6.5-point advantage for Format A was observed, but with these sample sizes and this much spread a gap at least that large, in either direction, would arise about 7 times in 100 even if the two formats were truly identical (t = 1.90, df ≈ 24.9, p ≈ 0.07). The data do not provide convincing evidence of a difference." Note what we did not say: we did not say the formats are equal. Failing to reject is a failure to convict, not a verdict of innocence — the study may simply be too small.

Section 2Paired samples

What makes a pair a pair

Sometimes the two columns of numbers are not two separate crowds of people but the same people twice, or two people deliberately matched to each other. Each observation in column 1 has a natural, non-arbitrary partner in column 2. That is a matched-pairs (or paired) design, and it shows up constantly:

Here is the beautiful part: a paired analysis is not a new theory. You compute each pair's difference, d = (value 1) − (value 2), throw away the original columns, and run an ordinary one-sample t-test on the single column of differences against a null value of 0.

H₀: μ_d = 0     H_a: μ_d ≠ 0     t = d̄ / ( s_d / √n )     df = n − 1

where n is the number of pairs, not the number of measurements. Eight people measured twice give you n = 8 and df = 7, not n = 16.

Because the procedure is a one-sample t-test, its conditions are the one-sample conditions — but applied to the column of differences, not to the original two columns. This is the point students most often skip.

In the worked example below there are only eight pairs, so normality of the differences carries real weight. The eight differences run from −3 to 14 with no gaps or outliers — consistent with a roughly symmetric population — so the t-procedure is reasonable. With a clearly skewed or outlier-ridden difference column and this few pairs, the Wilcoxon signed-rank test of Section 4 would be the safer choice.

Why pairing is worth doing

People differ enormously from one another, and that person-to-person variation is pure noise when you are trying to detect a treatment effect. Pairing subtracts that noise away: a naturally slow responder is slow in both columns, so the difference is unaffected by how slow they are. What is left in the difference column is the effect plus measurement noise — a much quieter signal to listen to. That is why a paired design can detect an effect with eight subjects that an independent design would miss with forty. Section 4 shows this on real numbers.

A worked example, all the way through

Eight participants completed a task; their reaction time in milliseconds was recorded before and after a short training module. Same eight people both times, so the data are paired. Test at α = 0.05, two-sided, with d = before − after (so a positive d̄ means they got faster).

Step 1 — mean difference. The differences sum to 12 + 5 − 3 + 9 + 14 + 2 + 7 + 10 = 56, so d̄ = 56/8 = 7.00 ms. (Reassuringly, this equals 412.25 − 405.25 — the mean of the differences is always the difference of the means.)

Step 2 — standard deviation of the differences. Deviations from 7: 5, −2, −10, 2, 7, −5, 0, 3. Squares: 25, 4, 100, 4, 49, 25, 0, 9, summing to 216.
s_d² = 216 / (8 − 1) = 30.857, so s_d = 5.555 ms.

Step 3 — standard error. SE = 5.555 / √8 = 5.555 / 2.8284 = 1.964 ms.

Step 4 — test statistic and df. t = 7.00 / 1.964 = 3.564, with df = 8 − 1 = 7.

Step 5 — p-value and decision. On 7 degrees of freedom the two-sided cutoff at α = 0.05 is 2.365, and the cutoff at α = 0.01 is 3.499. Our 3.564 clears both, so the two-sided p-value is a little under 0.01 — about 0.009. We reject H₀.

Step 6 — say it in English. "Reaction times improved by an average of 7.0 ms after training. A mean change at least this large, in either direction, would occur in fewer than 1% of studies if training truly had no effect (paired t = 3.56, df = 7, p ≈ 0.009), so there is convincing evidence that reaction time changed." Because this was a before/after study with no control group, we should stop short of crediting the training itself — practice effects and simple familiarity with the task are unaddressed rival explanations. The test is valid; the causal claim is limited by the design.

Section 3Two proportions

When the outcome is yes-or-no

Now the outcome is not a number but a category with two levels: clicked or did not click, recovered or did not recover, passed or failed. Each group is summarized by a sample proportion, p̂ = (number of successes) / (group size). "Success" simply labels the outcome you are counting; it carries no cheerful connotation.

H₀: p₁ − p₂ = 0     H_a: p₁ − p₂ ≠ 0

Here the reference curve is the standard normal, so the statistic is a z, not a t. The reason is that a proportion's standard deviation is determined entirely by the proportion itself — √(p(1−p)) — so once you assume a value for p there is no separate variance to estimate from data, and no degrees of freedom to lose.

The pooled proportion — and why it appears only in the test

The null hypothesis says the two population proportions are equal to some common value. If that is true, the best estimate of that common value uses every observation you have, ignoring group labels:

p̂_pool = (x₁ + x₂) / (n₁ + n₂)

and the standard error used for the test is built from that single pooled value:

SE_pool = √( p̂_pool(1 − p̂_pool) × (1/n₁ + 1/n₂) )     z = (p̂₁ − p̂₂) / SE_pool

A confidence interval, by contrast, is not built under the null — it is an estimate of whatever the difference actually is — so it uses the unpooled standard error with each group's own proportion. That distinction is the single most common slip in this section, and Section 5 returns to it.

The conditions are independence (as before) plus the success–failure condition: the normal approximation only works if you have enough of both outcomes. For the test, check it with the pooled proportion; all four of n₁p̂_pool, n₁(1−p̂_pool), n₂p̂_pool, n₂(1−p̂_pool) should be at least 10.

A worked example, all the way through

An email is sent in two versions. Version 1: 45 of 150 recipients responded. Version 2: 30 of 160 responded. Recipients were randomly assigned to a version. Test at α = 0.05, two-sided.

Step 1 — the sample proportions.
p̂₁ = 45/150 = 0.300   p̂₂ = 30/160 = 0.1875
Observed difference = 0.300 − 0.1875 = 0.1125, that is, 11.25 percentage points.

Step 2 — the pooled proportion. p̂_pool = (45 + 30)/(150 + 160) = 75/310 = 0.24194.

Step 3 — check success–failure. 150(0.24194) = 36.3 and 150(0.75806) = 113.7; 160(0.24194) = 38.7 and 160(0.75806) = 121.3. All four exceed 10. ✓

Step 4 — pooled standard error.
p̂_pool(1 − p̂_pool) = 0.24194 × 0.75806 = 0.18340
1/150 + 1/160 = 0.006667 + 0.006250 = 0.012917
Product = 0.18340 × 0.012917 = 0.0023688, so SE_pool = √0.0023688 = 0.04867.

Step 5 — the test statistic. z = 0.1125 / 0.04867 = 2.31.

Step 6 — p-value and decision. For the standard normal, the area beyond z = 2.31 in one tail is 0.0104. Doubling it for a two-sided test gives p = 0.021. Since 0.021 < 0.05, we reject H₀.

Step 7 — say it in English. "Version 1 drew responses from 30.0% of recipients versus 18.8% for Version 2, an 11.3-percentage-point difference. If the two versions were equally effective, a gap this large or larger, in either direction, would appear in about 2% of experiments (z = 2.31, p = 0.021), so there is convincing evidence that response rates differ." Because assignment was random, a causal reading is defensible here in a way it was not in Section 2.

Section 4Choosing the right test

Two questions get you there

Students often try to memorize a dozen tests. You do not need to. For the two-sample world, two questions settle it almost every time.

Question 1: what kind of outcome did you measure? If each subject produced a number you could sensibly average — minutes, dollars, millimetres of mercury — you are comparing means. If each subject produced a yes/no label you are counting, you are comparing proportions. (A 1–5 satisfaction rating is a genuine grey area; treating it as numeric is common and usually defensible, but say what you did.)

Question 2: how are the two groups related? If every value in one group has a specific partner in the other, the data are paired. Otherwise they are independent. This is answered by reading the study design, never by staring at the numbers.

What the choice actually costs you

This is not bookkeeping; the wrong choice changes the answer. Take the exact eight participants from Section 2 and pretend, wrongly, that the "before" and "after" columns came from two unrelated groups of eight people. The two column means are unchanged: 412.25 and 405.25, a difference of 7.00 ms. But now the person-to-person spread is back in the denominator. Working from those columns, s_before = 19.33 and s_after = 19.08, so

SE = √(19.33²/8 + 19.08²/8) = √(46.71 + 45.49) = √92.20 = 9.602

t = 7.00 / 9.602 = 0.73, df ≈ 14, p ≈ 0.48

The same seven-millisecond effect goes from p ≈ 0.009 (paired, correct) to p ≈ 0.48 (independent, wrong) — from convincing evidence to nothing at all. The paired standard error was 1.96 ms; the independent one was 9.60 ms, nearly five times larger, because it is dominated by the fact that participant 8 is simply slower than participant 2. Pairing sweeps that irrelevance out of the calculation. Analyze paired data as independent and you will routinely fail to find effects that are really there.

The reverse mistake is worse. If the data are genuinely independent, there is no way to line them up into pairs, and forcing a pairing (by sort order, or by row number in a spreadsheet) manufactures a relationship that does not exist and produces a p-value with no meaning at all.

Three decisions to make before you look

Fix these in advance, because deciding them after seeing the data invalidates the p-value:

• One-sided or two-sided? Two-sided is the honest default. A one-sided test is legitimate only when a difference in the opposite direction would be as uninteresting as no difference at all — and it must be chosen before the data are seen. Switching to one-sided because your two-sided p came out 0.08 is a well-known way of manufacturing a false positive.
• What is α? Conventionally 0.05, but that is a convention, not a law of nature. It is your tolerance for a Type I error — rejecting a true null.
• How many tests will you run? Every additional test is another chance at a false positive. Twenty independent tests at α = 0.05 on data with no real effects yield, on average, one "significant" finding. Decide your comparisons in advance, and correct for multiplicity if there are many.

Section 5Two-sample confidence intervals

From a verdict to a measurement

A test returns a verdict: reject, or fail to reject. That is a thin answer. It does not say how big the difference is, and "significant" and "important" are entirely different words. A confidence interval for the difference answers the better question — how big, give or take how much? — and every two-sample interval in this chapter has one shape:

(point estimate) ± (critical value) × (standard error)

The product of the last two columns is the margin of error. Nothing else changes from procedure to procedure.

Worked: the interval for Section 1's data

Return to the coaching study: x̄₁ − x̄₂ = 6.5, SE = 3.4274, df ≈ 24.9.

Step 1 — critical value. For 95% confidence with about 25 degrees of freedom, t* ≈ 2.06. (Rounding df down to 24 gives 2.064 — the difference is invisible at two decimals.)

Step 2 — margin of error. 2.06 × 3.4274 = 7.06 points.

Step 3 — the interval. 6.5 − 7.06 = −0.56 and 6.5 + 7.06 = 13.56, giving (−0.56, 13.56) points.

Step 4 — read it. The interval contains 0, so "no difference" is among the plausible values — exactly matching the earlier decision to fail to reject at α = 0.05. But it says far more than the test did: it says Format A could plausibly be a half-point worse or as much as thirteen and a half points better. That is an enormous range. The honest conclusion is not "the formats are the same"; it is "this study was too small to tell us anything useful, and it should be repeated with more participants."

The other two examples work identically. For the paired reaction times: 7.00 ± 2.365 × 1.964 = 7.00 ± 4.65, giving (2.35, 11.65) ms — entirely above zero, matching p ≈ 0.009. For the two email versions, using the unpooled SE = √(0.300×0.700/150 + 0.1875×0.8125/160) = √(0.001400 + 0.000952) = 0.04850, the interval is 0.1125 ± 1.96 × 0.04850 = 0.1125 ± 0.0951, giving (0.017, 0.208) — between 1.7 and 20.8 percentage points, all positive.

Duality — and the one place it leaks

For the two t-procedures, the interval and the two-sided test are two views of one calculation. A 95% interval that excludes 0 corresponds exactly to a two-sided test rejecting at α = 0.05; an interval containing 0 corresponds exactly to failing to reject. Both use the same point estimate, the same standard error, and the same critical value.

For two proportions this correspondence is only approximate, because the test uses the pooled standard error and the interval uses the unpooled one. When a result sits very near the boundary the two can disagree — a p-value of 0.048 alongside an interval whose lower limit is a hair below zero. Neither is a mistake; they are answering slightly different questions. Report both and describe the finding as borderline, which is what it is.

What a confidence interval actually means

Here is the sentence to memorize, because the tempting version is wrong. The parameter μ₁ − μ₂ is a fixed, unknown number. Your interval is what randomly varied — a different sample would have produced a different one. So the 95% describes the procedure: if you repeated this entire study endlessly, about 95% of the intervals it produced would cover the true difference. It does not say there is a 95% chance the true difference lies in the one interval you happen to be holding. That interval either covers the parameter or it does not; the coin has already landed.

The p-value needs the same care. p = 0.021 means: if the null were true, data at least this extreme would occur about 2.1% of the time. It is not the probability that the null is true, not the probability the result was a fluke, and not a measure of effect size. A trivial difference becomes "significant" with a large enough sample; a large difference stays "non-significant" with a small one. This is precisely why you report the interval alongside the p-value — the p-value tells you whether you can see the effect, and the interval tells you whether it is worth caring about.

ReferenceWords worth knowing

A quick reference for the terms in this chapter — skim it now, or circle back whenever a word trips you up.

Alternative hypothesis (H_a)

The claim that there is a difference; in this chapter usually μ₁ − μ₂ ≠ 0 or p₁ − p₂ ≠ 0.

Confidence interval

A range of plausible values for a parameter, built as (estimate) ± (critical value)(standard error). Its confidence level describes the long-run success rate of the procedure.

Degrees of freedom

The parameter that sets the shape of a t-curve; n − 1 for a paired test, Welch's approximation for two independent means.

Independent samples

Two groups in which no subject appears twice and no observation is naturally matched to another.

Margin of error

The half-width of a confidence interval: the critical value times the standard error.

Matched pairs

A design in which each observation in one condition has one specific partner in the other — the same subject twice, or deliberately matched subjects.

Nonparametric test

A procedure that does not assume a normal population, typically working from ranks; e.g., Wilcoxon rank-sum and Wilcoxon signed-rank.

Null hypothesis (H₀)

The claim of no difference between the two populations, against which the data are weighed.

Paired t-test

A one-sample t-test applied to the within-pair differences, testing μ_d = 0.

Point estimate

The single best guess at a parameter from the sample: here x̄₁ − x̄₂, d̄, or p̂₁ − p̂₂.

Pooled proportion

(x₁ + x₂)/(n₁ + n₂), the common proportion estimated under the null; used in the two-proportion test only.

Pooled two-sample t-test

The older variant that assumes equal population variances and combines them into one estimate; Welch's test is the safer default.

p-value

The probability of observing data at least as extreme as yours if the null hypothesis were true. It is not the probability that the null is true.

Sampling distribution

The distribution of a statistic across all possible samples; for a difference of means it is approximately a t-curve centered on the true difference.

Significance level (α)

The threshold, fixed in advance, at which a p-value counts as convincing; equivalently, the tolerated rate of Type I error.

Standard error

The estimated standard deviation of a statistic — how much the difference between the two samples would wobble from study to study.

Success–failure condition

The requirement of at least 10 expected successes and 10 expected failures in each group before trusting the normal approximation for proportions.

t-distribution

A bell curve with heavier tails than the normal, used when the standard deviation is estimated from data; it approaches the normal as df grows.

Test statistic

(Observed − hypothesized) divided by the standard error — the observed difference expressed in standard errors.

Two-sample t-test

The test of μ₁ − μ₂ = 0 from two independent samples of a numeric outcome.

Type I error

Rejecting a null hypothesis that is actually true — a false positive; its long-run rate is α.

Welch's t-test

The two-sample t-test that does not assume equal variances, using an approximate degrees-of-freedom formula; the recommended default.