Hypothesis Testing
The null hypothesis, the p-value, and the four decisions that follow: what a test establishes and what it never does.
- State a null and an alternative hypothesis for a concrete comparison
- Compute a t statistic by hand from two group means and their errors
- Interpret a p-value correctly, including the three common misreadings
- Explain the difference between statistical and practical significance
What it means
A hypothesis test asks how surprising the data would be if there were no effect. You state two hypotheses:
- , the null: there is no difference, the effect is zero;
- , the alternative: there is a difference, in a direction you state in advance.
Then you compute a statistic measuring how far the observed data sits from what predicts, convert it to a p-value — the probability of data at least this extreme if were true — and compare it to a threshold you set beforehand, conventionally 0.05.
For comparing two means, the statistic is a ratio:
The numerator is the effect you observed. The denominator is how much that difference would wobble under . A large ratio means the effect is large relative to the noise.
Why it adds something
It gives a disciplined answer to “could this just be chance?” — the question every comparison raises. It forces the direction and the threshold to be decided before seeing the data, which is what protects against the human talent for finding a story in noise.
It is also where a great deal of published statistical error lives, so knowing it is defensive as much as constructive.
Worked by hand
Two groups of deliveries, one with a new routing system:
- Current route: , mean 18.0 min, standard deviation 6.0.
- New route: , mean 15.0 min, standard deviation 5.0.
Step 1 — the hypotheses. : the means are equal. : the new route is faster.
Step 2 — standard errors.
Step 3 — the standard error of the difference. For independent groups the variances add:
Step 4 — the statistic.
Step 5 — the p-value. With roughly 58 degrees of freedom, a of 2.10 is exceeded in one tail with probability about 0.020. Since says “faster”, the one-tailed p-value is about 0.02; two-tailed it is about 0.04. Either way it is below 0.05.
Step 6 — the decision, stated honestly. “The new route is associated with a 3-minute shorter mean delivery time, , . The confidence interval for the difference is roughly 0.1 to 5.9 minutes.” Note how much more the interval says than the verdict: the effect could be trivial or substantial.
🧩 Challenge, think first, then reveal
The same 3-minute difference is found with per group. What happens to the p-value, and does the finding become more important?
💡 Answer: The standard errors shrink by a factor of about 3.2, so $t$ rises to roughly 6.7 and the p-value becomes tiny. The effect is unchanged at 3 minutes — the finding is more certain, not larger. This is the difference between statistical significance (evidence that an effect exists) and practical significance (whether it is worth acting on).
What a p-value is not
- Not the probability that is true. It is computed assuming is true.
- Not the probability you are wrong. It is a statement about data, not about your conclusion.
- Not a measure of effect size. A tiny effect with a huge sample yields a tiny p-value.
- Not a licence to fish. Testing many hypotheses and reporting the smallest p-value guarantees false positives; with 20 tests at 0.05 you expect one spurious “significant” result.
The same thing in Python
from scipy import stats
current = [17, 18, 19, 18, 16, 20] # illustrative samples
new = [14, 15, 16, 15, 13, 17]
res = stats.ttest_ind(new, current, alternative='less')
print(round(res.statistic, 2), round(res.pvalue, 4))Always print the effect and its interval alongside the test:
import statistics as st
diff = st.mean(current) - st.mean(new)
se = (st.stdev(current)**2 / len(current) + st.stdev(new)**2 / len(new)) ** 0.5
print(round(diff, 2), round(diff - 2*se, 2), round(diff + 2*se, 2))Seeing it geometrically
The t statistic is the distance between two sampling distributions measured in their own spread. In the GeoGebra t-test explorer you can slide the group means apart or enlarge the samples and watch the overlap shrink — the same picture as the p-value falling.
When to use it
- A pre-specified comparison between two groups or against a benchmark.
- Checking whether an observed difference is larger than what sampling noise would produce.
- As one input to a decision that also weighs cost, effect size and context.
Limitations
- It answers a narrow question — “is the effect zero?” — and says nothing about size or importance.
- Assumptions matter. Independence, and roughly normal data or large samples; violating them invalidates the p-value.
- Multiple comparisons inflate false positives unless corrected.
- It rewards large samples with significance for effects nobody cares about.
- It cannot rescue a biased design. A test on a biased sample gives a precise answer to the wrong question.
Common pitfalls
- Choosing the hypothesis after seeing the data, which quietly converts an exploratory finding into a spurious confirmation.
- Reporting p < 0.05 as “proven”; it is evidence against a null model, not proof of a mechanism.
- Hiding the effect size. A p-value without a mean difference and an interval is uninterpretable.
- Confusing one- and two-tailed tests, which changes the threshold without changing the data.
🧩 Challenges
🧩 Challenge, think first, then reveal
A study reports “p = 0.049, significant” and another “p = 0.051, not significant” for the same effect size and sample size. Is one finding real and the other not?
💡 Answer: No. The two p-values are essentially the same number, and the 0.05 line is a convention, not a fact about the world. Treating them as opposite conclusions is the "difference between significant and not significant is not itself significant" error. Report both with their intervals and let the reader see how similar the evidence is.
🤔 Socratic Questions
- Why must the hypothesis be stated before looking at the data?
- If a test is not significant, have you shown there is no effect?
- What does a 95% confidence interval add that a p-value does not?
- Why does testing twenty hypotheses guarantee roughly one false positive at the 0.05 level?
✅ Quick check
1. What does a p-value of 0.03 mean?
2. A difference is statistically significant with a huge sample. What else must you check?