Regression and Correlation Done Right
Fit a line, read its slope as a rate, check the four assumptions, and see why r and the slope are two views of one thing.
- Fit a least-squares line by hand on a small dataset
- Read the slope as a rate with units, and the intercept as a conditional mean
- Compute r and explain its exact relationship to the slope and the standard deviations
- Check the four regression assumptions on a residuals plot
- Explain when a fitted line should not be extrapolated
What it means
A regression line summarises how a response changes with a predictor . The least-squares line minimises the sum of squared vertical distances from the points to the line:
The slope is the correlation rescaled into the units of the data: multiply by the ratio of the two standard deviations and you get “how many units moves per unit of ”. This is why and always share a sign: they are the same relationship expressed on different scales.
Why it adds something
Correlation says that two variables move together; regression says how much, in units you can act on. “Correlation 0.7” is hard to use; “each extra hour of study is associated with 6.3 more points” is a sentence a decision can be built on.
Regression also produces the residuals — the parts of the line fails to explain — and those residuals are where every assumption check happens.
Worked by hand
Five students again: hours ; scores .
From the correlation lesson: , , , , .
Step 1 — the slope.
Step 2 — the intercept.
Step 3 — the line. .
Step 4 — read it. Each additional hour of study is associated with about 7.5 more points. The intercept, 41.65, is the predicted score for zero hours — a conditional mean, not a claim that a student who does nothing scores 41.65 in a causal sense.
Step 5 — residuals. Predicted versus actual:
| residual | |||
|---|---|---|---|
| 1 | 50 | 49.1 | +0.9 |
| 2 | 55 | 56.6 | −1.6 |
| 3 | 65 | 64.0 | +1.0 |
| 4 | 70 | 71.4 | −1.4 |
| 5 | 80 | 78.9 | +1.1 |
The residuals sum to zero (up to rounding), alternate sign, and stay small — a healthy fit for a straight line.
🧩 Challenge, think first, then reveal
Predict the score for 10 hours of study. What is the prediction, and why should you be cautious about it?
💡 Answer: $\hat{y} = 41.65 + 74.5 = 116.15$ — above the maximum possible score. This is extrapolation: the line was fitted on 1–5 hours and there is no evidence the relationship stays straight beyond that range. A score cannot exceed 100, so the model is already outside its valid domain.
The four assumptions
Check them on the residuals, not on the raw scatter:
- Linearity — residuals show no curve. A U-shaped pattern means the relationship bends.
- Equal spread — residuals fan out (or narrow) as grows, which is heteroscedasticity; the standard errors from the fit will then be wrong.
- Independence — residuals are not correlated with order of collection, group, or time.
- Approximate normality of residuals — needed for the p-values and intervals, less critical for the slope estimate itself.
import numpy as np
x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([50, 55, 65, 70, 80], dtype=float)
b1, b0 = np.polyfit(x, y, 1)
print(round(b1, 2), round(b0, 2)) # 7.45 41.65
resid = y - (b0 + b1 * x)
print(np.round(resid, 2)) # [ 0.9 -1.6 1.0 -1.4 1.1]With pandas and scipy you get the same slope plus the standard error you would report:
from scipy import stats
res = stats.linregress(x, y)
print(res.slope, res.intercept, res.rvalue, res.pvalue, res.stderr)Seeing it geometrically
The least-squares line is the one that minimises the total area of the squares drawn on each residual. Drag a point in the GeoGebra regression explorer and watch the line swing — and watch a single far-away point at the end of the range lever it much harder than a point near the middle.
When to use it
- Quantifying a relationship in the data’s own units, with a slope you can quote.
- Adjusting for a second variable, which is what multiple regression does and what saves you from Simpson’s paradox.
- Producing predictions inside the observed range, with the uncertainty attached.
Limitations
- A line cannot represent a curve. If the residuals show curvature, the model is wrong however high is.
- Correlation still is not causation. A regression slope inherits every confounder the correlation had.
- Influential points can set the slope. One point at an extreme can move the line dramatically; check by refitting without it.
- Extrapolation is dangerous beyond the fitted range, as the 10-hour example shows.
- Restricted range attenuates the slope. Fit only among high performers and the slope flattens.
Common pitfalls
- Reporting when you fitted a slope (or the reverse) without saying which.
- Reading the intercept as a real-world baseline when is outside the data.
- Ignoring the residual plot because was high.
- Calling the slope causal when the design was observational.
🧩 Challenges
🧩 Challenge, think first, then reveal
Two variables have , and . What is the slope of on ? And of on ? Why are they not reciprocals?
💡 Answer: $b_{y \mid x} = 0.6 \times 10/2 = 3$; $b_{x \mid y} = 0.6 \times 2/10 = 0.12$. Their product is $r^2 = 0.36$, not 1. Least squares minimises vertical distances, so swapping which variable is on the y-axis fits a different line; only when $r = \pm1$ do the two lines coincide.
🤔 Socratic Questions
- Why does the least-squares line pass through the point ?
- If , what is the least-squares slope, and what does the line predict for every ?
- Why do the residuals always sum to zero for a model with an intercept?
- What does a fan-shaped residual plot imply about the confidence interval for the slope?
✅ Quick check
1. A regression of score on hours gives a slope of 7.45. What does it mean?
2. The residual plot shows a clear U shape. What is the problem?
3. Why is extrapolating a fitted line beyond the data range risky?