Lesson M02.L03: Algebraic Properties of OLS and Decomposition of Variation
| Field | Detail |
|---|---|
| Module | M02 — Simple Linear Regression (Wooldridge Chapter 2) |
| Level | Graduate |
| Duration | 35 minutes |
| Learning Objective | State and verify the three algebraic properties of OLS residuals; derive and apply the SST = SSE + SSR decomposition; calculate R² from component sums. |
| Provenance | Wooldridge, Introductory Econometrics, 7th ed., Chapter 2, Section 2.3 |
Explanation
1. Recap: What OLS Gives Us
From M02.L02, the OLS estimators satisfy:
The fitted values and residuals are:
These algebraic properties hold in every sample — they follow directly from the OLS FOCs and require no distributional assumptions.
2. Three Algebraic Properties of OLS
Property 1: Residuals Sum to Zero
Proof: From FOC1 of the OLS minimisation:
Implication: The average residual is zero: \(\bar{\hat{u}} = 0\). OLS never systematically over- or under-predicts — by construction, the errors cancel out in aggregate.
Practical check: If your residuals don't sum to zero (up to floating-point rounding), you've made an error in computation.
Property 2: Residuals are Orthogonal to the Regressor
Proof: From FOC2:
Implication: The sample covariance between \(x\) and \(\hat{u}\) is zero. OLS ensures there is no linear pattern left between the regressor and the residuals. If there were, we could exploit it to improve the fit — OLS has already extracted all the linear information \(x\) contains about \(y\).
Note: This does not mean \(x\) and \(\hat{u}\) are independent — it only means they are linearly uncorrelated in the sample.
Property 3: The Regression Line Passes Through \((\bar{x}, \bar{y})\)
Proof: This is precisely what the intercept formula \(\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}\) guarantees. Rearranging: \(\bar{y} = \hat{\beta}_0 + \hat{\beta}_1 \bar{x}\). \(\checkmark\)
Implication: The OLS line is "anchored" at the sample centre of mass \((\bar{x}, \bar{y})\). Any candidate line through a different centre of mass would violate FOC1.
3. Decomposition of Total Variation: SST = SSE + SSR
Define three sums of squares:
| Term | Formula | Meaning |
|---|---|---|
| SST — Total Sum of Squares | \(\displaystyle\sum_{i=1}^n (y_i - \bar{y})^2\) | Total variation in \(y\) around its mean |
| SSE — Explained Sum of Squares | \(\displaystyle\sum_{i=1}^n (\hat{y}_i - \bar{y})^2\) | Variation in \(y\) explained by the model |
| SSR — Residual Sum of Squares | \(\displaystyle\sum_{i=1}^n \hat{u}_i^2\) | Variation in \(y\) left unexplained |
⚠️ Notation Warning: Wooldridge uses SSR for the residual (unexplained) sum of squares. Many other textbooks (e.g. Greene, Stock & Watson) use RSS for the same object and reserve SSR for a different quantity. Always check a textbook's conventions before using these acronyms.
The Fundamental Decomposition:
Derivation sketch: Write \(y_i - \bar{y} = (\hat{y}_i - \bar{y}) + \hat{u}_i\). Square both sides and sum over \(i\):
The cross-product term equals zero because of Properties 1 and 2 (specifically: \(\sum \hat{y}_i \hat{u}_i = \hat{\beta}_0 \sum \hat{u}_i + \hat{\beta}_1 \sum x_i \hat{u}_i = 0 + 0 = 0\), and \(\sum \bar{y} \hat{u}_i = \bar{y} \sum \hat{u}_i = 0\)). Hence SST = SSE + SSR. \(\checkmark\)
Key insight: This decomposition is a property of OLS, not of all possible regression methods. For example, minimising \(\sum |\hat{u}_i|\) (LAD regression) does not in general yield SST = SSE + SSR.
4. R-squared (Preview)
From the decomposition, we immediately define:
This is the fraction of total variation in \(y\) explained by the regression. Full treatment in M02.L04.
Worked Example
We continue with the Education–Wage dataset from M02.L02.
Recall: \(\bar{x} = 12\), \(\bar{y} = 12\), \(\hat{\beta}_1 = 1.35\), \(\hat{\beta}_0 = -4.20\).
Estimated equation: \(\widehat{\text{Wage}} = -4.20 + 1.35 \times \text{Educ}\)
Step 1: Compute fitted values and residuals
| \(i\) | \(x_i\) | \(y_i\) | \(\hat{y}_i = -4.20 + 1.35x_i\) | \(\hat{u}_i = y_i - \hat{y}_i\) |
|---|---|---|---|---|
| 1 | 8 | 7.25 | \(-4.20 + 10.80 = 6.60\) | \(7.25 - 6.60 = 0.65\) |
| 2 | 10 | 9.00 | \(-4.20 + 13.50 = 9.30\) | \(9.00 - 9.30 = -0.30\) |
| 3 | 12 | 11.50 | \(-4.20 + 16.20 = 12.00\) | \(11.50 - 12.00 = -0.50\) |
| 4 | 14 | 14.00 | \(-4.20 + 18.90 = 14.70\) | \(14.00 - 14.70 = -0.70\) |
| 5 | 16 | 18.25 | \(-4.20 + 21.60 = 17.40\) | \(18.25 - 17.40 = 0.85\) |
Step 2: Verify Property 1 — \(\sum \hat{u}_i = 0\)
Step 3: Verify Property 2 — \(\sum x_i \hat{u}_i = 0\)
| \(i\) | \(x_i\) | \(\hat{u}_i\) | \(x_i \hat{u}_i\) |
|---|---|---|---|
| 1 | 8 | 0.65 | 5.20 |
| 2 | 10 | −0.30 | −3.00 |
| 3 | 12 | −0.50 | −6.00 |
| 4 | 14 | −0.70 | −9.80 |
| 5 | 16 | 0.85 | 13.60 |
| Sum | 0.00 |
Step 4: Verify Property 3 — line passes through \((\bar{x}, \bar{y})\)
Step 5: Compute SST, SSE, SSR
| \(i\) | \(y_i - \bar{y}\) | \((y_i-\bar{y})^2\) | \(\hat{y}_i - \bar{y}\) | \((\hat{y}_i-\bar{y})^2\) | \(\hat{u}_i^2\) |
|---|---|---|---|---|---|
| 1 | −4.75 | 22.5625 | −5.40 | 29.16 | 0.4225 |
| 2 | −3.00 | 9.0000 | −2.70 | 7.29 | 0.0900 |
| 3 | −0.50 | 0.2500 | 0.00 | 0.00 | 0.2500 |
| 4 | 2.00 | 4.0000 | 2.70 | 7.29 | 0.4900 |
| 5 | 6.25 | 39.0625 | 5.40 | 29.16 | 0.7225 |
| Sum | 0 | 74.875 | 0 | 72.90 | 1.975 |
Verify decomposition:
R-squared:
Education explains approximately 97.4% of the total variation in wages in this sample. (This is very high; real-world wage regressions with a single regressor typically achieve \(R^2 \approx 0.10\)–\(0.30\).)
Common Misconception
"SST = SSE + SSR is a general mathematical identity for any fitted line."
This is false. The decomposition \(SST = SSE + SSR\) holds only when OLS is used to fit the line. The algebra relies on the OLS residuals satisfying Properties 1 and 2 — specifically, that \(\sum(\hat{y}_i - \bar{y})\hat{u}_i = 0\). This cross-product is zero because OLS residuals are orthogonal to \(x\) (and hence to \(\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\)).
If you impose any other line — say, a line chosen by eye, or a constrained line like \(\hat{\beta}_1 = 0\) forced in — the residuals will not satisfy Properties 1 and 2, and the cross-product term will not vanish. In that case, \(SSE + SSR \neq SST\).
A corollary: \(R^2\) as defined above (\(SSE/SST\)) is only meaningful for OLS regressions. Some software reports a "pseudo-\(R^2\)" for other estimators, but these are not the same quantity and are not bounded between 0 and 1 in the same way.
Practice Prompts
Problem 1
Using the commute-time dataset from M02.L02 Problem 1 (Population vs. Commute Time):
| City | \(x_i\) | \(y_i\) | \(\hat{y}_i\) | \(\hat{u}_i\) |
|---|---|---|---|---|
| A | 100 | 22 | ? | ? |
| B | 200 | 28 | ? | ? |
| C | 300 | 35 | ? | ? |
| D | 400 | 40 | ? | ? |
| E | 500 | 45 | ? | ? |
Recall from M02.L02: \(\hat{\beta}_0 = 16.6\), \(\hat{\beta}_1 = 0.058\), \(\bar{x} = 300\), \(\bar{y} = 34\).
(a) Fill in \(\hat{y}_i\) and \(\hat{u}_i\).
(b) Verify that \(\sum \hat{u}_i = 0\) and \(\sum x_i \hat{u}_i = 0\).
(c) Compute SST, SSE, SSR. Verify SST = SSE + SSR.
(d) Compute \(R^2\) and interpret it.
Show Answer
(a)
| City | \(\hat{y}_i = 16.6 + 0.058x_i\) | \(\hat{u}_i = y_i - \hat{y}_i\) |
|---|---|---|
| A | \(16.6 + 5.8 = 22.4\) | \(22 - 22.4 = -0.4\) |
| B | \(16.6 + 11.6 = 28.2\) | \(28 - 28.2 = -0.2\) |
| C | \(16.6 + 17.4 = 34.0\) | \(35 - 34.0 = 1.0\) |
| D | \(16.6 + 23.2 = 39.8\) | \(40 - 39.8 = 0.2\) |
| E | \(16.6 + 29.0 = 45.6\) | \(45 - 45.6 = -0.6\) |
(b) \(\sum \hat{u}_i = -0.4 - 0.2 + 1.0 + 0.2 - 0.6 = 0.0\) ✓
\(\sum x_i \hat{u}_i = 100(-0.4) + 200(-0.2) + 300(1.0) + 400(0.2) + 500(-0.6)\)
\(= -40 - 40 + 300 + 80 - 300 = 0\) ✓
(c)
\(SST = (22-34)^2 + (28-34)^2 + (35-34)^2 + (40-34)^2 + (45-34)^2\)
\(= 144 + 36 + 1 + 36 + 121 = 338\)
\(SSR = 0.16 + 0.04 + 1.00 + 0.04 + 0.36 = 1.60\)
\(SSE = 338 - 1.60 = 336.40\) ✓ (SST = SSE + SSR = 336.40 + 1.60 = 338 ✓)
(d) \(R^2 = 336.40/338 \approx 0.9953\). Population size explains about 99.5% of the variation in commute times in this sample. (With only 5 near-perfectly-linear observations, this is expected to be very high.)
Problem 2
A student claims: "I fitted a line to my data by eye and got SSE = 80. Then I calculated SST = 100. So my \(R^2 = 0.80\) and SSR must be 20."
Identify the flaw in this reasoning.
Show Answer
The student is applying \(SST = SSE + SSR\) to a line not fitted by OLS. This decomposition only holds for OLS. For an arbitrary line fitted by eye, the cross-product term \(2\sum(\hat{y}_i - \bar{y})\hat{u}_i\) does not equal zero, so it is entirely possible that \(SSE + SSR \neq SST\).
Furthermore, \(R^2 = SSE/SST\) is only meaningful for OLS. For a non-OLS line, SSE can actually exceed SST (i.e. you can compute a "negative \(R^2\)" if you're not careful), because the OLS line is the unique minimiser of SSR.
The student should either: (a) fit the line by OLS and use the standard formulas, or (b) explicitly compute SSR = \(\sum(y_i - \hat{y}_i)^2\) from their by-eye residuals and acknowledge that \(R^2 = 1 - SSR/SST\) is not the same as \(SSE/SST\) for non-OLS lines.
Problem 3
True or False — justify each:
(a) If \(R^2 = 1\), then \(\hat{u}_i = 0\) for all \(i\).
(b) If \(\sum \hat{u}_i = 0\), then \(\sum x_i \hat{u}_i = 0\).
(c) Adding an observation exactly on the regression line leaves \(R^2\) unchanged.
Show Answer
(a) True. \(R^2 = 1\) implies \(SSR = 0\), i.e. \(\sum \hat{u}_i^2 = 0\). Since each \(\hat{u}_i^2 \geq 0\), this forces every \(\hat{u}_i = 0\) — all observations lie exactly on the regression line.
(b) False. Property 1 (\(\sum \hat{u}_i = 0\)) and Property 2 (\(\sum x_i \hat{u}_i = 0\)) are separate conditions that both follow from OLS optimality. Property 1 alone does not imply Property 2. For example, a regression through the origin (no intercept) has Property 2 by construction but may not satisfy Property 1.
(c) False (in general). Adding an observation changes \(\hat{\beta}_0\), \(\hat{\beta}_1\), \(\bar{y}\), SST, SSE, and SSR — even if the new point happens to lie on the current regression line, re-estimating OLS with the new sample will generally shift the line and therefore change \(R^2\). The statement would only be exactly true if the new observation lay on the new regression line (i.e. the line re-estimated with \(n+1\) observations), which is a different condition.
Further Resources
- Wooldridge: Chapter 2, Section 2.3 (Properties of OLS on any sample), particularly the derivation of SST = SSE + SSR and the definition of \(R^2\)
- R: After
fit <- lm(y ~ x), useanova(fit)to see SSR (residual SS) and SSE (model SS) directly;sum(resid(fit))verifies Property 1 - Python:
results = ols(...).fit()→results.ssr(SSR),results.ess(SSE),results.centered_tss(SST) - Note on notation: If using Stock & Watson or Greene alongside Wooldridge, always check which symbol they use for residual SS vs explained SS — the naming conventions differ and can cause confusion on exams