Lesson M03.L02: OLS Estimation in Multiple Regression
Module: M03 — Multiple Regression Analysis
Level: Graduate (ECMT5001, USYD S1 2027)
Duration: 35 min
Learning Objective: Derive OLS estimators in multiple regression; interpret \(R^2\) and adjusted \(\bar{R}^2\); understand why \(R^2\) alone is insufficient for model selection.
Provenance: Wooldridge, Introductory Econometrics, Chapter 3 (Sections 3.2–3.3)
Explanation
OLS in Multiple Regression: The Objective
Given the model \(y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u\), OLS chooses estimates \(\hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_k\) to minimise the sum of squared residuals (SSR):
In simple regression (one regressor) the first-order conditions give explicit formulas. With \(k > 1\) regressors the FOCs are a system of \(k+1\) equations — solved most cleanly in matrix notation.
Matrix Notation (Awareness Level)
Stack the data into matrices:
The OLS estimator in closed form is:
Requirements for this formula to exist:
- \((\mathbf{X}'\mathbf{X})\) must be invertible — equivalent to MLR.3 (no perfect collinearity)
- \(n > k + 1\) — more observations than parameters
You are not expected to derive this result from scratch in ECMT5001, but you should recognise the notation and understand that the scalar formulas for \(k=1\) are a special case.
Fitted Values and Residuals
Once \(\hat{\boldsymbol{\beta}}\) is obtained:
These definitions are identical to simple regression — the OLS residual is always the actual minus fitted value.
Algebraic Properties of OLS
These properties follow mechanically from the first-order conditions and hold for any OLS regression (simple or multiple):
| Property | Statement |
|---|---|
| Zero mean residuals | \(\sum_{i=1}^n \hat{u}_i = 0\) |
| Orthogonality | \(\sum_{i=1}^n x_{ij}\hat{u}_i = 0\) for each \(j = 1,\ldots,k\) |
| Sample means on surface | \((\bar{x}_1, \ldots, \bar{x}_k, \bar{y})\) lies on the OLS regression surface |
| SST decomposition | \(SST = SSE + SSR\) where \(SSE = \sum(\hat{y}_i - \bar{y})^2\) |
The orthogonality conditions say the residuals are uncorrelated with every included regressor in the sample. This is a mechanical property — it does not imply \(E(u|x_j) = 0\) in the population (that requires Assumption MLR.4).
\(R^2\) in Multiple Regression
This is identical to the simple-regression formula. \(R^2\) is the fraction of the total variation in \(y\) explained by all regressors jointly.
Key mechanical fact: \(R^2\) never decreases when you add a regressor. Why? OLS can always set the new coefficient to zero and reproduce the old fit; any nonzero coefficient can only reduce SSR further.
This makes \(R^2\) a poor model-selection criterion — it will always favour the kitchen-sink model.
Adjusted \(R^2\) (Penalised Fit)
To penalise for adding extra parameters, define:
| Component | Meaning |
|---|---|
| \(SSR/(n-k-1)\) | Estimated residual variance \(\hat{\sigma}^2\) — penalises for each parameter used |
| \(SST/(n-1)\) | Sample variance of \(y\) — fixed |
Properties of \(\bar{R}^2\):
- Can be negative (when the regression fits worse than the intercept-only model)
- Decreases when an added variable contributes less than its "fair share" to fit reduction
- \(\bar{R}^2 \leq R^2\) always (equality when \(k=0\))
- A regressor raises \(\bar{R}^2\) if and only if its \(|t\text{-stat}| > 1\) (approximately)
where \(s_y^2 = SST/(n-1)\) is the sample variance of \(y\). This framing makes clear: adding a regressor raises \(\bar{R}^2\) iff it reduces the estimated residual variance \(\hat{\sigma}^2\).
Which to Report?
- \(R^2\): useful for describing how much variation is explained; always report it.
- \(\bar{R}^2\): use for comparing models with different numbers of regressors on the same sample and same \(y\).
- Neither is appropriate for comparing models with different transformations of \(y\) (e.g., \(\ln y\) vs \(y\)), because \(SST\) changes.
Worked Example
A researcher estimates two models on \(n = 100\) observations of Australian workers:
Model 1: \(\widehat{\ln(\text{wage})} = \hat{\beta}_0 + \hat{\beta}_1\,\text{educ}\)
| Statistic | Value |
|---|---|
| \(SSR_1\) | 48.6 |
| \(SST\) | 72.0 |
| \(R^2\) | 0.325 |
| \(\bar{R}^2\) | ? |
Model 2: \(\widehat{\ln(\text{wage})} = \hat{\beta}_0 + \hat{\beta}_1\,\text{educ} + \hat{\beta}_2\,\text{exper} + \hat{\beta}_3\,\text{tenure}\)
| Statistic | Value |
|---|---|
| \(SSR_2\) | 40.3 |
| \(SST\) | 72.0 |
| \(R^2\) | 0.440 |
| \(\bar{R}^2\) | ? |
Step 1: Compute \(\bar{R}^2\) for Model 1 (\(k=1\), \(n=100\))
Step 2: Compute \(\bar{R}^2\) for Model 2 (\(k=3\), \(n=100\))
Step 3: Compare
| Model | \(k\) | \(R^2\) | \(\bar{R}^2\) |
|---|---|---|---|
| 1 (educ only) | 1 | 0.325 | 0.318 |
| 2 (educ + exper + tenure) | 3 | 0.440 | 0.423 |
Both \(R^2\) and \(\bar{R}^2\) rise from Model 1 to Model 2, confirming that experience and tenure are jointly contributing meaningfully to explained variation — they are not just soaking up degrees of freedom.
Step 4: Compute \(\hat{\sigma}^2\) for Model 2
This is the estimated variance of the error term \(u\), used in constructing standard errors for each \(\hat{\beta}_j\).
Common Misconception
\(R^2\) increasing when you add a variable means the variable belongs
False. \(R^2\) mechanically increases with every added regressor, even noise. A variable with a \(t\)-statistic of 0.1 will still raise \(R^2\). Always check (1) \(\bar{R}^2\) — did it rise? (2) the \(t\)-statistic — is the coefficient statistically different from zero? (3) economic theory — does the variable belong in the model on substantive grounds? All three should support inclusion.
Practice Prompts
-
Algebraic property. Prove that \(\sum_{i=1}^n \hat{u}_i = 0\) in any OLS regression that includes an intercept. (Hint: The intercept column in \(\mathbf{X}\) is a column of ones; use the orthogonality condition \(\sum x_{i0}\hat{u}_i = 0\) where \(x_{i0} = 1\).)
-
Adjusted \(R^2\) calculation. A researcher estimates a model with \(n = 50\), \(k = 5\), and reports \(R^2 = 0.72\). Calculate \(\bar{R}^2\). Then suppose they add 5 more irrelevant regressors and \(R^2\) rises to 0.76. Calculate the new \(\bar{R}^2\) and explain whether the additional regressors improved the model.
-
Model comparison. Two models are estimated on the same \(n=200\) sample:
- Model A: \(y = \beta_0 + \beta_1 x_1 + u\); \(\bar{R}^2 = 0.41\)
- Model B: \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_4 x_4 + u\); \(\bar{R}^2 = 0.39\)
Which model do you prefer and why? What does it imply about \(x_2, x_3, x_4\)?
Further Resources
- Wooldridge, Ch 3.2–3.3 (pp. 85–102)
- Greene, Econometric Analysis, Ch 3 (full matrix derivation)
- R/Python:
lm()/statsmodels.OLS— both report \(R^2\) and adjusted \(R^2\) automatically