Lesson M03.L05: Multicollinearity and Variance of OLS

Module: M03 — Multiple Regression Analysis
Level: Graduate (ECMT5001, USYD S1 2027)
Duration: 35 min
Learning Objective: Derive the variance formula for OLS in multiple regression; define the Variance Inflation Factor (VIF); distinguish perfect from imperfect multicollinearity; explain consequences for inference without confusing multicollinearity with bias.
Provenance: Wooldridge, Introductory Econometrics, Chapter 3 (Section 3.4)

Explanation

Variance of \(\hat{\beta}_j\) in Multiple Regression

Under MLR.1–MLR.5, the sampling variance of the OLS estimator for \(\beta_j\) is:

\[\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{SST_j \cdot (1 - R_j^2)}\]

where:

Symbol	Definition
\(\sigma^2\)	Population error variance (\(\text{Var}(u \mid \mathbf{x})\))
\(SST_j\)	\(\displaystyle\sum_{i=1}^n (x_{ij} - \bar{x}_j)^2\) — total sample variation in \(x_j\)
\(R_j^2\)	\(R^2\) from regressing \(x_j\) on all other regressors \(x_1,\ldots,x_{j-1},x_{j+1},\ldots,x_k\)

The standard error (square root of estimated variance) is:

\[\widehat{SE}(\hat{\beta}_j) = \frac{\hat{\sigma}}{\sqrt{SST_j \cdot (1 - R_j^2)}}\]

where \(\hat{\sigma}^2 = SSR/(n-k-1)\).

Three Ways to Get Small Variance

From the formula, \(\text{Var}(\hat{\beta}_j)\) is small when:

\(\sigma^2\) is small — the regression model fits well; not much unexplained variation in \(y\)
\(SST_j\) is large — plenty of variation in \(x_j\) in the sample; more signal
\(R_j^2\) is small — \(x_j\) is not well explained by the other regressors; no multicollinearity

The third factor is what makes multicollinearity costly.

Variance Inflation Factor (VIF)

Define the Variance Inflation Factor for \(\hat{\beta}_j\):

\[VIF_j = \frac{1}{1 - R_j^2}\]

So:

\[\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{SST_j} \cdot VIF_j\]

The factor \(\sigma^2/SST_j\) is what the variance would be if \(x_j\) were orthogonal to all other regressors (\(R_j^2 = 0\), so \(VIF_j = 1\)). Multicollinearity inflates this baseline by the factor \(VIF_j \geq 1\).

Rule of thumb:

\(VIF_j\)	Interpretation
\(1\)	No collinearity; baseline variance
\(1\)–\(5\)	Mild; generally acceptable
\(5\)–\(10\)	Moderate; monitor closely
\(> 10\)	Severe; consider remedies

Some textbooks use VIF \(> 10\) as the threshold; others use VIF \(> 5\). Judgement and context matter.

Perfect Multicollinearity (MLR.3 Violated)

Perfect multicollinearity: \(R_j^2 = 1\) — one regressor is an exact linear combination of others.

\[\text{Var}(\hat{\beta}_j) = \frac{\sigma^2}{SST_j \cdot (1 - 1)} = \frac{\sigma^2}{0} = \infty\]

OLS breaks down completely — the matrix \(\mathbf{X}'\mathbf{X}\) is singular and \((\mathbf{X}'\mathbf{X})^{-1}\) does not exist. Software will drop one of the collinear columns or throw an error.

Common causes:

Dummy variable trap: Including \(d_{\text{male}}\) and \(d_{\text{female}}\) with an intercept (they sum to 1)
Exact definitional relationship: Including both total_tax and income_tax + GST + other_tax when the latter three sum exactly to the former
Redundant transformation: Including \(x\) and \(2x\) simultaneously

Imperfect Multicollinearity

Imperfect multicollinearity: \(R_j^2\) close to (but not equal to) 1.

OLS is still computable, unbiased, and BLUE — MLR.3 is not violated. However:

\(VIF_j\) is large → \(\text{Var}(\hat{\beta}_j)\) is large
Standard errors are large
\(t\)-statistics are small → it is hard to detect that \(\beta_j \neq 0\) even when it truly is (Type II error)
Coefficient estimates are sensitive to small changes in the sample (overfitting)

The core problem: with highly correlated regressors, the data contain insufficient independent information to precisely estimate the separate effects.

Economic Causes of Multicollinearity

Aggregate macroeconomic variables: GDP, consumption, and investment all trend together — high pairwise correlations in time series.
Overlapping demographic controls: Age, years of experience, and years since leaving school are all highly correlated life-stage variables.
Physical characteristics: Height and weight are highly correlated for human subjects.
Lagged variables: Including \(x_t\), \(x_{t-1}\), \(x_{t-2}\) in a time-series model creates near-perfect collinearity if \(x\) is a smooth series.

Remedies for Multicollinearity

Remedy	How	Caution
Collect more data	Larger \(n\) reduces \(\text{Var}(\hat{\beta}_j)\) directly	Not always feasible; may not solve collinearity if correlation is structural
Drop a variable	Remove one collinear regressor	Only valid if the dropped variable doesn't belong — otherwise introduces OVB
Principal components	Replace correlated \(x\)'s with uncorrelated linear combinations	Components lose economic interpretability
Ridge regression	Add penalty \(\lambda\\|\boldsymbol{\beta}\\|^2\) to SSR	Introduces bias; requires choice of tuning parameter \(\lambda\)

Note: If the collinear variable belongs in the model (non-zero \(\beta_j\)), dropping it trades multicollinearity for OVB — often a worse problem. The preferred solution is more data.

Worked Example

Setup: A researcher regresses log-housing prices on three regressors: \(x_1\) = floor area (m²), \(x_2\) = number of bedrooms, \(x_3\) = number of bathrooms. All are likely correlated.

Auxiliary regressions (to compute \(R_j^2\)):

\(j\)	Regress \(x_j\) on others	\(R_j^2\)	\(VIF_j\)
1 (floor area)	\(x_2, x_3\)	0.82	\(1/(1-0.82) = 5.56\)
2 (bedrooms)	\(x_1, x_3\)	0.75	\(1/(1-0.75) = 4.00\)
3 (bathrooms)	\(x_1, x_2\)	0.88	\(1/(1-0.88) = 8.33\)

Baseline variance (if \(R_j^2 = 0\)):

Suppose \(\hat{\sigma}^2 = 0.04\) and \(SST_3 = 200\) (variation in bathrooms). Then:

\[\text{Var}(\hat{\beta}_3)_{\text{baseline}} = \frac{0.04}{200} = 0.0002\]

\[\text{Var}(\hat{\beta}_3) = 0.0002 \times VIF_3 = 0.0002 \times 8.33 = 0.001666\]

\[SE(\hat{\beta}_3) = \sqrt{0.001666} \approx 0.0408 \quad \text{vs baseline } \sqrt{0.0002} \approx 0.0141\]

The standard error for \(\hat{\beta}_3\) is 2.89 times larger than it would be with no multicollinearity (since \(\sqrt{VIF_3} = \sqrt{8.33} \approx 2.89\)). This makes the \(t\)-statistic for \(\beta_3\) roughly 2.89 times smaller — harder to achieve statistical significance even if bathrooms genuinely affect price.

Diagnosis: \(VIF_3 = 8.33\) is in the "moderate to severe" range. The researcher should check whether floor area and bathrooms are nearly perfectly collinear, possibly because large homes always have many bathrooms. Adding the number of floors or age of building as an alternative dimension of heterogeneity may help.

Common Misconception

Multicollinearity biases OLS estimates

False. Under MLR.1–MLR.4, OLS is unbiased regardless of multicollinearity (as long as MLR.3 holds, i.e., no perfect collinearity). High \(R_j^2\) inflates \(\text{Var}(\hat{\beta}_j)\) and standard errors — affecting efficiency and inference — but does not bias \(E(\hat{\beta}_j)\). Confusing multicollinearity with OVB is one of the most common errors in applied econometrics.

Dropping a correlated variable fixes multicollinearity without cost

Dropping a variable reduces multicollinearity but only "fixes" the problem costlessly if the dropped variable has \(\beta_j = 0\) in the true model. If \(\beta_j \neq 0\), you have traded a precision problem (large SE) for a bias problem (OVB) — and bias is generally worse because it cannot be fixed by collecting more data.

Practice Prompts

VIF calculation. You have three regressors: income (\(x_1\)), consumption (\(x_2\)), savings (\(x_3\)), where by definition \(\text{savings} = \text{income} - \text{consumption}\).
(a) What is \(R_3^2\) from regressing savings on income and consumption?
(b) What is \(VIF_3\)?
(c) What does this imply for \(\text{Var}(\hat{\beta}_3)\)?
(d) How would you fix this problem?
Variance formula. In a regression of test scores on class size (\(x_1\)) and school funding (\(x_2\)) with \(n = 200\) Australian schools: \(\hat{\sigma}^2 = 100\), \(SST_1 = 400\), \(R_1^2 = 0.65\).
(a) Calculate \(\text{Var}(\hat{\beta}_1)\) and \(SE(\hat{\beta}_1)\).
(b) If the sample grew to \(n = 800\) (quadrupling), roughly how would \(SE(\hat{\beta}_1)\) change? (Hint: \(SST_1\) scales approximately with \(n\).)
Remedies trade-off. A researcher finds VIF = 12 for one regressor. List two remedies and, for each, state the condition under which it is valid without introducing new bias. Then explain which remedy is theoretically preferable if the researcher has access to a much larger dataset.

Further Resources

Wooldridge, Ch 3.4 (pp. 96–105)
Belsley, Kuh & Welsch, Regression Diagnostics (1980) — definitive reference on collinearity diagnostics
R: car::vif() computes VIF automatically after lm()
Python: statsmodels.stats.outliers_influence.variance_inflation_factor()

The Frisch-Waugh-Lovell (FWL) Theorem

The FWL theorem reveals the exact geometric meaning of "controlling for other variables" in multiple regression — one of the most important insights in applied econometrics.

Theorem: In the regression \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + u\), the OLS estimate \(\hat{\beta}_1\) is numerically identical to the slope from the simple regression of \(\tilde{y}\) on \(\tilde{x}_1\), where:

\[\tilde{x}_1 = M_2 x_1, \qquad \tilde{y} = M_2 y\]

and \(M_2 = I - X_2(X_2'X_2)^{-1}X_2'\) is the residual maker that projects out \(x_2\) (and the constant).

\[\hat{\beta}_1^{\text{MLR}} = \frac{\sum \tilde{x}_{1i} \tilde{y}_i}{\sum \tilde{x}_{1i}^2}\]

In plain words:

\(\tilde{x}_{1i}\) = the part of \(x_1\) not explained by \(x_2\) (residuals from regressing \(x_1\) on \(x_2\))
\(\tilde{y}_i\) = the part of \(y\) not explained by \(x_2\) (residuals from regressing \(y\) on \(x_2\))
\(\hat{\beta}_1\) = slope relating these two "cleaned" variables

This is literally and algebraically what "holding \(x_2\) constant" means.

Worked Example — Wages, Education, Experience

Suppose we regress \(\ln(\text{wage})\) on education (\(\text{educ}\)) and experience (\(\text{exper}\)):

\[\ln(\text{wage}) = \beta_0 + \beta_1 \cdot \text{educ} + \beta_2 \cdot \text{exper} + u\]

By FWL, \(\hat{\beta}_\text{educ}\) can be computed as:

Regress \(\text{educ}\) on \(\text{exper}\): \(\text{educ} = \delta_0 + \delta_1 \cdot \text{exper} + v\) → get residuals \(\tilde{\text{educ}}_i\)
Regress \(\ln(\text{wage})\) on \(\text{exper}\): get residuals \(\tilde{y}_i\)
Simple regression \(\tilde{y}_i\) on \(\tilde{\text{educ}}_i\): slope \(= \hat{\beta}_\text{educ}\) (identical to multiple regression)

\(\tilde{\text{educ}}_i\) captures variation in education that is orthogonal to experience — education differences that cannot be explained by differences in experience. This is the "clean" variation the multiple regression uses to estimate the education return.

Why FWL matters for ECMT5001

FWL explains exactly why adding a control variable changes a coefficient. If education and experience are positively correlated, and experience also raises wages (\(\beta_2 > 0\)), then the simple regression of wages on education overstates the education return (upward OVB). Multiple regression removes this bias by using only \(\tilde{\text{educ}}_i\) — variation in education orthogonal to experience — so the estimated \(\hat{\beta}_\text{educ}\) is no longer contaminated by the experience channel.

Any ECMT5001 exam question asking "what does controlling for X mean?" or "why does the coefficient change when you add Z?" is fundamentally asking about the FWL mechanism.

Source

Frisch & Waugh (1933), Lovell (1963). Proven in Heinze (2026, Ch. 1.3.8) and Wooldridge (Appendix E). Generalises to any number of partialled-out variables.