Lesson M02.L05: Unbiasedness of OLS — Assumptions and Expected Values
| Field | Detail |
|---|---|
| Module | M02 — Simple Linear Regression (Wooldridge Chapter 2) |
| Level | Graduate |
| Duration | 35 minutes |
| Learning Objective | State and interpret the SLR assumptions; prove OLS unbiasedness; derive the variance of OLS estimators; state the Gauss-Markov theorem; identify common violations of SLR.4. |
| Provenance | Wooldridge, Introductory Econometrics, 7th ed., Chapter 2, Sections 2.5–2.6 |
Explanation
1. From Algebra to Statistics
Lessons M02.L02–L04 established algebraic properties of OLS — things that are true by definition, in every sample, regardless of how the data were generated.
Now we shift to statistical properties: what can we say about OLS estimators across repeated samples drawn from the same population? For this, we need to specify how the data were generated — i.e., we need assumptions about the data generating process (DGP).
2. The Four Core Assumptions (SLR.1–SLR.4)
SLR.1 — Linear in Parameters
Assumption: The population model is:
where \(\beta_0\) and \(\beta_1\) are the unknown population parameters we want to estimate.
Key clarification: "Linear in parameters" does NOT mean the relationship between \(y\) and \(x\) must be linear. For example:
is linear in the parameters \(\beta_0\) and \(\beta_1\) even though it implies a nonlinear (log-log) relationship between \(y\) and \(x\). SLR.1 is satisfied.
What would violate SLR.1? A model like \(y = \beta_0 \cdot e^{\beta_1 x} + u\) — here \(\beta_1\) enters nonlinearly and OLS cannot be directly applied.
SLR.2 — Random Sampling
Assumption: \(\{(x_i, y_i)\}_{i=1}^n\) is a random sample of size \(n\) drawn from the population model.
Implication: Each observation is independently and identically distributed (i.i.d.) from the same joint distribution of \((x, y)\). No observation is special or selected based on the outcome.
Common violations: Time-series data (observations are correlated over time); panel data (same person observed repeatedly); choice-based sampling (e.g. only surveying employed workers when studying wages).
SLR.3 — Sample Variation in x (No Perfect Collinearity)
Assumption: In the sample, \(x\) is not constant:
Why it's needed: The OLS slope formula \(\hat{\beta}_1 = \sum(x_i-\bar{x})(y_i-\bar{y}) / \sum(x_i-\bar{x})^2\) requires a non-zero denominator. If all \(x_i\) are identical, we cannot estimate the effect of changing \(x\) — there's no variation to learn from.
Common violation: Multicollinearity (near-perfect collinearity among regressors in multiple regression; simple regression only has one \(x\) so exact collinearity is obvious).
SLR.4 — Zero Conditional Mean Error
Assumption: The expected value of the error, conditional on \(x\), is zero:
This is the crucial assumption. It means that, for any given value of \(x\), the unobservable factors in \(u\) are on average zero. Equivalently, \(u\) and \(x\) are mean-independent — knowing \(x\) tells you nothing about the average of \(u\).
Implications of SLR.4:
- \(E(u) = 0\) (unconditionally)
- \(\text{Cov}(x, u) = 0\) (zero covariance between \(x\) and \(u\))
- \(E(y \mid x) = \beta_0 + \beta_1 x\) (the population regression function is the conditional expectation of \(y\) given \(x\))
What violates SLR.4?
| Violation | Explanation |
|---|---|
| Omitted Variable Bias (OVB) | If a variable \(z\) affects \(y\) and is correlated with \(x\), then \(z\) is part of \(u\), and \(\text{Cov}(x, u) \neq 0\). E.g., regressing wages on education while omitting ability (ability is in \(u\), and ability correlates with education). |
| Reverse Causality (Simultaneity) | If \(y\) also causes \(x\), then \(x\) and \(u\) are jointly determined — \(\text{Cov}(x, u) \neq 0\). E.g., regressing police per capita on crime rate, but crime may also cause police allocation. |
| Measurement Error in x | If \(x\) is measured with error (\(\tilde{x} = x + \text{error}\)), the measurement error is part of \(u\) and correlates with \(\tilde{x}\) — violating SLR.4. This causes attenuation bias (estimates biased toward zero). |
3. Unbiasedness of OLS (Theorem 2.1)
Theorem: Under SLR.1–SLR.4:
Proof for \(\hat{\beta}_1\):
Start from the OLS formula:
Substitute \(y_i = \beta_0 + \beta_1 x_i + u_i\) into the numerator. Using the fact that \(\sum(x_i-\bar{x})(\beta_0 + \beta_1 \bar{x}) = 0\) (deviations sum to zero), the numerator simplifies to:
Therefore:
Taking the conditional expectation given \(\{x_1, \ldots, x_n\}\) and applying SLR.4 (\(E(u_i \mid x_i) = 0\)):
By the law of iterated expectations, \(E(\hat{\beta}_1) = \beta_1\). \(\blacksquare\)
The proof for \(\hat{\beta}_0\) follows similarly from \(\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}\) and unbiasedness of \(\hat{\beta}_1\).
4. SLR.5 — Homoskedasticity
Assumption: The variance of \(u\) is constant across all values of \(x\):
What this means: The spread of \(y\) around the conditional mean \(E(y \mid x) = \beta_0 + \beta_1 x\) is the same for all \(x\) — the error variance does not depend on \(x\).
Violation (Heteroskedasticity): If \(\text{Var}(u \mid x)\) varies with \(x\), the standard OLS standard errors are wrong (but OLS is still unbiased under SLR.1–SLR.4 alone). Heteroskedasticity-robust standard errors are the fix (covered in later modules).
5. Variance of OLS Estimators (Under SLR.1–SLR.5)
Theorem 2.2: Under SLR.1–SLR.5:
where \(SST_x = \sum_{i=1}^n (x_i - \bar{x})^2\).
Intuitions for \(\text{Var}(\hat{\beta}_1)\):
| Factor | Effect on \(\text{Var}(\hat{\beta}_1)\) | Intuition |
|---|---|---|
| Larger \(\sigma^2\) (noisier \(u\)) | Increases variance | More noise → harder to estimate the signal \(\beta_1\) |
| Larger \(SST_x\) (more variation in \(x\)) | Decreases variance | More variation in \(x\) → better identification of slope |
| Larger \(n\) | Decreases variance (via \(SST_x \propto n\)) | More data → more precise estimates |
6. Estimating \(\sigma^2\): The Unbiased Estimator
Since \(\sigma^2 = E(u^2)\) (under \(E(u) = 0\)), a natural estimator is to average the squared residuals. However, the sample residuals \(\hat{u}_i\) are not the same as the true errors \(u_i\) — they satisfy two constraints (Properties 1 and 2 from M02.L03), which "use up" 2 degrees of freedom.
The unbiased estimator of \(\sigma^2\) is:
Why \(n - 2\)? We estimated 2 parameters (\(\hat{\beta}_0\) and \(\hat{\beta}_1\)), leaving \(n - 2\) degrees of freedom. It can be shown that \(E(\hat{\sigma}^2) = \sigma^2\) — \(\hat{\sigma}^2\) is unbiased for \(\sigma^2\).
Using \(\hat{\sigma}^2\), the standard error of \(\hat{\beta}_1\) is:
where \(\hat{\sigma} = \sqrt{\hat{\sigma}^2}\) is called the standard error of the regression (also called the root MSE).
7. The Gauss-Markov Theorem (Preview)
Theorem (Gauss-Markov): Under SLR.1–SLR.5, the OLS estimators \(\hat{\beta}_0\) and \(\hat{\beta}_1\) are BLUE — Best Linear Unbiased Estimators.
- Linear: \(\hat{\beta}_1\) is a linear function of \(y_1, \ldots, y_n\)
- Unbiased: \(E(\hat{\beta}_1) = \beta_1\) (from Theorem 2.1 above)
- Best: \(\hat{\beta}_1\) has the smallest variance among all linear unbiased estimators of \(\beta_1\)
What this means: If you insist on estimating \(\beta_1\) with a linear unbiased estimator, OLS is the most efficient choice — it extracts the maximum information from the data.
What this does NOT mean: If you allow biased estimators (e.g. ridge regression) or non-linear estimators, you might achieve lower mean squared error (MSE = bias² + variance) in finite samples. The Gauss-Markov theorem is about BLUE, not BUE (Best Unbiased Estimator).
The full proof of Gauss-Markov is in Wooldridge Appendix E (for the multiple regression case).
Worked Example
Dataset: Education and Wages (continued from M02.L02)
Recall: \(n = 5\), \(\hat{\beta}_0 = -4.20\), \(\hat{\beta}_1 = 1.35\), \(SST_x = \sum(x_i-\bar{x})^2 = 40\), \(SSR = 1.975\).
Step 1: Estimate \(\sigma^2\) and \(\hat{\sigma}\).
Interpretation: The typical residual (deviation from the fitted line) is about $0.81 per hour. Given that wages in this dataset range from $7.25 to $18.25, this is a small typical error — consistent with the high \(R^2 = 0.974\).
Step 2: Compute \(SE(\hat{\beta}_1)\).
Interpretation: The estimated standard deviation of \(\hat{\beta}_1\) across hypothetical repeated samples is about $0.13/hour per year. The estimated slope of $1.35 is about \(1.35 / 0.128 \approx 10.5\) standard errors away from zero — very strongly statistically significant (we'll formalise this in Module 4).
Step 3: Unbiasedness — conceptual illustration.
Suppose the true model is \(\text{Wage}_i = -3.00 + 1.20 \times \text{Educ}_i + u_i\) (i.e. \(\beta_0 = -3\), \(\beta_1 = 1.20\)). Under SLR.1–SLR.4, if we drew many random samples of size 5 and estimated OLS in each, the average of the resulting \(\hat{\beta}_1\) estimates would converge to \(1.20\). The single estimate from our data (\(\hat{\beta}_1 = 1.35\)) may deviate from \(1.20\) — unbiasedness is a property across all possible samples, not a guarantee for any individual sample.
This is why standard errors matter: they quantify how much we expect \(\hat{\beta}_1\) to vary from sample to sample.
Step 4: What violates SLR.4 here?
The regression \(\text{Wage} = \beta_0 + \beta_1 \text{Educ} + u\) violates SLR.4 if, for example, more-able workers both earn higher wages and tend to get more education. In that case, ability is in \(u\) and is correlated with \(\text{Educ}\), so \(E(u \mid \text{Educ}) \neq 0\). The OLS estimate of \(\hat{\beta}_1\) would then over-estimate the true causal effect of education, because it partially picks up the effect of ability. This is classic omitted variable bias — addressed in Module 3 (Multiple Regression).
Common Misconception
"Unbiasedness means my OLS estimate is close to the true β."
This is false. Unbiasedness is a property of the estimator across the distribution of possible samples, not a guarantee about any single estimate.
Formally: \(E(\hat{\beta}_1) = \beta_1\) means that if you repeatedly drew samples from the population and computed OLS, the average of your estimates would equal the true \(\beta_1\). Any single sample might give an estimate far from \(\beta_1\).
Analogy: A scale that is perfectly calibrated (unbiased — the average reading equals the true weight) can still give a very different reading each time you weigh the same object if the scale is very noisy. The precision of the scale is captured by its variance, not its unbiasedness.
In practice, we care about Mean Squared Error (MSE):
Under SLR.1–SLR.4, OLS is unbiased, so \(\text{MSE} = \text{Var}(\hat{\beta}_1)\). The Gauss-Markov theorem (SLR.1–SLR.5) then guarantees this variance is as small as possible among linear unbiased estimators. But a biased estimator with much lower variance might have lower MSE overall — this is the bias-variance tradeoff.
Practice Prompts
Problem 1
For a sample of \(n = 30\) observations, you are given:
(a) Compute \(\hat{\sigma}^2\).
(b) Compute \(SE(\hat{\beta}_1)\).
(c) If \(n\) were doubled to 60 (with the same \(\hat{\sigma}^2\) and \(SST_x\) approximately doubling), what happens to \(SE(\hat{\beta}_1)\)?
Show Answer
(a) \(\hat{\sigma}^2 = SSR/(n-2) = 45/28 \approx 1.607\).
(b) \(SE(\hat{\beta}_1) = \hat{\sigma}/\sqrt{SST_x} = \sqrt{1.607}/\sqrt{120} = 1.268/10.954 \approx 0.1158\).
(c) With \(n = 60\), roughly \(SST_x \approx 240\) (doubling if \(\hat{\sigma}^2\) stays similar). \(\hat{\sigma}^2 \approx 45 \times 2/(58) \approx 1.552\) (SSR roughly doubles with double the observations).
\(SE(\hat{\beta}_1) \approx \sqrt{1.552}/\sqrt{240} \approx 1.246/15.49 \approx 0.0804\).
Approximately: \(SE\) scales as \(1/\sqrt{n}\), so doubling \(n\) reduces \(SE\) by a factor of \(\sqrt{2} \approx 1.41\), from 0.116 to about 0.082. ✓
Problem 2
State whether each of the following violates SLR.4 (\(E(u|x) = 0\)), and if so, specify the likely direction of bias in \(\hat{\beta}_1\):
(a) Regressing salary on years of experience, omitting whether the person has a graduate degree (and graduate degrees are correlated with experience).
(b) Regressing crop yield on rainfall, where rainfall is assigned by nature independently of soil quality.
(c) Regressing a country's growth rate on its initial GDP, using data only on countries with GDP above $5,000 (sample selection).
(d) Regressing hours worked on the reported wage rate, when workers misreport their wages by adding random noise (classical measurement error in \(x\)).
Show Answer
(a) Violates SLR.4. Graduate degree is in \(u\) (omitted), and it is likely correlated with experience (\(x\)) — people with graduate degrees may have more years of schooling and thus different experience profiles. If more experienced workers are more likely to have graduate degrees and graduate degrees increase salary, \(\hat{\beta}_1\) is positively biased (over-estimates the pure return to experience).
(b) Does NOT violate SLR.4 (likely). If rainfall is assigned by nature independently of soil quality (and other determinants of yield), then \(E(u \mid \text{rainfall}) = 0\) is plausible. This is analogous to a natural experiment — random assignment of the regressor satisfies the zero conditional mean assumption.
(c) Violates SLR.2 (random sampling) more than SLR.4 directly, but the effect propagates to bias. Restricting the sample to high-GDP countries introduces sample selection bias — the sample is not a random draw from all countries. OLS on this selected sample may be biased relative to the population relationship.
(d) Violates SLR.4. Classical measurement error in \(x\) causes attenuation bias: \(\hat{\beta}_1\) is biased toward zero (i.e. \(|E(\hat{\beta}_1)| < |\beta_1|\)). The measured \(\tilde{x} = x + e\) (where \(e\) is measurement noise) is correlated with the error in \(u\) (which contains \(-\beta_1 e\)), violating SLR.4.
Problem 3
True or False — justify each:
(a) Under SLR.1–SLR.4, if we run OLS on 1000 different random samples and average all the \(\hat{\beta}_1\) estimates, we will get exactly \(\beta_1\).
(b) If \(E(\hat{\beta}_1) = \beta_1\), then \(\hat{\beta}_1 = \beta_1\) in every sample.
(c) Dividing SSR by \(n\) instead of \(n-2\) gives a biased estimator of \(\sigma^2\), and specifically it under-estimates \(\sigma^2\).
(d) If SLR.5 (homoskedasticity) is violated, OLS is biased for \(\beta_1\).
Show Answer
(a) True in expectation (approximately true in practice). By the law of large numbers, the sample average of 1000 \(\hat{\beta}_1\) values converges to \(E(\hat{\beta}_1) = \beta_1\) as the number of replications grows. With 1000 replications, the average is very close to \(\beta_1\) but not exactly \(\beta_1\) due to sampling variation.
(b) False. Unbiasedness (\(E(\hat{\beta}_1) = \beta_1\)) is a statement about the expected value across samples. In any individual sample, \(\hat{\beta}_1\) will differ from \(\beta_1\) due to sampling error. Unbiasedness does not guarantee that any single estimate equals the true parameter.
(c) True. It can be shown that \(E\!\left[\frac{SSR}{n}\right] = \frac{n-2}{n}\sigma^2 < \sigma^2\). Dividing by \(n\) systematically under-estimates \(\sigma^2\). Dividing by \(n-2\) corrects for the 2 degrees of freedom consumed by estimating \(\hat{\beta}_0\) and \(\hat{\beta}_1\), giving an unbiased estimate.
(d) False. OLS unbiasedness (Theorem 2.1) only requires SLR.1–SLR.4. SLR.5 (homoskedasticity) is not needed for unbiasedness. If SLR.5 is violated: OLS is still unbiased, but the formula \(\text{Var}(\hat{\beta}_1) = \sigma^2/SST_x\) is no longer correct, making standard errors and hypothesis tests invalid. The fix is to use heteroskedasticity-robust standard errors (covered in later modules).
Further Resources
- Wooldridge: Chapter 2, Sections 2.5 (Expected values of OLS estimators) and 2.6 (Variances of OLS estimators); Theorem 2.1 (Unbiasedness) and Theorem 2.2 (Variances)
- Wooldridge: Appendix C, Section C.3 (Properties of estimators: unbiasedness, efficiency); this is the probability/statistics background assumed for Theorem 2.1
- Omitted Variable Bias: Wooldridge Chapter 3 derives the OVB formula precisely for multiple regression — highly recommended follow-up
- Gauss-Markov: Wooldridge Appendix E (formal proof); the simpler statement is in Chapter 2, Section 2.6
- R:
summary(lm(y ~ x))$sigmagives \(\hat{\sigma}\);vcov(fit)gives the variance-covariance matrix of \((\hat{\beta}_0, \hat{\beta}_1)\);coef(summary(fit))[, "Std. Error"]extracts standard errors - Python:
results.bsegives standard errors;results.scalegives \(\hat{\sigma}^2\);np.sqrt(results.scale / np.sum((x - x.mean())**2))manually computes \(SE(\hat{\beta}_1)\) - Simulation: A compelling way to understand unbiasedness is to simulate it in R/Python — generate data from a known DGP, run OLS 1000 times, and histogram \(\hat{\beta}_1\) to see it centred at \(\beta_1\)