Lesson M02.L01: The Simple Linear Regression Model
| Field | Detail |
|---|---|
| Module | Module 02 — Simple Regression Analysis |
| Level | Graduate (ECMT5001, USYD) |
| Duration | 30–35 minutes |
| Learning Objective | State and interpret the simple linear regression model; articulate the role of the error term and the zero conditional mean assumption; distinguish the population model from sample estimates; correctly frame regression as association rather than causation |
| Provenance | Wooldridge Introductory Econometrics, Chapter 2 (intro) |
Explanation
The Simple Linear Regression Model
The simple linear regression model describes a linear relationship between a dependent variable \(y\) and a single explanatory variable \(x\):
This is the population model — it describes the true (unknown) data-generating process.
| Component | Name | Role |
|---|---|---|
| \(y\) | Dependent variable (regressand) | The outcome we want to explain |
| \(x\) | Independent variable (regressor, covariate) | The explanatory variable |
| \(\beta_0\) | Intercept | Expected value of \(y\) when \(x = 0\) |
| \(\beta_1\) | Slope coefficient | Change in \(y\) for a one-unit increase in \(x\), ceteris paribus |
| \(u\) | Error term (disturbance) | Everything that affects \(y\) beyond \(x\) |
USYD notation note: The ECMT1020 notes also write this as \(Y = \beta_0 + \beta_1 X + \epsilon\), using \(\epsilon\) or \(u\) interchangeably for the error term. At ECMT5001 level we follow Wooldridge and use \(u\) throughout.
Interpreting the Coefficients
The intercept \(\beta_0\):
This is the expected value of \(y\) when \(x = 0\). In many economic applications the intercept has no direct behavioural interpretation (e.g., "wage when education = 0 years"), but it is essential for the model to fit the data correctly — it ensures the regression line passes through the correct point in the \((x, y)\) plane.
The slope \(\beta_1\):
This is the ceteris paribus effect of \(x\) on \(y\) — the change in \(y\) per one-unit increase in \(x\), holding everything else constant.
Wage-education example: In the model $\(\text{wage} = \beta_0 + \beta_1 \cdot \text{education} + u\)$ \(\beta_1\) is interpreted as: "One additional year of education is associated with a change of \(\beta_1\) dollars in hourly wage, holding all other factors constant." If \(\beta_1 = 2.5\), each extra year of schooling predicts a $2.50/hour higher wage.
GDP growth example: In the model $\(\text{GDP growth} = \beta_0 + \beta_1 \cdot \text{investment rate} + u\)$ \(\beta_1\) measures how much GDP growth changes (in percentage points) for each one-percentage-point increase in the investment rate.
The Error Term \(u\)
The error term \(u\) is not noise to be dismissed — it is a substantive part of the model. It captures:
- Omitted factors: All other variables that influence \(y\) but are not included in the model. For wages: ability, experience, industry, location, social networks, …
- Measurement error: If \(x\) or \(y\) is measured imperfectly, measurement error enters \(u\).
- Inherent randomness: Human behaviour is not fully deterministic; even a perfect model would have irreducible randomness.
Because we cannot directly observe \(u\), we must impose assumptions that make the model tractable.
The Zero Conditional Mean Assumption
The single most important assumption in simple regression:
This says: the average value of the unobserved error is zero for every value of \(x\).
What it implies: - The error is uncorrelated with \(x\): \(\text{Cov}(u, x) = 0\) (a weaker condition, necessary but not sufficient) - None of the omitted factors in \(u\) are systematically related to \(x\) - The relationship between \(y\) and \(x\) is correctly specified as linear
From the law of iterated expectations: If \(E(u \mid x) = 0\), then \(E(u) = E[E(u \mid x)] = 0\). So zero conditional mean implies zero unconditional mean — but not vice versa.
Economic example of violation: In the wage-education model, if more-able workers both earn higher wages and choose more schooling, then ability (in \(u\)) is positively correlated with education (\(x\)), violating \(E(u \mid x) = 0\). This causes omitted variable bias — a central problem in empirical economics.
Population Model vs Sample Estimates
The population model \(y = \beta_0 + \beta_1 x + u\) contains the true parameters \(\beta_0\) and \(\beta_1\), which we can never observe. From a sample \(\{(x_i, y_i)\}_{i=1}^{n}\), we compute estimates \(\hat{\beta}_0\) and \(\hat{\beta}_1\) (the OLS estimators, derived in M02.L02).
The fitted value for observation \(i\) is the model's predicted \(y\):
The residual for observation \(i\) is the difference between the actual and fitted value:
Residuals (\(\hat{u}_i\)) are the sample analogues of the unobservable error terms (\(u_i\)). They can be computed once we have estimates; the errors \(u_i\) can never be directly observed.
| Population | Sample | |
|---|---|---|
| Model | \(y = \beta_0 + \beta_1 x + u\) | \(y_i = \hat{\beta}_0 + \hat{\beta}_1 x_i + \hat{u}_i\) |
| Parameters | \(\beta_0, \beta_1\) (unknown, fixed) | \(\hat{\beta}_0, \hat{\beta}_1\) (computed, random) |
| Disturbances | \(u_i\) (unobservable) | \(\hat{u}_i\) (residuals, observable) |
| Prediction | \(E(y \mid x) = \beta_0 + \beta_1 x\) | \(\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\) |
Worked Example
Economic question: Do countries with higher investment rates experience faster GDP growth?
Step 1: Define variables and set up the model.
Let: - \(y_i\) = average annual GDP growth rate (%) for country \(i\) over 1990–2020 - \(x_i\) = average gross fixed capital formation as % of GDP (investment rate) for country \(i\)
Population model:
Step 2: Interpret \(\beta_0\) and \(\beta_1\) before estimation.
- \(\beta_0\): the expected GDP growth rate for a country with a 0% investment rate. This is economically implausible (no real-world country has zero investment), so \(\beta_0\) serves mainly as a mathematical anchor for the line.
- \(\beta_1\): the expected change in GDP growth (in percentage points) for each one-percentage-point increase in the investment rate, holding all else constant. Economic theory (capital accumulation, Solow model) predicts \(\beta_1 > 0\).
Step 3: What does \(u_i\) capture?
The error term \(u_i\) for country \(i\) includes: - Institutional quality (rule of law, corruption) - Human capital and education levels - Technology and productivity shocks - Political stability - Natural resource endowments - Trading partner conditions
Step 4: Assess the zero conditional mean assumption.
Is \(E(u_i \mid \text{investment rate}_i) = 0\) plausible? Potentially not — richer countries with better institutions (captured in \(u_i\)) may also have higher investment rates (\(x_i\)). If so, \(\beta_1\) would be biased upward (part of what we estimate as the "investment effect" would actually reflect institutional quality). This motivates moving to multiple regression (Module 03).
Step 5: Once data are obtained, compute fitted values and residuals.
Suppose OLS estimation yields \(\hat{\beta}_0 = 1.2\) and \(\hat{\beta}_1 = 0.15\). Then for a country with an investment rate of 25%:
If that country's actual GDP growth was 4.20%, its residual is:
The country grew 0.75 percentage points slower than the model predicted — meaning factors in \(u\) (perhaps poor institutions) dragged growth below its predicted level.
Common Misconception
"The regression line shows that \(x\) causes \(y\)."
This is false. An estimated regression line shows the conditional mean of \(y\) given \(x\) — a statement about association, not causation. The model
describes how \(y\) tends to vary with \(x\) in the data. It does not prove that changing \(x\) would cause \(y\) to change.
Why the distinction matters: If we regress wages on education and find \(\hat{\beta}_1 = 2.5\), this reflects correlation in the data. If more-educated people are also more innately able (an omitted variable in \(u\)), then the "effect" of education is overstated — the regression captures both the true causal effect of education and the omitted ability effect.
When does correlation become causation in regression?
Only under the zero conditional mean assumption \(E(u \mid x) = 0\). If this holds (e.g., in a randomised experiment where \(x\) is randomly assigned), then \(\hat{\beta}_1\) consistently estimates the causal effect. In observational data, we cannot guarantee \(E(u \mid x) = 0\) without additional structural or experimental arguments.
Tools for establishing causation (covered in Modules 06–08): randomised experiments, instrumental variables (IV), regression discontinuity design (RDD), difference-in-differences (DiD). All of these provide ways to make \(E(u \mid x) = 0\) (or an equivalent) more credible.
Practical rule: When reporting regression results, say "a one-unit increase in \(x\) is associated with a \(\hat{\beta}_1\) change in \(y\)" rather than "causes" — unless you have a credible identification strategy.
Practice Prompts
Prompt 1
A researcher models house prices using the simple regression:
where price is in $000s and sqft is square footage.
(a) Interpret \(\beta_0\) and \(\beta_1\) in economic terms. (b) List three economic factors that would be captured in \(u_i\). (c) OLS yields \(\hat{\beta}_0 = 25.0\) and \(\hat{\beta}_1 = 0.12\). A house has 1,800 sqft and sold for $260,000. Compute the fitted value and the residual. (d) Is the zero conditional mean assumption \(E(u \mid \text{sqft}) = 0\) likely to hold? Why or why not?
Answer
**(a)** - $\beta_0 = 25.0$: The expected price of a house with 0 square feet — not economically meaningful, but mathematically the y-intercept. - $\beta_1 = 0.12$: Each additional square foot of floor area is associated with a \$120 higher house price, holding all else constant. **(b)** Examples: neighbourhood quality/location, number of bedrooms and bathrooms, age and condition of the property, school district quality, proximity to public transport, lot size, renovation status. **(c)** $$\hat{y} = 25.0 + 0.12 \times 1{,}800 = 25.0 + 216.0 = 241.0 \quad (\$241{,}000)$$ $$\hat{u} = 260.0 - 241.0 = 19.0 \quad (\$19{,}000)$$ The house sold for \$19,000 more than predicted — likely reflecting unobserved quality (e.g., premium location, recent renovation). **(d)** Probably **not**. Larger houses are more likely to be in high-income suburbs (better schools, lower crime), which are also correlated with price. Location quality is in $u$ but correlated with sqft, violating $E(u \mid \text{sqft}) = 0$.Prompt 2
An economist writes down the model:
where \(\ln(\text{wage})\) is the natural log of hourly wage and educ is years of schooling.
(a) If \(\hat{\beta}_1 = 0.08\), give the approximate percentage-change interpretation of this coefficient. (b) What does \(\hat{\beta}_0\) estimate? Is this interpretable? (c) Name two variables likely captured in \(u_i\) and explain whether each would bias \(\hat{\beta}_1\) upward or downward.
Answer
**(a)** Since the dependent variable is in logs, $\hat{\beta}_1 = 0.08$ implies that one additional year of education is associated with approximately an **8% increase** in hourly wages (the exact statement is $100(\hat{\beta}_1)\%$, valid for small values of $\hat{\beta}_1$). **(b)** $\hat{\beta}_0 = \ln(\widehat{\text{wage}})$ when educ $= 0$. Exponentiating: $e^{\hat{\beta}_0}$ estimates the predicted hourly wage for a worker with zero years of schooling. In practice this is extrapolation far outside the data range and has little economic meaning. **(c)** - **Ability (IQ, non-cognitive skills):** More able people earn higher wages *and* tend to stay in school longer → positive correlation with educ → $\hat{\beta}_1$ biased **upward** (overestimates the return to schooling). - **Family background/wealth:** Wealthier families can invest more in schooling and also provide career networks → positive correlation with educ → $\hat{\beta}_1$ again biased **upward**. - (Counter-example) **Physical labour requirements:** Jobs requiring heavy physical labour may pay more but require less formal schooling → negative correlation with educ → $\hat{\beta}_1$ biased **downward**.Prompt 3
For each scenario below, (i) write the simple regression model, (ii) interpret \(\beta_1\), and (iii) identify the most important variable likely in \(u\) that would violate \(E(u \mid x) = 0\):
(a) Modelling a firm's total cost (\(y\)) as a function of output quantity (\(x\)). (b) Modelling a country's infant mortality rate (\(y\)) as a function of per-capita healthcare spending (\(x\)). (c) Modelling student exam scores (\(y\)) as a function of hours studied (\(x\)).
Answer
**(a)** $$\text{total cost}_i = \beta_0 + \beta_1 \cdot \text{output}_i + u_i$$ - $\beta_1$: the marginal cost — the expected increase in total cost from producing one additional unit of output. - Key omitted variable: input prices (wages, materials costs). Firms with higher input prices face higher costs *and* may produce less (negative correlation with output) → if input prices vary across firms in the data, $E(u \mid x) \neq 0$. **(b)** $$\text{infant mortality}_i = \beta_0 + \beta_1 \cdot \text{health spending}_i + u_i$$ - $\beta_1$: the expected change in infant mortality rate (per 1,000 live births) for a \$1 increase in per-capita healthcare spending. Theory predicts $\beta_1 < 0$. - Key omitted variable: income per capita. Richer countries both spend more on healthcare *and* have lower infant mortality for reasons beyond healthcare (sanitation, nutrition, education) → positive omitted-variable bias, understating the true healthcare effect. **(c)** $$\text{score}_i = \beta_0 + \beta_1 \cdot \text{hours}_i + u_i$$ - $\beta_1$: the expected increase in exam score (points) for each additional hour studied. - Key omitted variable: prior academic ability. Stronger students may study more efficiently (or strategically study more) *and* score higher → positive correlation with hours → $\hat{\beta}_1$ overstates the causal effect of study time.Further Resources
- Wooldridge — Introductory Econometrics: A Modern Approach, Chapter 2, Sections 2.1–2.2 (the simple regression model and OLS setup)
- Nick Huntington-Klein — The Effect: An Introduction to Research Design and Causality, Chapters 4–5 — free online textbook with outstanding treatment of why correlation ≠ causation and identification strategies
- USYD ECMT1020 Notes — Sections 2–3 (OLS estimators, CLRM assumptions) for USYD-specific formula derivations