Lesson M02.L02: OLS Estimation — Deriving the Slope and Intercept

Field	Detail
Module	M02 — Simple Linear Regression (Wooldridge Chapter 2)
Level	Graduate
Duration	35 minutes
Learning Objective	Derive the OLS estimators $\hat{\beta}_0$ and $\hat{\beta}_1$ from first principles, interpret them in context, and calculate them by hand from a small dataset.
Provenance	Wooldridge, Introductory Econometrics, 7th ed., Chapter 2, Sections 2.2–2.3

Explanation

1. The Regression Model

We postulate a population linear regression model:

\[y = \beta_0 + \beta_1 x + u\]

where:

$y$ is the dependent variable (outcome we want to explain)
$x$ is the independent variable (regressor, explanatory variable)
$\beta_0$ is the population intercept
$\beta_1$ is the population slope (the causal or associational effect of interest)
$u$ is the error term (everything else that affects $y$, unobserved)

We observe $n$ data points $\{(x_i, y_i)\}_{i=1}^n$. The goal of Ordinary Least Squares (OLS) is to find estimates $\hat{\beta}_0$ and $\hat{\beta}_1$ such that the fitted line:

\[\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\]

is "as close as possible" to the observed data.

2. The OLS Objective: Minimise Squared Residuals

For any candidate line $(\hat{\beta}_0, \hat{\beta}_1)$, define the residual for observation $i$ as:

\[\hat{u}_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i\]

This is the vertical distance between the observed $y_i$ and the fitted value $\hat{y}_i$.

OLS chooses $\hat{\beta}_0$ and $\hat{\beta}_1$ to minimise the sum of squared residuals (SSR):

\[\min_{\hat{\beta}_0,\, \hat{\beta}_1} \sum_{i=1}^n \hat{u}_i^2 = \sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2\]

Why squared residuals?
Squaring penalises large deviations heavily and treats positive/negative deviations symmetrically. Critically, squaring yields a smooth, convex objective function — this means the minimum is unique and can be found analytically by taking derivatives.

3. Derivation via First-Order Conditions

Let $Q(\hat{\beta}_0, \hat{\beta}_1) = \sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2$.

Step 1 — Differentiate with respect to $\hat{\beta}_0$ and set to zero:

\[\frac{\partial Q}{\partial \hat{\beta}_0} = -2 \sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0\]

Dividing by $-2$:

\[\sum_{i=1}^n (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0 \tag{FOC1}\]

Expanding and dividing by $n$:

\[\bar{y} = \hat{\beta}_0 + \hat{\beta}_1 \bar{x} \quad \Longrightarrow \quad \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} \tag{Intercept}\]

Step 2 — Differentiate with respect to $\hat{\beta}_1$ and set to zero:

\[\frac{\partial Q}{\partial \hat{\beta}_1} = -2 \sum_{i=1}^n x_i(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0\]

Dividing by $-2$:

\[\sum_{i=1}^n x_i(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = 0 \tag{FOC2}\]

Step 3 — Substitute the expression for $\hat{\beta}_0$ into FOC2.

Substituting $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$:

\[\sum_{i=1}^n x_i \bigl(y_i - \bar{y} - \hat{\beta}_1(x_i - \bar{x})\bigr) = 0\]

Expanding:

\[\sum_{i=1}^n x_i(y_i - \bar{y}) = \hat{\beta}_1 \sum_{i=1}^n x_i(x_i - \bar{x})\]

Using the identity $\sum x_i(x_i - \bar{x}) = \sum (x_i - \bar{x})^2$ and $\sum x_i(y_i - \bar{y}) = \sum (x_i - \bar{x})(y_i - \bar{y})$:

\[\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \hat{\beta}_1 \sum_{i=1}^n (x_i - \bar{x})^2\]

Solving for $\hat{\beta}_1$ (provided $\sum(x_i-\bar{x})^2 \neq 0$, i.e. $x$ is not constant):

\[\boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}}\]

And the intercept:

\[\boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}\]

4. Algebraic Intuition: Slope as Scaled Covariance

The OLS slope can be written compactly in population-moment notation:

\[\hat{\beta}_1 = \frac{\widehat{\text{Cov}}(x, y)}{\widehat{\text{Var}}(x)}\]

where $\widehat{\text{Cov}}(x,y) = \frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})$ and $\widehat{\text{Var}}(x) = \frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2$ (the $n$-denominator versions cancel).

Intuition:

If $x$ and $y$ move together (positive covariance), the slope is positive.
The denominator $\text{Var}(x)$ normalises the slope — it answers: "how much does $y$ move for each unit of $x$, accounting for how spread out $x$ itself is?"
If you multiply $x$ by 2, both numerator and denominator scale up, but denominator scales by 4 while numerator only by 2, so $\hat{\beta}_1$ halves. (We'll revisit this in M02.L04.)

5. Interpretation

Given the estimated equation $\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x$:

$\hat{\beta}_1$: A one-unit increase in $x$ is associated with a $\hat{\beta}_1$-unit change in $y$, on average (holding the rest of the model constant — though in simple regression there is nothing else to hold constant).
$\hat{\beta}_0$: The predicted value of $y$ when $x = 0$. Often not economically meaningful; interpret cautiously if $x = 0$ is outside the data range.

Caution on "causal" language: OLS gives the best linear predictor of $y$ given $x$. Whether $\hat{\beta}_1$ has a causal interpretation depends on additional assumptions (SLR.4: zero conditional mean error), not on the OLS algebra itself.

Worked Example

Dataset: Education and Hourly Wages

Suppose we observe $n = 5$ workers:

$i$	Years of Education ($x_i$)	Hourly Wage, $ ($y_i$)
1	8	7.25
2	10	9.00
3	12	11.50
4	14	14.00
5	16	18.25

Goal: Estimate $\hat{\beta}_0$ and $\hat{\beta}_1$ for $\text{Wage}_i = \beta_0 + \beta_1 \text{Educ}_i + u_i$.

Step 1: Compute means.

\[\bar{x} = \frac{8 + 10 + 12 + 14 + 16}{5} = \frac{60}{5} = 12\]

\[\bar{y} = \frac{7.25 + 9.00 + 11.50 + 14.00 + 18.25}{5} = \frac{60.00}{5} = 12.00\]

Step 2: Compute deviations and cross-products.

$i$	$x_i - \bar{x}$	$y_i - \bar{y}$	$(x_i-\bar{x})(y_i-\bar{y})$	$(x_i-\bar{x})^2$
1	$8 - 12 = -4$	$7.25 - 12 = -4.75$	$(-4)(-4.75) = 19.00$	16
2	$10 - 12 = -2$	$9.00 - 12 = -3.00$	$(-2)(-3.00) = 6.00$	4
3	$12 - 12 = 0$	$11.50 - 12 = -0.50$	$(0)(-0.50) = 0.00$	0
4	$14 - 12 = 2$	$14.00 - 12 = 2.00$	$(2)(2.00) = 4.00$	4
5	$16 - 12 = 4$	$18.25 - 12 = 6.25$	$(4)(6.25) = 25.00$	16
Sum	0	0	54.00	40

(Deviations always sum to zero — a useful check.)

Step 3: Calculate $\hat{\beta}_1$.

\[\hat{\beta}_1 = \frac{\sum(x_i - \bar{x})(y_i - \bar{y})}{\sum(x_i - \bar{x})^2} = \frac{54.00}{40} = 1.35\]

Step 4: Calculate $\hat{\beta}_0$.

\[\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} = 12.00 - 1.35 \times 12 = 12.00 - 16.20 = -4.20\]

Step 5: Write the estimated equation.

\[\widehat{\text{Wage}} = -4.20 + 1.35 \times \text{Educ}\]

Step 6: Interpretation.

Slope ($\hat{\beta}_1 = 1.35$): Each additional year of education is associated with, on average, $1.35 more per hour in wages.
Intercept ($\hat{\beta}_0 = -4.20$): A worker with zero years of education is predicted to earn $-\$4.20$/hour — this is outside the data range (minimum $x = 8$) and is economically meaningless. Do not over-interpret.
Predicted value check: For $x = 12$ (the mean): $\hat{y} = -4.20 + 1.35 \times 12 = -4.20 + 16.20 = 12.00 = \bar{y}$. ✓ The line passes through $(\bar{x}, \bar{y})$.

Common Misconception

"OLS minimises the sum of absolute residuals."

This is false. OLS minimises $\sum \hat{u}_i^2$ — the sum of squared residuals. The method that minimises $\sum |\hat{u}_i|$ is called Least Absolute Deviations (LAD) regression, and it does not yield closed-form analytical formulas like OLS does.

Squaring matters for three reasons:

Unique solution: The squared objective is strictly convex, guaranteeing a unique minimum found by setting derivatives to zero. The absolute-value objective is piecewise linear and may have multiple solutions.
Analytical tractability: The closed-form formulas $\hat{\beta}_1 = \sum(x_i-\bar{x})(y_i-\bar{y}) / \sum(x_i-\bar{x})^2$ arise directly from the quadratic FOCs.
Statistical theory: The squared-residual criterion links naturally to the normal distribution and the Gauss-Markov theorem (covered in M02.L05).

The tradeoff is that squaring makes OLS sensitive to outliers (a single large $\hat{u}_i$ contributes $\hat{u}_i^2$ to the objective). LAD is more robust to outliers but lacks clean analytical properties.

Practice Prompts

Problem 1

A researcher collects data on 5 cities: population (in thousands) and average commute time (minutes):

City	Population $x_i$	Commute $y_i$
A	100	22
B	200	28
C	300	35
D	400	40
E	500	45

(a) Compute $\bar{x}$ and $\bar{y}$.
(b) Compute $\hat{\beta}_1$ and $\hat{\beta}_0$.
(c) Interpret $\hat{\beta}_1$ in context.
(d) What is the predicted commute time for a city of 350,000 people?

Show Answer

(a) $\bar{x} = (100+200+300+400+500)/5 = 300$; $\bar{y} = (22+28+35+40+45)/5 = 34$.

(b)

City	$x_i - 300$	$y_i - 34$	Product	$(x_i-300)^2$
A	-200	-12	2400	40000
B	-100	-6	600	10000
C	0	1	0	0
D	100	6	600	10000
E	200	11	2200	40000
Sum			5800	100000

$\hat{\beta}_1 = 5800/100000 = 0.058$

$\hat{\beta}_0 = 34 - 0.058 \times 300 = 34 - 17.4 = 16.6$

(c) Each additional thousand residents in a city is associated with, on average, 0.058 additional minutes of commute time.

(d) $\hat{y} = 16.6 + 0.058 \times 350 = 16.6 + 20.3 = 36.9$ minutes.

Problem 2

You are given the following summary statistics for a sample of $n = 20$ students:

\[\bar{x} = 5.0 \text{ (hours studied)}, \quad \bar{y} = 70.0 \text{ (exam score)}$$ $$\sum_{i=1}^{20}(x_i - \bar{x})(y_i - \bar{y}) = 180, \quad \sum_{i=1}^{20}(x_i - \bar{x})^2 = 60\]

(a) Compute $\hat{\beta}_1$ and $\hat{\beta}_0$.
(b) Predict the score for a student who studies 8 hours.
(c) If the score variable is changed to a percentage of a 150-point scale (i.e. $y^* = y/150 \times 100$), how do $\hat{\beta}_0$, $\hat{\beta}_1$, and the fitted line change?

Show Answer

(a) $\hat{\beta}_1 = 180/60 = 3.0$; $\hat{\beta}_0 = 70.0 - 3.0 \times 5.0 = 55.0$.
Estimated equation: $\widehat{\text{Score}} = 55.0 + 3.0 \times \text{Hours}$.

(b) $\hat{y} = 55.0 + 3.0 \times 8 = 55.0 + 24.0 = 79.0$ points.

(c) $y^* = (100/150) \cdot y$. Multiplying $y$ by the constant $c = 100/150 \approx 0.6\overline{6}$ scales both $\hat{\beta}_0$ and $\hat{\beta}_1$ by $c$:
$\hat{\beta}_0^* = 55.0 \times (100/150) \approx 36.67$; $\hat{\beta}_1^* = 3.0 \times (100/150) = 2.0$.
The fitted values simply rescale; $R^2$ is unchanged (covered in M02.L04).

Problem 3

True or False — explain your reasoning for each:

(a) If $\hat{\beta}_1 = 0$, the OLS regression line is horizontal and $\hat{y}_i = \bar{y}$ for all $i$.
(b) The OLS line always passes through the point $(0, 0)$.
(c) You can have $\hat{\beta}_1 = 0$ even when $x$ and $y$ are strongly correlated.

Show Answer

(a) True. If $\hat{\beta}_1 = 0$, then $\hat{\beta}_0 = \bar{y} - 0 \cdot \bar{x} = \bar{y}$, so $\hat{y}_i = \bar{y}$ for all $i$. The regression line is flat at the sample mean of $y$.

(b) False. The OLS line passes through $(\bar{x}, \bar{y})$, not $(0, 0)$ — unless it happens that $\bar{x} = 0$ and $\bar{y} = 0$. The intercept $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}$ is specifically designed to enforce the $(\bar{x}, \bar{y})$ constraint.

(c) False. $\hat{\beta}_1 = \widehat{\text{Cov}}(x,y)/\widehat{\text{Var}}(x) = 0$ requires $\widehat{\text{Cov}}(x,y) = 0$, which means $x$ and $y$ are sample-uncorrelated. If $x$ and $y$ are strongly correlated, $\widehat{\text{Cov}}(x,y) \neq 0$, so $\hat{\beta}_1 \neq 0$.

Further Resources

Wooldridge: Chapter 2, Sections 2.2 (Definition of OLS) and 2.3 (Properties of OLS on any sample)
R: lm(y ~ x, data = df) fits the OLS line; coef() extracts $\hat{\beta}_0$ and $\hat{\beta}_1$; summary(lm(...)) shows everything
Python: statsmodels.formula.api.ols('y ~ x', data=df).fit() — or sklearn.linear_model.LinearRegression() (note: sklearn doesn't include intercept SE/t-stats)
Manual check: Verify your hand calculations with np.cov(x, y)[0,1] / np.var(x) for the slope
Video: Ben Lambert's OLS Derivation (YouTube) — clear walkthrough of the FOC approach

\(i\)	\(x_i - \bar{x}\)	\(y_i - \bar{y}\)	\((x_i-\bar{x})(y_i-\bar{y})\)	\((x_i-\bar{x})^2\)
1	\(8 - 12 = -4\)	\(7.25 - 12 = -4.75\)	\((-4)(-4.75) = 19.00\)	16
2	\(10 - 12 = -2\)	\(9.00 - 12 = -3.00\)	\((-2)(-3.00) = 6.00\)	4
3	\(12 - 12 = 0\)	\(11.50 - 12 = -0.50\)	\((0)(-0.50) = 0.00\)	0
4	\(14 - 12 = 2\)	\(14.00 - 12 = 2.00\)	\((2)(2.00) = 4.00\)	4
5	\(16 - 12 = 4\)	\(18.25 - 12 = 6.25\)	\((4)(6.25) = 25.00\)	16
Sum	0	0	54.00	40