Lesson M01.L04: Joint Distributions, Conditional Expectation, and Covariance
| Field | Detail |
|---|---|
| Module | Module 01 — Probability & Statistics Review |
| Level | Graduate (ECMT5001, USYD) |
| Duration | 30–35 minutes |
| Learning Objective | Understand joint and conditional distributions; compute covariance and correlation; correctly interpret the relationship between independence and zero covariance |
| Provenance | Wooldridge Introductory Econometrics, Appendix B.2–B.4 |
Explanation
Joint Distributions
When two random variables \(X\) and \(Y\) are observed together, their behaviour is described by a joint distribution.
Discrete case — Joint PMF:
The joint PMF satisfies \(\sum_x \sum_y f(x,y) = 1\).
Continuous case — Joint PDF:
Marginal distributions recover the behaviour of each variable alone by summing (or integrating) over the other:
In economics, a joint distribution is natural whenever two outcomes are simultaneously determined — for example, an individual's wage and their years of schooling both emerge from the same labour-market process.
Conditional Distributions and Conditional Expectation
The conditional PMF/PDF of \(Y\) given \(X = x\) is:
The conditional expectation \(E(Y \mid X = x)\) is the expected value of \(Y\) once we fix \(X\) at the value \(x\):
Economic interpretation: \(E(\text{wage} \mid \text{education} = 16)\) is the average wage we would expect for a worker with exactly 16 years of schooling — the conditional mean of wages within that schooling group.
Law of Iterated Expectations
One of the most useful results in probability theory:
The "outer" expectation is taken over the distribution of \(X\). In words: if we first compute the conditional mean of \(Y\) for each value of \(X\), and then average those conditional means using the distribution of \(X\), we recover the unconditional mean of \(Y\).
Why it matters in econometrics: The law of iterated expectations underpins the zero conditional mean assumption \(E(u \mid x) = 0\) in regression — if the error has zero conditional mean for every value of \(x\), it also has unconditional mean zero: \(E(u) = E[E(u \mid x)] = 0\).
Covariance
Covariance measures the direction and degree to which \(X\) and \(Y\) move together:
Computational shortcut (USYD notation):
Sample covariance (used in estimation):
Key properties:
| Property | Formula |
|---|---|
| Symmetry | \(\text{Cov}(X,Y) = \text{Cov}(Y,X)\) |
| Cov with constant | \(\text{Cov}(X, a) = 0\) |
| Cov with itself | \(\text{Cov}(X, X) = \text{Var}(X)\) |
| Scaling | \(\text{Cov}(aX, bY) = ab\,\text{Cov}(X, Y)\) |
| Linearity | \(\text{Cov}(X + Z, Y) = \text{Cov}(X, Y) + \text{Cov}(Z, Y)\) |
Variance of a sum (crucial for regression asymptotics):
If \(X\) and \(Y\) are uncorrelated, \(\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)\).
Correlation Coefficient
Covariance is scale-dependent. The Pearson correlation coefficient standardises it to the range \([-1, 1]\):
where \(\sigma_X = \sqrt{\text{Var}(X)}\) and \(\sigma_Y = \sqrt{\text{Var}(Y)}\).
- \(\rho = +1\): perfect positive linear relationship
- \(\rho = -1\): perfect negative linear relationship
- \(\rho = 0\): no linear relationship (but see the common misconception below)
Sample correlation (USYD formula):
Independence and Zero Covariance
If \(X\) and \(Y\) are independent, then \(E(XY) = E(X)E(Y)\), which implies:
Independence \(\Rightarrow\) zero covariance (always).
Zero covariance \(\not\Rightarrow\) independence (in general).
The exception: if \((X, Y)\) follow a bivariate normal distribution, then \(\rho_{XY} = 0\) does imply independence. This is a special property of the normal distribution, not a general rule.
Worked Example
Setup: A survey of 5 workers records years of education \(X\) and (log) hourly wage \(Y\). The joint distribution (summarised as a table of population probabilities) is:
| \(Y = 2.5\) | \(Y = 3.0\) | \(Y = 3.5\) | \(P(X=x)\) | |
|---|---|---|---|---|
| \(X = 12\) | 0.20 | 0.10 | 0.05 | 0.35 |
| \(X = 16\) | 0.10 | 0.25 | 0.15 | 0.50 |
| \(X = 18\) | 0.02 | 0.05 | 0.08 | 0.15 |
| \(P(Y=y)\) | 0.32 | 0.40 | 0.28 | 1.00 |
Step 1: Compute marginal means.
Step 2: Compute \(E(XY)\).
Step 3: Covariance.
Step 4: Standard deviations.
Step 5: Correlation.
Step 6: Conditional expectation \(E(Y \mid X = 16)\).
Interpretation: Workers with 16 years of education are expected to earn a log wage of 3.05, compared to the unconditional mean of 2.98. The positive covariance (0.34) and correlation (0.39) confirm that education and wages move in the same direction in this population.
Common Misconception
"If Cov(X, Y) = 0 (or ρ = 0), then X and Y are independent."
This is false in general. Zero covariance means there is no linear relationship between \(X\) and \(Y\), but a nonlinear relationship can still exist.
Classic counterexample: Let \(X \sim N(0,1)\) and \(Y = X^2\). Then:
(since \(E(X^3) = 0\) for a symmetric distribution). Yet \(Y\) is completely determined by \(X\) — they are maximally dependent. Covariance and correlation only capture linear association.
The one valid exception: For the bivariate normal distribution, \(\rho = 0 \Leftrightarrow\) independence. This is a special property of the normal family and does not extend to other distributions.
Econometric relevance: The assumption \(\text{Cov}(u, x) = 0\) in OLS is weaker than the assumption \(E(u \mid x) = 0\). The latter (zero conditional mean) implies the former (zero covariance), but not vice versa. OLS unbiasedness requires the stronger condition.
Practice Prompts
Prompt 1
Consider the joint distribution of \(X\) (hours studied per week) and \(Y\) (exam score, out of 100):
| \(Y = 60\) | \(Y = 80\) | \(Y = 95\) | \(P(X=x)\) | |
|---|---|---|---|---|
| \(X = 5\) | 0.15 | 0.10 | 0.05 | 0.30 |
| \(X = 10\) | 0.10 | 0.25 | 0.10 | 0.45 |
| \(X = 20\) | 0.05 | 0.10 | 0.10 | 0.25 |
(a) Compute \(E(X)\), \(E(Y)\), and \(E(XY)\). (b) Compute \(\text{Cov}(X, Y)\). (c) Compute \(\rho_{XY}\).
Answer
**(a)** $$E(X) = 5(0.30) + 10(0.45) + 20(0.25) = 1.50 + 4.50 + 5.00 = 11.00$$ $$E(Y) = 60(0.30) + 80(0.45) + 95(0.25) = 18 + 36 + 23.75 = 77.75$$ $$E(XY) = 5(60)(0.15) + 5(80)(0.10) + 5(95)(0.05)$$ $$+ 10(60)(0.10) + 10(80)(0.25) + 10(95)(0.10)$$ $$+ 20(60)(0.05) + 20(80)(0.10) + 20(95)(0.10)$$ $$= 45 + 40 + 23.75 + 60 + 200 + 95 + 60 + 160 + 190 = 873.75$$ **(b)** $$\text{Cov}(X,Y) = 873.75 - (11.00)(77.75) = 873.75 - 855.25 = 18.50$$ **(c)** Compute variances: $$E(X^2) = 25(0.30)+100(0.45)+400(0.25) = 7.5+45+100=152.5; \quad \text{Var}(X)=152.5-121=31.5; \quad \sigma_X \approx 5.612$$ $$E(Y^2) = 3600(0.30)+6400(0.45)+9025(0.25) = 1080+2880+2256.25=6216.25; \quad \text{Var}(Y)=6216.25-6045.0625=171.19; \quad \sigma_Y \approx 13.084$$ $$\rho_{XY} = \frac{18.50}{5.612 \times 13.084} \approx \frac{18.50}{73.43} \approx 0.252$$ A moderate positive correlation — more study hours is associated with higher scores.Prompt 2
Let \(X \sim U\{-1, 0, 1\}\) (discrete uniform) and \(Y = X^2\).
(a) Find the joint distribution of \((X, Y)\). (b) Compute \(\text{Cov}(X, Y)\). (c) Are \(X\) and \(Y\) independent? Explain.
Answer
**(a)** Since $Y$ is a deterministic function of $X$: | $X$ | $Y$ | $P(X=x, Y=y)$ | |---|---|---| | $-1$ | 1 | 1/3 | | 0 | 0 | 1/3 | | 1 | 1 | 1/3 | **(b)** $$E(X) = (-1)(1/3) + 0(1/3) + 1(1/3) = 0$$ $$E(Y) = 1(1/3) + 0(1/3) + 1(1/3) = 2/3$$ $$E(XY) = (-1)(1)(1/3) + (0)(0)(1/3) + (1)(1)(1/3) = -1/3 + 0 + 1/3 = 0$$ $$\text{Cov}(X,Y) = E(XY) - E(X)E(Y) = 0 - 0 \cdot (2/3) = 0$$ **(c)** $X$ and $Y$ are **not independent** — knowing $X$ completely determines $Y = X^2$. This is the classic counterexample to "zero covariance implies independence."Prompt 3
A portfolio holds two assets with returns \(R_1\) and \(R_2\): - \(E(R_1) = 0.08\), \(\text{Var}(R_1) = 0.04\) - \(E(R_2) = 0.12\), \(\text{Var}(R_2) = 0.09\) - \(\text{Cov}(R_1, R_2) = -0.02\)
The portfolio return is \(R_P = 0.5 R_1 + 0.5 R_2\).
(a) Find \(E(R_P)\). (b) Find \(\text{Var}(R_P)\). (c) Interpret the effect of the negative covariance on portfolio risk.
Answer
**(a)** $$E(R_P) = 0.5 E(R_1) + 0.5 E(R_2) = 0.5(0.08) + 0.5(0.12) = 0.10$$ **(b)** $$\text{Var}(R_P) = 0.5^2 \text{Var}(R_1) + 0.5^2 \text{Var}(R_2) + 2(0.5)(0.5)\text{Cov}(R_1, R_2)$$ $$= 0.25(0.04) + 0.25(0.09) + 0.5(-0.02)$$ $$= 0.01 + 0.0225 - 0.01 = 0.0225$$ **(c)** Without diversification, a naïve average of individual variances would be $0.25(0.04)+0.25(0.09) = 0.0325$. The negative covariance reduces portfolio variance to 0.0225, illustrating the diversification benefit: assets that tend to move in opposite directions lower total portfolio risk.Further Resources
- Wooldridge — Introductory Econometrics: A Modern Approach, Appendix B.2–B.4 (joint distributions, covariance, correlation)
- Khan Academy — Covariance and the regression line — accessible visual introduction to covariance
- USYD ECMT1020 Notes — Section 1 (Expectation, Variance, Covariance Rules) for USYD-specific formula sheet notation