Lesson M01.L06: Hypothesis Testing
| Field | Detail |
|---|---|
| Module | Module 01 — Probability & Statistics Review |
| Level | Graduate (ECMT5001, USYD) |
| Duration | 30–35 minutes |
| Learning Objective | Set up and execute a formal hypothesis test; interpret the p-value correctly; distinguish Type I and Type II errors; connect confidence intervals to hypothesis tests; avoid the most common conceptual errors about "accepting" the null |
| Provenance | Wooldridge Introductory Econometrics, Appendix C.6 |
Explanation
The Framework
Hypothesis testing is a formal procedure for deciding, on the basis of sample data, whether a claim about a population parameter is plausible.
Null hypothesis \(H_0\): The maintained claim, assumed true until evidence overturns it. Usually a specific value:
Alternative hypothesis \(H_1\): The claim we would accept if \(H_0\) is rejected. Can be:
| Type | Form | When to Use |
|---|---|---|
| Two-sided | \(H_1: \mu \neq \mu_0\) | No prior directional belief |
| One-sided (upper) | \(H_1: \mu > \mu_0\) | Theory predicts parameter is larger |
| One-sided (lower) | \(H_1: \mu < \mu_0\) | Theory predicts parameter is smaller |
Significance level \(\alpha\): The maximum tolerated probability of incorrectly rejecting a true \(H_0\). Common choices: \(\alpha = 0.05\) (5%) or \(\alpha = 0.01\) (1%).
Errors in Hypothesis Testing
| \(H_0\) is TRUE | \(H_0\) is FALSE | |
|---|---|---|
| Reject \(H_0\) | Type I Error (probability = \(\alpha\)) | Correct Decision (power = \(1-\beta\)) |
| Fail to reject \(H_0\) | Correct Decision | Type II Error (probability = \(\beta\)) |
- Type I error (false positive): Rejecting \(H_0\) when it is actually true. Controlled by choosing \(\alpha\).
- Type II error (false negative): Failing to reject \(H_0\) when \(H_1\) is true. Its probability \(\beta\) depends on the true parameter value, sample size, and \(\alpha\).
- Power \(= 1 - \beta\): the probability of correctly rejecting a false \(H_0\). A good test has high power.
There is an inherent trade-off: lowering \(\alpha\) (stricter standard for rejection) reduces Type I errors but increases Type II errors (lower power).
Test Statistic
For testing \(H_0: \mu = \mu_0\) with a sample of size \(n\), the test statistic is:
- Numerator: observed distance of the sample mean from the null value
- Denominator: \(SE(\bar{X}) = s/\sqrt{n}\), the standard error (estimated standard deviation of \(\bar{X}\))
Under \(H_0\) and either (a) normality of \(X_i\), or (b) large \(n\) (CLT), this statistic follows a \(t\)-distribution with \(n-1\) degrees of freedom:
As \(n \to \infty\), \(t_{n-1} \to N(0,1)\).
The p-value
The p-value is the probability, computed assuming \(H_0\) is true, of observing a test statistic at least as extreme as the one actually observed:
Decision rule: Reject \(H_0\) if:
Equivalently, using critical values: reject \(H_0\) if \(|t_{\text{obs}}| > t_{n-1,\, \alpha/2}\) (two-sided) or \(t_{\text{obs}} > t_{n-1,\, \alpha}\) (one-sided upper).
A small p-value means the data would be very unlikely if \(H_0\) were true — this is evidence against \(H_0\).
Two-sided vs One-sided Tests
For a two-sided test at \(\alpha = 0.05\) with large \(n\):
- Critical value: \(\pm 1.96\) (from \(N(0,1)\); for \(t_{n-1}\), use the appropriate table value)
- Rejection region: \(|t| > 1.96\)
For a one-sided upper test at \(\alpha = 0.05\) with large \(n\):
- Critical value: \(+1.645\)
- Rejection region: \(t > 1.645\)
Use one-sided tests only when there is a strong a priori theoretical reason to believe the parameter can only deviate in one direction. Otherwise, use two-sided tests to avoid inflating the apparent significance.
Confidence Intervals as the Dual of Hypothesis Tests
A 95% confidence interval for \(\mu\) is:
For large \(n\), \(t_{n-1,\, 0.025} \approx 1.96\):
Duality result: A two-sided test at significance level \(\alpha\) rejects \(H_0: \mu = \mu_0\) if and only if \(\mu_0\) falls outside the \((1-\alpha)\times 100\%\) confidence interval.
The confidence interval is therefore the set of all parameter values that would not be rejected by the data. It gives richer information than a simple reject/fail-to-reject decision.
Worked Example
Economic question: Is the average annual wage of economics graduates equal to $50,000?
Setup:
| Value | |
|---|---|
| Sample size | \(n = 49\) |
| Sample mean | \(\bar{X} = \$52{,}500\) |
| Sample std dev | \(s = \$10{,}500\) |
| Null value | \(\mu_0 = \$50{,}000\) |
| Significance level | \(\alpha = 0.05\) |
Step 1: State hypotheses.
Two-sided, because we are simply asking whether the true mean differs from $50,000 in either direction.
Step 2: Compute the test statistic.
Step 3: Find the p-value.
With \(df = 48\), the critical value at \(\alpha = 0.05\) (two-sided) is approximately \(t_{48,\, 0.025} \approx 2.011\).
Using tables or software: \(P(|T| \geq 1.667) \approx 0.102\).
Step 4: Make the decision.
Step 5: Interpret.
At the 5% significance level, we do not have sufficient evidence to conclude that the average wage of economics graduates differs from $50,000. The sample mean of $52,500 is about 1.67 standard errors above the null value — this could reasonably occur by chance even if \(\mu = \$50,000\).
Confidence interval check:
Since $50,000 lies inside the interval, we confirm: fail to reject \(H_0\) at \(\alpha = 0.05\). ✓
Common Misconception
There are two critical conceptual errors, both extremely common even among researchers:
Misconception 1: "Fail to reject \(H_0\)" means "Accept \(H_0\)".
This is wrong. When we fail to reject \(H_0\), we are saying: "The data are not inconsistent with \(H_0\) at this significance level." This is a fundamentally weaker claim than saying \(H_0\) is true. We simply do not have enough evidence to rule it out.
The hypothesis-testing framework is asymmetric by design: we set up \(H_0\) as the default, and reject only when the evidence is strong. Absence of evidence is not evidence of absence.
Analogy: A not-guilty verdict in court means "not proven guilty beyond reasonable doubt" — it does not mean the defendant is innocent.
Misconception 2: "The p-value is the probability that \(H_0\) is true."
This is wrong. The p-value is a statement about the data conditional on \(H_0\) being true. In the frequentist framework, \(H_0\) is either true or false — it does not have a probability. The p-value answers: "If \(H_0\) were true, how surprising would it be to see data as extreme as ours?"
Correct interpretations: - ✅ "The p-value of 0.102 means that, if the true mean wage were $50,000, there would be a 10.2% chance of observing a sample mean at least $2,500 away from $50,000." - ❌ "There is a 10.2% probability that \(H_0\) is true."
Econometric relevance: These errors lead to overconfidence (treating failure to reject as confirmation) and misreporting (describing p-values as posterior probabilities). At the graduate level, precision in language is essential.
Practice Prompts
Prompt 1
A researcher tests whether the return on a portfolio is zero (no abnormal returns). From 100 monthly observations: \(\bar{R} = 0.8\%\), \(s = 4.5\%\).
(a) State \(H_0\) and \(H_1\) (two-sided). (b) Compute the test statistic. (c) At \(\alpha = 0.05\), use the critical value \(\pm 1.984\) (\(df = 99\)). Make a decision. (d) Compute the 95% confidence interval for the population mean return.
Answer
**(a)** $$H_0: \mu = 0\% \qquad H_1: \mu \neq 0\%$$ **(b)** $$SE = \frac{4.5}{\sqrt{100}} = 0.45$$ $$t = \frac{0.8 - 0}{0.45} \approx 1.778$$ **(c)** Since $|1.778| < 1.984$, we **fail to reject $H_0$** at $\alpha = 0.05$. There is insufficient evidence of abnormal returns. **(d)** $$0.8\% \pm 1.984 \times 0.45\% = 0.8\% \pm 0.893\% = [-0.093\%,\ 1.693\%]$$ The interval contains 0%, confirming failure to reject. The researcher cannot claim statistically significant abnormal returns.Prompt 2
A government economist claims that a new training programme raises average weekly earnings by more than $100. A sample of 36 programme participants yields \(\bar{X} = \$120\) gain, \(s = \$180\).
(a) Set up a one-sided hypothesis test appropriate for this question. (b) Compute the test statistic. (c) At \(\alpha = 0.05\), the critical value is \(t_{35,\, 0.05} \approx 1.690\). What is the conclusion? (d) Would the conclusion change at \(\alpha = 0.10\) (critical value \(\approx 1.306\))?
Answer
**(a)** We want to test whether the gain exceeds \$100, so: $$H_0: \mu = 100 \qquad H_1: \mu > 100 \qquad \text{(one-sided upper)}$$ **(b)** $$SE = \frac{180}{\sqrt{36}} = 30$$ $$t = \frac{120 - 100}{30} = \frac{20}{30} \approx 0.667$$ **(c)** Since $0.667 < 1.690$, we **fail to reject $H_0$** at $\alpha = 0.05$. The sample provides insufficient evidence that the programme raises earnings by more than \$100. **(d)** Since $0.667 < 1.306$, we still fail to reject at $\alpha = 0.10$. The result is not statistically significant at conventional levels — the \$120 observed gain is well within what we'd expect by chance under $H_0: \mu = 100$.Prompt 3
In a sample of 25 firms, average R&D expenditure as a percentage of revenue is \(\bar{X} = 4.2\%\) with \(s = 1.8\%\). An industry report claims the true mean is 3.5%.
(a) Test \(H_0: \mu = 3.5\%\) vs \(H_1: \mu \neq 3.5\%\) at \(\alpha = 0.05\) (\(df = 24\), critical value \(\approx \pm 2.064\)). (b) Compute the p-value bracket using t-tables (hint: where does \(t_{\text{obs}}\) fall between critical values for various \(\alpha\)?). (c) Construct a 99% confidence interval for \(\mu\) (\(t_{24,\, 0.005} \approx 2.797\)). Does it contain 3.5%? (d) Write a one-sentence non-technical conclusion for a policy report.
Answer
**(a)** $$SE = \frac{1.8}{\sqrt{25}} = 0.36$$ $$t = \frac{4.2 - 3.5}{0.36} = \frac{0.7}{0.36} \approx 1.944$$ Since $|1.944| < 2.064$, we **fail to reject $H_0$** at $\alpha = 0.05$. **(b)** From $t_{24}$ tables, $t_{0.025} \approx 2.064$ and $t_{0.05} \approx 1.711$. Our $t_{\text{obs}} \approx 1.944$ falls between them, so the two-sided p-value is between 0.05 and 0.10. We have weak evidence against $H_0$ but not at the 5% level. **(c)** $$4.2 \pm 2.797 \times 0.36 = 4.2 \pm 1.007 = [3.193\%,\ 5.207\%]$$ The interval contains 3.5%, consistent with failure to reject at the 5% level (and also at the 1% level, since 3.5% is inside the 99% CI). **(d)** "Based on our sample of 25 firms, we cannot rule out that the true average R&D expenditure rate equals the industry benchmark of 3.5% of revenue, though the observed average of 4.2% suggests actual spending may be somewhat higher."Further Resources
- Wooldridge — Introductory Econometrics: A Modern Approach, Appendix C.6 (hypothesis testing) and Section 4.2 (t-tests for OLS coefficients)
- Khan Academy — Significance Tests (Hypothesis Testing) — step-by-step introduction with interactive exercises
- ASA Statement on p-values — Wasserstein & Lazar (2016), The American Statistician — a concise authoritative guide to correct p-value interpretation (freely available online)