Simple Linear Regression (OLS)

You already know how to draw a line through a cloud of points. This chapter answers the harder, more useful question: which line exactly, how much should you believe its slope — and why regression turns out to be the same conditional expectation and covariance you've been building since Tier 100.

The model

You have paired observations: for each point an input $x$ and an output $y$, and you suspect $y$ moves with $x$ in a roughly straight-line way. The honest model is a line plus noise:

$Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i, \qquad \mathbb{E}[\varepsilon_i \mid X_i]=0.$

$\beta_0$ is the intercept (where the line crosses the vertical axis), $\beta_1$ is the slope — the thing you actually care about, because it says how much $y$ changes per one-unit change in $x$. The error $\varepsilon$ is your admission that the world is not a perfect line. That scatter is not your enemy; it is the very thing that lets you ask whether the slope you found is real.

Ordinary least squares

For any candidate line, each point has a vertical gap to it — a residual. A good line makes the residuals small. Ordinary least squares (OLS) picks the unique line minimizing the sum of squared residuals:

$(\hat\beta_0, \hat\beta_1) = \arg\min_{\beta_0,\beta_1} \sum_{i=1}^{n}\big(Y_i - \beta_0 - \beta_1 X_i\big)^2.$

We square for two reasons you've seen before (ch3 of Tier 100): squaring stops a big positive gap from cancelling a big negative one, and it punishes large misses far more than small ones, so the fit refuses to shrug off an outlier.

The estimates — and an old friend

Grind through that minimization (it is worth seeing done by hand once) and two clean formulas drop out:

$\hat\beta_1 = \frac{\sum_i (X_i-\bar X)(Y_i-\bar Y)}{\sum_i (X_i-\bar X)^2} = \frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(X)}, \qquad \hat\beta_0 = \bar Y - \hat\beta_1 \bar X.$

Read the slope slowly, because it is not a new object. It is the covariance of Tier 100 ch3 — how $X$ and $Y$ move together — divided by the variance of $X$ — how $X$ moves on its own. Regression's slope is the covariance, rescaled by the input's own spread. The intercept then simply forces the line through the centre of mass: the fitted line always passes through $(\bar X, \bar Y)$.

And the line itself is an old friend too. In 102 ch3 you met the regression function $\mathbb{E}[Y\mid X]$ — the best possible prediction of $Y$ given $X$. There it was an abstract conditional expectation; here OLS estimates it from data, under the working assumption that it is a straight line. So regression is not a new idea bolted on — it is conditional expectation made concrete and covariance turned into a testable slope. The whole statistical arc closes here.

How good is the fit? R²

A slope tells you the direction and strength of the relationship, but not how tightly the data follows it. For that we reach back, again, to 102 ch3 — the law of total variance, $\mathrm{Var}(Y) = \text{explained} + \text{unexplained}$. The coefficient of determination is exactly the explained fraction:

$R^2 = 1 - \frac{\text{RSS}}{\text{TSS}} = \frac{\text{variance explained by the line}}{\text{total variance of }Y},$

where RSS is the residual sum of squares and TSS the total sum of squares of $Y$ about its mean. An $R^2$ of 0.9 means the line accounts for 90% of $Y$'s variation and only 10% is residual scatter; an $R^2$ of 0.1 means the relationship, even if real, explains almost nothing.

Least squares is maximum likelihood

Why squared residuals, and not, say, absolute ones? There's a deep answer from 201 ch1: if the errors $\varepsilon$ are normally distributed, then minimizing the sum of squared residuals is exactly maximizing the likelihood. OLS is the maximum-likelihood estimator under Gaussian noise — not an arbitrary convention but the principled estimate when the scatter is bell-shaped. That also surfaces the assumptions OLS quietly rests on, and which you will learn to check in ch3: the relationship is genuinely linear, the errors have zero mean and constant variance, they are independent, and (for exact small-sample inference) roughly normal. As Wasserman puts it, the inferences are most trustworthy "when the residuals behave like random normal numbers." When they don't, the fit can still be computed — it just lies about its own certainty.

Is the slope real? Standard error, t, and CI

Here is what separates a quant from someone who merely owns a spreadsheet. Your estimate $\hat\beta_1$ is almost never exactly zero, even when $X$ and $Y$ are unrelated, because random scatter always conjures some accidental tilt. The whole discipline of inference (201) exists to ask: is this slope evidence, or noise that survived by luck? You need the standard error of the slope:

$\mathrm{se}(\hat\beta_1) = \frac{\hat\sigma}{\sqrt{\sum_i (X_i-\bar X)^2}},$

and staring at it tells you exactly when to trust a slope. The error shrinks — trust grows — when the residuals are small (points hug the line), when you have more data, and when the $X$ values are spread out (a wide lever arm pins the line down far better than a cramped cluster). Noisy data, few points, or bunched-up $x$'s all inflate the error and tell you to hold your conclusion loosely.

Then inference is mechanical, exactly as in 201 ch3. The t-statistic is the slope over its own standard error, $t = \hat\beta_1/\mathrm{se}(\hat\beta_1)$ — the effect measured in units of its own uncertainty. A $t$ near 2 sits about two standard errors from zero, the rough line for 5% significance; equivalently the confidence interval is $\hat\beta_1 \pm \approx 2\,\mathrm{se}$. If it comfortably excludes zero, you have evidence; if it straddles zero, the data is consistent with no relationship at all, and honesty demands you say so.

A bonus: regression toward the mean

The word "regression" is a historical accident worth keeping, because it names a real trap. Galton noticed that unusually tall fathers tend to have sons closer to average height — the fitted slope is less than one, so predictions are pulled toward the mean. The same thing governs markets: an extreme reading — a huge IV spike, an outlier return — tends to be followed by a more average one, not because of any restoring force, but because extremes are partly luck, and luck doesn't repeat. Mistake that statistical reversion for a tradeable signal (or for the failure of a strategy) and you will misread the most ordinary thing in data. Extremes regress; plan for it.

Application: the wing slope