Multiple Regression & Its Assumptions

In the last chapter you fit a line to one predictor: a single slope, a single intercept, a cloud of points. That model answered one question — as $x$ moves, how does $y$ tend to move with it? A good question. But the world rarely hands you one predictor at a time, and when you force a one-variable line onto a many-variable reality, the slope you get back can lie to you in a very specific way. This chapter is about adding predictors — not because more is better (it usually isn't), but because the move from one to several changes what a slope means.

The confounding problem

Suppose you want to know how steeply implied vol rises as you walk into the deep tail of a smile. You could draw a line through two points — the near wing and the far wing — and read its slope. A perfectly good single-variable answer. But the tail's steepness is not the only thing moving: the overall level of volatility drifts up and down, and the smile gets more or less curved. Measure tail steepness with one crude slope on a day when the entire surface has lifted, and you cannot tell whether the tail is genuinely more fearful or whether everything simply got more expensive at once. The single number has confounded two stories. Multiple regression pulls them apart.

The model and the matrix form

In multiple regression you model $y$ as a linear combination of several predictors plus noise:

$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots + \beta_p X_{ip} + \varepsilon_i.$

Stack all $n$ observations and it collapses to one clean matrix equation,

$\mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},$

where $\mathbf{Y}$ is the column of outcomes, $\mathbf{X}$ is the design matrix (one row per observation, one column per predictor, plus a leading column of ones for the intercept), and $\boldsymbol{\beta}$ is the vector of coefficients. We estimate it the same way as before — minimize the squared residuals — and the solution has a closed form worth committing to memory:

$\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{Y}.$

Read it slowly. $\mathbf{X}^\top\mathbf{X}$ is a small square matrix capturing how the predictors covary with each other; inverting it is precisely the step that de-correlates them, so each coefficient reflects its own contribution and not its neighbour's. That inversion is the mathematical machinery of "holding the others fixed."

The partial effect

Here is the sentence that matters most in the chapter: each slope $\beta_j$ is the expected change in $y$ for a one-unit increase in $x_j$, holding every other predictor fixed. That phrase — holding the others fixed — is what a single-variable regression cannot give you, and it is exactly what cures the confounding problem. $\beta_j$ is a partial effect: the steepness of the tail after the overall level has been accounted for.

The five assumptions — and omitted-variable bias

The formula always returns an answer. Whether it means anything rests on five classical assumptions a serious analyst can recite:

• Linearity — $y$ really is a linear function of the predictors plus noise.
• No perfect multicollinearity — no predictor is an exact linear combination of the others; if two columns of $\mathbf{X}$ carry the same information, $\mathbf{X}^\top\mathbf{X}$ cannot be inverted.
• Exogeneity — $\mathbb{E}[\varepsilon \mid \mathbf{X}] = 0$; the errors are uncorrelated with the predictors.
• Homoskedasticity — the error variance is constant across observations.
• No autocorrelation — the errors are uncorrelated with each other (the one that bites hardest in time series — the bridge to the rest of this course).

The first three buy you unbiased coefficients; the last two buy you trustworthy standard errors. The exogeneity failure has a name you met in Tier 100 ch6: omitted-variable bias. If you leave out a variable that affects $y$ and is correlated with an included predictor, that predictor's coefficient absorbs the missing effect and becomes biased — it is no longer a clean partial effect but a confounded one. A steep, impressive slope estimated on data that violates exogeneity is not evidence of anything except your own optimism.

Multicollinearity, quantified

"No perfect multicollinearity" is the hard requirement; near multicollinearity is the practical menace. When two predictors are highly correlated, $\mathbf{X}^\top\mathbf{X}$ is nearly singular and its inverse blows up — so the coefficient standard errors balloon. The diagnostic is the variance inflation factor: regress predictor $j$ on all the others, get its $R_j^2$, and

$\mathrm{VIF}_j = \frac{1}{1 - R_j^2}.$

As $R_j^2 \to 1$ (predictor $j$ is well-explained by the others), $\mathrm{VIF}_j \to \infty$ and that coefficient's variance explodes. The symptom: coefficients that are individually insignificant, wildly unstable, and prone to flip sign when you nudge the data — because the regression genuinely cannot separate two predictors that move together. (Note: collinearity damages the individual coefficients, not necessarily the model's predictions.)

More isn't better: adjusted R² and overfitting

Why not just throw in every predictor you can find? Because R² can only rise when you add a predictor — even a column of pure noise. In-sample fit is a downward-biased, optimistic measure of real performance: the model can always contort itself to hug the data it was fit on. This is the bias-variance tradeoff (Tier 100 ch5) in model-selection clothing: too few predictors underfits (high bias, missing real structure); too many overfits (high variance, fitting noise that won't repeat), and it is the very backtest illusion of ch6 — spectacular in-sample, useless out-of-sample.

The honest fix is to penalize complexity. The simplest is adjusted R², which docks the fit for each predictor used:

$\bar R^2 = 1 - \frac{\mathrm{RSS}/(n - p - 1)}{\mathrm{TSS}/(n - 1)},$

so it rises only if a new predictor earns its keep against the degrees of freedom it costs. More principled criteria — Mallows' $C_p$, AIC, BIC, and cross-validation — formalize the same instinct: reward fit, punish complexity, and aim for the model that will predict, not just the one that flatters the past.

The F-test: are the predictors jointly useful?

The individual t-tests ask, one at a time, "is this coefficient nonzero?" The F-test asks a different, joint question: "taken together, do these predictors explain $y$ better than the intercept alone?" This matters precisely under collinearity, where each coefficient's t can look insignificant (their effects can't be cleanly separated) even though the group is jointly, decisively significant. Read the F first for "is there anything here at all," then the t's for "which pieces."

Application: the crash-o-meter