05 Linear Models

Welcome to Chapter 5: Linear Models

• The intersection of machine learning and econometrics.
• The art and science of finding linear relationships in data.
• An indispensable foundation for building more complex models.

Goal 1: Build Powerful Predictive Capabilities

Linear models are the workhorses of economic forecasting.

• Macroeconomics: Predicting GDP growth, inflation, and unemployment rates.
• Financial Markets: Forecasting stock returns and asset price volatility.
• Micro-level Behavior: Predicting consumer purchasing habits and corporate sales.

Mastering them means you possess the fundamental tools to quantitatively forecast the future.

Goal 2: Separate Predictive Association from Causal Effects

Linear regression first describes a conditional-mean association; it does not automatically answer ‘why’. Causal analysis must define the treatment, outcome, estimand, and a defensible identification design.

• Identification checklist: establish timing and an assignment/exogeneity source; require consistency/SUTVA; measure treatment and key confounders; check overlap/positivity when relevant; choose in advance the estimand and specification.
• Design before regression: tax, education, and pricing effects need a randomized or credible quasi-experimental design—not merely a broad ‘no omitted variables’ phrase.
• Without identification: report a predictive association, not a policy or pricing effect.

This chapter trains prediction; its causal examples mark the boundary and do not turn an ordinary regression into an identified effect.

Goal 3: Establish the Foundation for Advanced Models

Nearly all modern, advanced machine learning models incorporate linear transformations at their core.

• Neural Networks: Each neuron performs a weighted sum (a linear transformation) followed by a non-linear activation function.
• Factor Models: In finance, asset returns are modeled as linear exposures to various risk factors.
• Generalized Linear Models (GLMs): Extend the linear predictor to data with various distributions via a link function.

A solid understanding of linear models is the gateway to the broader world of machine learning.

This Chapter’s Learning Roadmap

We will follow a path from simple to complex to comprehensively master the ‘family’ of linear models.

The Core Idea: What is a Linear Model?

The central assumption of a linear model is that the target variable can be expressed as a weighted sum of the input features.

For a sample \(\mathbf{x} = (x_1, x_2, \ldots, x_d)\) with d features, the prediction function is:

\[ \large{f(\mathbf{x}; \mathbf{w}, b) = w_1x_1 + w_2x_2 + \ldots + w_dx_d + b} \]

Using vector notation, this can be written concisely as:

\[ \large{f(\mathbf{x}; \mathbf{w}, b) = \mathbf{w}^T\mathbf{x} + b} \]

• \(\mathbf{w} = (w_1, \ldots, w_d)\): The weight vector, which determines the importance of each feature.
• \(b\): The bias or intercept term, which acts as the model’s baseline.

Anatomy of a Linear Model

This diagram illustrates how a linear model combines input features to produce a single predictive value.

The Geometry: It Defines a Decision Hyperplane

The model’s equation, \(\mathbf{w}^T\mathbf{x} + b = 0\), geometrically defines a hyperplane.

• In 2D space (d=2): This is a straight line (\(w_1x_1 + w_2x_2 + b = 0\)).
• In 3D space (d=3): This is a flat plane (\(w_1x_1 + w_2x_2 + w_3x_3 + b = 0\)).

This hyperplane divides the feature space into two halves, forming the decision boundary for all linear classifiers.

Hyperplanes Take Different Forms in Different Dimensions

The Weight Vector w Determines the Hyperplane’s Orientation

The weight vector \(\mathbf{w}\) is not just for weighting features; geometrically, it is always perpendicular to the decision hyperplane.

• The direction of \(\mathbf{w}\) points in the direction of the fastest increase in the function \(f(\mathbf{x})\).
• Orthogonality argument:

  • For two hyperplane points, \(\mathbf{w}^T(\mathbf{x}_A - \mathbf{x}_B) = 0\).

  • Therefore \(\mathbf{w}\) is orthogonal to every direction lying within the hyperplane.

In the previous slide, the vector \(\mathbf{w}\) (teal arrow) is perpendicular to the decision boundary (grey line).

From Geometry to Prediction: The Decision Rule

Once we have the hyperplane \(\mathbf{w}^T\mathbf{x} + b = 0\), classification is straightforward.

For a new data point \(\mathbf{x}\), we compute the value of \(f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b\):

• If \(f(\mathbf{x}) > 0\), the point lies on the side of the hyperplane pointed to by \(\mathbf{w}\). We predict it as the positive class (e.g., label +1).
• If \(f(\mathbf{x}) < 0\), the point lies on the other side. We predict it as the negative class (e.g., label -1).

This decision function is often written as: \(\hat{y} = \text{sign}(\mathbf{w}^T\mathbf{x} + b)\).

Key Math: Distance from a Point to the Hyperplane

What is the distance from a sample point \(\mathbf{x}\) to the decision boundary \(\mathbf{w}^T\mathbf{x} + b = 0\)? This concept is crucial for Support Vector Machines (SVMs).

From analytic geometry, the distance \(r\) is given by:

\[ \large{r = \frac{|\mathbf{w}^T\mathbf{x} + b|}{\|\mathbf{w}\|}} \]

Where \(\|\mathbf{w}\|\) is the L2 norm (Euclidean length) of the weight vector, \(\|\mathbf{w}\| = \sqrt{w_1^2 + w_2^2 + \ldots + w_d^2}\).

Within one fixed model, \(|f(\mathbf{x})|\) is proportional to geometric distance; across models, use \(|f(\mathbf{x})|/\|\mathbf{w}\|\). A geometric margin is not a calibrated class probability.

5.1 From Classification to Regression: Linear Regression

If our goal is not to predict a discrete class (like ‘Expansion/Recession’) but a continuous value (like house prices, stock prices), the linear model becomes Linear Regression.

We directly use the model’s output as the predicted value:

\[ \large{\hat{y} = f(\mathbf{x}; \mathbf{w}, b) = \mathbf{w}^T\mathbf{x} + b} \]

The objective now becomes: Find the best \(\mathbf{w}\) and \(b\) such that the predicted value \(\hat{y}\) is as close as possible to the true observed value \(y\).

Measuring ‘Closeness’: Mean Squared Error (MSE)

We use a Loss Function to quantify the ‘error’ of our predictions. For linear regression, the most common loss function is the Mean Squared Error (MSE).

For a dataset of \(N\) samples \(\{(\mathbf{x}_n, y_n)\}_{n=1}^N\), the MSE is defined as:

\[ \large{J(\mathbf{w}, b) = \frac{1}{N} \sum_{n=1}^N (y_n - \hat{y}_n)^2 = \frac{1}{N} \sum_{n=1}^N (y_n - (\mathbf{w}^T\mathbf{x}_n + b))^2} \]

Our goal is to find the \(\mathbf{w}\) and \(b\) that minimize this \(J(\mathbf{w}, b)\). This is the famous Least Squares Method.

Geometric Intuition of the MSE Loss Function

Least squares finds the line that minimizes the sum of squared vertical residuals.

The Normal Equation Has Pros and Cons

Problem: What if Features are Numerous or Correlated?

Standard linear regression (OLS) runs into trouble in certain situations:

1. Overfitting:

• When the number of features \(d\) is close to or exceeds the number of samples \(N\), the model can become overly complex.

• It may fit the training data perfectly but perform poorly on new data.

2. Multicollinearity:
   • Pattern: features are highly correlated, such as house area and number of rooms.
   • Mechanism: \(\mathbf{X}^T\mathbf{X}\) becomes nearly singular.
   • Consequence: weight estimates \(\mathbf{w}\) become unstable and unreliable.

A Visual Example of Overfitting

An overfit model learns the ‘noise’ in the training data, not just the underlying ‘signal’.

The Core Trade-off: Bias vs. Variance

• Bias: The systematic difference between a model’s predictions and the true values. High bias means the model is too simple (underfitting).
• Variance: The variability of a model’s predictions across different training sets. High variance means the model is too sensitive to the training data (overfitting).

Our goal is to find a model that achieves a good balance between bias and variance.

The Solution: Regularization Penalizes Complexity

The core idea of regularization is to penalize model complexity while minimizing training error.

We achieve this by adding a Penalty Term to the loss function, which is related to the magnitude of the weights \(\mathbf{w}\).

\[ \large{J_{\text{reg}}(\mathbf{w}, b) = \text{Training Error (e.g., MSE)} + \lambda \cdot \text{Complexity Penalty}} \]

• \(\lambda \ge 0\) is the regularization parameter, a hyperparameter we set. It controls the strength of the penalty.
• \(\lambda = 0\): No penalty; this reverts to standard linear regression.
• \(\lambda \to \infty\): The penalty is extreme, forcing all weights toward zero.

5.2 L2 Regularization: Ridge Regression

Ridge Regression uses the squared L2 norm of the weight vector, \(\|\mathbf{w}\|_2^2 = \sum_{j=1}^d w_j^2\), as its penalty term.

The objective function is:

\[ \large{J_{\text{Ridge}}(\mathbf{w}, b) = \text{MSE}(\mathbf{w},b) + \lambda \sum_{j=1}^d w_j^2} \]

Effect:

• It causes shrinkage of the coefficients, pulling them towards zero but rarely making them exactly zero.
• By penalizing large weights, it makes the model smoother and reduces variance.
• It effectively handles multicollinearity, making the model more stable.

L1 Regularization: Lasso Regression

Lasso (Least Absolute Shrinkage and Selection Operator) Regression uses the L1 norm of the weight vector, \(\|\mathbf{w}\|_1 = \sum_{j=1}^d |w_j|\), as its penalty.

The objective function is:

\[ \large{J_{\text{Lasso}}(\mathbf{w}, b) = \text{MSE}(\mathbf{w},b) + \lambda \sum_{j=1}^d |w_j|} \]

Effect:

• Lasso not only shrinks coefficients but can force the coefficients of some unimportant features to be exactly zero.
• Therefore, Lasso performs automatic Feature Selection, producing a sparser, more interpretable model, which is highly valuable in economic analysis.

5.3 Logistic Regression

Let’s return to classification. What are the problems with using the raw output of a linear model, \(\mathbf{w}^T\mathbf{x}+b\), directly for classification?

1. Mismatched Output Range: The output is \((-\infty, +\infty)\), whereas the Sigmoid maps every finite input strictly into \((0, 1)\).
2. Sensitivity to Outliers: A single outlier far from the decision boundary can drastically shift the regression line, thereby altering the classification outcome.

Logistic regression solves these issues with a clever ‘squashing’ function.

Probabilistic Interpretation of Logistic Regression

The logistic regression model assumes the probability of a sample belonging to the positive class (y=1) is:

\[ \large{P(y=1 | \mathbf{x}; \mathbf{w}, b) = \sigma(\mathbf{w}^T\mathbf{x} + b)} \]

Therefore, the probability of it belonging to the negative class (y=0) is:

\[ \large{P(y=0 | \mathbf{x}; \mathbf{w}, b) = 1 - P(y=1 | \mathbf{x}; \mathbf{w}, b)} \]

A decision threshold of 0.5 is typically used: if \(P(y=1 | \mathbf{x}) > 0.5\) (which means \(\mathbf{w}^T\mathbf{x} + b > 0\)), we predict 1; otherwise, we predict 0.

Intuition Behind the Cross-Entropy Loss

The SVM Optimization Problem (Hard Margin)

In summary, the optimization problem for a linearly separable (hard-margin) SVM is:

\[ \large{\min_{\mathbf{w}, b} \quad \frac{1}{2}\|\mathbf{w}\|^2} \]

\[ \large{\text{subject to} \quad y_n(\mathbf{w}^T\mathbf{x}_n + b) \ge 1, \quad \forall n=1, \ldots, N} \]

This is a convex quadratic programming problem with inequality constraints, which can be solved using methods like Lagrange duality.

The Real World is Messy: What if Data isn’t Linearly Separable?

In real-world economic data, perfect linear separability is almost never the case. There will always be some noise or outliers.

If we force a hard-margin SVM on such data, we might either find no solution or find a poor boundary that overfits to the noisy points.

Solution: Introduce the Soft Margin, which allows the model to make a few mistakes.

Soft-Margin SVMs Tolerate Errors via Slack Variables

We introduce a slack variable \(\xi_n \ge 0\) for each data point.

The constraint is relaxed to:

\[ \large{y_n(\mathbf{w}^T\mathbf{x}_n + b) \ge 1 - \xi_n} \]

• If \(\xi_n = 0\), the point is correctly classified and outside the margin.
• If \(0 < \xi_n < 1\), the point is within the margin but still correctly classified.
• If \(\xi_n = 1\), the constraint permits the point to lie exactly on the decision boundary, so strict correct classification is not guaranteed.
• If \(\xi_n > 1\), the constraint permits the point to cross the boundary and be misclassified.

The Hyperparameter C Balances Margin Width and Errors

\(C\) is a crucial hyperparameter that controls the penalty for slack variables. Think of it as the inverse of the regularization parameter \(\lambda\).

• Small \(C\): Low penalty for errors. The model prioritizes a wide margin, even if it means some points are inside the margin or misclassified. High tolerance, strong regularization, may underfit.
• Large \(C\): High penalty for errors. The model tries very hard to classify every point correctly, which can lead to a narrow margin and overfitting to the training data. Low tolerance, weak regularization, may overfit.

5.5 Multiclass Linear Models

We have focused on binary classification, but many real-world tasks involve multiple categories. For example:

• Segmenting customers into ‘High-Value’, ‘Mid-Value’, and ‘Low-Value’.
• Recognizing handwritten digits (0-9, a 10-class problem).
• Predicting the state of the economy: ‘Recovery’, ‘Boom’, ‘Recession’, or ‘Depression’.

How can we extend binary classifiers to handle multiclass scenarios?

Strategy 1: One-vs-Rest (OvR)

This strategy involves training one binary classifier for each class, which is trained to distinguish that class from all other classes combined.

• For K classes, you train K classifiers.
• Prediction compares K decision scores. They are directly comparable only under a consistent training/scaling scheme or after calibration, and they are not automatically probability confidences.

OvR vs. OvO: A Comparison

5.6 The Class Imbalance Problem

In many important real-world applications, the event of interest is very rare.

• Financial Fraud Detection: The vast majority of transactions are legitimate.
• Rare Disease Diagnosis: Most people are healthy.
• Ad Click-Through Prediction: A very small fraction of users click on an ad.

This situation is known as Class Imbalance. For example, a dataset might contain 99% negative samples and only 1% positive samples.

A Better Evaluation Tool: The Confusion Matrix

The Confusion Matrix is a table that visualizes the performance of a classification model.

Solution 1: Data-Level Resampling

The most direct approach is to fix the imbalance at the data level. There are two main strategies:

Pros and Cons of Resampling Methods

Main lesson: SMOTE is Extension; continue from the resampling comparison to algorithm-level adjustment, then the main case.

How the SMOTE Algorithm Works

Solution 2: Algorithm-Level Adjustments

Besides modifying the data, we can also adjust the learning algorithm itself.

• Adjusting Class Weights: We can assign a higher penalty for misclassifying minority class samples in the loss function.
  • For example, setting class_weight='balanced' in scikit-learn models.
  • This forces the model to pay more attention to correctly classifying the minority class during optimization.
• Changing the Decision Threshold: Logistic regression uses a 0.5 probability threshold by default. For imbalanced problems, we can lower this threshold (e.g., to 0.3) to increase recall for the minority class.

The Linear Model Family at a Glance

This diagram summarizes the models we’ve discussed, showing how different combinations of loss functions and regularization penalties lead to different models.

Thank You!

Q & A