Introduction to Linear Regression

What is Data Mining? 🤔

Data mining is the process of discovering patterns, insights, and knowledge from large datasets. It involves using various techniques from statistics, computer science, and database management to extract valuable information. It’s like being a detective, but instead of solving crimes, you’re uncovering hidden clues within data.

What is Machine Learning? 🤖

Machine learning is a field of artificial intelligence (AI) that focuses on enabling computers to learn from data without being explicitly programmed (i.e., without being given specific rules for every situation). It involves developing algorithms that can identify patterns, make predictions, and improve their performance over time as they are exposed to more data.

What is Statistical Learning? 📈

Statistical learning is a framework for understanding data using statistical methods. It encompasses a vast set of tools for modeling and understanding complex datasets. It’s a recently developed area in statistics and blends with parallel developments in computer science, such as machine learning. It provides the theoretical underpinnings for many machine learning techniques.

Relationship Between Data Mining, Machine Learning, and Statistical Learning

graph LR
    A[Data Mining] --> C(Common Ground)
    B[Machine Learning] --> C
    D[Statistical Learning] --> C
    C --> E[Insights & Predictions]

Introduction to Linear Regression

Linear regression is a fundamental approach in supervised learning, a type of machine learning where we have labeled data (input features and corresponding output values). It’s used primarily for predicting a quantitative response – a numerical value (e.g., sales, price, height).

Introduction to Linear Regression (Continued)

• Linear regression is a cornerstone in statistical learning. It’s widely used, extensively studied, and forms the basis for many other, more advanced techniques.
• Many advanced statistical learning methods can be seen as extensions or generalizations of linear regression. It’s often the first technique you learn, and it’s a building block for more complex methods.

Visualizing Linear Regression

Relationship between advertising spending and sales

Simple Linear Regression: The Basics

Simple linear regression predicts a quantitative response, \(Y\), using a single predictor variable, \(X\), assuming a linear relationship:

\[ Y \approx \beta_0 + \beta_1X \]

• \(\beta_0\): Intercept (value of \(Y\) when \(X = 0\)). This is where the line crosses the Y-axis.
• \(\beta_1\): Slope (change in \(Y\) for a one-unit increase in \(X\)). This determines how steep the line is.
• \(\beta_0\) and \(\beta_1\) are the model coefficients or parameters. These are the values we need to estimate from the data.

Simple Linear Regression: Equation Visualization

The equation \(Y \approx \beta_0 + \beta_1X\) represents a straight line:

graph LR
    subgraph Linear Equation
    A[Y] --> B(β₀ + β₁X)
    end
    B --> C[β₀: Intercept]
    B --> D[β₁: Slope]
    C --> E[Value of Y when X = 0]
    D --> F[Change in Y for a one-unit increase in X]

Simple Linear Regression: Example

For instance, let’s regress sales onto TV advertising:

\[ \text{sales} \approx \beta_0 + \beta_1 \times \text{TV} \]

• \(Y\): Sales (in thousands of units). This is the response variable – what we’re trying to predict.
• \(X\): TV advertising budget (in thousands of dollars). This is the predictor variable – what we’re using to make the prediction.

Simple Linear Regression: Prediction

Once we estimate the coefficients \(\beta_0\) and \(\beta_1\) (denoted as \(\hat{\beta_0}\) and \(\hat{\beta_1}\)), we can predict sales for a given TV advertising budget (x):

\[ \hat{y} = \hat{\beta_0} + \hat{\beta_1}x \]

Estimating the Coefficients: Least Squares

Our goal is to find \(\hat{\beta_0}\) and \(\hat{\beta_1}\) that best fit the data. “Best fit” means the line should be as close as possible to the data points \((x_i, y_i)\). We use the least squares method. Intuitively, we want to find the line that minimizes the overall “error” between the observed data and the predicted values.

Estimating the Coefficients: Residual Sum of Squares (RSS)

The least squares method minimizes the residual sum of squares (RSS):

\[ \text{RSS} = \sum_{i=1}^{n} (y_i - \hat{y_i})^2 = \sum_{i=1}^{n} (y_i - \hat{\beta_0} - \hat{\beta_1}x_i)^2 \]

• \(y_i\): Actual sales for the \(i\)-th observation. The observed value.
• \(\hat{y_i}\): Predicted sales for the \(i\)-th observation. The value predicted by our linear model.
• \(e_i = y_i - \hat{y_i}\): The residual for the \(i\)-th observation (the difference between the actual and predicted values). This is the “error” for a single data point.

Visualizing Residuals

graph LR
    subgraph Residual Calculation
    A[Observed Data (yᵢ)] --> C{Residual (eᵢ)}
    B[Predicted Data (ŷᵢ)] --> C
    C --> D[eᵢ = yᵢ - ŷᵢ]
    end

Estimating the Coefficients: Least Squares Explained

Estimating the Coefficients: Formulas

Using calculus (specifically, taking partial derivatives and setting them to zero), we find the values of \(\hat{\beta_0}\) and \(\hat{\beta_1}\) that minimize RSS:

\[ \hat{\beta_1} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \]

\[ \hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} \]

• \(\bar{x}\): Sample mean of \(X\). The average value of the predictor.
• \(\bar{y}\): Sample mean of \(Y\). The average value of the response.

Interpreting the Coefficient Formulas (1/2)

The formula for the slope, \(\hat{\beta_1}\), can be interpreted as:

\[ \hat{\beta_1} = \frac{\text{Covariance}(X, Y)}{\text{Variance}(X)} \]

Interpreting the Coefficient Formulas (2/2)

The formula for the intercept, \(\hat{\beta_0}\), ensures that the regression line passes through the point (\(\bar{x}\), \(\bar{y}\)).

\[ \hat{\beta_0} = \bar{y} - \hat{\beta_1}\bar{x} \]

Visualizing the Least Squares Fit

Least Squares Fit

Visualizing the Least Squares Fit: Explanation

Assessing Coefficient Accuracy: Population vs. Sample

• Population Regression Line: The “true” (but usually unknown) relationship: \(Y = \beta_0 + \beta_1X + \epsilon\). This represents the ideal, underlying relationship in the entire population. We almost never know this.
• Least Squares Line: The estimated relationship based on our sample: \(\hat{y} = \hat{\beta_0} + \hat{\beta_1}x\). This is our best guess at the true relationship, based on the data we have. It’s an estimate of the population regression line.
• \(\epsilon\): random error term, which captures all the factors that influence Y but are not included in the model. It represents the inherent randomness in the relationship.

Population vs. Sample Regression Lines: Visualization

Population vs. Sample Regression Lines

Population vs Sample Regression Lines: Left Panel Explanation

Population vs Sample Regression Lines: Right Panel Explanation

Population vs. Sample Regression Lines

Assessing Coefficient Accuracy: Unbiasedness

• \(\hat{\beta_0}\) and \(\hat{\beta_1}\) are estimates of the true (unknown) parameters \(\beta_0\) and \(\beta_1\).
• These estimates are unbiased: on average, they will equal the true values.

Unbiasedness Explained

Assessing Coefficient Accuracy: Standard Error

• Standard Error: Measures the average amount that an estimate (\(\hat{\beta_0}\) or \(\hat{\beta_1}\)) differs from the true value (\(\beta_0\) or \(\beta_1\)). It quantifies the typical uncertainty in our estimates. It’s like a measure of the “spread” of the estimates we would get if we took many samples.

Standard Error: Formulas

Formulas for standard errors:

\[ \text{SE}(\hat{\beta_0})^2 = \sigma^2 \left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \right] \]

\[ \text{SE}(\hat{\beta_1})^2 = \frac{\sigma^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \] - \(\sigma^2\): Variance of the error term \(\epsilon\) (usually unknown, estimated by the residual standard error, RSE). This represents the variability of the data around the true regression line.

Standard Error: Interpretation (1/2)

\[ \text{SE}(\hat{\beta_0})^2 = \sigma^2 \left[ \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \right] \]

Standard Error: Interpretation (2/2)

\[ \text{SE}(\hat{\beta_1})^2 = \frac{\sigma^2}{\sum_{i=1}^{n}(x_i - \bar{x})^2} \]

Assessing Coefficient Accuracy: Confidence Intervals

• Confidence Interval: A range of values that is likely to contain the true unknown value of a parameter, with a certain level of confidence (e.g., 95%). It gives us a sense of how much our estimate might vary if we took different samples.

Confidence Interval: Formula and Interpretation

• Approximate 95% confidence interval for \(\beta_1\):

\[ \hat{\beta_1} \pm 2 \cdot \text{SE}(\hat{\beta_1}) \]

Hypothesis Testing

• Null Hypothesis (H₀): There is no relationship between \(X\) and \(Y\) (\(\beta_1 = 0\)). This is the “skeptical” viewpoint – we assume there’s no relationship unless the data provide strong evidence otherwise.
• Alternative Hypothesis (Hₐ): There is some relationship between \(X\) and \(Y\) (\(\beta_1 \neq 0\)). This is what we’re trying to find evidence for.

Hypothesis Testing: t-statistic and p-value

• t-statistic: Measures how many standard deviations \(\hat{\beta_1}\) is away from 0:

\[ t = \frac{\hat{\beta_1} - 0}{\text{SE}(\hat{\beta_1})} \]

• p-value: The probability of observing a t-statistic as extreme as, or more extreme than, the one calculated, assuming \(H_0\) is true. It tells us how likely it is to see data like ours if there really is no relationship between X and Y.

Hypothesis Testing: Interpretation

Hypothesis Testing: Example (Advertising Data)

Hypothesis Testing: Example Interpretation

Assessing Model Accuracy: RSE

• Residual Standard Error (RSE): An estimate of the standard deviation of the error term \(\epsilon\). It represents the average amount that the response will deviate from the true regression line. It’s a measure of the model’s lack of fit – how much the data points tend to scatter around the regression line.

RSE: Formula

\[ \text{RSE} = \sqrt{\frac{1}{n-2}\text{RSS}} = \sqrt{\frac{1}{n-2}\sum_{i=1}^{n}(y_i - \hat{y_i})^2} \]

Assessing Model Accuracy: R²

• R² Statistic: Measures the proportion of variance explained by the model. It’s a measure of how well the model captures the variability in the response. It always falls between 0 and 1.

R²: Formula

\[ R^2 = \frac{\text{TSS} - \text{RSS}}{\text{TSS}} = 1 - \frac{\text{RSS}}{\text{TSS}} \]

• Total Sum of Squares (TSS): \(\sum(y_i - \bar{y})^2\) - Measures the total variance in the response \(Y\) before considering the predictor. It represents the total variability in the response.

R²: Interpretation

Assessing Model Accuracy: Example (Advertising Data)

Assessing Model Accuracy: Example Interpretation

Multiple Linear Regression

Extends simple linear regression to handle multiple predictors:

\[ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_pX_p + \epsilon \]

• \(\beta_j\): The average effect on \(Y\) of a one-unit increase in \(X_j\), holding all other predictors fixed. This is a crucial point – the interpretation of each coefficient is conditional on the other predictors being in the model.

Multiple Linear Regression: Example (Advertising Data)

\[ \text{sales} = \beta_0 + \beta_1 \times \text{TV} + \beta_2 \times \text{radio} + \beta_3 \times \text{newspaper} + \epsilon \]

Here, we’re trying to predict sales using TV, radio, and newspaper advertising budgets.

Multiple Linear Regression: Example Results

Multiple Linear Regression: Example Interpretation

Correlation Between Predictors

Correlation Matrix: Explanation

One: Is There a Relationship? (F-test)

• Null Hypothesis (H₀): All coefficients are zero (\(\beta_1 = \beta_2 = \dots = \beta_p = 0\)). This means none of the predictors are related to the response.
• Alternative Hypothesis (Hₐ): At least one coefficient is non-zero. This means at least one predictor is related to the response.

F-statistic: Formula

• F-statistic:

\[ F = \frac{(\text{TSS} - \text{RSS})/p}{\text{RSS}/(n - p - 1)} \]

F-test: Example (Advertising Data)

F-test: Example Interpretation

Two: Deciding on Important Variables (Variable Selection)

• Goal: Identify the subset of predictors that are most strongly related to the response. We want to find the most important predictors and exclude those that don’t contribute meaningfully to the model. We aim for a parsimonious model – one that is as simple as possible while still explaining the data well.

• Methods:

  • Forward Selection: Start with the null model (intercept only) and add predictors one by one, based on which improves the model fit the most (e.g., largest decrease in RSS or increase in R²).
  • Backward Selection: Start with all predictors and remove them one by one, based on which has the least impact on the model fit (e.g., smallest decrease in RSS or decrease in R²).
  • Mixed Selection: Combination of forward and backward selection, allowing for both adding and removing predictors at each step.

Variable Selection: Explanation

Three: Model Fit (RSE and R²)

• RSE and R²: Same interpretations as in simple linear regression. RSE measures the average prediction error, and R² measures the proportion of variance explained.
• Important Note: R² will always increase when more variables are added to the model, even if those variables are only weakly associated with the response. This is because adding variables always reduces the RSS on the training data.

Model Fit: Caveat

Addressing Prediction Uncertainty

• Coefficient Uncertainty: Addressed with confidence intervals.
• Model Bias: Addressed by considering more complex models (e.g., non-linear models, interaction terms).
• Irreducible Error: Addressed with prediction intervals.

Confidence vs. Prediction Intervals

• Confidence Interval: Quantifies uncertainty around the average response value for a given set of predictor values. It tells us where the average response is likely to fall, given the predictor values.
• Prediction Interval: Quantifies uncertainty around a single response value for a given set of predictor values. It tells us where an individual response is likely to fall, given the predictor values.

Confidence vs. Prediction Intervals: Comparison

Qualitative Predictors

• Qualitative Predictor (Factor): A variable with categorical values (levels). Examples: gender (male/female), region (North/South/East/West), type of product (A/B/C).
• Dummy Variable: A numerical variable used to represent a qualitative predictor in a regression model. We convert categorical values into numerical codes.

Dummy Variables: How They Work

-   For a predictor with two levels: Create *one* dummy variable.
-   For a predictor with more than two levels: Create *one fewer dummy variable than the number of levels.
-   One level will serve as a *baseline* (reference) level.

Qualitative Predictors: Example (Credit Data)

We want to predict balance using the own variable (whether someone owns a house).

• Create a dummy variable:

\[ x_i = \begin{cases} 1 & \text{if person } i \text{ owns a house} \\ 0 & \text{if person } i \text{ does not own a house} \end{cases} \]

Qualitative Predictors: Regression Model

• Regression model:

\[ y_i = \beta_0 + \beta_1x_i + \epsilon_i = \begin{cases} \beta_0 + \beta_1 + \epsilon_i & \text{if person } i \text{ owns a house} \\ \beta_0 + \epsilon_i & \text{if person } i \text{ does not own a house} \end{cases} \]

Qualitative Predictors: Interpretation

Qualitative Predictors: More than Two Levels

Suppose we have a qualitative predictor, region, with three levels: North, South, and West. We create two dummy variables:

\[ x_{i1} = \begin{cases} 1 & \text{if person } i \text{ is from the North} \\ 0 & \text{otherwise} \end{cases} \]

\[ x_{i2} = \begin{cases} 1 & \text{if person } i \text{ is from the South} \\ 0 & \text{otherwise} \end{cases} \]

Qualitative Predictors: Multi-Level Regression

The regression model would be:

\[ y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \epsilon_i = \begin{cases} \beta_0 + \beta_1 + \epsilon_i & \text{if North} \\ \beta_0 + \beta_2 + \epsilon_i & \text{if South} \\ \beta_0 + \epsilon_i & \text{if West (baseline)} \end{cases} \]

Qualitative Predictors: Multi-Level Interpretation

Interactions

• Additive Assumption: The effect of one predictor on the response does not depend on the values of other predictors. The effect of each predictor is independent of the others. This is a simplifying assumption, but it may not always be true.
• Interaction Effect (Synergy): The effect of one predictor on the response does depend on the values of other predictors. The combined effect of two predictors is different from the sum of their individual effects.
• Interaction Term: Include the product of two predictors in the model to capture the interaction effect.

Interactions: Explanation

Interactions: Example (Advertising Data)

\[ \text{sales} = \beta_0 + \beta_1 \times \text{TV} + \beta_2 \times \text{radio} + \beta_3 \times (\text{TV} \times \text{radio}) + \epsilon \]

This model includes an interaction term between TV and radio advertising.

Interactions: Rewritten Equation

The model with the interaction term can be rewritten as:

\[ \text{sales} = \beta_0 + (\beta_1 + \beta_3 \times \text{radio}) \times \text{TV} + \beta_2 \times \text{radio} + \epsilon \]

Interactions: Qualitative and Quantitative Predictors

We can also include interactions between qualitative and quantitative predictors. For example, we could model the relationship between balance, income, and student (a qualitative variable indicating whether someone is a student) as:

\[ \text{balance} = \beta_0 + \beta_1 \times \text{income} + \beta_2 \times \text{student} + \beta_3 \times (\text{income} \times \text{student}) + \epsilon \]

where student is a dummy variable (1 if student, 0 otherwise).

Interaction: Qualitative and Quantitative Interpretation

Non-linear Relationships: Polynomial Regression

• Linearity Assumption: The relationship between the predictors and the response is linear. This is another simplifying assumption that may not always hold.
• Polynomial Regression: Include polynomial terms (e.g., \(X^2\), \(X^3\)) of the predictors in the model to capture non-linear relationships.

Polynomial Regression: Example

\[ \text{mpg} = \beta_0 + \beta_1 \times \text{horsepower} + \beta_2 \times \text{horsepower}^2 + \epsilon \]

Visualizing Polynomial Regression

## Polynomial Regression: degree = 1

Polynomial Regression: degree = 2

Polynomial Regression

Potential Problems: Impact

Problem 1: Non-linearity

• Detection: Residual plots (plot of residuals vs. predicted values or predictors). A non-linear pattern in the residual plot suggests non-linearity.
• Solution: Non-linear transformations of the predictors (log, square root, polynomial terms), or use non-linear regression models.

Problem 2: Correlation of Error Terms

• Detection: Examine the context of the data (e.g., time series data, clustered data). Autocorrelation plots for time series data.
• Solution: Use time series methods (e.g., autoregressive models) or generalized least squares.

Problem 3: Heteroscedasticity

• Detection: Residual plots. A “funnel” shape (increasing or decreasing spread of residuals) suggests heteroscedasticity.
• Solution: Transformations of the response variable (e.g., log(Y)), weighted least squares.

Problem 4: Outliers

• Detection: Residual plots, studentized residuals. Observations with very large residuals (e.g., |studentized residual| > 3) are potential outliers.
• Solution: Investigate the outliers. If they are due to data entry errors, correct them. If not, consider robust regression methods or removing the outliers (with caution).

Problem 5: High Leverage Points

• Detection: Leverage statistics. Observations with high leverage have a large influence on the regression line.
• Solution: Investigate the points. If they are errors, correct them. If not, consider their impact on the model.

Problem 6: Collinearity

• Detection: Correlation matrix of predictors. Variance inflation factor (VIF). High correlation (> 0.7 or 0.8) or VIF values (> 5 or 10) suggest collinearity.
• Solution: Remove one or more of the collinear predictors, combine collinear predictors, or use regularization techniques (ridge regression, lasso).

Summary

• Linear regression is a fundamental and versatile tool for predicting a quantitative response.
• It relies on assumptions about linearity, additivity, and the error terms. It’s important to check these assumptions.
• We can assess model fit (RSE, R²), coefficient significance (t-statistics, p-values), and overall significance (F-test).
• Multiple linear regression handles multiple predictors, allowing us to isolate the effect of each predictor while controlling for others.
• Extensions (qualitative predictors, interactions, polynomial regression) enhance flexibility, allowing us to model more complex relationships.
• Diagnosing and addressing potential problems is crucial for ensuring the reliability and validity of the model.

Thoughts and Discussion

• How do we choose the “best” model among a set of possible models? (Model selection criteria, cross-validation, adjusted R², AIC, BIC)
• What are the limitations of linear regression, and when might other methods be more appropriate? (Non-linear relationships, complex interactions, non-normal errors, high dimensionality)
• How can we effectively diagnose and address the potential problems in linear regression? (Residual plots, transformations, robust regression, generalized linear models)
• How can the insights from a linear regression model be used to inform real-world decisions? (Marketing: optimizing advertising spend, finance: predicting stock prices, healthcare: identifying risk factors for diseases)
• How does the size and quality of the data affect the model outcome? (More data is generally better; garbage in, garbage out; consider potential biases in the data)
• What are some ethical considerations when using linear regression (or any statistical model) for decision-making? (Fairness, transparency, accountability)