02 Representation Learning: Extracting Core Insights from High-Dimensional Data

Welcome to Chapter 2: Representation Learning

Today’s Agenda:

1. Introduction: Why do Economics and Finance need ‘dimensionality reduction’?
2. Data Preprocessing: The cornerstone of success
3. Part 1: Linear Methods (PCA, LDA, MDS)
4. Part 2: Nonlinear Manifold Learning (Isomap, LLE, t-SNE)
5. Part 3: Advanced Topics (Sparse Representation)
6. Conclusion: How to choose the right tool for your problem

Core Question: Why Study ‘Dimensionality Reduction’ in Economics?

Imagine we want to predict a company’s stock return. How many variables might we have?

We Face a ‘Data-Rich, Insight-Poor’ Dilemma

The explosive growth in data volume does not directly translate to an increase in insight. The goal of representation learning is to extract the signal from the noise.

This is the so-called ‘Curse of Dimensionality’.

What is the ‘Curse of Dimensionality’?

As dimensionality d increases, a fixed sample becomes progressively sparser. Maintaining the same grid resolution can require exponentially many cells or samples; whether distances concentrate still depends on the distribution, metric, and scaling.

The Curse of Dimensionality is an Enemy of Modeling and Analysis

When data dimensionality d is too high, a series of serious problems arise:

Representation learning (or dimensionality reduction) is the key to solving this problem.

Before We Begin: Preprocessing is the Foundation of Success

Before applying any complex dimensionality reduction algorithm, we must clean the raw data. This is like laying the foundation before building a house.

Preprocessing Issue 1: Outliers

Extreme values can severely distort a model’s variance calculation (e.g., in PCA), pulling it towards the direction of the outlier.

Common Treatments: Winsorization, log transformation, or direct removal.

Preprocessing Issue 2: Missing Data

Most algorithms cannot handle missing values (NaN).

• Common Strategies:
  1. Deletion: If the missing proportion is small, delete the row or column.
  2. Imputation: Fill with the mean, median, or more complex models (like K-Nearest Neighbors).

Preprocessing Issue 3: Inconsistent Scales

If ‘Market Cap’ (trillions) and ‘P/E Ratio’ (tens) are analyzed together, market cap will completely dominate the results.

• Solution: Feature Scaling. The most common is Standardization, which transforms data to have a mean of 0 and a variance of 1.
• Formula: \(x'_{i} = \large{\frac{x_i - \mu_i}{\sigma_i}}\)

Part 1: Linear Dimensionality Reduction Methods

Principal Component Analysis (PCA): Finding the Directions of Maximum Variance

PCA is the most classic and commonly used linear dimensionality reduction method.

• Core Idea: Rotate the coordinate system so that the new axes (principal components) explain the maximum possible variance in the data.
• Goal: Preserve the largest sample variance; this does not automatically preserve information most relevant to prediction, causal identification, or economic interpretation.

PCA Geometry: PC1 Follows the Maximum-Variance Direction

PCA’s Objective Function: Maximizing Projected Variance

PCA seeks a projection direction (a unit vector \(w\)) that maximizes the variance of the projected data.

• One-direction projection: Under the feature-by-sample convention \(X\in\mathbb{R}^{d\times N}\), \(\mathbf z=X^T\mathbf w\in\mathbb{R}^{N}\).
• Projected variance: \(\operatorname{Var}(\mathbf z)=\mathbf w^T S\mathbf w\), where \(S=XX^T/N\in\mathbb{R}^{d\times d}\) is the covariance matrix of centered data.

The optimization problem is:

\[ \large{\max_{w} \quad w^T S w} \]

\[ \large{\text{s.t.} \quad w^T w = 1} \]

PCA’s Derivation: The Lagrangian

We use the method of Lagrange multipliers to solve this constrained optimization problem.

1. Formulate the Lagrangian: The goal is to maximize \(w^T S w\) subject to the constraint that \(w\) is a unit vector, i.e., \(w^T w = 1\).

   \[ \large{L(w, \lambda) = w^T S w - \lambda(w^T w - 1)} \]

PCA’s Derivation: The First-Order Condition

2. Take the derivative with respect to \(w\) and set it to zero: This finds the critical points of the Lagrangian function.

   \[ \large{\frac{\partial L}{\partial w} = 2Sw - 2\lambda w = 0} \]

LDA’s Objective: The Within-Class Scatter Matrix

• Within-class Scatter Matrix (\(S_w\)): Measures the scatter of data points within each class.
  • \(S_w = \sum_{c=1}^{C} \sum_{x_i \in c} (x_i - \mu_c)(x_i - \mu_c)^T\)
  • We want to minimize this. It represents how compact each class is.

LDA’s Objective: The Between-Class Scatter Matrix

• Between-class Scatter Matrix (\(S_b\)): Measures the scatter of the class means around the overall mean.
  • \(S_b = \sum_{c=1}^{C} N_c (\mu_c - \mu)(\mu_c - \mu)^T\)
  • We want to maximize this. It represents how far apart the classes are from each other.

LDA’s Objective Function: Maximizing the Ratio

For one direction \(w\), Fisher’s criterion is a generalized Rayleigh quotient. For a multidimensional projection, take leading generalized eigen-directions; equivalently, maximize projected between-class scatter under an \(S_w\)-orthonormality constraint:

\[ \begin{aligned} J(w)&=\frac{w^T S_b w}{w^T S_w w},\\ \max_W\;&\operatorname{tr}(W^T S_b W) \quad\text{s.t.}\quad W^T S_w W=I. \end{aligned} \]

If \(S_w\) is singular, use regularization or a generalized-eigenvalue/SVD solver rather than a literal inverse.

Multidimensional Scaling (MDS): Reconstructing a ‘Map’ from Distances

Optional topic: after the PCA/LDA comparison, Core goes directly to the Yangtze Delta main case. MDS and the following nonlinear/sparse methods form a returnable extension branch.

MDS has a completely different starting point from PCA and LDA. It doesn’t work with the feature matrix \(X\) directly but starts from a known distance (or dissimilarity) matrix \(D\).

• Core Idea: Find a set of points \(Z\) in a low-dimensional space such that the Euclidean distances between these points are as close as possible to the original distance matrix \(D\).
• Use Case: When we don’t have the original features but can measure the dissimilarity between objects. For example, survey data on ‘brand similarity’ or the ‘edit distance’ between genetic sequences.

MDS Analogy: Reconstructing a City Map

Imagine you only know the straight-line flight distances between major cities, but you have no latitude or longitude information.

The goal of MDS is to find the optimal 2D coordinates for each city based on this distance matrix.

Part 2: Nonlinear Dimensionality Reduction (Manifold Learning)

The Core Idea of Manifold Learning: Data Lives on a Low-Dimensional Manifold

• Manifold Hypothesis: The high-dimensional data we observe is actually generated by a few latent variables (the intrinsic dimension), and these data points lie on a low-dimensional manifold embedded in the high-dimensional space.
• Goal: To ‘unroll’ this manifold and find low-dimensional coordinates that reflect the true neighborhood relationships of the data.
• Difference from Linear Methods: Manifold learning focuses on local structure, assuming that Euclidean distances are only reliable between nearby points.

Locally Linear Embedding (LLE): Preserving Local Linear Relationships

LLE assumes that each data point can be linearly reconstructed by its neighbors, and this local geometric relationship should be preserved in the low-dimensional space.

The Heart of LLE: Preserve Reconstruction Weights, Not Distances

t-SNE: The Swiss Army Knife of Data Visualization

t-SNE (t-distributed Stochastic Neighbor Embedding) is a widely used exploratory visualization tool that embeds local neighborhoods into 2D or 3D; it does not establish a “most powerful” ranking or clustering correctness.

• Core Idea (Probabilistic Matching):
  1. In high-D space, convert Euclidean distances between points into conditional probabilities that represent the likelihood that point \(i\) would pick point \(j\) as its neighbor (using a Gaussian distribution).
  2. In low-D space, define a similar conditional probability (using a heavier-tailed t-distribution).
  3. Adjust the positions of points in the low-D space to make the two probability distributions as similar as possible (by minimizing the KL divergence).
• Advantage: It can reveal local neighborhoods and possible group structure; confirm any apparent clusters with a validation measure chosen before viewing the result.

t-SNE Key Parameter: Perplexity

• Perplexity:
  • This is the most important parameter. It can be loosely interpreted as the ‘effective number of neighbors’ each point considers.
  • Typical values are between 5 and 50.
  • A lower value focuses on local structure, while a higher value considers more of the global structure.

Important Cautions for Interpreting t-SNE plots

1. Do not over-interpret distances between clusters: The distance between two clusters on a t-SNE plot does not meaningfully represent how ‘far apart’ they are in the original space.
2. Do not over-interpret the size of clusters: The area of a cluster on the plot does not mean it contains more data points or has a larger variance.
3. t-SNE is an exploratory visualization tool, not a rigorous clustering analysis method.

Part 3: Advanced Topic: Sparse Representation

Sparse Representation: Building Signals with the Fewest ‘Blocks’

Previous methods aimed to ‘compress’ data, but sparse representation has a different starting point.

• Core Idea: Any signal (e.g., an economic time series) can be represented as a linear combination of a few ‘atoms’ (basis vectors) from a ‘dictionary’ \(\Psi\).

  \[ \large{x = \Psi s} \]

  Here, \(s\) is a sparse vector, meaning most of its elements are zero.

Reconstruction Algorithm: Matching Pursuit

Finding the sparse vector \(s\) from \(z\) and \(\Theta\) is an NP-hard problem. Matching Pursuit is a greedy algorithm that approximates the solution iteratively.

1. Initialize: Residual \(r_0 = z\), sparse solution \(s=0\).
2. Find Most Correlated Atom: Find the atom in the dictionary \(\Theta\) that is most correlated with the current residual \(r\) (has the largest inner product).
3. Update Solution: Add the contribution of this atom to the solution \(s\).
4. Update Residual: Subtract the contribution of this atom from the current residual \(r\).
5. Iterate: Repeat steps 2-4 until the residual is small enough or a maximum number of iterations is reached.

Final Thought: Reduction is a Means to an End, Not the Goal Itself

In this chapter, we have explored a range of representation learning (dimensionality reduction) methods, from linear to nonlinear.

• They are powerful tools for exploratory data analysis, helping us discover hidden structures like factors, clusters, and low-dimensional manifolds in seemingly chaotic high-dimensional data.
• They are also a crucial preprocessing step for building predictive models, effectively improving model stability and generalization.

The key is to always combine your domain knowledge (economics, finance) to interpret the results of dimensionality reduction, giving real-world meaning to these abstract dimensions and structures.

Thank You!

Questions & Discussion