10 Artificial Neural Networks - An Economic Perspective

Artificial Neural Networks: An Economic Perspective

A journey from the limits of linear models to the power of deep learning for economic analysis.

Today’s Agenda: Moving Beyond Linearity

1. Core · Review & Reflection: limitations of linear models.
2. Core · Core Idea: biological inspiration for a mathematical model.
3. Core · Building Block: start with one neuron.
4. Core · Key Innovation: activation functions introduce non-linearity.
5. Core · Build a Network: from Perceptron to MLP.
6. Core · Model Learning: gradient descent and backpropagation.
7. Core · China-Market Practice: chronological split, threshold choice on validation, and one final test evaluation.
8. Extension (optional) · CNNs and Model History: return to the Core leakage check and transfer.

The Beauty and Burden of the Linear Assumption

A linear relationship means that for every one-unit increase in an independent variable \(X\), the change in the dependent variable \(Y\) is constant (\(\beta\)).

But is the real world always this simple?

The Real World: A Web of Complex, Non-Linear Relationships

Many economic phenomena cannot be perfectly described by a straight line.

• Diminishing Marginal Utility: The happiness gained from an increase in income diminishes at higher income levels.
• The Laffer Curve: The relationship between tax rates and tax revenue is an ‘inverted U-shape’.
• ‘Fear’ and ‘Greed’ in Financial Markets: Asset prices react non-linearly to news, exhibiting thresholds and sharp fluctuations.

When faced with these complex non-linear relationships, traditional econometric models may fall short.

Example 1: Diminishing Marginal Utility

The higher the income, the smaller the increase in happiness from the same amount of money.

Example 2: The Laffer Curve

Higher tax rates are not always better. Excessively high rates can stifle economic activity, leading to a decrease in tax revenue.

The Mathematical Abstraction: The McCulloch-Pitts Neuron

In 1943, Warren McCulloch and Walter Pitts proposed the first mathematical model of a neuron, known as the ‘M-P model’.

It simulates two key processes of a biological neuron:

1. Signal Aggregation: It receives input signals from multiple upstream neurons and calculates their weighted sum.
2. Activation Decision: It compares this weighted sum to a threshold. If the sum exceeds the threshold, the neuron ‘fires’ and outputs a signal; otherwise, it remains ‘inhibited’ and outputs nothing.

M-P Model Step 1: Signal Aggregation

Assume a neuron receives p input signals \(x_1, x_2, \dots, x_p\) from other neurons.

First, a linear transformation (weighted sum) is performed:

\[ \large{u = \sum_{i=1}^{p} w_i x_i} \]

Here, \(w_i\) represents the ‘weight’ of the \(i\)-th connection, simulating the strength of a synapse. A higher weight means the corresponding input signal is more important.

M-P Model Step 2: Activation Decision

Next, the weighted sum \(u\) is compared with a threshold \(\theta\):

\[ \large{y = \begin{cases} 1, & \text{if } u \ge \theta \quad \text{(Fires)} \\ 0, & \text{if } u < \theta \quad \text{(Inhibited)} \end{cases}} \]

This is an ‘all-or-nothing’ response pattern, like a switch that is either on (1) or off (0).

Graphical Representation of the M-P Model

We can represent the M-P model with a simple computation graph.

A More Convenient Formulation: Introducing the Bias Term

Working with a threshold \(\theta\) is algebraically inconvenient. We can perform a simple transformation.

Let \(b = -\theta\). This \(b\) is called the bias.

Then, the condition \(u \ge \theta\) is equivalent to \(u - \theta \ge 0\), which is \(u + b \ge 0\).

This allows us to treat the bias \(b\) as a special weight whose corresponding input is always 1.

\[ \large{z = \left(\sum_{i=1}^{p} w_i x_i\right) + b} \]

The activation process then becomes checking if \(z\) is greater than or equal to 0.

Saturated Activation 1: The Sigmoid Function

The Sigmoid function, also known as the Logistic function, was one of the most common activation functions in early neural networks.

\[ \large{\sigma(z) = \frac{1}{1 + e^{-z}}} \]

• Role: Squeezes any real-valued input into the range \((0, 1)\).
• Probability condition: With a binary output layer and an appropriate Bernoulli likelihood/cross-entropy objective, Sigmoid can parameterize a probability; empirical calibration must still be checked on validation data.

Pros and Cons of the Sigmoid Function

Visualizing the Sigmoid Function and Its Derivative

Derivative: \(\sigma'(z) = \sigma(z)(1 - \sigma(z))\)

Saturated Activation 2: The Tanh Function

The hyperbolic tangent (Tanh) function is a variant of the Sigmoid.

\[ \large{\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2\sigma(2z) - 1} \]

• Role: Squeezes input into the range \((-1, 1)\).
• Core Advantage:

  • Zero-centered means the range and graph are symmetric around zero; it does not guarantee zero-mean realized activations.

  • With a roughly symmetric preactivation distribution this can improve gradient directions, but actual convergence depends on the distribution and optimizer.

Pros and Cons of the Tanh Function

Visualizing the Tanh Function and Its Derivative

Derivative: \(\tanh'(z) = 1 - \tanh^2(z)\)

The Modern Default: ReLU (Rectified Linear Unit)

The Rectified Linear Unit (ReLU) is currently the most popular activation function, especially in deep learning.

\[ \large{\text{ReLU}(z) = \max(0, z) = \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{if } z \le 0 \end{cases}} \]

It acts like a gatekeeper: negative values are blocked (set to zero), while positive values pass through unchanged.

Pros and Cons of the ReLU Function

From a Single Neuron to a Network: The Perceptron

In 1957, Frank Rosenblatt introduced the Perceptron, which can be considered the first complete, learnable neural network model.

• Structure: A single M-P model neuron.

• Activation Function: The sign function, which outputs -1 or 1.

  \[ \large{\hat{y} = \text{sign}(\mathbf{w}^T \mathbf{x} + b)} \]

• Capability: The Perceptron is a linear classifier. It can find a line (or hyperplane) in the feature space to separate data points into two classes.

The Perceptron Learning Algorithm: Error-Driven

The Perceptron’s learning rule is very intuitive: ‘Correct mistakes as you see them’.

1. Initialize weights \(\mathbf{w}\) and bias \(b\).
2. For each training example \((\mathbf{x}, y)\):

   1. Make a prediction \(\hat{y}\) using the current parameters.

   2. If the prediction is wrong (\(y \neq \hat{y}\)), update the parameters:

      \[ \large{\mathbf{w} \leftarrow \mathbf{w} + \eta y \mathbf{x}} \]

      \[ \large{b \leftarrow b + \eta y} \]

      where \(\eta\) is the learning rate.

   3. If the prediction is correct, do nothing.

3. If the data are linearly separable, convergence is finite. Otherwise cap epochs or use an error tolerance and retain the best iterate.

Visualizing the XOR Problem: Linearly Inseparable

It is impossible to draw a single straight line to separate the blue squares (y=0) from the orange triangles (y=1).

Network Architecture: Depth vs. Width

How to Train an MLP: The Core Idea

We have the network structure, but how do we find the optimal parameter values for the thousands (or millions) of parameters (all the W’s and b’s)?

1. Define a Loss Function: First, we need a function to measure how ‘bad’ the model’s predictions are.
   • Regression: Mean Squared Error (MSE)
   • Classification: Cross-Entropy
2. Objective: Find the set of parameters \((\mathbf{W}, \mathbf{b})\) that minimizes the total loss over the entire training set.
3. Method: Use the Gradient Descent algorithm.

The Mathematics of Gradient Descent

The parameter update rule is:

\[ \large{\theta_{\text{new}} = \theta_{\text{old}} - \eta \nabla_{\theta} J(\theta)} \]

• \(\theta\): Represents all model parameters (W, b).
• \(J(\theta)\): The loss function.
• \(\nabla_{\theta} J(\theta)\): The gradient of the loss function with respect to the parameters. It points in the direction of the steepest ascent.
• \(-\nabla_{\theta} J(\theta)\): Points in the direction of the steepest descent.
• \(\eta\): The learning rate, which determines the size of each step.

The Biggest Challenge: How to Compute the Gradient?

For a deep network, the loss function is an extremely complex composite function of thousands or millions of parameters.

\[ \large{L = f_L(f_{L-1}(\dots f_1(\mathbf{x}; \mathbf{W}^{(1)}, \mathbf{b}^{(1)}); \dots); \mathbf{W}^{(L)}, \mathbf{b}^{(L)})} \]

Taking the derivative of this directly is nearly impossible. We need an efficient algorithm to compute this gradient.

In Practice: Alerting Next-Month HS300 Downside with Local Data

Using local HS300 daily observations, we construct a monthly task: at month-end (t), use only information then available to predict whether month (t+1) has a negative return.

• File/key: data/index/hs300_index_only.h5 / hs300
• Raw fields: datetime, close, volume, total_turnover; prices are index points and turnover follows the local dictionary
• Sample: 2005-01-01 to 2024-12-31; the target only denotes next-month HS300 return direction
• Evaluation: first 70% train, next 15% validation, final 15% test, strictly chronological

Step 1: Acquiring and Preparing the Data

from pathlib import Path  # Locating Local Index Files Using Path Objects
import numpy as np  # Calculate Logarithmic Change vs. Finite Value
import pandas as pd  # Read daily rows and aggregate them by month
# Public download: https://assets.qiufei.site/data/index/hs300_index_only.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/index/hs300_index_only.h5")
# Windows: Path(r"C:\qiufei\data\index\hs300_index_only.h5")
# macOS: Path("/Users/your_name/data/index/hs300_index_only.h5")
# Linux: Path("/home/your_name/data/index/hs300_index_only.h5")
index_path = Path("/home/ubuntu/r2_data_mount/data/index/hs300_index_only.h5")
daily_index = pd.read_hdf(index_path, key='hs300')  # Reading real daily quotes from fixed HDF key
daily_index['date'] = pd.to_datetime(daily_index['datetime'].astype(str), format='%Y%m%d%H%M%S')  # Parse local timestamp
daily_index = daily_index.query("'2005-01-01' <= date <= '2024-12-31'").set_index('date').sort_index()  # Fixed Sample Period and Sequence
monthly_index = daily_index.resample('ME').agg(close=('close', 'last'), turnover=('total_turnover', 'sum'))  # Summarize end-of-month point and monthly turnover
monthly_index['monthly_return'] = monthly_index['close'].pct_change()  # Calculate the current monthly return
monthly_index['momentum_3m'] = monthly_index['close'].pct_change(3)  # Compute three-month momentum at month-end
monthly_index['realized_volatility'] = daily_index['close'].pct_change().rolling(20).std().resample('ME').last() * np.sqrt(20)  # Calculate 20th month Volatility
monthly_index['turnover_growth'] = monthly_index['turnover'].pct_change()  # Calculate current-month turnover growth
monthly_index['next_month_return'] = monthly_index['monthly_return'].shift(-1)  # Aligning next-month realized return
monthly_index['target_date_t1'] = monthly_index.index.to_series().shift(-1)  # Preserve the next-month label-realization date
feature_columns = ['monthly_return', 'momentum_3m', 'realized_volatility', 'turnover_growth']  # Fix four features available at month-end
analysis_frame = monthly_index.dropna(subset=feature_columns + ['next_month_return', 'target_date_t1']).copy()  # Drop the final month whose next-month return is unknown
analysis_frame['next_month_down'] = analysis_frame['next_month_return'].lt(0).astype(int)  # Create binary labels only for observed future returns
label_realization_date = analysis_frame.index + pd.offsets.MonthEnd(1)  # Record label realization at the following month-end
assert (analysis_frame['target_date_t1'].dt.to_period('M') == analysis_frame.index.to_period('M') + 1).all()  # Verify exact next-calendar-month realization
display(analysis_frame.head())  # Display real input fields and label construction

Step 2: Define Features and Target, and Split the Dataset

input_feature_matrix = analysis_frame[feature_columns]  # Construct Feature Matrix Sorted by Date
target_values = analysis_frame['next_month_down']  # Extract the next-month decline target
train_end = int(len(analysis_frame) * .70)  # Use the earliest 70% month as the training period
validation_end = int(len(analysis_frame) * .85)  # Use the next 15% month as the validation period
validation_start_date = analysis_frame.index[train_end]
test_start_date = analysis_frame.index[validation_end]
train_mask = (analysis_frame.index < validation_start_date) & (analysis_frame['target_date_t1'] < validation_start_date)  # Purge training labels realized in validation
validation_mask = (analysis_frame.index >= validation_start_date) & (analysis_frame.index < test_start_date) & (analysis_frame['target_date_t1'] < test_start_date)  # Purge validation labels realized in test
test_mask = analysis_frame.index >= test_start_date
training_features, y_train = input_feature_matrix.loc[train_mask], target_values.loc[train_mask]  # Keep the purged training window
validation_features, y_validation = input_feature_matrix.loc[validation_mask], target_values.loc[validation_mask]  # Keep the purged validation window
testing_features, y_test = input_feature_matrix.loc[test_mask], target_values.loc[test_mask]  # Keep the held-out test window
assert analysis_frame.loc[train_mask, 'target_date_t1'].max() < validation_features.index.min() and analysis_frame.loc[validation_mask, 'target_date_t1'].max() < testing_features.index.min()  # Verify label-realization boundaries
split_summary = pd.DataFrame({'start': [training_features.index.min(), validation_features.index.min(), testing_features.index.min()], 'end': [training_features.index.max(), validation_features.index.max(), testing_features.index.max()], 'size': [len(training_features), len(validation_features), len(testing_features)], 'down_rate': [y_train.mean(), y_validation.mean(), pd.NA]}, index=['train', 'validation', 'test'])  # Report boundaries while keeping test prevalence held-out until the threshold is fixed
display(split_summary)  # Verify that the time boundary is strictly incremented

Step 4: Building and Training the MLP Model

from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(
    hidden_layer_sizes=(50, 50),
    activation='relu',
    solver='adam',
    max_iter=500,
    random_state=42
)
print('Starting model training...')  # 展示当前步骤的结果。
mlp.fit(scaled_training_features, y_train)
print(f'Iterations={mlp.n_iter_}; converged before cap={mlp.n_iter_ < mlp.max_iter}')  # Clarify whether this optimization is converging

Starting model training...
Iterations=500; converged before cap=False

Step 6: Open One Test check with the fixed Threshold

from sklearn.metrics import average_precision_score, classification_report, roc_auc_score  # Evaluating Categories and Sorting Performance at the Same Time
y_prob = mlp.predict_proba(scaled_testing_features)[:, 1]  # Generate held-out test decline probabilities for the first time
y_pred = (y_prob >= selected_cost_threshold).astype(int)  # Apply the threshold fixed before test access
test_cost_confusion = confusion_matrix(y_test, y_pred)  # Save the cost confusion matrix in the same check
print('Classification Report (Test Set):')  # Mark the classification indicators from the test window that never participated in parameter tuning
print(classification_report(y_test, y_pred, target_names=['Up/Flat', 'Down']))  # 展示当前步骤的结果。
print(f'Validation ROC-AUC: {roc_auc_score(y_validation, y_validation_prob):.3f}')  # Report Validation Window ranking performance
print(f'Test ROC-AUC: {roc_auc_score(y_test, y_prob):.3f}')
print(f'Test AP (average precision): {average_precision_score(y_test, y_prob):.3f}')  # Report Metrics More Sensitive to Downstream Categories
print(f'fixed-threshold test cost: {4 * test_cost_confusion[1, 0] + test_cost_confusion[0, 1]}')  # Report the complete cost within the same check

Classification Report (Test Set):
              precision    recall  f1-score   support

     Up/Flat       0.36      0.71      0.48        14
        Down       0.50      0.18      0.27        22

    accuracy                           0.39        36
   macro avg       0.43      0.45      0.37        36
weighted avg       0.44      0.39      0.35        36

Validation ROC-AUC: 0.343
Test ROC-AUC: 0.412
Test AP (average precision): 0.566
fixed-threshold test cost: 76

Optional Extension: Convolutional Neural Networks (CNNs)

The MLPs we’ve discussed are fully-connected, meaning every neuron in a layer is connected to every neuron in the previous layer.

When processing data with spatial or temporal structure, like images or time series, the number of parameters in a fully-connected network explodes, and it fails to leverage the local structure of the data.

A Convolutional Neural Network (CNN) is a special type of feedforward network that addresses these issues through local connectivity and weight sharing.

The Core Idea of CNNs: Analyzing Data Like a Visual System

CNNs are inspired by the biological visual cortex.

1. Receptive Field: Each neuron focuses only on a small region of the input (local connectivity).
2. Feature Map: A ‘filter’ or ‘kernel’ slides across the entire input, searching for a specific pattern (like an edge or corner) and generating a feature map (weight sharing).

This is like how we look at a photo: we don’t process every pixel at once, but rather identify local lines and shapes first, then combine them into more complex objects.

The Key Layer of a CNN: The Convolutional Layer

At each position, a kernel computes an element-wise product sum (plus bias) to extract a local feature.

The Key Layer of a CNN: The Pooling Layer

A pooling layer (or downsampling layer) typically follows a convolutional layer.

Purpose:

1. Downsampling: Reduces feature-map size. Pooling itself has no learned parameters and can reduce computation and parameters in downstream layers.
2. Local translation tolerance: Makes responses less sensitive to small shifts within a pooling window. This is not rotation invariance and does not guarantee global translation invariance.

Common Methods:

• Max Pooling: Takes the maximum value from a region.
• Average Pooling: Calculates the average value of a region.

Visualizing Max Pooling

The diagram below shows a 2x2 max pooling operation on a 4x4 feature map.

CNN Applications in Economics?

Although CNNs were born from image recognition, their core idea of recognizing local patterns can be applied to economics:

• Time Series Analysis: A financial time series (e.g., stock prices) can be treated as a 1D ‘image’. CNNs can be used to identify technical analysis patterns like ‘head and shoulders’ or ‘double bottoms’.
• Textual Analysis: A matrix of word vectors from a sentence can be treated as a 2D image. CNNs can extract local semantic features for analyzing the sentiment or topics of financial reports and news articles.
• Satellite Imagery Analysis: Using satellite data like nighttime lights or ships in ports to predict regional economic activity.

A Brief History of Neural Networks: A Tour of Famous Models

Since 2012, the field of deep learning has seen a surge of landmark CNN architectures. Understanding them helps us appreciate how networks have become progressively deeper and more powerful.

• LeNet-5 (1998): The ancestor of modern CNNs.
• AlexNet (2012): Popularized the combination of ReLU, dropout, and GPU training at ImageNet scale.
• VGGNet (2014): Demonstrated the importance of network depth.
• GoogLeNet (2014): Introduced the ‘Inception module’, improving network width and efficiency.
• ResNet (2015): Introduced ‘residual connections’, solving the training problem for extremely deep networks.

LeNet-5 (1998): The Founder of a Classic Architecture

Proposed by Yann LeCun for recognizing handwritten digits on checks. Its classic architecture [CONV -> POOL -> CONV -> POOL -> FC -> OUTPUT] is still influential today.

AlexNet (2012): The ‘Big Bang’ of Deep Learning

AlexNet won the 2012 ImageNet competition by a massive margin, heralding the dawn of the deep learning era.

Key Contributions:

• Successful training of a comparatively deep CNN at ImageNet scale.
• Widespread use of ReLU, which accelerated training.
• Dropout and data augmentation to reduce overfitting.
• Multi-GPU computation that made large-scale training a reproducible major advance.

Sources and Further Reading

• Goodfellow, I., Bengio, Y., and Courville, A. (2016), Deep Learning, MIT Press.
• Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986), “Learning Representations by Back-propagating Errors,” Nature.
• scikit-learn User Guide:
  • MLPClassifier for fitting.
  • TimeSeriesSplit for chronological evaluation.
  • Probability calibration for reliability diagnostics.
• Data: local hs300_index_only.h5 / hs300; all evaluation and confusion-matrix results come from the same completed analysis.

Extension Summary: CNNs and Model History

CNNs use local connectivity, shared weights, and pooling for spatial structure; model history shows how these components evolved into deeper architectures.

Conclusion: A New Paradigm for Economic Modeling

• Capturing Non-linearity: The core strength of neural networks is their powerful ability to fit non-linear relationships, helping us understand complex economic phenomena that linear models cannot explain.
• Data-Driven: They are highly data-driven models capable of automatically learning features and patterns from large-scale datasets.
• A Powerful Toolkit: Core covers MLP training and evaluation; CNNs extend the same ideas to structured inputs in the optional section.

Future Outlook and Caveats

• Explainability (XAI):

  • Neural networks are often called ‘black box’ models because their decision-making processes are not transparent.

  • This is a major hurdle for their application in high-stakes areas like policy advice and credit scoring, and it is a hot research topic.

• Causal Inference:

  • Neural networks excel at prediction (finding correlations) but cannot be directly used for causal inference.

  • Combining neural networks with causal inference frameworks (like Diff-in-Diff or Instrumental Variables) is a frontier research area.

• More Models: We only introduced feedforward networks today. For time series data, Recurrent Neural Networks (RNNs) and their variants (like LSTM, GRU) are a more natural choice.

Thank You!

Q & A