07: Ensemble Learning

The Wisdom of the Crowd in Action

Ensemble Learning Concept Three simple 'weak learners' (a circle, a square, and a triangle) are combined via an aggregation process to form a single, complex 'strong learner'. Ensemble Learning Many simple models create one strong model Learner A Learner B Learner C + + Strong Learner

In Finance and Economics, Prediction is Everywhere

We rely on various models to make critical decisions:

Prediction Applications in Finance and Economics Four icons representing a bank, hedge fund, government, and insurance company, each paired with its typical prediction task. Key Decisions Rely on Prediction Banks Credit-card default risk Hedge Funds Next-day stock direction Government Next-quarter GDP growth Insurers Claim-filing risk

The Core Question: A Single Model vs. The Crowd

Any single model can have flaws, biases, or errors.

The question this chapter explores is:

If we combine many ‘pretty good’ models, can we get a ‘very powerful’ super-model?

Spoiler alert: the answer is yes. This method is called Ensemble Learning.

Today’s Learning Objectives

By the end of this lecture, you can:

  1. Core: Diagnose a single decision tree’s overfitting from training and validation errors.
  2. Core: Choose between Bagging and Boosting using bias, variance, learner correlation, and evidence.
  3. Extension: Compute one entropy, information-gain, Gini, and AdaBoost-weight update by hand; the 90-minute Core only requires explaining how these quantities drive splitting and reweighting.
  4. Core: Explain why this lesson chooses in advance random forest and treats tree/AdaBoost test results only as descriptive mechanism contrasts.
  5. Core: Interpret feature importance produced by the fitted model while explaining why importance is not causality.
  6. Extension: Explain out-of-fold meta-features in Stacking and compare its mechanism with the two core families.

Metric Convention: AP Is Not Trapezoidal PR-AUC

Every average_precision_score output is named AP (average precision). AP weights precision by recall increments and is not trapezoidal area under the empirical PR curve; tables, exercises, and reference outputs use AP consistently.

Before You Begin: Start with the Single-Tree Failure

  1. If a tree keeps splitting until each leaf holds one observation, what usually happens to training and test error?
  2. Does bootstrap sampling draw with or without replacement?
  3. What can accuracy conceal when the positive class is rare?

Write all three judgments before revealing the answer.

Reveal and remediation

  • Training error approaches zero while test error may rise; bootstrap sampling is with replacement; accuracy can conceal failure on the minority class, so inspect AP (average precision), recall, and the confusion matrix.

  • If item 1 was wrong, return to tree overfitting; for items 2 or 3, return to the Bagging workflow or majority baseline.

Why This Learning Order?

  • Diagnose: identify the single tree’s high variance first.

  • Repair: motivate Bagging, then study Boosting for bias reduction.

  • Extend: use Stacking for heterogeneous-model fusion.

Our Learning Roadmap

Chapter Learning Roadmap A four-stage horizontal roadmap: diagnose single-tree failure, explain why ensemble mechanisms work, repair with ensembles, then use real-data evidence for a decision. Our Path to Ensemble Mastery 1 Tree Failure Diagnose first 2 Why & How Mechanisms 3 Ensemble Repair Models 4 Evidence Decision

90-Minute Main lesson: Failure Before Repair

  • 0–12 min | Single-tree failure: learn how a tree splits, why it has high variance, and predict minority-class recall at the default threshold.
  • 12–42 min | Core mechanisms: Bagging reduces variance; Boosting reweights errors. Stacking is extension material.
  • 42–70 min | Real data: compare one tree, random forest, and AdaBoost on the same A-share chronological split.
  • 70–85 min | Decision: majority baseline, AP (average precision), confusion matrix, and a validation-chosen threshold.
  • 85–90 min | Lesson review: explain why better ranking does not imply a useful default threshold.

Study sequence

Establish the Failure First: Training Fit Is Not Selection Evidence

  • Fit on training data, diagnose on validation, and keep test held-out.

  • If a tree’s training AUC is near 1 while validation AUC drops sharply, the evidence indicates high-variance risk, not readiness for practical use.

  • Before seeing validation or test results, this lesson chooses in advance random forest because its mechanism targets high variance while retaining feature-importance diagnostics.

  • Validation selects only its operating threshold. held-out test tree and AdaBoost results are descriptive mechanism contrasts and cannot revise that choice.

Predict first: if random-forest validation ROC-AUC is close to the tree’s, will recall automatically become useful at 0.5?

Check 1 reveal

No. AUC measures ranking; 0.5 is a separate operating point. If you answered yes, return to “ranking is not an operating point” and trace how moving a threshold changes FP and FN.

learning path Preview: Two Repairs After a Single Tree Fails

Failure diagnosis Core repair Validation expectation
Strong training fit, weak validation, instability under perturbation Bagging: resample in parallel and average trees Ranking should stabilize; recall at 0.5 is not guaranteed
Constrained trees still miss difficult observations systematically Boosting: sequentially increase weight on errors Difficult-case ranking may improve; choose the operating point on validation

Continue Core: make a mechanism prediction here, then move forward to why a single tree fails. The Hoeffding argument, full AdaBoost derivation, and Stacking below remain skippable Extensions.

Extension A: The Theoretical Foundation

Main-lesson note: the following Hoeffding argument is extension material; continue the main path at why a single tree fails.

Why can we trust the ‘wisdom of the crowd’?

Hoeffding’s Inequality Provides the Proof

Hoeffding’s Inequality gives us a probabilistic guarantee:

The average of many independent random variables will converge to its true expected value with extremely high probability.

In other words, if you have enough samples, the sample average is a very good estimate of the true average.

This sounds abstract, so let’s use a classic coin toss experiment to build intuition.

An Intuitive Analogy: The Coin Toss Experiment

  • Problem: We have a potentially biased coin. The true probability of heads is p (unknown). How can we estimate p?
  • Method: Toss it n times and calculate the frequency of heads, h (the sample mean).
  • Intuition: The larger n is, the closer h should be to the true p.

Hoeffding’s Inequality precisely quantifies the degree of this ‘closeness’.

Hoeffding’s Inequality in Mathematical Terms

It states that the probability of the sample mean h being far from the true mean p by more than any small amount ε decreases exponentially as the sample size n increases.

\[ \large{P(|h - p| > \epsilon) \le 2e^{-2n\epsilon^2}} \]

This negative exponential term is the key. It means that for every additional observation (coin toss), the probability of making a large error shrinks dramatically.

The Leap from Coin Tosses to Ensemble Learning

Analogy: Coin Tossing and Ensemble Learning A diagram illustrating the analogy between a coin toss experiment on the left and ensemble learning on the right, with arrows connecting corresponding concepts. Coin Toss Experiment A single coin One result: H or T Toss the coin T times Frequency of heads Ensemble Learning One weak learner One result: right or wrong Train T independent learners Vote for the final prediction

Ensemble Error Falls Exponentially

Based on this analogy, a corollary of Hoeffding’s Inequality tells us:

  • Base-learner conditions: there are \(T\) independent binary classifiers, each with error rate \(\epsilon<0.5\).

  • Ensemble rule: combine them by simple majority voting to form \(H(x)\).

  • Theoretical result: under these strong conditions, the upper bound on ensemble error decreases exponentially with \(T\):

\[ \large{P(H(x) \ne y) \le \exp(-2T(0.5 - \epsilon)^2)} \]

Two Conditions for Ensemble Success

Ensemble learning works because it relies on two key assumptions:

  1. Independence: Each ‘weak learner’ needs to be different. If they all make the same mistakes, the ensemble provides no benefit.
  2. Better than Random: Each ‘weak learner’ must have an accuracy slightly better than guessing (for binary classification, > 50%).

As long as these two conditions are met, we can almost always build a powerful learner by creating an ensemble.

Extension B: Three Ensemble Mechanisms

Main-lesson note: the full mechanism diagrams and Stacking are extensions; the core summary follows the single-tree and pruning sequence.

In practice, how do we create a group of ‘weak learners’ that satisfy those two conditions?

The Three Schools of Ensemble Learning

Comparison of Ensemble Methods A three-card comparison of Bagging, Boosting, and Stacking, detailing their learner relationship, the main problem they solve, and a real-world analogy. Ensemble Methods: A Comparison Bagging Parallel learners Reduces variance Expert vote Boosting Sequential learners Targets errors Corrects mistakes Stacking Layered learners Combines predictions Meta-learner

Mechanism 1: Bagging (Bootstrap Aggregating)

The workflow for Bagging is very intuitive: Bootstrap + Aggregating.

  1. Bootstrap: From the original training set D, create T new training sets D_1, D_2, ..., D_T of the same size by sampling with replacement.
  2. Train: On each new training set D_t, independently and in parallel, train a base learner h_t.
  3. Aggregate:
    • Classification: Simple voting.
    • Regression: Simple averaging.

The Bagging Workflow

Bagging (Bootstrap Aggregating) Workflow A three-step flowchart showing Bootstrap sampling from original data, parallel training of base learners, and aggregation into a final model. Bagging (Bootstrap Aggregating) Workflow Original Data D 1. Bootstrap Sampling Training Set D₁ Training Set D₂ Training Set Dₙ 2. Parallel Training Learner h₁ Learner h₂ Learner hₙ 3. Aggregation Final Model H

Bagging’s Magic: Reducing Variance

  • Variance: The degree to which a model’s predictions fluctuate on different training sets. High variance means the model is too sensitive to the training data and is prone to overfitting.
  • Why Bagging Works:

    • Each base learner sees only a subset of the data, so their individual overfitting directions may differ.

    • By averaging or voting out these different errors, the overall volatility is smoothed out, thus reducing variance.

  • Most Successful Application: Random Forest.

Mechanism 2: Boosting

Boosting is a family of algorithms that ‘boosts’ weak learners into strong ones using a sequential, iterative approach.

  1. Initialize: Assign equal weights to all training samples.
  2. Iterative Training (t=1 to T):
    1. Train a weak learner h_t on the currently weighted sample set.
    2. Increase the weights of samples that h_t misclassified.
    3. Decrease the weights of samples that h_t classified correctly.
  3. Final Combination: The final strong learner is a weighted combination of all the weak learners.

Boosting Workflow

Core Workflow of the Boosting Algorithm A sequential flowchart of the Boosting algorithm, showing iterative training of weak learners (h1, h2, ... hn) on re-weighted data, followed by a weighted combination into a final strong learner H(x). Core Workflow of Boosting Initial Data equal weights Weak learnerh₁ Weak learnerh₂ … Learnerhₙ Errors from h₁ set weights for h₂ Final strong learner H(x) H(x) = sign(Σ αᵢhᵢ(x)) Weighted combination

Boosting’s Core: Reducing Bias

  • Bias: The systematic gap between a model’s predictions and the true values. High bias means the model is underfitting and hasn’t learned the data’s fundamental patterns.
  • Why Boosting Works: Each new learner is forced to focus on the ‘difficult’ samples that previous learners got wrong. This process continuously corrects the model’s systematic errors, gradually reducing bias.
  • Famous Algorithms: AdaBoost, Gradient Boosting Machines (GBM), XGBoost.

Mechanism 3: Stacking

Stacking is a more sophisticated combination strategy that tries to learn how to ‘intelligently’ combine the predictions of base learners, rather than simply voting or averaging.

  1. Layer 0: Train several different base learners. Use their predictions as new features.
  2. Layer 1: Train a ‘Meta-Learner’ whose input is the predictions from the Layer 0 models and whose output is the final prediction.

The Stacking Workflow

Stacking Ensemble Method Workflow A two-level flowchart of the Stacking method. Level 0 shows base learners (SVM, KNN, RF) making predictions on the original data. Level 1 shows a meta learner (Logistic Regression) training on those predictions to make the final output. Stacking Workflow Original Training Set Level 0: Base Learners Model A (SVM) Model B (KNN) Model C (RF) New Features from Predictions Level 1: Meta Learner Meta Model: Logistic Regression

Stacking’s Advantage: Model Fusion

  • Core Idea: Stacking doesn’t just combine predictions; it trains a meta-model to learn when to trust which base model more.
  • Example:
    • Models A and B agree while Model C differs sharply.
    • The meta-model can learn to lean toward A and B in that region.
  • Use Case: Very popular in data science competitions (like Kaggle) for squeezing out the last bit of performance by blending the strengths of multiple high-performing models.

Part 3: The Favorite Building Block—Trees

Why spend so much time on decision trees? Because they are by far the most common and successful base learners for ensemble methods.

Decision Trees: A Natural Fit for Ensembles

  • Pros:
    • Non-linear, capable of capturing complex relationships.
    • Interpretable (a single tree).
    • Relatively fast to train.
  • Cons:
    • Very prone to overfitting. A single decision tree’s performance is often unstable (high variance).

A Perfect Match

  • The high variance of decision trees is exactly what Bagging (like Random Forest) is designed to combat through averaging!

  • And the ‘weakness’ of a tree (by limiting its depth) makes it the perfect ‘raw material’ for Boosting.

How Does a Decision Tree Make Decisions?

A decision tree continuously splits a complex dataset into purer subsets by asking a series of ‘yes/no’ questions.

  • Root Node: Represents the entire dataset.
  • Internal Node: Represents a test on a feature (a question).
  • Branch: Represents the outcome of the test (the answer).
  • Leaf Node: Represents the final decision class or predicted value.

Anatomy of a Decision Tree

Decision Tree Structure A diagram showing the components of a decision tree: a root node ('Is the setting a classroom?'), an internal node ('Is a teacher present?'), branches ('Yes'/'No'), and leaf nodes ('Attend Class'/'Self-study'). Decision Tree Structure RootClassroom setting? Internal nodeTeacher present? LeafSelf-study Attend Self-study YesNo YesNo Branches carry the Yes/No outcomes

Choosing the Best Split

The key to growing a decision tree is to select the optimal feature to split the data at each step.

The ‘optimal’ standard is: after the split, the resulting subsets are the ‘purest’.

Higher ‘purity’ means less uncertainty and a clearer classification.

An Intuitive Look at ‘Purity’

Illustration of Dataset Purity Three jars illustrate data purity. The 'Low Purity' jar has a 50/50 mix of blue and orange balls. The 'Medium Purity' jar has mostly blue balls. The 'High Purity' jar has only blue balls. Goal: Increase Purity with Each Split Low Purity Medium Purity High Purity

How Do We Quantify ‘Purity’?

We use two main metrics to measure ‘impurity’ or ‘disorder’:

  1. Information Entropy (used in ID3, C4.5 algorithms)
  2. Gini Impurity (used in the CART algorithm)

The goal is the same: choose a split that results in the minimum weighted impurity in the child nodes.

Purity Metric 1: Information Entropy

  • Information Entropy H(D) measures the uncertainty or disorder of a dataset D.
  • The higher the entropy, the more chaotic the dataset (more mixed classes).
  • The lower the entropy, the purer the dataset (most samples belong to one class).

For a dataset D with K classes, its information entropy is defined as:

\[ \large{H(D) = - \sum_{k=1}^{K} p_k \log_2(p_k)} \]

where p_k is the proportion of samples belonging to class k.

Numerical Properties of Entropy

Imagine a dataset with two classes: Positive (+) and Negative (-).

Scenario p_+ p_- Entropy H(D) Purity
Perfectly Pure 1.0 0.0 -1*log2(1) - 0 = 0 Highest
Mixed 0.8 0.2 -0.8*log2(0.8) - 0.2*log2(0.2) ≈ 0.72 Lower
Most Chaotic 0.5 0.5 -0.5*log2(0.5) - 0.5*log2(0.5) = 1 Lowest

When positive and negative cases are equally likely, uncertainty is at its maximum, and entropy is 1.

Splitting Criterion 1: Information Gain

The ID3 algorithm uses Information Gain as its splitting criterion.

  • Idea: Calculate how much the system’s uncertainty (entropy) decreases after splitting dataset D by attribute A. The larger the decrease, the better A is for classification.

  • Formula:

    \[ \large{\text{Gain}(D, A) = H(D) - H(D|A)} \]

    where H(D) is the entropy before the split, and H(D|A) is the weighted average of the entropy of the subsets after the split (called conditional entropy).

  • Decision: Choose the attribute A that maximizes Gain(D, A) as the splitting node.

Worked Information Gain (1/2)

Use the same 15 observations as the Chinese deck: 9 “class” positives and 6 “self-study” negatives.

\[ \large{H(D)=-\frac{9}{15}\log_2\frac{9}{15}-\frac{6}{15}\log_2\frac{6}{15}\approx0.971} \]

Split by setting: classroom has (5+, 2−), dorm has (1+, 2−), and outdoors has (3+, 2−). Their entropies are 0.863, 0.918, and 0.971.

Worked Information Gain (2/2)

\[ \large{H(D\mid \text{setting})=\frac{7}{15}(0.863)+\frac{3}{15}(0.918)+\frac{5}{15}(0.971)\approx0.910} \]

\[ \large{\operatorname{Gain}(D,\text{setting})=0.971-0.910=0.061} \]

ID3 repeats this calculation for all candidate attributes and selects the largest gain. High-cardinality identifiers can overfit, which motivates gain ratio or stronger regularization.

Purity Metric 2: Gini Impurity

The CART (Classification and Regression Tree) algorithm uses the Gini Index to select the splitting attribute.

  • Gini Impurity Gini(D): The probability of misclassifying a randomly chosen element from dataset D if it were randomly labeled according to the distribution of labels in D.
  • The smaller the Gini index, the higher the purity of the dataset.

\[ \large{\text{Gini}(D) = \sum_{k=1}^{K} p_k (1 - p_k) = 1 - \sum_{k=1}^{K} p_k^2} \]

Splitting Criterion 2: Gini Index Gain

  • Idea: Choose an attribute A and a split point that results in the minimum weighted Gini index after the split.

  • Formula: For a split on attribute A into V subsets:

    \[ \large{\text{GiniIndex}(D|A) = \sum_{v=1}^{V} \frac{|D^v|}{|D|} \text{Gini}(D^v)} \]

  • Decision: Choose the attribute A and split that minimizes GiniIndex(D|A).

  • Advantage: Compared to entropy, Gini index calculation does not involve logarithms, making it computationally more efficient.

Worked Gini Split (1/2)

For the binary split “setting = classroom,” \(D_1\) has 7 observations (5+, 2−) and \(D_2\) has 8 observations (4+, 4−).

\[ \large{\operatorname{Gini}(D_1)=1-\left(\frac57\right)^2-\left(\frac27\right)^2\approx0.408} \]

\[ \large{\operatorname{Gini}(D_2)=1-\left(\frac48\right)^2-\left(\frac48\right)^2=0.5} \]

Worked Gini Split (2/2)

\[ \large{\operatorname{GiniIndex}(D\mid\text{classroom})=\frac7{15}(0.408)+\frac8{15}(0.5)=0.457} \]

CART evaluates candidate binary split points and selects the one with the smallest weighted Gini impurity. This is the same numerical example and conclusion as in the Chinese deck.

Decision Trees’ Realistic Problem: Overfitting

  • If a tree is allowed to grow without limits, it will continue to split until each leaf node contains only one sample.

  • The training error will be zero, but the model will be extremely complex and generalize poorly to new data.

A Realistic Problem with Decision Trees: Overfitting A structured diagram supporting the concept explained on this slide. The Overfitting Process in a Decision Tree Underfitting X1 AB High bias ·simple Good Fit X1 X2X3 AB CD Balanced fit Overfitting X1 High variance ·noise Model Complexity Increases → Overfitting: strong training fit, weak generalization.

The Solution: Pruning

To prevent overfitting, we need to ‘prune’ the decision tree.

  • Pre-pruning: During the tree’s growth, if a split does not improve generalization performance (e.g., performance on a validation set decreases), stop splitting early.
    • Pros: Faster, produces smaller trees.
    • Cons: Can be ‘short-sighted’, missing good split combinations.
  • Post-pruning: First, grow a full decision tree. Then, from the bottom up, examine nodes. If removing a subtree improves generalization performance, prune it.
    • Pros: Usually more effective, less likely to miss good structures.
    • Cons: Higher computational cost.

Mechanism Summary: From Failure to Evidence

Failure diagnosis Core repair Testable expectation
High variance under sample perturbations Bagging: resample in parallel and average trees Ranking should stabilize; recall at 0.5 is not guaranteed
Systematic misses on difficult observations Boosting: sequentially increase weight on errors Difficult-case ranking may improve; validate the threshold
  • Transition prediction： If random-forest AUC rises slightly, is that enough to call the downside alert useful? Answer before entering the same-split comparison.

  • Reveal： No. Also inspect AP (average precision), positive recall, the confusion matrix, and the validation-fixed threshold.

  • Local learning path choice： Core goes directly to the shared A-share check. Learners who enter the full random-forest construction and AdaBoost derivation below return forward to that check.

Extension C: Full Random-Forest Construction and AdaBoost Derivation

Main lesson: for the 90-minute learning path, go now to the shared A-share check. This mechanism branch also returns only forward to that check.

Part 4: Powerful Ensemble Models

Now, let’s combine our ‘building block’ (Decision Tree) with our ‘construction methods’ (Bagging/Boosting).

Bagging + Decision Trees = Random Forest

Random Forest is a highly successful extension of Bagging. It adds an extra layer of randomness on top of Bagging: feature randomness.

Construction Process:

  1. Perform T rounds of Bootstrap sampling to get T training subsets.
  2. For each subset, train a decision tree. When splitting each node of this tree:
    • Do not select the best feature from all d features.
    • Instead, randomly select l features (l < d), and then choose the best one from that smaller set.
  3. Combine the T trees through voting or averaging.

Random Forest’s Two Sources of Randomness

The Dual Randomness of Random Forest Shows the two sources of randomness in Random Forest: row sampling (bootstrap) and column sampling (random feature selection). Random Forest's Dual Randomness Original Dataset F1F2...Fk... 1. Row Sampling(Bootstrap) Subset for one tree F1F2...Fk... 2. Column Sampling (Features)

Why is Random Forest More Powerful? ‘Diversity’

  • Role of Feature Randomness: It de-correlates the trees in the forest.

  • Why De-correlation Matters:

    • Without feature randomness, every tree in the forest would likely choose the same strongest feature to split on at the root node.

    • This would lead to very similar tree structures, diminishing the benefit of ensembling.

  • By forcing each tree to consider only a subset of features at each split, Random Forest ensures that each tree learns from a different ‘perspective’.

  • This makes them ‘specialized’ in different ways.

  • When combined, they form a more powerful and complementary team, further reducing the overall variance.

Boosting + Decision Trees = AdaBoost

AdaBoost (Adaptive Boosting) is the classic algorithm of the Boosting family.

The Core Iterative Loop:

  1. Train a weak learner h_t.
  2. Evaluate h_t’s performance and assign it a weight α_t (better models get higher weights).
  3. Update the training sample weights w based on h_t’s predictions (misclassified samples get higher weights).
  4. Repeat.

The final model is a weighted combination of all weak learners.

AdaBoost Algorithm Explained (1/4): Initialization

Goal: Train a strong classifier \(\large{H(x) = sign(\sum \alpha_t h_t(x))}\)

1. Initialize: The weights for all N training samples are initialized equally:

\[ \large{w_{1,n} = 1/N} \quad \text{for } n=1, \dots, N \]

AdaBoost Algorithm Explained (2/4): Training & Evaluation

For \(t=1,\ldots,T\):

  1. Train: Fit h_t(x) to minimize error under the current sample weights.

  2. Weighted error: Sum the weights of samples misclassified by h_t.

    \[ \large{\epsilon_t = \sum_{n=1}^{N} w_{t,n} I(h_t(x_n) \ne y_n)} \]

  3. Learner weight:

    \[ \large{\alpha_t = \frac{1}{2} \ln\left(\frac{1 - \epsilon_t}{\epsilon_t}\right)} \]

    A lower ε_t gives h_t a larger vote α_t.

AdaBoost Algorithm Explained (3/4): Updating Weights

  1. Update Sample Weights w: This is the core adaptive step.

    \[ \large{w_{t+1, n} = \frac{w_{t,n} \exp(-\alpha_t y_n h_t(x_n))}{Z_t}} \]

    (Z_t is a normalization factor to ensure the new weights sum to 1)

    Intuitive Explanation:

    • If sample n is classified correctly (\(y_n h_t(x_n) = 1\)), the exponent is negative, and w decreases.
    • If sample n is misclassified (\(y_n h_t(x_n) = -1\)), the exponent is positive, and w increases.

AdaBoost Algorithm Explained (4/4): Final Combination

3. Final Output: Combine all T weak learners via a weighted vote using their respective weights α_t to form the final strong classifier:

\[ \large{H(x) = \text{sign}\left(\sum_{t=1}^{T} \alpha_t h_t(x)\right)} \]

Part 5: Practice—Next-Quarter Downside Risk

After the optional topic / Core continue: both routes now join the same validation decision and one held-out test check.

Task definition

  • Use valuation features observed in quarter (t) to predict whether total market capitalization, represented by local field market_cap, falls by at least 20% in quarter (t+1).

  • The RQData financial-field documentation distinguishes two measures:

    • market_cap: total shares × unadjusted A-share close.
    • a_share_market_val_in_circulation: circulating A-share value.

  • the local file preserves both fields.

  • This is a teaching market-downside proxy, not credit default, financial distress, or a regulatory designation.

Practice Data and Evaluation Plan

  • Local file: local data/stock/valuation_factors_quarterly_15_years.h5
  • HDF key: valuation_factors; index: order_book_id, date
  • Sample: 2015-01-01 to 2024-12-31; units follow the local data dictionary and market capitalization is log-transformed
  • Split: train through 2021, validate on 2022, and test on 2023–2024 without random shuffling
  • Models: one tree, random forest, and AdaBoost share exactly the same observations and features

Step 1: Load the Data and Construct the Target

The target is formed from the next quarterly observation; every predictor comes only from the current quarter, separating the available information set from the future outcome.

Code 
from pathlib import Path  # Locating Local Data Using Cross-Platform Path Objects
import numpy as np  # Provides logarithmic transformation and finite-value inspection
import pandas as pd  # Read HDF5 and Construct Company — Quarterly Panel
# Public download: https://assets.qiufei.site/data/stock/valuation_factors_quarterly_15_years.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/stock/valuation_factors_quarterly_15_years.h5")
# Windows: Path(r"C:\qiufei\data\stock\valuation_factors_quarterly_15_years.h5")
# macOS: Path("/Users/your_name/data/stock/valuation_factors_quarterly_15_years.h5")
# Linux: Path("/home/your_name/data/stock/valuation_factors_quarterly_15_years.h5")
valuation_path = Path("/home/ubuntu/r2_data_mount/data/stock/valuation_factors_quarterly_15_years.h5")
valuation_raw = pd.read_hdf(valuation_path, key='valuation_factors')  # Read local observations from the fixed HDF key
valuation_panel = valuation_raw.reset_index()  # Restore Company Code and Date Index to Normal Fields
valuation_panel['date'] = pd.to_datetime(valuation_panel['date'])  # Unifying Quarter Date Types for Time-Splitting
valuation_panel = valuation_panel.query("'2015-01-01' <= date <= '2024-12-31'").copy()  # Fix the easy to check sample period
valuation_panel = valuation_panel.sort_values(['order_book_id', 'date'])  # Sort by Company and Time to Construct Next Quarter's Label
valuation_panel['target_date'] = valuation_panel.groupby('order_book_id')['date'].shift(-1)  # Preserve the date on which each label is actually realized
valuation_panel['next_market_cap'] = valuation_panel.groupby('order_book_id')['market_cap'].shift(-1)  # Obtain the firm's next available record first
current_quarter = valuation_panel['date'].dt.to_period('Q')  # Map feature dates to calendar quarters
target_quarter = valuation_panel['target_date'].dt.to_period('Q')  # Map label dates to calendar quarters
valuation_panel['is_exact_next_quarter'] = target_quarter == current_quarter + 1  # Keep exact adjacency and reject skipped quarters
valuation_panel['next_quarter_return'] = valuation_panel['next_market_cap'] / valuation_panel['market_cap'] - 1  # Calculate the market capitalization change rate for the next quarter
valuation_panel['downside_risk'] = (valuation_panel['next_quarter_return'] <= -0.20).astype(int)  # Define Proxy Labels with a decline of at least 20%
Code 
feature_columns = ['pe_ratio_ttm', 'pb_ratio_lf', 'dividend_yield_ttm', 'ev_to_ebitda_ttm', 'market_cap']  # Use only fields visible in the current quarter
model_panel = valuation_panel.loc[valuation_panel['is_exact_next_quarter']].dropna(subset=feature_columns + ['next_market_cap', 'target_date']).copy()  # Reject next-available records that are not next-quarter
model_panel['log_market_cap'] = np.log(model_panel['market_cap'].clip(lower=1))  # Reduce magnitude skew with a logarithmic market value
feature_columns = ['pe_ratio_ttm', 'pb_ratio_lf', 'dividend_yield_ttm', 'ev_to_ebitda_ttm', 'log_market_cap']  # Update Model Field List
model_panel[feature_columns] = model_panel[feature_columns].replace([np.inf, -np.inf], np.nan)  # Mark the infinite valuation multiple as missing
model_panel = model_panel.dropna(subset=feature_columns)  # Keep full observations of accessible models
input_feature_matrix = model_panel[feature_columns]  # Construct the feature matrix with actual business fields
target_values = model_panel['downside_risk']  # Extract the next-quarter large-decline target
assert (model_panel['target_date'].dt.to_period('Q') == model_panel['date'].dt.to_period('Q') + 1).all()  # Verify exact next-quarter realization

The chronological split is separated to emphasize that it is part of the empirical design, not an interchangeable utility setting.

Code 
train_mask = (model_panel['date'] <= '2021-12-31') & (model_panel['target_date'] < '2022-01-01')  # Require training labels to be realized before validation
validation_mask = model_panel['date'].between('2022-01-01', '2022-12-31') & (model_panel['target_date'] < '2023-01-01')  # Require validation labels to be realized before test
test_mask = model_panel['date'].between('2023-01-01', '2024-12-31')  # Keep the last two years for out-of-sample testing of pseudo samples
training_features, y_train = input_feature_matrix.loc[train_mask], target_values.loc[train_mask]  # Extract Training Sample without Scrambling Time
validation_features, y_validation = input_feature_matrix.loc[validation_mask], target_values.loc[validation_mask]  # Extract Validation Sample
testing_features, y_test = input_feature_matrix.loc[test_mask], target_values.loc[test_mask]  # Extract Final Test Sample
assert model_panel.loc[train_mask, 'target_date'].max() < model_panel.loc[validation_mask, 'date'].min()  # Prevent training-label overlap with validation
assert model_panel.loc[validation_mask, 'target_date'].max() < model_panel.loc[test_mask, 'date'].min()  # Prevent validation-label overlap with test
split_summary = pd.DataFrame({'n': [train_mask.sum(), validation_mask.sum(), test_mask.sum()], 'feature_end': [model_panel.loc[train_mask, 'date'].max(), model_panel.loc[validation_mask, 'date'].max(), model_panel.loc[test_mask, 'date'].max()], 'target_end': [model_panel.loc[train_mask, 'target_date'].max(), model_panel.loc[validation_mask, 'target_date'].max(), model_panel.loc[test_mask, 'target_date'].max()]}, index=['train', 'validation', 'test'])  # Report feature and label-realization boundaries together
display(split_summary)  # Make adjacency filtering and label-date purging easy to check
n feature_end target_end
train 50811 2021-09-30 2021-12-31
validation 8880 2022-06-30 2022-09-30
test 9695 2024-09-30 2024-12-31

Evaluation plan

Choose the random forest before viewing the test results, then use the 2022 validation set to maximize F2, which gives more weight to recall. The tree and AdaBoost test results are included only for comparison.

Step 2: Training the Baseline - A Single Decision Tree

First, we’ll train a single decision tree as our performance baseline. To prevent severe overfitting, we’ll limit its maximum depth to 5.

Code 
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, average_precision_score, roc_auc_score  # Simultaneous Evaluation of Sorting Ability and Minority Recognition
dt_clf = DecisionTreeClassifier(max_depth=5, random_state=42)
dt_clf.fit(training_features, y_train)
dt_clf.get_params()['max_depth']  # Verify only the fitted baseline specification while test labels remain held-out
5

Step 3: Training a Bagging Model - Random Forest

Now, let’s see how a ‘forest’ of 100 decision trees performs. n_estimators is the number of base learners T.

Code 
from sklearn.ensemble import RandomForestClassifier
rf_clf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42, n_jobs=-1)
rf_clf.fit(training_features, y_train)
from sklearn.metrics import precision_recall_curve
validation_probability = rf_clf.predict_proba(validation_features)[:, 1]  # Generate validation probabilities before any test access
validation_precision, validation_recall, candidate_thresholds = precision_recall_curve(y_validation, validation_probability)  # Enumerate validation PR operating points
validation_f2 = 5 * validation_precision[:-1] * validation_recall[:-1] / (4 * validation_precision[:-1] + validation_recall[:-1] + 1e-12)  # Weight recall more than precision
selected_threshold = candidate_thresholds[np.argmax(validation_f2)]
print(f'Validation-fixed forest threshold: {selected_threshold:.4f}')  # Record the only selection before test access
Validation-fixed forest threshold: 0.0504

Observation rule: Call it an improvement only if the executed output is higher; one chronological split is not statistical evidence of “significant superiority.”

Step 4: Training a Boosting Model - AdaBoost

Finally, let’s try AdaBoost. It also uses decision trees but employs a sequential strategy focused on correcting errors.

Code 
from sklearn.ensemble import AdaBoostClassifier
base_estimator = DecisionTreeClassifier(max_depth=1)
ada_clf = AdaBoostClassifier(
    estimator=base_estimator,
    n_estimators=100,
    random_state=42
)
ada_clf.fit(training_features, y_train)
ada_clf.get_params()['n_estimators']  # Verify only the descriptive contrast specification while test labels remain held-out
100

Observation rule: Compare the executed ROC-AUC and AP (average precision). If their rankings differ, explain the error trade-off rather than choose in advanceing a winner.

Step 5: Threshold fixed, Open One Test check

  • The previous step fixed the random-forest threshold using 2022 validation only.

  • Now, for the first time, generate any test prediction: the forest is the teaching check before practical use, while tree and AdaBoost results are post-hoc mechanism contrasts that cannot change the fixed specification.

Code 
from sklearn.dummy import DummyClassifier  # Establish minimum baseline for predicting majority only
from sklearn.metrics import confusion_matrix, recall_score  # Quantify event recognition in the one test check
majority_model = DummyClassifier(strategy='most_frequent')  # Fixed Majority Class Prediction Rule
majority_model.fit(training_features, y_train)  # Learn Most Classes Only from Training Period
majority_prediction = majority_model.predict(testing_features)  # Generate Baseline Prediction on Same fixed Test Period
majority_probability = majority_model.predict_proba(testing_features)[:, 1]  # Generate the baseline positive-class probability
y_pred_dt = dt_clf.predict(testing_features)
y_prob_dt = dt_clf.predict_proba(testing_features)[:, 1]
y_pred_rf = rf_clf.predict(testing_features)
y_prob_rf = rf_clf.predict_proba(testing_features)[:, 1]
y_pred_ada = ada_clf.predict(testing_features)
y_prob_ada = ada_clf.predict_proba(testing_features)[:, 1]
threshold_prediction = (y_prob_rf >= selected_threshold).astype(int)  # Apply the fixed threshold to the testing period
auc_dt = roc_auc_score(y_test, y_prob_dt)  # Save the tree test AUC for the later post-hoc chart
auc_rf = roc_auc_score(y_test, y_prob_rf)  # Save the forest test AUC for the later post-hoc chart
auc_ada = roc_auc_score(y_test, y_prob_ada)  # Save the AdaBoost test AUC for the later post-hoc chart
evaluation_rows = []  # Summarize directly comparable test evidence
for model_name, prediction, probability in [('Majority baseline', majority_prediction, majority_probability), ('[email protected]', y_pred_dt, y_prob_dt), ('[email protected]', y_pred_rf, y_prob_rf), ('[email protected]', y_pred_ada, y_prob_ada), ('Forest@validation threshold', threshold_prediction, y_prob_rf)]:  # Compare the same test sample
    true_negative, false_positive, false_negative, true_positive = confusion_matrix(y_test, prediction).ravel()  # Expand the confusion matrix four times
    evaluation_rows.append([model_name, roc_auc_score(y_test, probability), average_precision_score(y_test, probability), recall_score(y_test, prediction, zero_division=0), true_negative, false_positive, false_negative, true_positive])  # Save Sorting and Classification Metrics
evaluation_table = pd.DataFrame(evaluation_rows, columns=['model/threshold', 'ROC-AUC', 'AP (average precision)', 'recall', 'TN', 'FP', 'FN', 'TP'])  # Form a checkable results table
print(f'Validation-fixed threshold: {selected_threshold:.4f}')  # Report Threshold Sources instead of Recall Test Sets
display(evaluation_table.round(4))  # Demonstrate Majority Baseline, Confusion Matrix and Minority Metrics

Step 5 Evidence: Ranking Is Not an Operating Point

Model/threshold ROC-AUC AP (average precision) positive recall
Majority baseline 0.5000 0.0483 0.0000
Tree @ 0.5 0.7284 0.0972 0.0000
Forest @ 0.5 0.7258 0.1051 0.0000
AdaBoost @ 0.5 0.7304 0.1077 0.0000
Forest @ validation threshold 0.0504 0.7258 0.1051 0.8996
  • Executed confusion matrices:

    • majority and forest @ 0.5 both give TN=9,227, FP=0, FN=468, TP=0.

    • At the validation threshold: TN=3,005, FP=6,222, FN=47, TP=421; accuracy falls to 0.3534.

  • Decision meaning:

    • this example chooses the threshold that maximizes validation F2, which favors recall; the result shows that higher recall comes with many false alerts.

    • This is not monetary cost minimization and need not transfer to other data.

    • A cost-based rule would require an explicit FP/FN cost ratio and a new choice on validation data.

    • Use test only for final evaluation.

Step 6: Post-hoc Description, Not Model Selection

AUC summarizes ranking across thresholds; this test comparison is descriptive, not model selection.

Code 
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
sns.set_theme(style='whitegrid', context='talk')  # 展示当前步骤的结果。
fig, ax = plt.subplots(figsize=(10, 6), dpi=100)
results = pd.DataFrame({
    'Model': ['Single Decision Tree', 'Random Forest', 'AdaBoost'],
    'AUC': [auc_dt, auc_rf, auc_ada]
}).sort_values('AUC', ascending=True)
colors = ['#A4AAAB', '#00A4E6', '#A91F2B']
bars = ax.barh(results['Model'], results['AUC'], color=colors, height=0.6)
ax.set_xlim(0.45, 1.0)
ax.set_xlabel('ROC AUC Score', fontsize=20, labelpad=10)
ax.set_title('Model Comparison: Next-Quarter Downside Risk', fontsize=20, pad=20, weight='bold')
for bar in bars:  # 遍历当前教学对象以完成重复计算。
    width = bar.get_width()
    ax.text(width + 0.0005, bar.get_y() + bar.get_height()/2, f'{width:.4f}',
            ha='left', va='center', fontsize=20, weight='bold')
ax.spines[['top', 'right', 'bottom']].set_visible(False)
ax.xaxis.grid(True, linestyle='--', which='major', color='grey', alpha=0.5)
ax.yaxis.grid(False)
ax.tick_params(axis='y', labelsize=14, length=0)
ax.tick_params(axis='x', labelsize=12)
plt.tight_layout()  # 展示当前步骤的结果。
plt.show()  # 展示当前步骤的结果。
Horizontal bars compare the ROC-AUC of one decision tree, random forest, and AdaBoost during the 2023–2024 test period.
Figure 1: Test-period ROC-AUC for three models predicting next-quarter downside risk among Chinese listed firms

Result Analysis: Test Comparisons Cannot Choose a Winner

  • Figure 1 shows the ROC-AUC of three models on the same test period.

  • These final results are not used to choose the model: the practical example continues with the random forest and the threshold selected from validation data.

This comparison does not prove that ensembles always outperform a single tree, nor does one observed difference establish statistical significance. It shows how to make the comparison on the same data and time periods.

Another Major Advantage of Ensembles: Interpretability

Ensemble models, especially tree-based ones, have another huge benefit: they can tell us which input features are most important for making the final decision.

This is crucial in economics and finance. We don’t just want to predict; we want to understand the drivers behind the prediction.

Visualizing Feature Importance

Read the fields actually used by the fitted forest; do not insert variable names before observing the output.

Code 
import numpy as np
importances = rf_clf.feature_importances_
feature_names = input_feature_matrix.columns
df_importance = pd.DataFrame({'feature': feature_names, 'importance': importances})
df_importance = df_importance.sort_values('importance', ascending=False).head(15)
fig, ax = plt.subplots(figsize=(10, 7), dpi=100)
sns.barplot(
    x='importance',
    y='feature',
    data=df_importance,
    palette='viridis',
    ax=ax
)
ax.set_title('Feature Importance Analysis (from Random Forest)', fontsize=20, pad=20, weight='bold')
ax.set_xlabel('Relative Importance (Mean Decrease in Impurity)', fontsize=20, labelpad=10)
ax.set_ylabel('')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.tick_params(axis='both', which='major', labelsize=12)
plt.tight_layout()  # 展示当前步骤的结果。
plt.show()  # 展示当前步骤的结果。
Horizontal bars rank the five real valuation fields by mean decrease in impurity from the fitted random forest.
Figure 2: Random-forest MDI importance for next-quarter downside risk

Business Interpretation: From Actual Fields to Cautious Claims

  • Figure 2 can contain only pe_ratio_ttm, pb_ratio_lf, dividend_yield_ttm, ev_to_ebitda_ttm, and log_market_cap.

  • Read the executed top-k values, then use the local dictionary to define them.

MDI can favor continuous features with many split points and behave unstably among correlated predictors. It gives neither direction nor a causal effect. Any interpretation must state the sample period, proxy target, and these limitations.

Formative Check: Repair High Variance

  • Question： A single tree has training AUC 0.99 and validation AUC 0.66, while 100 trees make highly correlated errors. What should you try first?

Choose “more trees / constrain trees and randomize features / Boosting,” then justify it with bias–variance language.

  • Check 2 reveal and remediation:

    • Constrain tree complexity and use feature randomization to reduce inter-tree correlation, then reassess Bagging.

    • If you chose only “more trees,” revisit Bagging’s correlation condition; if you chose Boosting, revisit the high-training/low-validation AUC variance diagnosis.

Formative Check: When Boosting Comes First

  • Question： A constrained shallow tree persistently misses the same difficult cases in both training and validation; both scores are low and close, and Bagging repeats the same systematic misses. Try Bagging or Boosting first?
  • Reveal and remediation:

    • Try Boosting first so later weak learners up-weight persistently misclassified cases.

    • This is high bias/systematic miss, not high variance from strong training and weak validation.

    • If you still choose Bagging, return to the variance-repair versus bias-repair table.

Step-by-Step Exercise: Explain the Executed Top-k

  • Task： Run the importance code and submit the top two fields, their definitions, one testable non-causal interpretation, and one MDI limitation.

  • Complete answer:

    • pb_ratio_lf: latest price-to-book ratio, price divided by book value per share; dimensionless.

    • log_market_cap: natural log of market capitalization in the local file’s native currency unit.

    • A testable non-causal interpretation asks whether ranges of these fields are associated with different next-quarter large-decline rates in this sample.

    • MDI can favor continuous variables with many split points and split importance among correlated variables; it supplies neither direction nor a causal effect.

Five-Minute Lesson Review

  • Check your understanding：

    • Choose Bagging or Boosting from high variance or systematic misses.

    • Explain why test-period comparisons cannot select a model after the fact.

    • State that MDI describes predictive association, not causal direction.

  • Answer independently： Diagnose first, then interpret the same held-out test evidence:
  1. If training AUC is 0.99 and validation AUC is 0.66, what does that imply, and which repair should come first?
  2. On test, the forest has higher AP (0.1051 > 0.0972) but slightly lower ROC-AUC (0.7258 < 0.7284). What does that support—and not support?
  3. Which validation criterion selects this threshold? Why is F2 maximization not the same as minimizing monetary FP/FN cost, and why can the threshold not be reselected on test?
  • Reference answer and remediation:

    • Strong training and weak validation indicate high variance; first constrain the tree and reduce correlated errors with Bagging/feature randomization.

    • The ranking metrics disagree, so the evidence supports higher test AP here, not uniform superiority.

    • This example locks the validation-F2 maximum; F2 is a recall-weighted summary, not an unstated FP/FN cost function. check test once.

Core summary: Failure—Repair—Evidence—Decision

  1. A tree with near-perfect training performance but weaker validation performance raises a high-variance warning.
  2. Bagging reduces variance through sample/feature randomization; Boosting reduces bias through sequential error correction.
  3. This lesson chooses in advance random forest from its high-variance repair and interpretability mechanism; same-window tree/AdaBoost test metrics are descriptive contrasts only.
  4. Choose the forest operating point by validation-F2 maximization; evaluate on test once, and treat MDI as associational evidence only. A business-cost rule requires separately stated costs and a new choice on validation data.

Extension: Independent Rolling Stability check

  • Task： Use 2021, 2022, 2023, and 2024 in turn as the test year, training only on prior quarters. Submit yearly ROC-AUC, AP (average precision), class prevalence, and top-three fields.

  • Complete-answer reminder:

    • refit chronologically for every year;

    • report ROC-AUC, AP (average precision), prevalence and each model’s top three fields for all four years;

    • interpret MDI as predictive rather than causal;

    • mention its bias and shared-importance limitation;

    • and compare stability only as far as the evidence allows.

Code 
rolling_rows = []  # Save Four Years of Pseudo-Sample Outside Evidence
for test_year in [2021, 2022, 2023, 2024]:  # Scroll the test window forward year by year
    rolling_test_start = pd.Timestamp(f'{test_year}-01-01')
    rolling_train_mask = (model_panel['date'] < rolling_test_start) & (model_panel['target_date'] < rolling_test_start)  # Purge training labels realized in the test year
    rolling_test_mask = model_panel['date'].between(f'{test_year}-01-01', f'{test_year}-12-31')
    rolling_model = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42, n_jobs=-1)  # Refit the same specification in each time fold
    rolling_model.fit(input_feature_matrix.loc[rolling_train_mask], target_values.loc[rolling_train_mask])  # Exclude future years from training
    rolling_probability = rolling_model.predict_proba(input_feature_matrix.loc[rolling_test_mask])[:, 1]  # Generate Out-of-Sample Probability for Current Year
    rolling_target = target_values.loc[rolling_test_mask]  # Retrieve the realized labels for the same year
    assert model_panel.loc[rolling_train_mask, 'target_date'].max() < model_panel.loc[rolling_test_mask, 'date'].min()  # Verify each rolling fold's label boundary
    importance_order = np.argsort(rolling_model.feature_importances_)[::-1][:3]  # Obtain top three positions from that fitted model for the Year
    yearly_top_three = ', '.join(input_feature_matrix.columns[importance_order])  # Save the first three fields of the year as verifiable text
    rolling_rows.append([test_year, len(rolling_target), rolling_target.mean(), roc_auc_score(rolling_target, rolling_probability), average_precision_score(rolling_target, rolling_probability), yearly_top_three])  # Summarize annual metrics vs. actual top three fields
rolling_table = pd.DataFrame(rolling_rows, columns=['test year', 'n', 'prevalence', 'ROC-AUC', 'AP (average precision)', 'top-3 fields'])  # Generate Answer Table covering All Deliverables
display(rolling_table.round(4))  # Display the executed results
test year n prevalence ROC-AUC AP (average precision) top-3 fields
0 2021 16319 0.1243 0.7381 0.2456 pb_ratio_lf, log_market_cap, pe_ratio_ttm
1 2022 8880 0.1541 0.6413 0.2260 pb_ratio_lf, log_market_cap, pe_ratio_ttm
2 2023 4678 0.0782 0.7213 0.1635 pb_ratio_lf, log_market_cap, pe_ratio_ttm
3 2024 5017 0.0203 0.7148 0.0447 pb_ratio_lf, log_market_cap, pe_ratio_ttm

Rolling check: Executed Reference Output

  • Complete reference output:

    • 2021–2024 ROC-AUC values are 0.7381, 0.6413, 0.7213, and 0.7148;

    • AP (average precision) values are 0.2456, 0.2260, 0.1635, and 0.0447, against prevalences 0.1243, 0.1541, 0.0782, and 0.0203.

    • Every yearly model’s executed top three is pb_ratio_lf, log_market_cap, pe_ratio_ttm.

    • Weak 2024 AP combines rare events with limited ranking value.

  • Answer note：

    • No yearly refit or full-sample top three → return to chronological splitting and “refit each fold.”

    • Causal interpretation of importance → return to the MDI limitation.

Sources and Further Reading

  • Breiman, L. (2001), “Random Forests”, Machine Learning, 45, 5–32; an adjacent authoritative source for the random-forest mechanism.
  • Freund, Y. and Schapire, R. E. (1997), “A Decision-Theoretic Generalization of On-Line Learning,” Journal of Computer and System Sciences, 55, 119–139.
  • scikit-learn User Guide: Decision Trees, Ensembles, and permutation importance.
  • Data: local valuation_factors_quarterly_15_years.h5 / valuation_factors; figures are generated by the code on these slides.

Chapter Summary

  1. Extension theory: under independence and individual accuracy above chance, Hoeffding’s inequality can bound majority-vote error exponentially; Core does not assess this result.
  2. Two Main Paths:
    • Bagging (e.g., Random Forest): Parallel training, uses voting/averaging to reduce variance and make the model more stable.
    • Boosting (e.g., AdaBoost): Sequential training, iteratively focuses on errors, uses a weighted combination to reduce bias and make the model more accurate.
  3. Core Component: Decision trees are the ideal ‘building blocks’ for ensembles. Their inherent high variance or controllable ‘weakness’ makes them a perfect match for both Bagging and Boosting.
  4. Model choice in this lesson:

  • use random forest to reduce high variance while retaining importance diagnostics; choose its threshold on validation data; then use the test period once, with tree/AdaBoost metrics as additional descriptions.

  • In any project, model selection must finish before test access.

  • MDI importance is associational, not causal.