08 Cluster Analysis

Welcome to the World of Cluster Analysis

Imagine you lead Tencent Video, holding viewing data for hundreds of millions of users in the same Chinese-platform setting as the Chinese deck.

How would you recommend movies that users might like?
How would you produce original content that caters to local tastes in different markets?
How would you discover and serve emerging, niche viewing communities?

Measurable Objectives and Before You Begin

90-minute Core:
- Explain why scaling changes distances;
- compute one silhouette value;
- calculate inertia and silhouette for $K=2\ldots10$ on one dataset;
- profile clusters from centroids rather than numeric labels;
- and explain whether the result is stable.
Extension： Compare the assumptions, membership semantics, and suitable use cases of hierarchical clustering, DBSCAN, and GMM. The core completion claim does not include this objective.
Check： If market capitalization is measured in hundreds of millions while dividend yield is a decimal, which dominates raw Euclidean distance? Can K-Means choose K automatically?

Write both judgments before revealing the answer.

Reveal and remediation:
- The higher-scale market-capitalization field dominates, so scale first; K-Means requires an external K-selection rule.
- If item 1 was wrong, return to feature scaling; if item 2 was wrong, return to “choose in advance the K rule.”

90-Minute Main lesson

0–15 min: classification versus clustering, distance, and scaling.
15–35 min: centroids, metrics, and silhouette.
35–60 min: K-Means and a K-selection rule stated before viewing the results.
60–82 min: Yangtze River Delta real-data analysis and stability.
82–90 min: application exercise and feedback.

Study sequence

scaling → silhouette → real case → lesson review. Hierarchical clustering, DBSCAN, and GMM are optional topics.

This Chapter’s Learning Roadmap

We will embark on a comprehensive journey to master the core concepts and practices of cluster analysis.

The Core Task is to Discover Structure in Unstructured Data

A better strategy is to identify groups of users with similar tastes and then make targeted recommendations.

This process of ‘grouping like with like’ is the core of what we will study today: Clustering. It is the art and science of automatically grouping similar data objects into clusters without predefined categories.

Clustering is a Form of Unsupervised Learning

Supervised learning predicts supplied labels; clustering receives no labels and searches for structure in the feature representation.

The task of clustering is to partition similar data points into the same group (cluster).

Clustering vs. Classification: Labels Define the Task

Before diving into the world of clustering, it’s essential to clarify the difference between Clustering and Classification.

Classification (Supervised Learning)

Goal: Predict known labels.
Input: Labeled data (e.g., this email is/is not spam).
Process: Learn the mapping rules from features to labels.
Output: A model that can predict labels for new data.

Classification Visual: Learn a Boundary from Known Labels

Clustering (Unsupervised Learning)

Goal: Discover hidden groups in the data.
Input: Unlabeled data.
Process: Group data based on its intrinsic similarity.
Output: A partition of the data into clusters and an interpretation of these clusters.

Hard Clustering as a Rigorous Partition

Given a dataset $X = \{x_1, x_2, \ldots, x_N\}$, hard clustering methods such as K-Means divide it into K clusters $\{C_1, C_2, \ldots, C_K\}$.

The next three properties define a hard partition only.
DBSCAN may also return a noise set $C_0$, while GMM uses responsibilities $r_{nk}\in[0,1]$ for soft membership; neither should be forced into this universal definition.

Partition Property 1/3: Non-empty

Each defined cluster must actually contain at least one data point. We don’t make meaningless divisions.

\[ \large{\forall k \in \{1, \dots, K\}, C_k \neq \emptyset} \]

Partition Property 2/3: Mutually Exclusive

Each data point can only belong to one single cluster.

\[ \large{\forall k \neq l, C_k \cap C_l = \emptyset} \]

Partition Property 3/3: Exhaustive

Every single data point in the dataset must be assigned to some cluster.

\[ \large{\bigcup_{k=1}^{K} C_k = X} \]

The Core Idea: Maximize Intra-cluster Similarity

The intuitive goal of clustering is straightforward:

Samples within a cluster should be as ‘similar’ as possible, while samples between clusters should be as ‘dissimilar’ as possible.

The Two Foundational Components of Clustering

To achieve this goal, we must first address two fundamental questions:

Component 1: Defining ‘Similarity’ with Mathematics

In clustering algorithms, ‘dissimilarity’ is typically measured by ‘distance’. The farther the distance, the more dissimilar the points.

An effective distance metric $DM(x, y)$ must satisfy the following four properties:

Property	Mathematical Expression	Economic Intuition
Non-negativity	$DM(x, y) \geq 0$	The difference between two customers cannot be negative.
Identity	$DM(x, y) = 0 \iff x=y$	Distance is zero if and only if the two feature vectors are identical.
Symmetry	$DM(x, y) = DM(y, x)$	The difference from customer A to B is the same as from B to A.
Triangle Inequality	$DM(x, z) \leq DM(x, y) + DM(y, z)$	Indirect difference via a third party is never less than the direct difference.

Distance Metric 1: Euclidean Distance

This is the most common and intuitive definition of distance: the straight-line distance between two points in space.

For two d-dimensional vectors $x=(x_1, \dots, x_d)$ and $y=(y_1, \dots, y_d)$, their Euclidean distance is:

\[ \large{DM_{euc}(x, y) = \sqrt{\sum_{i=1}^{d} (x_i - y_i)^2}} \]

Use Case

When the dimensions of the data are comparable and we care about the absolute physical distance, Euclidean distance is the top choice.

Distance Metric 2: Manhattan Distance

Also known as ‘city block distance’, it calculates the sum of the absolute differences of their Cartesian coordinates.

\[ \large{DM_{man}(x, y) = \sum_{i=1}^{d} |x_i - y_i|} \]

Use Case

Manhattan distance is useful for grid-constrained movement or when absolute deviations should react less strongly than squared deviations to outliers.
It is still unit- and scale-dependent, so it does not replace scaling or feature weights chosen for a clear business reason.

Visualizing Distance: Euclidean vs. Manhattan

A Critical Practical Step: Feature Scaling

Warning: Scale the data before distance-based clustering (such as K-Means) unless all features already share meaningful units or business knowledge provides suitable distance weights.

Customer	Age (Years)	Annual Income (USD)
A	25	50,000
B	30	50,100

Problem: Without scaling, the squared difference of $100 in income will far outweigh the squared difference of 5 years in age, causing the clustering result to be completely dominated by income.
Solution: Use tools like StandardScaler to bring all features to a similar scale (e.g., mean of 0, variance of 1).

Scaling Check

Check 1： If market cap changes from CNY hundreds of millions to CNY while other fields stay fixed, should unscaled K-Means remain unchanged? Decide first.

Reveal： No; Euclidean distance changes with units. If you answered yes, revisit the distance formula and multiply that coordinate difference by $10^8$.

The Impact of Feature Scaling

Without scaling, distance calculations are ‘hijacked’ by features with large numerical ranges.

Component 2: Center or Density Connectivity

K-Means represents a cluster by a centroid; DBSCAN instead uses local density connectivity.

1. Mean-based Center

Algorithm: K-Means
Definition: the arithmetic mean of the cluster’s points.
Formula: \[ \large{\mu_k = \frac{1}{|C_k|} \sum_{x_n \in C_k} x_n} \]
Limitation: sensitive to outliers.

2. Density-connected Cluster

Algorithm: DBSCAN
Definition: start at a core point with at least min_samples points in its $\varepsilon$-neighborhood, then expand by density reachability.
Membership: core, border, or noise; there is no unique centroid.
Contrast: density-peak methods such as CFSFDP select high-density, well-separated centers.

K-Means Centroid vs DBSCAN Density Connectivity

Evaluating Performance: Is Our Grouping Meaningful?

We need a set of objective metrics to evaluate the effectiveness of our clustering. Evaluation methods fall into two main categories:

External Measures

Prerequisite: You have ‘ground truth’ class labels (for evaluation only).
Goal: Measure the consistency between the clustering result and the true labels.
Representative Metrics:
- Purity
- Entropy
- Homogeneity
- Completeness

Internal Measures

Prerequisite: You do not have true labels.
Goal: Evaluate the quality of the clustering based only on the data itself.
Core Idea: Are clusters sufficiently compact (cohesive) and well-separated?
Representative Metric:
- Silhouette Coefficient

External Metric (1): Purity

Idea： How ‘pure’ is a cluster? We check the proportion of the most frequent ‘true class’ within that cluster.

Calculation:

For each cluster $C_k$, find the true class $L_s$ that is most dominant.
Calculate the proportion of samples from this class, $|C_k \cap L_s|$, relative to the total number of samples in the cluster, $|C_k|$.
The overall Purity is the weighted average over all clusters.

\[ \large{\text{Purity} = \sum_{k=1}^{K} \frac{|C_k|}{N} \max_s \left( \frac{|C_k \cap L_s|}{|C_k|} \right)} \]

Pro： simple and intuitive. Limit: Purity rises as clusters fragment; one point per cluster yields 1, so constrain K or report an internal metric too.

Purity Calculation Example

Suppose we have 10 samples, with true classes being 5 diamonds and 5 hexagons. A clustering algorithm divides them into two clusters, K1 and K2.

True Class	Diamond	Hexagon
Cluster K1	5 (dominant)	1
Cluster K2	0	4 (dominant)

Calculate Purity of K1: $Purity(K_1) = \max(\frac{5}{6}, \frac{1}{6}) = \frac{5}{6}$

Calculate Purity of K2: $Purity(K_2) = \max(\frac{0}{4}, \frac{4}{4}) = \frac{4}{4} = 1$

Calculate Overall Purity (Weighted Average):

\[ \large{\begin{aligned} Purity_{total} &= \frac{|K1|}{N} Purity(K_1) + \frac{|K_2|}{N} Purity(K_2) \\ &= \frac{6}{10}\frac{5}{6} + \frac{4}{10} \times 1 = 0.9 \end{aligned}} \]

External Metric (2): Entropy

Idea

Measures the ‘disorder’ or ‘impurity’ within a cluster. If a cluster contains a mix of samples from various true classes, its entropy is high. If it’s pure, its entropy is low.

Calculation: For a cluster $C_k$, its entropy is calculated as:

\[ \large{\text{Entropy}(C_k) = - \sum_{s=1}^{S} p_{ks} \log_2(p_{ks})} \]

where $p_{ks} = \frac{|C_k \cap L_s|}{|C_k|}$ is the probability of true class $s$ appearing in cluster $k$.

Entropy Calculation Example

Let’s use cluster K1 from the previous example to calculate its entropy.

K1 has 6 samples.
The probability of the ‘Diamond’ class is $p_{diamond} = 5/6$.
The probability of the ‘Hexagon’ class is $p_{hexagon} = 1/6$.

\[ \begin{aligned} \operatorname{Entropy}(K_1) &=-\left[\frac56\log_2\!\left(\frac56\right) +\frac16\log_2\!\left(\frac16\right)\right]\\ &\approx 0.65. \end{aligned} \]

The closer this value is to 0, the purer the cluster. K2 contains only hexagons, so $p_{hexagon} = 1$, and $\text{Entropy}(K_2) = -1 \log_2(1) = 0$, which is perfectly pure.

External Metrics (3): Homogeneity & Completeness

Purity and Entropy only tell part of the story. Homogeneity and Completeness provide a more comprehensive view.

Homogeneity: Measures whether each cluster contains only members of a single class. If every cluster perfectly contains members of only one true class, homogeneity is 1. This is analogous to ‘Precision’ in economics.
Completeness:
- Measures whether all members of a given class are assigned to the same cluster.
- If all members of a true class are perfectly grouped into one cluster, completeness is high.
- This is analogous to ‘Recall’.

These two metrics are often in a trade-off.

Homogeneity vs. Completeness: An Intuitive Example

Assume the ground truth is {3 Apples, 3 Pears}.

Perfect Clustering

Cluster 1: {🍎, 🍎, 🍎}
Cluster 2: {🍐, 🍐, 🍐}
Homogeneity = 1
Completeness = 1

High Homogeneity, Low Completeness

Cluster 1: {🍎}
Cluster 2: {🍎, 🍎}
Cluster 3: {🍐, 🍐, 🍐}
Homogeneity = 1 (each cluster is pure)
Completeness < 1 (Apples are split)

Low Homogeneity, High Completeness

Cluster 1: {🍎, 🍎, 🍎, 🍐, 🍐, 🍐}
Homogeneity < 1 (Cluster 1 is impure)
Completeness = 1 (all same-class items are together)

Internal Metric: Silhouette Coefficient

Without true labels, the Silhouette Coefficient balances within-cluster cohesion and between-cluster separation.

Idea: each point should be close to its own cluster and far from the nearest other cluster.

Calculation: For a sample $x_n$:

$a(x_n)$: The average distance from $x_n$ to all other points in its own cluster (measures intra-cluster cohesion).
$b(x_n)$: The average distance from $x_n$ to all points in the nearest other cluster (measures inter-cluster separation).

\[ SC(x_n) = \frac{b(x_n) - a(x_n)}{\max\{a(x_n), b(x_n)\}} \]

Core Numerical Check: Compute One Silhouette Value

Compute first： if $a(x_n)=2$ and the nearest-other-cluster distance is $b(x_n)=5$, what is $SC(x_n)$? Is it closer to 1, 0, or -1?

Reveal and remediation:
- $SC=(5-2)/\max(2,5)=3/5=0.6$, positive and closer to 1, so the point is more cohesive with its assigned cluster.
- If you used $a+b$ in the denominator, revisit the definition; if the result was negative, check the order $b-a$.

Visualizing the Silhouette Coefficient Calculation

Interpreting the Silhouette Coefficient

The value of the Silhouette Coefficient $SC(x_n)$ ranges from -1 to 1.

SC ≈ 1: Excellent! This means $a(x_n)$ is much smaller than $b(x_n)$. The point is well-matched to its own cluster and poorly-matched to neighboring clusters.
SC ≈ 0: The point is on or very close to the decision boundary between two neighboring clusters.
SC ≈ -1: Very bad! This means the point is likely assigned to the wrong cluster.

We can calculate the average silhouette coefficient over all data points to evaluate the entire clustering result.

Algorithm 1: K-Means Clustering

K-Means is one of the most famous and widely used clustering algorithms.

Core Idea: Iteratively partition the data into K clusters, such that each data point belongs to the cluster with the nearest mean (cluster center), and the within-cluster sum of squares is minimized.
Pros: Simple algorithm, fast computation, works extremely well for spherical, similarly sized clusters.
Cons: Requires the number of clusters K to be specified beforehand, sensitive to initial centroid placement, performs poorly on non-spherical clusters and with outliers.

The Objective Function of K-Means

K-Means aims to minimize the Sum of Squared Errors (SSE) within all clusters, also known as Inertia.

\[ \large{\text{minimize} \sum_{k=1}^{K} \sum_{x_n \in C_k} ||x_n - \mu_k||^2} \]

Where:

$K$ is the number of clusters.
$C_k$ is the $k$-th cluster.
$\mu_k$ is the centroid of the $k$-th cluster.
$||x_n - \mu_k||^2$ is the squared Euclidean distance from data point $x_n$ to the centroid of its assigned cluster.

Each step of the algorithm (assignment and update) is designed to reduce this total error value.

K-Means Iterative Process

The K-Means algorithm process is like a game of “capturing territory,” repeated in a four-step cycle:

Initialize: Randomly select K data points as the initial cluster centers.
Assign: For each data point, calculate its distance to all K cluster centers and assign it to the nearest one.
Update: For each cluster, recalculate its center (i.e., the mean of all data points in that cluster).
Repeat: Repeat steps 2 and 3 until the cluster centers no longer change significantly, or the assignment of data points no longer changes.

K-Means Step 1/4: Initialization

Assume K=3. We randomly select three points in the data space as the initial centroids.

K-Means Step 2/4: Assignment

Based on the distance of each point to these three initial centers, color them with the color of the nearest center.

K-Means Step 3/4: Update

For each colored cluster, calculate the mean of all its points and move the cluster center to this new location.

K-Means Step 4/4: Convergence

Repeat the ‘Assign-Update’ steps until the cluster centers no longer move. At this point, we have our final clustering result.

K-Means Achilles’ Heel: Initialization

Random initialization can lead to suboptimal or even completely wrong clustering results.

Problem: If the initial centers are chosen poorly (e.g., all within the same true cluster), the algorithm can get stuck in a local optimum and fail to find the global optimum.
Solution: K-Means++
- Choose the first center uniformly at random; each later sample is drawn with probability proportional to its squared distance $D(x)^2$ from the nearest selected center—not deterministically as the farthest point.
- This usually improves initial coverage, but it does not guarantee a global optimum. Use multiple restarts and report stability.

K-Means++ Initialization Strategy

The Most Critical Question: How to Choose the Value of K?

The K-Means algorithm itself cannot tell us what the optimal K is. This is a hyperparameter that we must determine using domain knowledge and data insights.

Fortunately, we have two common heuristic methods to aid this decision:

The Elbow Method
Silhouette Analysis

Choosing K: Choose in advance the Rule Before Seeing the Result

Inertia (within-cluster sum of squares): decreases mechanically as K grows; use it only to assess diminishing marginal improvement, not to “prove” a K.
Average silhouette: combines within-cluster cohesion and between-cluster separation; this lecture chooses in advance the maximum over $K=2,\ldots,10$.
Stability: compare common firms across another period’s data or resample. Cluster labels may permute, so numeric IDs are not business identities.
The next section calculates inertia and silhouette on one real dataset, chooses K, and then shows firms, centroids, and cluster profiles.
See Rousseeuw (1987) for the silhouette definition and interpretation.

Case Study: Valuation Profiles of Yangtze River Delta Listed Firms

We use local observations of non-financial listed firms in Shanghai, Jiangsu, Zhejiang, and Anhui at the latest available quarter no later than 2024-12-31. Clusters are exploratory structures, not “good/bad” ratings.

Case Data: Yangtze River Delta Valuation Fields

Code

from pathlib import Path  # Declaring Local Data Sources Using Path Objects
import numpy as np  # Provides logarithmic transformation and quantile cropping
import pandas as pd  # Read and Merge Company Basic Information and Valuation Factors
# Public download: https://assets.qiufei.site/data/stock/valuation_factors_quarterly_15_years.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/stock/valuation_factors_quarterly_15_years.h5")
# Windows: Path(r"C:\qiufei\data\stock\valuation_factors_quarterly_15_years.h5")
# macOS: Path("/Users/your_name/data/stock/valuation_factors_quarterly_15_years.h5")
# Linux: Path("/home/your_name/data/stock/valuation_factors_quarterly_15_years.h5")
valuation_path = Path("/home/ubuntu/r2_data_mount/data/stock/valuation_factors_quarterly_15_years.h5")
# Public download: https://assets.qiufei.site/data/stock/stock_basic_data.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/stock/stock_basic_data.h5")
# Windows: Path(r"C:\qiufei\data\stock\stock_basic_data.h5")
# macOS: Path("/Users/your_name/data/stock/stock_basic_data.h5")
# Linux: Path("/home/your_name/data/stock/stock_basic_data.h5")
basic_path = Path("/home/ubuntu/r2_data_mount/data/stock/stock_basic_data.h5")
valuation_data = pd.read_hdf(valuation_path, key='valuation_factors').reset_index()  # Read quarterly valuation factors
basic_data = pd.read_hdf(basic_path, key='stock_basic_info')  # Read company region and industry information
valuation_data['date'] = pd.to_datetime(valuation_data['date'])  # Convert Quarterly Index to Date
snapshot_date = valuation_data.loc[valuation_data['date'] <= '2024-12-31', 'date'].max()  # Select the latest available quarter no later than the deadline
yangtze_provinces = ['上海市', '江苏省', '浙江省', '安徽省']  # Clarify the scope of the Yangtze River Delta filter
company_filter = basic_data['province'].isin(yangtze_provinces) & basic_data['sector_code_name'].ne('金融')  # Exclude the financial industry to enhance comparability
company_columns = ['order_book_id', 'symbol', 'province', 'industry_name']  # Keep company field required for portrait
snapshot = valuation_data.loc[valuation_data['date'].eq(snapshot_date)].merge(basic_data.loc[company_filter, company_columns], on='order_book_id')  # Merging Company Information for the Same Quarter
cluster_features = ['pe_ratio_ttm', 'pb_ratio_lf', 'dividend_yield_ttm', 'market_cap']  # Define Four Clustering Fields with True Semantics
analysis_frame = snapshot.dropna(subset=cluster_features).copy()  # Drop rows with missing clustering features
analysis_frame['log_market_cap'] = np.log(analysis_frame['market_cap'].clip(lower=1))  # Logarithmic Decrease Bias for Market Cap
cluster_features = ['pe_ratio_ttm', 'pb_ratio_lf', 'dividend_yield_ttm', 'log_market_cap']  # Update the model input with a logarithmic market value
analysis_frame[cluster_features] = analysis_frame[cluster_features].clip(analysis_frame[cluster_features].quantile(.01), analysis_frame[cluster_features].quantile(.99), axis=1)  # Shrinking by 1% and 99%
display(analysis_frame[['symbol', 'province', 'industry_name'] + cluster_features].head())  # Display real fields rather than simulated customer data

Table 1: Sample fields for Yangtze River Delta valuation profiles

	symbol	province	industry_name	pe_ratio_ttm	pb_ratio_lf	dividend_yield_ttm	log_market_cap
0	中国天楹	江苏省	生态保护和环境治理业	32.461148	1.112704	0.004312	23.222964
1	丰原药业	安徽省	医药制造业	17.359177	1.426737	0.024470	21.770256
2	华数传媒	浙江省	广播、电视、电影和影视录音制作业	26.172157	0.904128	0.030556	23.314116
3	东方盛虹	江苏省	化学纤维制造业	-16.764315	1.630315	0.012180	24.717386
4	英特集团	浙江省	批发业	11.630537	1.248177	0.028592	22.439025

Choose K with the Same Rule for Every Candidate

Every candidate K uses the same 2024-12-31 firm data, scaled variables, and random seed. The rule selects the highest average silhouette over $K=2,\ldots,10$.

Case Implementation: Select and Fit K

Among $K=2\ldots10$, select the largest average silhouette score. Use the inertia curve to understand diminishing returns, not to replace the stated rule.

Code

from sklearn.cluster import KMeans  # Fit K-Means with Different K Values
from sklearn.metrics import silhouette_score  # Calculate Comprehensive Indicator for Intra-Cluster Cohesion and Inter-Cluster Separation
from sklearn.preprocessing import StandardScaler  # Eliminating the dominance of different dimensions on the Euclidean distance
scaler = StandardScaler()  # estimate scaling parameters only on current unsupervised snapshot
scaled_features = scaler.fit_transform(analysis_frame[cluster_features])  # Standardize the four valuation image fields
candidate_k_values = range(2, 11)  # Pre-Calculated Candidate K is 2 to 10
inertia_by_k = {}  # Save Cluster Flatness Per K Fang He
silhouette_by_k = {}  # Save Average Profile Coefficients Per K
for candidate_k in candidate_k_values:  # Execute the same fitting process for each candidate K
    candidate_model = KMeans(n_clusters=candidate_k, random_state=42, n_init=20)  # Fixed Seed and Multiple Initialization
    candidate_labels = candidate_model.fit_predict(scaled_features)  # Generate cluster labels produced by the fitted model under this K
    inertia_by_k[candidate_k] = candidate_model.inertia_  # Record Inertia
    silhouette_by_k[candidate_k] = silhouette_score(scaled_features, candidate_labels)  # Record Average Profile Coefficient
selected_k = max(silhouette_by_k, key=silhouette_by_k.get)  # Select the K with the highest contour coefficient by preset rules
kmeans = KMeans(n_clusters=selected_k, random_state=42, n_init=20)  # Build Final Model with Selected K
analysis_frame['Cluster'] = kmeans.fit_predict(scaled_features)  # Write the actual clustering results back to the company snapshot
cluster_profile = analysis_frame.groupby('Cluster')[cluster_features].agg(['mean', 'median', 'count'])  # Generating cluster portraits from real labels
display(pd.DataFrame({'inertia': inertia_by_k, 'silhouette': silhouette_by_k}))  # Show the cost of verifiable proofs selected by K
display(cluster_profile)  # Demonstrate Size, Mean and Median of Each Cluster

Result

for 2024-12-31, n=1,871.
Silhouette scores for K=2…10 are 0.2883, 0.3153, 0.3004, 0.3279, 0.3472, 0.2937, 0.2733, 0.2639, and 0.2501.
The rule therefore selects K=6.

Case Evidence: K-Selection Curves Generated from the Data

Code

import matplotlib.pyplot as plt  # Drawing K-Selection Evidence Generated by Actual Calculations
k_table = pd.DataFrame({'K': list(candidate_k_values), 'inertia': list(inertia_by_k.values()), 'silhouette': list(silhouette_by_k.values())})  # Compare both metrics for the same fitted models
figure, left_axis = plt.subplots(figsize=(10, 5.2))  # Create a projection-safe biaxial canvas
left_axis.plot(k_table['K'], k_table['inertia'], marker='o', markersize=9, linewidth=2.5, color='#33658A')  # Drawing inertia that Drops Monotonically with K
left_axis.set_xlabel('K', fontsize=30)  # Indicate the number of candidate clusters
left_axis.set_ylabel('Inertia', color='#33658A', fontsize=30)  # Indicate the left axis meaning
left_axis.tick_params(axis='both', labelsize=30)  # Keep ticks legible after slide projection
right_axis = left_axis.twinx()  # Establish an independent vertical axis for contour coefficients
right_axis.plot(k_table['K'], k_table['silhouette'], marker='s', markersize=9, linewidth=2.5, color='#F26419')  # Drawing the computed mean silhouette values
right_axis.axvline(selected_k, color='#F26419', linestyle='--', alpha=.6)  # Mark the K selected by the preset rules
right_axis.set_ylabel('Average silhouette', color='#F26419', fontsize=30)  # Indicate the right axis meaning
right_axis.tick_params(axis='y', labelsize=30)  # Keep ticks legible after slide projection
figure.tight_layout()  # Prevent axis labels from being cut
plt.show()  # Show graphs generated from current data

A dual-axis line chart shows inertia and average silhouette for K from 2 to 10 on the same company snapshot, marking the K with maximum silhouette. — Figure 1: Inertia and average silhouette for Yangtze River Delta valuation profiles

Case Result: Visualizing the Firms and Centroids

Let’s visualize the clustering results to see what customer segments we have discovered.

Code

import matplotlib.pyplot as plt
import seaborn as sns  # Drawing Real Company Scatter using Unified Color Scheme
centroids = scaler.inverse_transform(kmeans.cluster_centers_)  # Restore the centroid to the original valuation dimension
plt.style.use('seaborn-v0_8-whitegrid')  # Use a light grid suitable for projection
figure, axis = plt.subplots(figsize=(9, 5))  # Create single-page visible canvas
sns.scatterplot(data=analysis_frame, x='pe_ratio_ttm', y='pb_ratio_lf', hue='Cluster', palette='viridis', s=55, alpha=.7, ax=axis)  # Draw Company by fitted cluster label
axis.scatter(centroids[:, 0], centroids[:, 1], s=220, c='red', marker='X', edgecolor='black', label='Centroids')  # Mark the center of mass estimated by the model
axis.set_title(f'Executed clustering: K={selected_k}')  # Reading K from running results instead of hard-coded
axis.set_xlabel('Trailing PE ratio')  # Explain horizontal field
axis.set_ylabel('Latest PB ratio')  # Explain vertical field
axis.legend(title='Computed cluster', bbox_to_anchor=(1.02, 1), loc='upper left')  # Use neutral cluster numbers to avoid fixed business mappings
figure.tight_layout()  # Prevent Legend and Coordinates from being cropped
plt.show()  # Show graphs generated from current fit_predict results

Scatterplot of actual firms by trailing PE and latest PB, colored by computed cluster, with fitted centroids marked as X symbols. — Figure 2: Executed K-Means clusters in the PE–PB plane for Yangtze River Delta firms

Case Result: Rules for Business Interpretation

Read the executed means, medians, and counts in cluster_profile, then assign relative descriptions such as “high PB–low dividend yield–large capitalization.” Numeric cluster labels have no permanent business meaning.

Executed profiles (means, n)

Cluster 0 has positive PE 17.96, PB 1.91, dividend yield 4.8%, n=253;
Cluster 1 has high PE/PB and the largest capitalization, n=457;
Cluster 2 has moderate PE/PB and smaller capitalization, n=1,006;
Cluster 3 has extremely high positive PE, n=54;
Cluster 4 has negative PE and high PB, n=54;
Cluster 5 has extremely negative PE, n=47.
Extreme means should be labeled “unusual valuation shapes,” never quality ranks.

These clusters describe similarity in this quarter’s data. They do not prove higher future returns and should not mechanically trigger investment decisions.

Formative Check: Why Must This Case Report K=6?

Question： This case has silhouette 0.3472 at K=6 and 0.2937 at K=7. What if K=7 tells an easier business story?

Choose “keep K=6 / silently use K=7 / report a transparent override” before revealing.

Check 2 reveal： The stated rule selects K=6. Choosing K=7 for an easier business interpretation would require reporting both scores and the stability comparison. Choosing only the more appealing story creates selection bias.

Step-by-Step Exercise and Complete Answer

Task： Restrict candidates to K=3…6 and submit inertia, silhouette, selected K, and one profile.
Complete answer:
- with candidate_k_values = range(3, 7), silhouettes are K3=0.3153, K4=0.3004, K5=0.3279, and K6=0.3472, so K=6 remains selected.
- The six cluster sizes are 253, 457, 1,006, 54, 54, and 47.
- Read profiles from the executed cluster_profile; do not attach names to IDs in advance.

Apply It to a New Case

Task： Repeat the full analysis for the latest quarter no later than 2023-12-31 and compare common firms across dates using adjusted Rand index.
Complete-answer reminder：
- Design: fix the cutoff, choose the latest eligible quarter, and repeat the same preprocessing and K-selection rule.
- Alignment: compare only firms observed in both periods.
- Evidence: report ARI, sample sizes, both K values, and a non-causal stability interpretation.

Code

comparison_date = valuation_data.loc[valuation_data['date'] <= '2023-12-31', 'date'].max()  # Select the true latest quarter before the deadline
comparison_snapshot = valuation_data.loc[valuation_data['date'].eq(comparison_date)].merge(basic_data.loc[company_filter, company_columns], on='order_book_id')  # Reuse the same company filter
comparison_df = comparison_snapshot.dropna(subset=['pe_ratio_ttm', 'pb_ratio_lf', 'dividend_yield_ttm', 'market_cap']).copy()  # keep the field convention consistent
comparison_df['log_market_cap'] = np.log(comparison_df['market_cap'].clip(lower=1))  # Use identical logarithmic transformations
comparison_df[cluster_features] = comparison_df[cluster_features].clip(comparison_df[cluster_features].quantile(.01), comparison_df[cluster_features].quantile(.99), axis=1)  # Shrink tail within this quarter's data
comparison_scaler = StandardScaler()  # Generate Unsupervised Scaling for Comparison Snapshot Independent Estimation
comparison_scaled = comparison_scaler.fit_transform(comparison_df[cluster_features])  # Standardize the same four fields
comparison_scores = {}  # Save 2023 Candidate Profile Coefficients
for comparison_k in range(2, 11):  # Execute Same Candidate Range as 2024
    comparison_labels = KMeans(n_clusters=comparison_k, random_state=42, n_init=20).fit_predict(comparison_scaled)  # Fixed Initialization Rule
    comparison_scores[comparison_k] = silhouette_score(comparison_scaled, comparison_labels)  # Record the computed score
comparison_selected_k = max(comparison_scores, key=comparison_scores.get)  # Select K by Same Rule
comparison_df['Cluster'] = KMeans(n_clusters=comparison_selected_k, random_state=42, n_init=20).fit_predict(comparison_scaled)  # Fit 2023 Final Labels
common_firms = analysis_frame[['order_book_id', 'Cluster']].merge(comparison_df[['order_book_id', 'Cluster']], on='order_book_id', suffixes=('_2024', '_2023'))  # Only Align Two Periods of Common Company
from sklearn.metrics import adjusted_rand_score  # Use stability metrics unaffected by cluster number permutation
stability_ari = adjusted_rand_score(common_firms['Cluster_2023'], common_firms['Cluster_2024'])  # Calculate the common company Ari
print(comparison_date, len(comparison_df), comparison_selected_k, len(common_firms), round(stability_ari, 4))  # Report Full Key Output

2023-06-30 00:00:00 1765 6 1762 0.358

Apply It to a New Case: Result and Interpretation

Complete reference output:
- 2023-06-30 (the latest available quarter before the cutoff), n=1,765, K=6; n=1,762 firms appear in both snapshots and ARI=0.3580.
- The value is far below 1, showing material regime sensitivity.
- ARI is invariant to cluster-label permutation but does not establish an economic causal mechanism.

Answer note： Revisit label permutation, common-firm alignment, or the non-causal interpretation if any was mishandled.

Core summary: From Distance to an easy to check Claim

Define the representation: scale features without shared meaningful units; choose in advance any domain weights.
Choose K consistently: use the stated silhouette rule and disclose overrides and stability instead of choosing by story.
Bound interpretation: IDs are permutable and profiles come from executed output; clusters describe similarity, not return or causality.

Core ends here. Continue only if you need the Extension’s alternative cluster-shape assumptions.

Sources and Further Reading

MacQueen, J. (1967), “Some Methods for Classification and Analysis of Multivariate Observations.”
Rousseeuw, P. J. (1987), “Silhouettes”, Journal of Computational and Applied Mathematics, 20, 53–65; the adjacent source for this case’s K-selection rule.
scikit-learn User Guide: clustering performance evaluation.
Data: local valuation_factors_quarterly_15_years.h5 / valuation_factors and stock_basic_data.h5 / stock_basic_info; case figures are generated by slide code.

Algorithm 2: Hierarchical Clustering

Unlike K-Means, which partitions data into K clusters in one go, Hierarchical Clustering creates a hierarchy of clusters.

Core Idea: It doesn’t produce a single clustering but a tree-like structure (a Dendrogram) that shows how data points are successively merged (or split).
Two Main Strategies:
- Agglomerative: Bottom-up. Starts with each point as its own cluster, then progressively merges the most similar clusters.
- Divisive: Top-down. Starts with all points in one cluster, then progressively splits the most dissimilar ones.

The Agglomerative Hierarchical Clustering Workflow

Start: Treat each data point as an individual cluster.
Merge: Find the two closest clusters among all clusters and merge them into a new one.
Repeat: Repeat step 2 until all data points are merged into a single cluster.

The entire history of these merges forms a tree-like hierarchical structure.

Measuring Cluster-to-Cluster Distance

This is defined by the Linkage Criterion. There are three main types:

Visualizing Hierarchical Clustering: The Dendrogram

Vertical: merge height is cluster dissimilarity; horizontal: individual observations.
A horizontal cut selects the clusters present at that dissimilarity threshold.

Algorithm 3: Density-Based Clustering (DBSCAN)

When K-Means Fails: K-Means assumes clusters are spherical. It struggles with irregularly shaped clusters (like crescents or rings).

Enter DBSCAN (Density-Based Spatial Clustering of Applications with Noise):

Core Idea: Groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions.
Pros: Does not require specifying the number of clusters, can find arbitrarily shaped clusters, and is robust to noise.
Cons: Sensitive to the choice of its two core parameters (eps and min_samples), struggles with datasets of varying density.

The Core Concepts of DBSCAN

DBSCAN’s logic relies on three key definitions, controlled by two parameters: eps (a radius) and min_samples (a minimum number of points).

The Intuitive Flow of the DBSCAN Algorithm

The DBSCAN process is like a “snowball effect”:

Randomly select an unvisited point P.
Check if P is a core point.
- If it is:
  - Start a new cluster with P.
  - Then, find all density-reachable points through P’s ε-neighborhood (including other core and border points) and add them all to this cluster.
  - This process expands like a chain reaction until the entire density-connected region is covered.
- If it is not (i.e., it’s a border or noise point): Temporarily mark it as noise and move to the next point.
Repeat steps 1 and 2 until all points have been visited.

Case Study: DBSCAN Shines Where K-Means Fails

Let’s look at an example that K-Means cannot handle but DBSCAN solves perfectly: the “two moons” dataset.

Algorithm 4: Gaussian Mixture Model (GMM)

Beyond ‘Hard’ Clustering

K-Means performs a ‘black-or-white’ hard assignment for each point. But sometimes, a point might lie on the boundary of two clusters, and we’d prefer a probabilistic description.

Gaussian Mixture Model (GMM) offers a Soft Clustering approach.

Core Idea: Assumes that all data points are generated from a mixture of K different Gaussian (normal) distributions. The clustering process is about finding the parameters (mean, variance, weight) of these K distributions.
Result: GMM doesn’t directly tell us which cluster each point belongs to; instead, it gives us the probability that the point belongs to each cluster.

GMM vs. K-Means: A Comparison

Feature	K-Means	Gaussian Mixture Model (GMM)
Cluster Shape	Assumes spherical (circular)	Can adapt to elliptical shapes
Assignment	Hard Assignment	Soft Assignment (Probabilistic)
Mathematical Basis	Distance-based	Probability-based (Expectation-Maximization)
Flexibility	Lower	Higher, better at fitting complex data

GMM can be seen as a probabilistic generalization of K-Means.

Case Study: GMM Excels at Fitting Elliptical Clusters

When cluster shapes are not circular but elliptical, the advantages of GMM become apparent.

Extension Synthesis: How to Choose the Right Clustering Algorithm?

Today we’ve learned four major clustering algorithms, each with its own strengths and weaknesses. In practice, there is no ‘best’ algorithm, only the ‘most suitable’ one.

Algorithm	Key Advantages	Key Disadvantages	Best For…
K-Means	Fast, simple, scalable	Requires K, sensitive to spherical assumption	Large datasets with simple, well-separated clusters
Hierarchical	No K before fitting the hierarchy	A flat partition still needs a documented cut-height or cluster-count rule; expensive (O(N^2))	Small datasets, exploring data structure
DBSCAN	No need for K, finds any shape	Sensitive to parameters, struggles with varied density	Non-spherical clusters, datasets with noise
GMM	Soft clustering, flexible shapes	Complex, high computational cost	Probabilistic output needed, overlapping clusters

An Algorithm Selection Decision Tree

Here is a simplified decision-making flowchart to help you choose an appropriate clustering algorithm in practice.

The Business Value of Cluster Analysis

Cluster analysis is more than just a set of algorithms; it’s a powerful tool for business insight.

Marketing: Identify customer segments for targeted advertising and personalized recommendations (e.g., Amazon, Netflix).
Finance: Detect unusual transaction patterns to identify potential fraud (e.g., American Express).
Urban planning: group areas by resident travel patterns.
Social networks: discover communities and interest groups.

Mastering clustering means mastering the key to finding structure and creating value from vast amounts of data.

Thank You!

Q & A

Property	Mathematical Expression	Economic Intuition
Non-negativity	\(DM(x, y) \geq 0\)	The difference between two customers cannot be negative.
Identity	\(DM(x, y) = 0 \iff x=y\)	Distance is zero if and only if the two feature vectors are identical.
Symmetry	\(DM(x, y) = DM(y, x)\)	The difference from customer A to B is the same as from B to A.
Triangle Inequality	\(DM(x, z) \leq DM(x, y) + DM(y, z)\)	Indirect difference via a third party is never less than the direct difference.