By the end of this lecture, you can:
pgmpy query and state the extra assumptions needed for causal interpretation.Write all three answers before revealing.
\(P(A)P(B\mid A)P(C\mid A,B)\); no; no—causal interpretation requires structural assumptions and identification.
If item 1 was wrong, return to the chain rule; for item 2 or 3, return to d-separation or conditioning/intervention.
Foundations: BN → d-separation → BN factorization.
Application: conditioning and intervention → pgmpy observational query → real-data practice → lesson review.
HMMs and the material that follows are optional topics.
Imagine forecasting GDP growth. It’s a complex system of numerous interrelated variables.
How can we clearly represent and reason about the complex dependencies between these variables?
In theory, to fully describe a system, we need the joint probability distribution \(P(x_1, x_2, \dots, x_d)\).
But this is an almost impossible task.
Imagine we have d binary economic indicators (e.g., rate hike / no rate hike).
To store their joint probability distribution, we need a parameter table.
| # of Variables (d) | 2^d probability cells; 2^d − 1 free parameters | Analogy |
|---|---|---|
| 2 | 4 / 3 | A small note |
| 10 | 1,024 / 1,023 | A single page |
| 20 | 1,048,576 / 1,048,575 | A thick book |
| 30 | 2^30 / (2^30 − 1) | A small library |
| 50 | 2^50 / (2^50 − 1) | Already extremely large |
This is computationally intractable.
Most variables are not directly connected to everything else. A variable is typically only directly related to a few others.
Instead of storing one massive joint table, a graph supplies local factors. Directed and undirected models normalize differently, so a generic “product of local probabilities” is not a valid common definition.
\[ \begin{aligned} \text{BN: }&p(\mathbf x)=\prod_i p(x_i\mid x_{\mathrm{pa}(i)}),\\ \text{MRF: }&p(\mathbf x)=\frac{1}{Z}\prod_{C\in\mathcal C}\psi_C(x_C). \end{aligned} \]
The ‘graph’ structure is precisely what visualizes which variables are ‘locally’ related and which are conditionally independent.
We will primarily study two major classes of PGMs, which differ in how they handle dependencies.
| Model Type | Graph Structure | Meaning of Edges | Core Application Areas |
|---|---|---|---|
| Bayesian Networks | Directed Acyclic Graph (DAG) | Encodes factorization and conditional independence; causal only with extra assumptions | Expert systems, time-series, causal models |
| Markov Networks | Undirected Graph | Encodes potential factorization and conditional independence | Image processing, social networks, physical systems |
Today, we will begin our exploration with directed graphs: Bayesian Networks.
We will learn about their:
Three ways to represent the joint probability \(P(\mathbf{x})\) of a \(d\)-dimensional random vector \(\mathbf{x} = (x_1, \dots, x_d)\):
Assume all variables are mutually independent.
\[ \large{P(x_1, \dots, x_d) = P(x_1)P(x_2)\dots P(x_d)} \]
Using the definition of conditional probability, any joint probability can be precisely decomposed.
\[ \large{P(x_1, \dots, x_d) = P(x_1)P(x_2|x_1)\dots P(x_d|x_1, \dots, x_{d-1})} \]
PGMs offer a testable tradeoff: sparse structure can reduce parameter and inference costs, but the benefit depends on the graph, state cardinalities, and inference algorithm.
This is the secret to how probabilistic graphical models reduce computational complexity.
Bayesian Networks (BNs), also known as Probabilistic Directed Graphical Models (PDGMs), use a Directed Acyclic Graph (DAG) to encode dependencies among variables.
It consists of two parts:
A probabilistic BN commits to： factorization, the local Markov property, and conditional independences readable through d-separation.
Giving a DAG causal semantics requires:
structural-equation/intervention semantics and the causal Markov condition.
Faithfulness is an additional distributional assumption often used for structure learning, not part of the definition of a causal DAG.
Identification is query-specific: for example, back-door adjustment needs the relevant confounding and positivity conditions.
Arrow direction alone cannot establish these conditions from observational data.
Accordingly, this lecture’s pgmpy queries are observational conditioning unless a structural causal model is separately justified.
Let’s use a graph structure to build a simplified model of some process.
In a directed graph, we use family relationships to describe the connections between nodes:
parents(X3) is {X1, X2}.parents(X6) is {X4, X5}.children(X1) is {X3, X5}.X1, X2) is a root node in the factorization; calling it an exogenous cause requires an additional structural causal interpretation.The structure of the DAG directly defines a core conditional independence assumption: The Local Markov Property.
A node is conditionally independent of its non-descendants, given its parents.
Let’s examine the graph in Figure 1 again:
X3:
{X1, X2}{X4, X6}{X5} (Note: X1,X2 are its ancestors, also considered non-descendants)X3 is conditionally independent of X5 given X1 and X2.Based on the local Markov property, any joint probability distribution that is consistent with the DAG can be factorized into the product of the conditional probabilities of all nodes given their parents.
\[ \large{P(x_1, \dots, x_d) = \prod_{i=1}^d P(x_i | \text{parents}(x_i))} \tag{1}\]
This formula is the heart of Bayesian Networks. It decomposes a vast, complex joint probability distribution into a series of small, manageable local probability models (also known as Conditional Probability Distributions, CPDs).
Applying the formula Equation 1 to the graph structure in Figure 1:
\[ \large{ \begin{aligned} & P(X_1, X_2, X_3, X_4, X_5, X_6) \\ & = P(X_1) \cdot P(X_2) \cdot P(X_3|X_1, X_2) \cdot P(X_4|X_3) \\ & \quad \cdot P(X_5|X_1) \cdot P(X_6|X_4, X_5) \end{aligned} } \]
Now that we have the structure, we need to provide specific parameters for each local probability model. For discrete variables, this is typically done using Conditional Probability Tables (CPTs).
The local Markov property tells us about the independence of a node from its ‘non-descendants’. But we often need to answer more general questions:
Are any two arbitrary sets of nodes A and B independent, given a third set E?
D-Separation (Directed Separation) is a complete, general set of rules that allows us to ‘read’ any conditional independence directly from the graph structure.
The core idea of D-separation is to check if all paths between two sets of nodes are ‘blocked’.
If all paths from A to B are blocked by the set of observed variables E, then A and B are conditionally independent given E.
If at least one path is unblocked, the graph does not guarantee that A and B are conditionally independent. D-connection implies dependence only with an additional faithfulness/no-cancellation assumption.
Any path in a complex DAG can be broken down into three fundamental connection structures. Understanding these three is the key to mastering D-separation.
b <- a -> c
b and c are connected through their common parent a.a is observed (given), the path from b to c is blocked.Ice Cream Sales \(\leftarrow\) Hot Weather \(\rightarrow\) Heatstroke CasesIf we already know it’s hot today, observing high ice cream sales tells us nothing new about the number of heatstroke cases.
The weather information has ‘explained away’ the correlation.
Conversely, if the weather is unknown, high ice cream sales would indeed increase our belief in more heatstroke cases.
b -> a -> c
b influences c through the intermediate node a.a is observed (given), the path from b to c is blocked.Fed Policy \(\rightarrow\) Market Interest Rates \(\rightarrow\) Real Estate PricesOnce we directly observe the market interest rates, the specific Fed policy that led to this rate is no longer important for predicting real estate prices.
The interest rate itself contains all the necessary information.
b -> a <- c
b and c are both parents of a. a is their common effect.a (or any of its descendants) is unobserved, the path is blocked.a (or any of its descendants) is observed, the path is unblocked!Academic Talent \(\rightarrow\) Gets Tenure \(\leftarrow\) Political SkillThis phenomenon is known as ‘explaining away’.
Knowing a professor is talented tells us nothing about their political skills (they are independent beforehand).
However, if we know they got tenure and then find out they aren’t very talented, it strongly suggests they must be very politically skilled.
A path from a set of nodes A to B is blocked by a set of observed nodes E if there is a node v on the path such that either:
v is a tail-to-tail or head-to-tail node, and v is in the evidence set E.v is a head-to-head (collider) node, and neither v nor any of its descendants are in the evidence set E.If all paths from A to B are blocked by E, we say that A and B are D-separated given E, which implies conditional independence: A \(\perp\) B | E.
the collider blocks the path while B and its descendants are unobserved; conditioning on B opens it.
If you assumed “more controls always help,” return to the three basic structures and mark the collider.
Now, let’s apply these concepts to the classic ‘Student’ Bayesian Network. This network models the various factors that influence whether a student gets a good recommendation letter.
The structure of this graph encodes our prior beliefs about student performance.
first write \(P(I,D,G,S,L)\) using “each node conditions only on its parents,” then reveal.
. . .
Based on the graph structure and Equation 1, the factorized joint probability is:
\[ \large{P(I, D, G, S, L) = P(I) \cdot P(D) \cdot P(S|I) \cdot P(G|I, D) \cdot P(L|G)} \]
Note how we’ve simplified the problem! For example, we only need \(P(L|G)\), not the intractable \(P(L|I, D, G, S)\).
Now we need to populate each factor with a CPT. These probabilities are typically estimated from domain experts or historical data.
1. Root Nodes (Prior Probabilities)
2. Conditional Probabilities
This table shows the probability of a student getting an A/B/C grade for different combinations of intelligence and course difficulty.
| I (Intelligence) | D (Difficulty) | P(G=A) | P(G=B) | P(G=C) |
|---|---|---|---|---|
| 0 (low) | 0 (easy) | 0.30 | 0.40 | 0.30 |
| 0 (low) | 1 (hard) | 0.05 | 0.25 | 0.70 |
| 1 (high) | 0 (easy) | 0.90 | 0.08 | 0.02 |
| 1 (high) | 1 (hard) | 0.50 | 0.30 | 0.20 |
With a fully specified Bayesian Network, we can perform Inference. This means that after observing some variables (Evidence), we can compute the posterior probability distribution of other unobserved variables.
This answers \(P(Y\mid X=x)\): how Y is distributed among observations with \(X=x\). It does not answer what happens if we force X to x.
Let market sentiment \(U\) affect financing constraints \(X\) and investment \(Y\): \(U\rightarrow X, U\rightarrow Y\), with \(X\rightarrow Y\).
Conditioning: \(P(Y\mid X=x)\) mixes the X path with selection induced by common cause U.
Intervention: \(P(Y\mid do(X=x))\) cuts incoming arrows to X before comparing Y under forced X.
Conclusion: If U is unobserved and no alternative identification strategy exists, an ordinary BN query cannot substitute the first distribution for the second.
Check 2： Is deleting arrows into X conditioning or intervention? Does an ordinary pgmpy query perform that deletion? Answer first.
The Core Problem: Compute \(P(\text{Query Variables} | \text{Evidence Variables})\)
pgmpyEnough theory, let’s make this real with code. We will use a powerful Python library pgmpy to implement the student model.
run and inspect model validation and the observational query \(P(D=1\mid I=1,L=1)\). Explain why an ordinary query deletes no incoming arrows and therefore is not an intervention.
Steps:
pgmpy Hands-on: Building and Validating the ModelFirst, we’ll use a complete code block to build the student model.
from pgmpy.models import DiscreteBayesianNetwork
from pgmpy.factors.discrete import TabularCPD
student_model = DiscreteBayesianNetwork([('D', 'G'), ('I', 'G'), ('I', 'S'), ('G', 'L')])
cpd_d = TabularCPD('D', 2, [[0.6], [0.4]])
cpd_i = TabularCPD('I', 2, [[0.7], [0.3]])
cpd_s = TabularCPD('S', 2,
values=[[0.95, 0.2], [0.05, 0.8]],
evidence=['I'], evidence_card=[2])Now complete the remaining conditional tables and validate that they agree with the graph and normalize correctly.
# Status convention: D = 0 Simple/1 Difficult, I = 0 Low/1 High, G = 0 C, 1 B, 2 A, L = 0 Weak/1 Strong.
# evidence = ['D', 'I'] The column order is (D = 0, I = 0), (D = 0, I = 1), (D = 1, I = 0), (D = 1, I = 1).
cpd_g = TabularCPD('G', 3,
values=[[0.30, 0.02, 0.70, 0.20], # G=0：C。
[0.40, 0.08, 0.25, 0.30], # G=1：B。
[0.30, 0.90, 0.05, 0.50]], # G=2：A。
evidence=['D', 'I'], evidence_card=[2,2])
# The recommendation table gives weak/strong probabilities in the order of columns G = C, B, A.
cpd_l = TabularCPD('L', 2,
values=[[0.9, 0.4, 0.01], [0.1, 0.6, 0.99]], # L=0, L=1.
evidence=['G'], evidence_card=[3])
student_model.add_cpds(cpd_d, cpd_i, cpd_s, cpd_g, cpd_l)
assert student_model.check_model() # 确认概率表与网络结构一致。
print('Model built and validated successfully!') # 展示当前步骤的结果。Model built and validated successfully!pgmpy Hands-on: Creating an Inference Engine and QueryingNow for the exciting part! We create an inference engine (here, using Variable Elimination) and ask our question.
Suppose we observe a student with high intelligence (I=1) who received a strong recommendation letter (L=1).
What is the probability that the course was hard (D=1)? i.e., compute \(P(D=1 | I=1, L=1)\).
+------+----------+
| D | phi(D) |
+======+==========+
| D(0) | 0.6701 |
+------+----------+
| D(1) | 0.3299 |
+------+----------+The query result shows \(P(D=1 | I=1, L=1) \approx 0.3299\).
Optional topics: The following material covers HMMs, undirected models, and general inference. For the main lesson, continue to real-data practice and the lesson review.
If we ‘unroll’ a Bayesian Network over time, we get a Dynamic Bayesian Network (DBN).
The simplest and most famous DBN is the Hidden Markov Model (HMM).
Imagine you are trapped in a cave and can’t see the weather outside. But every day, some water seeps into the cave.
Your task is to infer the most likely sequence of weather patterns outside based on the sequence of water seepage you observe.
Modeling time-series data where the true state of the system is unobservable (hidden), and we can only infer it through some visible observations.
HMMs are built on two key simplifying assumptions, which make the model tractable.
The current true state \(z_t\) depends only on the true state at the previous time step, \(z_{t-1}\).
\[ \large{P(z_t | z_{t-1}, z_{t-2}, \dots, z_1) = P(z_t | z_{t-1})} \]
The current observation \(x_t\) depends only on the current true state \(z_t\).
\[ \large{P(x_t | z_t, x_{t-1}, z_{t-1}, \dots) = P(x_t | z_t)} \]
This quarter’s GDP growth rate depends only on whether the economy is currently in recession or expansion, and not on previous states or observations.
These two assumptions together define a very simple chain-like graphical structure.
To fully define an HMM, we need three parameters, usually denoted as \(\lambda = (A, B, \pi)\).
Let’s use a simplified model to identify whether the economy is in Expansion or Recession.
Our Task: Given an observed sequence of GDP growth, infer the most likely sequence of economic states (Expansion/Recession).
Matrix A represents the ‘inertia’ of the business cycle.
\[ \large{A = \begin{pmatrix} P(\text{Exp}|\text{Exp}) & P(\text{Rec}|\text{Exp}) \\ P(\text{Exp}|\text{Rec}) & P(\text{Rec}|\text{Rec}) \end{pmatrix} = \begin{pmatrix} 0.9 & 0.1 \\ 0.3 & 0.7 \end{pmatrix}} \]
Matrix B represents the typical economic output in different states.
\[ \begin{aligned} B_{jk}&=P(x_t=k\mid z_t=j),\\[2pt] B&=\begin{pmatrix}0.05&0.45&0.50\\0.60&0.35&0.05\end{pmatrix}. \end{aligned} \]
\(\pi\) represents our prior belief about the economic state at time t=1.
\[ \large{\pi = \begin{pmatrix} P(z_1=\text{Exp}) \\ P(z_1=\text{Rec}) \end{pmatrix} = \begin{pmatrix} 0.8 \\ 0.2 \end{pmatrix}} \]
Once we have a model \(\lambda = (A, B, \pi)\), HMM theory primarily addresses three problems.
BN arrows are useful for directed conditional-probability factorizations.
When interactions are symmetric, or we do not wish to impose a parent order, an undirected graph is more natural.
Causal semantics remain a separate modeling commitment.
Social Networks: If I am your friend, you are my friend.
Image Pixels: The color of a pixel is highly correlated with the colors of its neighbors, but none is the ‘cause’ of the others.
Spatial Economics: Housing prices in one region mutually influence prices in neighboring regions.
For these types of problems, Markov Random Fields (MRFs), also known as Probabilistic Undirected Graphical Models (PUGMs), are a more natural choice.
An MRF is defined by two parts:
Unlike the complex rules of D-separation, conditional independence in undirected graphs is very intuitive:
If all paths from a set of nodes A to a set of nodes B must pass through a set of nodes C, then A and B are conditionally independent given C.
In other words, C separates A and B.
For arbitrary distributions, the global property implies the local property, which implies the pairwise property.
The reverse implications—and hence equivalence—require \(p(\mathbf{x})>0\) for every joint configuration \(\mathbf{x}\) in the full product state space, so structural zeros are excluded.
How does the joint probability of an undirected graph factorize? The answer is through Cliques.
The joint probability distribution of an MRF can be factorized into a product of potential functions defined on its maximal cliques.
This important theorem provides the bridge between the joint probability distribution of an MRF and its potential functions:
Positivity: the joint distribution must satisfy \(P(\mathbf{x})>0\) throughout the full product state space.
Graph condition: it obeys the conditional independences encoded by the undirected graph \(G\).
Factorization: the joint probability is a product of potentials \(\psi_C\) over the maximal cliques \(C\) of \(G\):
\[ \large{P(\mathbf{x}) = \frac{1}{Z} \prod_{C \in \mathcal{C}} \psi_C(\mathbf{x}_C)} \]
where \(\mathcal{C}\) is the set of all maximal cliques.
The \(Z\) in the formula is called the partition function or normalization constant.
\[ \large{Z = \sum_{\mathbf{x}} \prod_{C \in \mathcal{C}} \psi_C(\mathbf{x}_C)} \]
To ensure that the probabilities of all possible states sum to 1.
Since the product of potential functions is not guaranteed to do so, we need to divide by the sum of this product over all possible configurations to normalize it.
Computing \(Z\) is often the hardest part of working with MRFs.
It requires summing over all possible configurations of all variables, which is an exponential computation.
This is the main bottleneck for inference and learning in MRFs.
| Feature | Bayesian Network (Directed) | Markov Network (Undirected) |
|---|---|---|
| Graph Structure | Directed Acyclic Graph (DAG) | Undirected Graph |
| Core Idea | Factorization of conditional probabilities | Factorization of potential/energy functions |
| Parameterization | Conditional Probability Distributions (CPDs) | Potential functions on maximal cliques |
| Normalization | Automatically satisfied (local normalization) | Requires a global partition function Z |
| Use Cases | Directed generative models; causal models only with causal assumptions | Symmetric interactions, spatial or discriminative models |
| Interpretability | Strong, easy to understand | Weaker, potential functions are less intuitive |
Conditional Random Fields (CRFs) are a special type of MRF that is a discriminative model.
CRFs can relax the strict observation independence assumption of HMMs.
In a CRF, the probability of the current state \(z_t\) can depend on features from the entire observation sequence \(X\), making the model much more powerful and flexible.
The most common type of CRF is the linear-chain CRF, which is specialized for sequence data.
Its conditional probability \(P(Z|X)\) takes the form:
\[ \large{P(Z|X) = \frac{1}{Z(X)} \exp\left( \sum_{t,k} \lambda_k f_k(z_{t-1}, z_t, X, t) \right)} \]
We have learned about directed and undirected graphs. Both represent factorizations of a joint probability, but in different ways.
Is there a more general representation that can unify both models and provide a basis for universal inference algorithms?
The answer is the Factor Graph.
A factor graph is a bipartite graph that explicitly represents how a global function (like a joint probability) is factorized into a product of local functions (factors).
Recall the student BN factorization: \(P(\cdot) = P(I) P(D) P(S|I) P(G|I,D) P(L|G)\)
Recall the MRF factorization: \(P(\mathbf{x}) \propto \psi_1(A,B,E) \psi_2(B,C) \psi_3(E,D)\)
once a factor set is chosen, the graph shows each local function and its variables.
The same global function can still be represented by merging, splitting, or reparameterizing factors, so factor graphs are not unique.
Belief Propagation (BP), also known as the Sum-Product Algorithm, is a general algorithm for performing efficient, exact inference on graphical models.
On graphs without cycles (i.e., trees), the algorithm converges after a finite number of message passes and yields exact marginal probabilities.
On graphs with cycles (Loopy BP), the algorithm is not guaranteed to converge or give exact solutions, but it often works very well in practice.
Why is this algorithm efficient? Because it cleverly uses the distributive law to avoid redundant computations.
Consider calculating the marginal probability \(P(x_1) = \sum_{x_2, x_3, x_4} P(x_1,x_2,x_3,x_4)\).
Assume \(P(\mathbf{x}) \propto f_1(x_1,x_2) f_2(x_2,x_3) f_3(x_3,x_4)\).
Brute-force calculation: For each value of \(x_1\), iterate through all combinations of \(x_2, x_3, x_4\).
Sum-product algorithm (smart calculation):
\[ \begin{aligned} m_{3\to2}(x_3)&=\sum_{x_4}f_3(x_3,x_4),\\ m_{2\to1}(x_2)&=\sum_{x_3}f_2(x_2,x_3)m_{3\to2}(x_3),\\ P(x_1)&\propto\sum_{x_2}f_1(x_1,x_2)m_{2\to1}(x_2). \end{aligned} \]
We push the summations as far inside as possible, summing out the innermost variables first and passing the result outward as a ‘message’. This is the mathematical essence of the sum-product message-passing algorithm.
Use local pre-adjusted daily prices for Jiangsu Hengrui Pharmaceuticals (600276.XSHG) to estimate the empirical probability of a next-day decline after observing today’s return direction and volume state.
from pathlib import Path # Locate the downloaded data file
import pandas as pd # Read HDF5 and Calculate Discrete Conditional Probability Table
# Public download: https://assets.qiufei.site/data/stock/stock_price_pre_adjusted.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/stock/stock_price_pre_adjusted.h5")
# Windows: Path(r"C:\qiufei\data\stock\stock_price_pre_adjusted.h5")
# macOS: Path("/Users/your_name/data/stock/stock_price_pre_adjusted.h5")
# Linux: Path("/home/your_name/data/stock/stock_price_pre_adjusted.h5")
price_path = Path("/home/ubuntu/r2_data_mount/data/stock/stock_price_pre_adjusted.h5")
price_columns = ['close', 'volume'] # Only Select Fields Required for This Exercise to Reduce Memory
hengrui_prices = pd.read_hdf(price_path, key='data', where='order_book_id == "600276.XSHG"', columns=price_columns).reset_index() # Select Hengrui Medicine on behalf of the company in the Yangtze River Delta
hengrui_prices['date'] = pd.to_datetime(hengrui_prices['date']) # Unifying Dealing Day Types
hengrui_prices = hengrui_prices.query("'2015-01-01' <= date <= '2024-12-31'").sort_values('date') # Fixed Sample Period and Maintained Timing
today_return = hengrui_prices['close'].pct_change() # Keep first day unavailable gains as missing
rolling_volume_median = hengrui_prices['volume'].rolling(20).median() # Retain warm-up window as missing first
next_return = today_return.shift(-1) # Direct Alignment of next-trading-day return and Retention of End-of-Day Missing
hengrui_prices['today_down'] = today_return.lt(0).where(today_return.notna()).astype('Int64') # Discretize only when benefits are available
hengrui_prices['volume_high'] = hengrui_prices['volume'].gt(rolling_volume_median).where(rolling_volume_median.notna()).astype('Int64') # Only Discretize at full 20 day window
hengrui_prices['next_day_down'] = next_return.lt(0).where(next_return.notna()).astype('Int64') # Construct results only when next-day returns are available
observational_table = hengrui_prices.dropna().groupby(['today_down', 'volume_high'])['next_day_down'].agg(['mean', 'size']) # Calculate Empirical Condition Probability and Sample Size
display(observational_table) # Demonstrate All Condition Combinations without Prefilling Run Values| mean | size | ||
|---|---|---|---|
| today_down | volume_high | ||
| 0 | 0 | 0.505942 | 589 |
| 1 | 0.496933 | 652 | |
| 1 | 0 | 0.465008 | 643 |
| 1 | 0.472486 | 527 |
mean estimates \(P(\text{next down}\mid\text{today state, volume state})\).
It is not \(P(\text{next down}\mid do(\text{volume high}))\), because news, volatility, and liquidity may be common causes that have not been identified.
(today_down, volume_high)=(0,0),(0,1),(1,0),(1,1), next-day downside rates are 0.5059, 0.4969, 0.4650, and 0.4725, with n=589, 652, 643, and 527.item 1 is yes; item 2 is separated before observation, while observing B or a descendant opens the path.
If either was wrong, return to the three basic structures and mark collider status path by path.
Repeat the table for another Yangtze River Delta non-financial firm.
Data: report the file, key, and date; construct states without looking ahead.
Evidence: show the full conditional table and use d-separation correctly.
Boundary: explain why cross-firm differences cannot be attributed causally to volume.
from pathlib import Path # Locate the downloaded data file
transfer_code = '600104.XSHG' # Choose Shanghai Automobile Group as a non-financial migration company in the Yangtze River Delta
# Public download: https://assets.qiufei.site/data/stock/stock_price_pre_adjusted.h5
# After downloading, change the next line to the file's actual location on your device.
# Course-relative option: Path("data/stock/stock_price_pre_adjusted.h5")
# Windows: Path(r"C:\qiufei\data\stock\stock_price_pre_adjusted.h5")
# macOS: Path("/Users/your_name/data/stock/stock_price_pre_adjusted.h5")
# Linux: Path("/home/your_name/data/stock/stock_price_pre_adjusted.h5")
price_path = Path("/home/ubuntu/r2_data_mount/data/stock/stock_price_pre_adjusted.h5")
transfer_prices = pd.read_hdf(price_path, key='data', where=f'order_book_id == "{transfer_code}"', columns=price_columns).reset_index() # Select select the same minimum field
transfer_prices['date'] = pd.to_datetime(transfer_prices['date']) # Unifying Dealing Day Types
transfer_prices = transfer_prices.query("'2015-01-01' <= date <= '2024-12-31'").sort_values('date') # Fixed Same Sample Period as Demonstration
transfer_return = transfer_prices['close'].pct_change() # Retain first day unavailable returns as missing
transfer_volume_median = transfer_prices['volume'].rolling(20).median() # Keep warm-up window for missing
transfer_next_return = transfer_return.shift(-1) # Align next-trading-day returns and Preserve End-of-Day Missing
transfer_prices['today_down'] = transfer_return.lt(0).where(transfer_return.notna()).astype('Int64') # Create the daily-return indicator only when the return is observed
transfer_prices['volume_high'] = transfer_prices['volume'].gt(transfer_volume_median).where(transfer_volume_median.notna()).astype('Int64') # Only Discrete Full Scroll Window
transfer_prices['next_day_down'] = transfer_next_return.lt(0).where(transfer_next_return.notna()).astype('Int64') # Create the next-day indicator only when its return is observed
transfer_table = transfer_prices.dropna().groupby(['today_down', 'volume_high'])['next_day_down'].agg(['mean', 'size']) # Calculate Four Empirical Condition Units
display(transfer_table.round(4)) # Demonstrate Full Reference Output| mean | size | ||
|---|---|---|---|
| today_down | volume_high | ||
| 0 | 0 | 0.5044 | 563 |
| 1 | 0.5046 | 656 | |
| 1 | 0 | 0.4817 | 683 |
| 1 | 0.4872 | 509 |
for (today_down, volume_high)=(0,0),(0,1),(1,0),(1,1), next-day downside rates are 0.5044, 0.5046, 0.4817, and 0.4872, with n=563, 656, 683, and 509.
High- versus low-volume differences are small.
Even a larger gap would remain observational because news, volatility, and liquidity may affect both volume and next-day returns.
stock_price_pre_adjusted.h5 / data; the empirical table is generated by slide code.do(·) and causal identification.Questions & Discussion