11 Reinforcement Learning - Optimal Decision-Making Under Uncertainty
The Core Question: How Do Machines ‘Learn’ Optimal Decisions?
How can a machine, operating in a complex and uncertain world, automatically learn an optimal sequence of decisions through interaction and trial-and-error to achieve its long-term goals?
This is not just a technical question; it touches the core of economics, including rational decision-making, dynamic optimization, and game theory.
The Concept of an ‘Agent’ in Economics
In economics, we constantly study ‘agents’—consumers, firms, governments—and how they make choices to maximize their utility or profit.
Reinforcement learning provides us with a novel, computational framework to simulate and optimize the decision-making processes of these agents, especially under conditions of incomplete information and dynamic environments.
A Business Case: Dynamic Pricing
Imagine you are a pricing algorithm (an agent) for an e-commerce platform.
State: Current inventory, competitor prices, user traffic, time of day…
Goal: Find a pricing Policy that maximizes total revenue over the next 30 days.
You don’t have a dataset of ‘correct answers’. You can only learn by trying different prices and observing the outcomes. This is the home ground of reinforcement learning.
This Chapter’s Goal: Build Economic Intuition for RL
By the end of this lecture, you can:
Express a sequential economic decision using states, actions, rewards, transitions, and discounting.
Compute one Bellman-expectation, Q-learning, and terminal-state update by hand.
Distinguish the on-policy Sarsa target from off-policy Q-learning.
Compute standard DQN and Double DQN targets term by term and explain the target network.
State the assumptions of the teaching environment and offline-evaluation risks when transferring RL to Chinese financial data.
Before You Begin
If two rewards are 2 and 4 with γ=0.5, what is the return?
Should a terminal transition bootstrap the next-state value?
If behavior and target policies differ, is learning on-policy or off-policy?
Write all three answers before revealing.
Reveal and remediation
\(2+0.5\times4=4\); no, the terminal mask is zero; off-policy. If an item was wrong, return respectively to discounted return, terminal updates, or on/off-policy definitions.
90-Minute Main lesson and Optional Topics
0–15 min: MDP tuple, return, and discount; complete the before-you-begin review.
15–38 min: state value, action value, and a Bellman-expectation calculation.
38–62 min: expectation to max, Sarsa versus Q-learning, and one hand update.
62–80 min: DQN, target network, and the uses and limits of replay.
80–90 min: local offline-RL support check and lesson review.
Optional topics: the cliff-walking visual, Dueling DQN, and application outlook.
The Three Families of Machine Learning
Machine learning is typically categorized into three main paradigms based on the learning style and data format.
Supervised Learning
Learns a mapping function from inputs to outputs, like a teacher providing correct answers.
Core Task: Prediction or Classification.
Data Format: Labeled data (X, y), where y is the correct answer.
Economics Analogy:
Regression: Predicting next year’s GDP based on historical data (GDP, interest rates, inflation).
Classification: Determining if a customer will default on a loan based on their information.
Limitation: Relies on large amounts of high-quality labeled data; cannot make sequential decisions.
Unsupervised Learning
Discovers hidden structures or patterns in data without any correct answers.
Core Task: Discovering data structure.
Data Format: Unlabeled data X.
Economics Analogy:
Clustering: Segmenting customers into different market segments.
Dimensionality Reduction: Extracting a few key factors from hundreds of macroeconomic indicators.
Limitation: Can only describe data, not guide decisions.
Reinforcement Learning
An agent learns how to make a sequence of decisions through ‘trial and error’ interaction with an environment to maximize cumulative reward.
Core Task: Learning an optimal sequence of decisions.
Data Format: Interaction data (state, action, reward).
Economics Analogy:
A firm continuously adjusts its advertising, R&D, and pricing strategies over multiple quarters to maximize its long-term market share and profit.
The Essence of RL: No ‘Correct Answer’
The most fundamental difference from supervised learning is that there is no predefined correct action in any given situation.
The Essence of RL: Delayed Feedback
Today’s action might have consequences that only become apparent far into the future. The feedback signal (reward) is often delayed.
The Essence of RL: The Explore vs. Exploit Trade-off
An agent must decide: should it exploit what it already knows to get a good reward, or should it explore new actions to potentially discover even better rewards?
Core Framework: Markov Decision Process (MDP)
Reinforcement learning problems are typically modeled mathematically as a Markov Decision Process (MDP).
This is a mathematical framework for modeling sequential interactions between a decision-maker (the agent) and an environment. Understanding the MDP is the foundation for understanding all of RL.
It consists of five core components: (S, A, P, R, γ).
The Protagonists of an MDP: Agent and Environment
MDP Component 1: State (S)
A State (\(s \in S\)) is a current description of the environment at a particular moment in time.
Definition: A set of information describing the current situation of the world, sufficient for making future decisions.
Key Property (Markov Property): The future depends only on the present state, not on the sequence of events that preceded it.
A company’s state could be (cash flow, inventory level, market share).
An economy’s state could be (GDP growth, inflation rate, unemployment rate).
MDP Component 2: Action (A)
An Action (\(a \in A\)) is an operation that the agent can perform.
Definition: The set of behaviors the agent can choose from in each state.
Types: Can be discrete (e.g., Buy, Sell, Hold) or continuous (e.g., set price to $10.53).
Economics Examples:
A company’s actions could be (set product price, determine advertising budget).
A central bank’s actions could be (raise rates by 25 bps, lower rates, maintain).
MDP Component 3: Reward (R)
A Reward (\(R_t\)) is the immediate feedback signal the environment gives to the agent after it takes action \(A_t\) in state \(S_t\).
Definition: A scalar value that measures the ‘goodness’ of an action. The Reward Hypothesis states that all goals can be described as the maximization of expected cumulative reward.
RL’s Goal: To maximize cumulative reward, not instantaneous reward.
Economics Examples:
For a trading algorithm, the reward is the daily portfolio return.
For a company, the reward is the quarterly profit.
MDP Component 4: Transition Probability (P)
State Transition Probability describes the dynamics of the environment.
Definition: The probability of transitioning to the next state \(s'\) after taking action \(a\) in state \(s\).
Mathematical Notation:
\[
\large{ p(s' | s, a) = P(S_{t+1} = s' | S_t = s, A_t = a) }
\]
Economics Example:
Given a current market share of 20% (s), if the company invests $1M in advertising (a), what is the probability that the market share will grow to 22% (s') next quarter? This is often stochastic.
MDP Component 5: Discount Factor (\(\gamma\))
The Discount Factor (\(\gamma\)) reflects the agent’s preference for present rewards over future rewards. We will discuss this in more detail shortly. It is a value between 0 and 1.
Summary: The MDP Quintuple
A complete Markov Decision Process is defined by a quintuple: \((\mathcal{S}, \mathcal{A}, P, R, \gamma)\)
The Interaction Loop: The Agent-Environment ‘Dance’
The core process of reinforcement learning is a continuous loop of interaction.
The Interaction Loop: A Step-by-Step Breakdown
Observe: At time t, the agent observes the environment’s state \(S_t\).
(e.g., A trading algorithm observes the current stock price is $100 and volume is 500k shares.)
Decide: The agent selects an action \(A_t\) based on its policy \(\pi\).
(e.g., The algorithm decides to ‘Buy 100 shares’.)
Execute: The environment receives the action \(A_t\).
Evolve: The environment transitions to a new state \(S_{t+1}\) and provides a reward \(R_{t+1}\) based on its internal dynamics.
(e.g., The market executes the trade, the price moves to $101, and the day’s unrealized profit is +$100.)
This loop repeats.
The Agent’s Brain: The Policy Function (\(\pi\))
The Policy (\(\pi\)) is the agent’s code of conduct, its ‘brain’. It defines a mapping from states to actions.
Definition: A mapping from states to actions. It tells the agent what to do in each state.
Types:
Deterministic Policy: \(a = \pi(s)\), a unique action for each state.
Stochastic Policy: \(\pi(a|s) = P(A_t = a | S_t = s)\), the probability of taking action \(a\) in state \(s\).
RL’s Goal: To find an optimal policy \(\pi^*\) that maximizes long-term return.
The Economic Intuition of a Stochastic Policy
Why might we need a policy that is random?
To Explore the Unknown
A deterministic policy might get stuck in a rut, missing out on better options forever.
Randomness allows the agent to explore new, untried actions.
This is like a company occasionally trying a completely new marketing channel.
To Counter Adversaries
In some situations, the optimal behavior is inherently random. This is common in game theory; the optimal strategy for ‘Rock, Paper, Scissors’ is to choose randomly, making you unpredictable to your opponent.
The Objective Function: Maximize Cumulative Return
The agent’s goal is not to maximize the immediate reward \(R_{t+1}\), but to maximize the future cumulative return starting from the current moment.
The \(\gamma\) (gamma) in the formula is the discount factor, where \(0 \le \gamma \le 1\).
Economic Analogy:
If each step has a fixed duration and rewards are period cash flows, \(\gamma\) can resemble \(1/(1+r)\).
In general RL it defines the task’s weighting of distant rewards and is not identical to NPV.
Why do we need to discount?
Time Value of Money: A dollar today is worth more than a dollar tomorrow.
Task Time Preference: \(\gamma\) specifies the relative weight of near and distant rewards; represent model uncertainty separately through states, transitions, or robustness analysis.
Mathematical Condition: With bounded rewards and \(0\le\gamma<1\), the infinite discounted return converges absolutely; \(\gamma=1\) needs additional conditions such as termination.
The Effect of the Discount Factor: An Example
Suppose the reward sequence is [+10, +2, +5, +8, ...]
\(\gamma = 0\) (Extremely Myopic)
\(G_t = 10 + 0 \cdot 2 + 0 \cdot 5 + \dots = 10\) Only cares about the immediate reward.
The choice of \(\gamma\) is a key modeling decision that defines the agent’s degree of ‘foresight’.
The Yardstick for a Good Policy: The Value Function
How do we determine if a state is ‘good’ or ‘bad’? Or if an action is ‘good’ or ‘bad’?
This is the role of the Value Function. It is the Expectation of future returns.
There are two core types of value functions:
State-Value Function (\(V_\pi(s)\))
Action-Value Function (\(Q_\pi(s, a)\))
State-Value Function V(s): How Good is My Current Situation?
The State-Value Function \(V_\pi(s)\) answers the question: ‘If I start from state \(s\) and always follow policy \(\pi\), how much total future return can I expect to get?’
Economics Example: Given the current macroeconomic conditions (state \(s\)), if the Federal Reserve follows its current interest rate policy (policy \(\pi\)), what is the expected value of future total economic output (return)?
Action-Value Function Q(s, a): How Good is This Choice?
Starting point: the agent is in state \(s\) and first chooses action \(a\).
Continuation rule: it follows policy \(\pi\) thereafter.
Quantity answered:\(Q_\pi(s,a)\) is the expected total future return under that choice and continuation.
Intuition: \(Q_\pi(s, a)\) is an estimate of the Quality of taking a specific action in a specific state.
Economics Example: Given the current inventory level (state \(s\)), if the company chooses to cut prices by 10% (action \(a\)), what is the expected value of the company’s future total profit (return)?
The Relationship Between V and Q Functions
The value of a state is the expected value of all possible actions that could be taken from that state, weighted by the policy’s probability of choosing them.
\(\pi(a|s)\) is the probability of choosing action \(a\) in state \(s\).
\(Q_\pi(s, a)\) is the value after choosing action \(a\).
This relationship is very intuitive and crucial for later algorithms.
The Core of Dynamic Programming: The Bellman Expectation Equation
The Bellman Equation is the most fundamental equation in reinforcement learning. It establishes a recursive relationship between the value of a state and the values of its successor states.
In words: The value of the current state = the expectation over all possible actions of the (immediate reward + the discounted expected value of the next state).
Decomposing the Bellman Expectation Equation (V-function)
Let’s break down the equation:
\[ \large{V_\pi(s) = \sum_{a \in A} \pi(a|s) \Big( \underbrace{R(s,a)}_{\text{What I get now}} + \gamma \underbrace{\sum_{s' \in S} p(s'|s,a) V_\pi(s')}_{\text{What I can expect later}} \Big)} \]
Outer Expectation \(\sum_{a \in A} \pi(a|s) \dots\): Because our policy might be stochastic, we must average over all possible actions.
Immediate Reward \(R(s,a)\): The reward received immediately after taking action \(a\).
Inner Expectation \(\sum_{s' \in S} p(s'|s,a) \dots\): Because the environment’s response (state transition) might be stochastic, we must average over all possible next states.
\(V_\pi(s')\): This is the magic of recursion. The value of the next state can be expanded in the same way.
Bellman Expectation Equation (Q-function)
For the action-value function \(Q_\pi(s, a)\), the Bellman equation has a slightly different form:
Solving the RL Problem: Finding the Optimal Policy \(\pi^*\)
The ultimate goal of reinforcement learning is to find an optimal policy \(\pi^*\) that achieves a higher or equal expected return than any other policy from any initial state.
\[
\large{ \pi^* = \arg\max_{\pi} V_\pi(s) \quad \text{for all } s \in S }
\]
The corresponding optimal value functions are denoted \(V^*(s)\) and \(Q^*(s, a)\).
The Bellman Optimality Equation
For an optimal policy, the Bellman equation takes a special form: the Bellman Optimality Equation. It no longer involves an expectation over actions but instead takes the maximum.
The max operator embodies optimality: the agent will always choose the action that leads to the best possible future.
How to Solve? Generalized Policy Iteration (GPI)
We know the target (the Bellman Optimality Equation), but how do we find the solution? Most RL algorithms follow a common pattern called Generalized Policy Iteration (GPI).
This is an iterative process involving two intertwined steps:
Policy Evaluation: How Good is My Current Policy?
Suppose we have a fixed policy \(\pi\) (e.g., an existing set of trading rules). We want to know its value.
Task: Compute \(V_\pi(s)\).
Method: Use the Bellman Expectation Equation and solve it iteratively.
We start with a random \(V_0\) and repeatedly use the old value function to compute the new one until \(V_k\) converges. This process is called iterative policy evaluation.
Policy Improvement: Can I Do Better?
Once we know the value function \(V_\pi\) for our current policy \(\pi\), we can try to improve it.
Method: For each state \(s\), instead of following \(\pi\), we act ‘greedily’ by choosing the action that leads to the highest Q-value:
For a finite MDP with exact policy evaluation and comparable action values, the new policy \(\pi'\) is no worse than \(\pi\).
This is the Policy Improvement Theorem; offline financial data without action support do not inherit that guarantee.
Algorithm Classification: Do We Need a Model of the Environment?
Algorithms that solve this GPI loop can be divided into two main categories:
Model-Based
The algorithm learns and/or uses an explicit transition-and-reward model for planning. The model may be given or learned, and it may be approximate.
Special Case: Dynamic programming with a known exact model.
Advantage: High data efficiency.
Limitation: learned models can be wrong and planning can amplify model bias; exact transition models are rarely available.
Model-Free
The algorithm does not need a model of the environment. It learns directly from samples (experience) generated by interacting with the environment.
Typical Algorithms: Q-Learning, Sarsa
Advantage: Much more applicable to complex, real-world problems.
Model-Free Methods: A Metaphor for Exploration
Model-Free Algorithm Classification: How Do We Learn?
Within model-free algorithms, we can further classify them based on their learning style:
On-policy
The agent learns from and improves the same policy it uses to make decisions.
Analogy: A novice driver learns to improve their own driving skills while they are actively driving.
Typical Algorithm: Sarsa.
Off-policy
The agent can learn from a policy that is different from the one it is currently using to explore. It can learn a target (greedy) policy while behaving according to an exploratory policy.
Analogy: An expert learns how to be a perfect driver by watching videos of a novice driver’s mistakes.
Typical Algorithm: Q-Learning.
Model-Free Method 1: Monte Carlo (MC)
MC methods are the most intuitive model-free learning approach.
Core Idea: Estimate the value function by simulating a large number of complete episodes. An episode is a full trajectory from a starting state to a terminal state.
How to estimate \(Q_\pi(s,a)\)?
Follow policy \(\pi\) and play many episodes.
For every episode that visited the pair \((s,a)\), record the actual return \(G_t\) that followed.
Average these returns to estimate \(Q_\pi(s,a)\).
\[
\large{ Q(s,a) \leftarrow \text{Average}(G_t \text{ for all visits to } (s,a)) }
\]
Limitation:
Full-return MC waits for the end of an episode, so updates are delayed and can have high variance.
Continuing tasks can still use truncation, regenerative states, or average-reward constructions; MC is not restricted to naturally episodic tasks.
Model-Free Method 2: Temporal-Difference (TD)
TD learning is one of the most central and innovative ideas in reinforcement learning. It combines the advantages of MC and dynamic programming.
Core Idea: Learn from every single step, without waiting for the episode to end. It updates the value of the current state using an estimate of the next state’s value.
Comparison with MC:
MC’s update target is the actual final return\(G_t\).
TD’s update target is an estimated future return\(R_{t+1} + \gamma V(S_{t+1})\).
This process of updating an estimate with another estimate is called bootstrapping.
The TD(0) Algorithm’s Update Rule
The simplest TD algorithm, TD(0), has the following update rule:
\(\alpha\): The learning rate, which determines how much we update our estimate based on the new information.
TD Target (\(R_{t+1} + \gamma V(S_{t+1})\)): A bootstrapped target combining one observed reward with the current next-state estimate; it is not guaranteed to be more accurate than \(V(S_t)\).
TD Error (\(\delta_t\)): The difference between the current estimate and that bootstrapped target, used for an incremental update; finite-sample bias and noise can remain.
A Vivid Example: Walking from Dorm to Classroom
Imagine you want to predict how long it takes to walk from your dorm to the classroom. Your initial guess is 18 minutes.
MC Method: Wait for the episode’s final return, then update the starting-state value.
TD Method: After each observation, update from the current reward and next-state estimate.
On-Policy TD Control Algorithm: Sarsa
Sarsa is an on-policy TD control algorithm that aims to learn the Q-function.
Name Origin: Its update rule requires a quintuple \((S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})\), which spells out S, A, R, S’, A’.
Key point: the update uses the action \(A_{t+1}\)actually taken in \(S_{t+1}\). Here \(d_t=1\) marks a terminal transition, so \((1-d_t)\) sets the bootstrap term to zero.
The Exploration-Exploitation Dilemma: \(\epsilon\)-greedy Policy
To retain exploration while exploiting the current best action, we typically replace a purely greedy policy with an \(\epsilon\)-greedy policy.
With probability \(1-\epsilon\), choose the best-estimated action: \(a^* = \arg\max_a Q(s,a)\). (Exploit)
With probability \(\epsilon\), choose an action randomly from all possible actions. (Explore)
The value of \(\epsilon\) is often decayed over time. Claims of sufficient state-action exploration additionally require reachability, persistent visitation, and an appropriate decay schedule; \(\epsilon\)-greedy alone gives no unconditional guarantee.
Off-Policy TD Control Algorithm: Q-Learning
Q-Learning is one of the most famous and widely used algorithms in reinforcement learning. It is an off-policy algorithm.
Core Idea: It follows a behavior policy (e.g., \(\epsilon\)-greedy) to explore, while simultaneously learning the Q-values of a different target policy (the purely greedy policy).
Key difference: a nonterminal transition uses \(\max_{a'} Q(S_{t+1},a')\) regardless of the realized next action; when \(d_t=1\), Q-learning does not bootstrap from the terminal state.
Sarsa vs. Q-Learning: A Cliff Walking Analogy
Imagine an agent walking along the edge of a cliff, trying to get from the start (S) to the goal (G). Falling off the cliff results in a large negative reward.
Sarsa vs. Q-Learning: Interpretation
Sarsa (On-policy):
It’s a ‘cautious’ learner.
Because it knows its next action is also exploratory (and might accidentally step off the cliff), it factors this risk into its Q-value updates.
It learns a safer, albeit slower, path that stays far from the cliff.
Q-Learning (Off-policy):
It’s a ‘bold’ learner.
It always assumes the optimal action will be taken next, regardless of the fact that it might make mistakes while exploring.
It learns the fastest path right along the cliff edge, because it believes it will eventually execute that path perfectly.
Feature
Sarsa (On-policy)
Q-Learning (Off-policy)
Objective
Learns the value of its current behavior policy
Learns the value of the optimal policy
Update
Uses the Q-value of the next actual action
Uses the Q-value of the next possible best action
Behavior
More conservative, avoids risky shortcuts
More aggressive, favors optimal but risky paths
Python in Practice: Solving the ‘FrozenLake’ Problem with Q-Learning
To make these concepts concrete, let’s look at a classic RL problem: FrozenLake.
Environment: A 4x4 grid, with some tiles being safe frozen ice (F) and others being holes (H).
Goal: Navigate from the start (S) to the goal (G) without falling into a hole.
States: 16 grid positions.
Actions: Up, Down, Left, Right.
Reward: +1 for reaching the goal, 0 otherwise.
Challenge: The ice is slippery. An action to move ‘right’ might have some probability of sliding somewhere else.
This is a classic MDP.
Q-Learning Code Framework: Initialization
We need a Q-table to store the value of each (state, action) pair. Initially, we know nothing, so we set all values to zero.
reward reaches the start and the verified path is 0 → 4 → 8 → 9 → 13 → 14 → 15. Terminal masking blocks cross-episode bootstrap; this deterministic execution check is not evidence for slippery, continuous, or market settings.
From Tables to Reality: The Curse of Dimensionality
The tabular method (Q-table) we just discussed works well for problems with small state and action spaces.
But what about real-world economic problems?
The state space of Chess is approximately \(10^{47}\).
The state space of Go is approximately \(10^{170}\).
The state of a self-driving car is continuous (position, velocity, orientation), making the state space infinite.
It is impossible to create a Q-table for these problems. This is where Deep Reinforcement Learning (DRL) comes in.
Deep Reinforcement Learning: Neural Networks for RL
The core idea of DRL is to use a deep neural network to approximate the value function or policy function, instead of a table.
Value Network: Takes a state \(s\) as input and outputs its value \(V(s)\), or the Q-value for each action \(Q(s,a)\).
\[
\large{ Q(s, a; \mathbf{w}) \approx Q^*(s, a) }
\]
Here, \(\mathbf{w}\) represents the weights of the neural network.
Policy Network: Takes a state \(s\) as input and outputs the probability of taking each action, \(\pi(a|s)\).
major advance: The Deep Q-Network (DQN)
The 2013 DQN preprint evaluated seven Atari games from raw pixels and exceeded the compared human expert on three.
The 2015 Nature study expanded the evaluation to 49 games and reported aggregate performance comparable to a professional human games tester; these scopes and claims should not be conflated.
The core of DQN is using a deep convolutional neural network (CNN) to approximate the optimal action-value function \(Q^*(s,a)\).
DQN’s Loss Function: How to Train the Network?
How do we train this Q-network? We want the network’s prediction \(Q(s, a; \mathbf{w})\) to be as close as possible to the ‘target value’ given by the Bellman Optimality Equation.
Let \(\theta\) denote online-network parameters, \(\theta^-\) frozen target-network parameters, and \(d_t\in\{0,1\}\) the terminal indicator. Standard target-network DQN uses:
We can then use gradient descent to optimize the network weights \(\mathbf{w}\).
Experience Replay Reduces Correlation
DQN stabilizes training by storing transitions and sampling them later.
Effect: replay reduces adjacent-transition correlation and improves reuse; a changing-policy buffer still has stale distributions and incomplete state-action coverage.
A Frozen Target Network Stabilizes Bootstrap Targets
If the target changes with the online parameters on every update, learning chases a moving target. DQN uses θ for the current prediction and a temporarily frozen θ⁻ for the bootstrap target.
Effect: The target network w- is ‘frozen’ for a period, making the learning target much more stable.
The Overestimation Problem in DQN
Standard DQN uses max for action selection and evaluation; with noisy estimates, selecting the maximum creates upward selection bias.
If the Q-value estimates themselves have noise, taking the maximum will amplify positive noise, leading to an optimistic bias. This bias can propagate and accumulate through the bootstrapping process.
Solution: Double DQN (DDQN)
DDQN addresses the overestimation problem by decoupling ‘action selection’ from ‘value evaluation’.
This decoupling is designed to reduce DQN’s maximization-induced overestimation bias. It does not guarantee conservative or more accurate estimates in every state or task; underestimation and accuracy must be evaluated empirically.
Numerical Check: Compute DQN and DDQN Targets First
Let \(R_{t+1}=1, \gamma=0.9, d_t=0\).
At the next state, the online network estimates actions A/B as \((5,4)\), while the target network estimates \((3,6)\).
Write both targets first, then decide whether a transition with \(d_t=1\) still bootstraps.
DQN takes the target-network maximum 6, so \(Y^{DQN}=1+0.9\times6=6.4\).
DDQN selects A with the online network and evaluates A as 3 with the target network, so \(Y^{DDQN}=1+0.9\times3=3.7\).
If \(d_t=1\), both equal immediate reward 1; the terminal mask prevents bootstrap.
Remediation
If DQN used the online-network maximum, revisit the target-network formula;
This architecture allows the network to learn the intrinsic value of states more efficiently, especially in situations where many actions have similar values.
Formative Check: Terminal State and Policy Semantics
Question 1： A transition yields −2 and terminates; γ=0.99 and the next-state network maximum is 100. What is the correct target?
Question 2： Sarsa uses the realized next action while Q-learning uses the maximizing action. Which is on-policy?
Reveal and remediation:
item 1 is −2 because the terminal mask removes bootstrap; item 2 is Sarsa on-policy and Q-learning off-policy.
If you wrote 97, revisit terminal transitions; if policies were reversed, revisit the two targets.
Step-by-Step Exercise: Compute One DDQN Target
Task： (R=0.5,γ=0.8,d=0); online A/B/C values are (2,5,4), target values are (6,1,3). Compute both targets.
Complete answer:
DQN takes the target-network maximum 6: \(0.5+0.8\times6=5.3\).
DDQN selects B online and evaluates B as 1 with the target network: \(0.5+0.8\times1=1.3\).
Apply It to a New Case: Offline RL with a Local Chinese Index
Task:
Using local hs300_index_only.h5 / hs300, design a monthly position teaching environment: month-end available states, actions {cash, hold}, and reward equal to next-month portfolio return minus transaction costs.
report the file, key and sample, construct states without looking ahead, include costs in the reward, explain behavior-policy coverage risk, and propose rolling out-of-sample evaluation with baselines.
Never present backtest returns as a live-performance guarantee.
Study note： This review completes the main lesson; study the Dueling DQN section afterward if time permits.
Apply It to a New Case: Executable Reference
Code
from pathlib import Path # Locate Local CSI 300 Minimum Filesimport numpy as np # Calculate return and volatility statesimport pandas as pd # Read and Construct Offline Transfer Table# Public download: https://assets.qiufei.site/data/index/hs300_index_only.h5# After downloading, change the next line to the file's actual location on your device.# Course-relative option: Path("data/index/hs300_index_only.h5")# Windows: Path(r"C:\qiufei\data\index\hs300_index_only.h5")# macOS: Path("/Users/your_name/data/index/hs300_index_only.h5")# Linux: Path("/home/your_name/data/index/hs300_index_only.h5")index_path = Path("/home/ubuntu/r2_data_mount/data/index/hs300_index_only.h5")daily_index = pd.read_hdf(index_path, key='hs300') # Reading real China index quotesdaily_index['date'] = pd.to_datetime(daily_index['datetime'].astype(str), format='%Y%m%d%H%M%S')monthly_close = daily_index.query("'2005-01-01' <= date <= '2024-12-31'").set_index('date')['close'].resample('ME').last().to_frame() # Fixed Sample Period and Aggregated Monthlymonthly_close['return_1m'] = monthly_close['close'].pct_change() # Construct the monthly returnmonthly_close['momentum_3m'] = monthly_close['close'].pct_change(3) # Construct Monthly End Visible Momentummonthly_close['volatility_3m'] = monthly_close['return_1m'].rolling(3).std() # Compute rolling three-month volatilitymonthly_close['high_vol'] = monthly_close['volatility_3m'].gt(monthly_close['volatility_3m'].expanding().median()).astype(int) # Use only the historical extended median discretizationmonthly_close['positive_momentum'] = monthly_close['momentum_3m'].gt(0).astype(int) # Create a no-look-ahead momentum statemonthly_close['state'] =2* monthly_close['high_vol'] + monthly_close['positive_momentum'] # Encode four discrete statesmonthly_close['behavior_action'] = monthly_close['positive_momentum'] # Clarify demonstration behavioral strategies rather than fictitious real transactionsmonthly_close['next_return'] = monthly_close['return_1m'].shift(-1) # Aligning Next Month Environmental Feedbackmonthly_close['reward'] = monthly_close['behavior_action'] * monthly_close['next_return'] -.001* monthly_close['behavior_action'].diff().abs().fillna(0) # Deduct 10bp Swap Costsmonthly_close['next_state'] = monthly_close['state'].shift(-1) # Aligning Next Stateoffline_batch = monthly_close.dropna().copy() # Form a complete offline transfer samplesupport_table = pd.crosstab(offline_batch['state'], offline_batch['behavior_action']) # check Each State Action Coveragedisplay(support_table) # Show whether counterfactual action values are identifiable
behavior_action
0
1
state
0
73
0
1
0
73
2
39
0
3
0
51
Apply It to a New Case: Evaluation and Support decision rule
Complete reference output:
236 transitions span 2005-04 to 2024-11.
Action counts by state are (73,0), (0,73), (39,0), and (0,51).
Every state lacks one action, so the deterministic behavior policy provides no counterfactual support.
The correct conclusion is do not estimate a policy-improvement return from this batch; obtain broader behavior coverage and use conservative offline RL plus rolling baselines.
Apply It to a New Case: Chronological Evaluation
Chronological evaluation:
use rolling-origin splits.
In each fold, fit only on earlier months, choose hyperparameters and a support threshold on the immediately following validation window, then evaluate once on the later test window before rolling forward.
Report cash-only, always-hold, and the observed behavior policy as cost-matched baselines in every test window.
Because every state currently lacks one action, an alternative-policy value cannot be estimated from these data; the result here is “not identifiable,” not an invented backtest return.
Sources and Further Reading
Sutton, R. S. and Barto, A. G. (2018), Reinforcement Learning: An Introduction, 2nd ed.; adjacent support for Bellman equations, TD, and convergence conditions.
Mnih, V. et al. (2015), “Human-level control through deep reinforcement learning,” Nature.
van Hasselt, H., Guez, A., and Silver, D. (2016), “Deep Reinforcement Learning with Double Q-learning,” AAAI.
FrozenLake is explicitly a teaching simulation; the transfer uses local real HS300 prices without inventing backtest results.
Conclusion: RL Provides a New Paradigm for Economic Decision-Making
Core Framework: MDP provides a unified mathematical language for sequential decision problems.
Core Idea: convergence requires conditions such as a finite MDP, sufficient exploration, suitable learning rates, and adequate value representation; deep RL has no unconditional optimal-convergence guarantee.
Core Algorithms: From Q-Learning to DQN and its variants, algorithmic advancements are making it possible to solve increasingly complex problems.
Economic Significance: RL is more than just a tool for game AI; it offers a powerful, data-driven toolkit for dynamic pricing, resource management, portfolio optimization, auction design, and many other fields in economics.
Future Directions: RL’s Application in Economics
Market Modeling: Simulating markets with multiple RL agents to study the emergence of market equilibria and complex dynamics.
Algorithmic Game Theory: Designing agents that perform optimally in competitive or cooperative environments, such as auctions or supply chain negotiations.
Personalized Policies: Developing dynamic, individualized strategies for each user (e.g., in e-commerce recommendations) or each asset (e.g., in a portfolio).
Reinforcement learning is evolving from a subfield of computer science into a general framework for understanding and optimizing intelligent decision-making in complex systems.