03 Bayesian Classifiers

Welcome to Chapter 3: Bayesian Classifiers

The Core Question: How to Make Optimal Classifications Under Uncertainty?

Imagine you are a credit manager at a bank, facing a critical decision every day:

Should this loan application be approved?

This links two distinct tasks: classify the applicant’s default state, then choose a lending action using the loss matrix.

• Input Data (X): Applicant’s features, such as annual income, credit score, debt level, etc.
• True State (\(Y\)): No Default or Default.
• Model Output: the posterior \(P(Y=\text{Default}\mid X)\) (or a predicted label \(\hat Y\)).
• Business Action (\(a\)): Approve or Reject; the posterior and loss matrix jointly determine the action.

Case Study: Credit Approval Decisions

Let’s look at some simplified applicant data.

Objective: Estimate \(P(Y\mid X)\) from applicant features, then select the action \(a^*(X)\) with minimum conditional risk. A class label is not a lending action.

Why Choose the Bayesian Approach?

Bayesian classifiers provide a powerful decision-making framework based on probability theory.

Our Learning Roadmap for This Chapter

We will build a complete Bayesian decision-making system step by step.

Part 1: Foundations of Probability

Core Concept 1: Prior Probability

The Prior Probability \(P(y=c)\) is our inherent belief or the historical frequency of a class c occurring, before observing any new data.

In the credit approval example, the prior probability refers to:

Without looking at any specific information about an applicant, what is the probability that a customer will default, based on the bank’s historical data?

It is the baseline or starting point for our decision.

Prior Probability: A Concrete Example

Suppose the bank has processed 10,000 loan applications in the past, of which 500 ultimately defaulted.

Core Concept 2: Probability Density Function (PDF)

For continuous variables (like income, age), we use a Probability Density Function (p(x)) to describe their distribution.

• The PDF itself is not a probability; p(x) can be greater than 1.
• The probability that a variable falls into an interval [a, b] is the integral of the PDF over that interval: \(P(a \le x \le b) = \int_a^b p(x) dx\).
• The total area under the curve is 1.

PDF (Continuous) vs. PMF (Discrete)

An important distinction:

Visualization: The Normal (Gaussian) Distribution

The Normal distribution is the most common PDF in finance and business, defined by its mean μ (center) and variance σ² (spread).

Core Concept 3: Class-Conditional Probability

The Class-Conditional Probability \(p(x | y=c)\) answers: “If we already know a sample belongs to class c, what is the probability density of observing data x?”

In our credit approval example:

• p(income | y = Default): Given a customer is a defaulter, what does their income distribution look like?
• p(income | y = No Default): Given a customer is a good client, what does their income distribution look like?

This is the bridge connecting our observation (data x) and the unknown state (class y). We call this the Likelihood.

Visualizing Income Distributions for Different Customer Classes

Suppose historical data shows that non-defaulting customers (blue) generally have higher incomes than defaulting customers (red).

Bayes’ Theorem: The Magic Formula from Prior to Posterior

Bayes’ Theorem is the mathematical cornerstone that connects these four core concepts.

\[ \large{P(y=c \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid y=c) P(y=c)}{p(\mathbf{x})}} \]

This formula tells us how to update our beliefs using data.

Visualizing the Components of Bayes’ Theorem

Calculating the ‘Evidence’ p(x): The Law of Total Probability

The denominator p(x) is the total probability of observing the data x, regardless of its class. We calculate it using the Law of Total Probability.

Part 2: Bayesian Decision Theory

With Posterior Probabilities, How Do We Decide?

We can now calculate P(y=Default | x) and P(y=No Default | x).

A seemingly simple and intuitive rule is: Choose the class with the highest posterior probability.

\[ \large{f(\mathbf{x}) = \underset{c}{\arg\max} \ P(y=c \mid \mathbf{x})} \]

This is known as the Maximum a Posteriori (MAP) decision rule.

The Hidden Assumption of the MAP Rule

The MAP rule implies a very strong assumption: the cost of all types of errors is the same.

In the real world, this assumption is often false.

The Costs of Misclassification Are Different: Introducing the Loss Function

In credit approval, there are two types of errors:

1. Type I Error (False Negative): Misclassifying a defaulting customer as ‘No Default’ (approving a bad loan).
   • Consequence: The bank loses the principal, a huge cost.
2. Type II Error (False Positive): Misclassifying a non-defaulting customer as ‘Default’ (rejecting a good customer).
   • Consequence: The bank loses potential interest income, a smaller cost.

Quantifying Costs: The Loss Matrix L(a, y)

We use a Loss Matrix L(a, y) to quantify the cost of taking action \(a\) when the applicant’s true state is \(y\).

Expected Loss: Measuring the Average Cost of a Decision

For a given observation \(x\), the Expected Loss or Conditional Risk \(R(a|x)\) is the posterior-weighted loss of taking action \(a\) over all possible true states \(y\).

\[ \large{R(a \mid \mathbf{x}) = \sum_y L(a, y) P(Y=y \mid \mathbf{x})} \]

This formula calculates: “If I take action \(a\), what average loss will I incur?”

Calculating Expected Loss: A Step-by-Step Example

Suppose for a certain applicant x, we have calculated the posterior probabilities:

• P(y=No Default | x) = 0.8
• P(y=Default | x) = 0.2

And recall our loss matrix (simplified to numbers):

• L(Approve, No Default) = -20 (gain 20k)
• L(Approve, Default) = 500 (lose 500k)
• L(Reject, No Default) = 1 (opportunity cost 1k)
• L(Reject, Default) = 0

Calculating R(Approve | x)

If we choose to ‘Approve’ this loan:

R(Approve | x) = L(Approve, No Default) * P(No Default|x) + L(Approve, Default) * P(Default|x)

\[ \large{R(\text{Approve} \mid \mathbf{x}) = (-20) \times 0.8 + (500) \times 0.2 = -16 + 100 = 84} \]

The expected loss for choosing ‘Approve’ is $84k.

Calculating R(Reject | x)

If we choose to ‘Reject’ this loan:

R(Reject | x) = L(Reject, No Default) * P(No Default|x) + L(Reject, Default) * P(Default|x)

\[ \large{R(\text{Reject} \mid \mathbf{x}) = (1) \times 0.8 + (0) \times 0.2 = 0.8} \]

The expected loss for choosing ‘Reject’ is $0.8k.

The Bayes Decision Rule: Minimize Expected Loss

The optimal action \(a^*(x)\) minimizes the conditional risk \(R(a|x)\) for each applicant.

\[ \large{a^*(\mathbf{x}) = \underset{a}{\arg\min} \ R(a \mid \mathbf{x})} \]

In our example: R(Approve|x) = 84 vs. R(Reject|x) = 0.8. Since 0.8 < 84, the optimal decision is to ‘Reject’.

Even though the applicant has an 80% chance of being a good customer! The 20% risk of default, combined with the high cost, makes rejection the more rational choice.

A Special Case: The 0-1 Loss Function

If the cost of all misclassifications is equal (e.g., 1) and the cost of correct classification is 0, this is the 0-1 loss function.

\[ \large{L(i, j) = \begin{cases} 0, & \text{if } i = j \\ 1, & \text{if } i \neq j \end{cases}} \]

Decision Rule Under 0-1 Loss

Under 0-1 loss, the expected loss R(i|x) simplifies to:

\[ \large{R(i \mid \mathbf{x}) = \sum_{j=1}^{C} L(i, j) P(y=j \mid \mathbf{x}) = \sum_{j \neq i} P(y=j \mid \mathbf{x})} \]

\[ \large{= 1 - P(y=i \mid \mathbf{x})} \]

Minimizing R(i|x) is equivalent to minimizing 1 - P(y=i|x), which means maximizing the posterior probability P(y=i|x).

Therefore, the MAP decision rule is a special case of minimizing expected loss under a 0-1 loss function.

Visualizing the Decision Boundary

The decision rule divides feature space into optimal-action regions. Under 0-1 loss, actions correspond one-to-one with class labels; only in that special case may the regions be called class regions directly.

For the MAP rule, the decision boundary is where the posterior probabilities are equal, e.g., P(y=1|x) = P(y=2|x).

Part 3: Parameter Estimation

The Real-World Challenge: We Don’t Know the True Probability Distributions

So far, we have assumed that P(y=c) and p(x|y=c) are known.

In reality, we don’t know them. All we have is a set of historical data (the training set).

Core Task: How can we estimate the parameters of these probability distributions from the data?

The Parameter Estimation Approach

1. Choose a Model: We first assume the data follows a specific form of probability distribution, such as a Gaussian distribution \(\mathcal{N}(\mu, \sigma^2)\).
2. Estimate Parameters: Our task then becomes estimating the model’s unknown parameters, \(\theta = (\mu, \sigma^2)\), from the data.

Method 1: Maximum Likelihood Estimation (MLE)

The core idea of Maximum Likelihood Estimation (MLE) is:

Choose the set of parameters \(θ\) that maximizes the joint probability of observing the data we have, \(X = {x_1, ..., x_N}\).

In other words, which set of parameters provides the “best explanation” for the data we see?

The Likelihood Function L(θ|X)

We define the likelihood function L(θ|X), which represents the probability of observing data X given parameters θ.

Assuming the samples are independent and identically distributed (i.i.d.), the joint probability is the product of individual sample probabilities:

\[ \large{L(\theta \mid X) = p(X \mid \theta) = \prod_{n=1}^{N} p(x_n \mid \theta)} \]

Our goal is to find the θ* that maximizes L(θ|X).

\[ \large{\theta^*_{MLE} = \underset{\theta}{\arg\max} \ L(\theta \mid X)} \]

MLE Trick: Use the Log-Likelihood Function

Products are difficult to differentiate. We typically maximize the log-likelihood function LL(θ|X) instead.

Since the logarithm is a monotonically increasing function, maximizing L is the same as maximizing log(L).

\[ \large{LL(\theta \mid X) = \log p(X \mid \theta) = \sum_{n=1}^{N} \log p(x_n \mid \theta)} \]

This turns the product into a sum, greatly simplifying the calculation.

The Limitation of MLE: Overfitting

MLE completely “trusts” the data. If the dataset is small or biased, the MLE result can be poor.

Classic example: Tossing a coin.

• You flip a coin 3 times and get Heads, Heads, Heads.
• MLE will estimate the probability of Heads as \(p(H) = 3/3 = 1\).
• This implies you will never predict Tails. This is obviously unreasonable.

We need a way to incorporate our prior knowledge about the world (e.g., that coins are usually fair).

Method 2: Maximum a Posteriori (MAP) Estimation

MAP (Maximum a Posteriori) estimation builds on MLE by introducing a prior distribution p(θ) over the parameters θ themselves.

It no longer maximizes the likelihood p(X|θ), but rather the posterior probability of the parameters p(θ|X).

\[ \large{p(\theta \mid X) \propto p(X \mid \theta) p(\theta)} \]

The goal of MAP is:

\[ \large{\theta^*_{MAP} = \underset{\theta}{\arg\max} \ [p(X \mid \theta) p(\theta)]} \]

The Log-Form of MAP

Again, using logarithms simplifies the calculation:

\[ \large{\theta^*_{MAP} = \underset{\theta}{\arg\max} \ [\log p(X \mid \theta) + \log p(\theta)]} \]

\[ \large{= \underset{\theta}{\arg\max} \left[ \sum_{n=1}^{N} \log p(x_n \mid \theta) + \log p(\theta) \right]} \]

Part 4: Handling Complex Distributions

The Limitation of a Single Model

So far, we have assumed that data from a single class can be described by one simple distribution (like a single Gaussian). But what if the data distribution is more complex?

Main lesson: GMM/EM is Extension; continue from MLE/MAP directly to Naive Bayes.

A Solution for Complexity: Gaussian Mixture Models (GMM)

Optional topic: Core proceeds from MLE/MAP directly to the high-dimensional Naive Bayes challenge. Return to GMM/EM only after the principal practice and summary.

What if our data distribution is a mix of several “clusters”?

For example, customer spending habits might be divided into ‘high-spending’, ‘medium-spending’, and ‘low-spending’ groups, with each group being approximately normally distributed.

The Gaussian Mixture Model (GMM) is designed precisely for this situation.

GMM Definition: A Weighted Sum of Multiple Gaussians

A GMM models a complex probability distribution as a weighted sum of K Gaussian components.

\[ \large{p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \mathcal{N}(\mathbf{x} \mid \mu_k, \Sigma_k)} \]

Where:

• \(K\): The number of mixture components (a hyperparameter).
• \(π_k\): The mixing coefficient (weight) of the \(k\)-th component, with \(\sum_{k=1}^{K} \pi_k = 1\).
• \(μ_k, Σ_k\): The mean and covariance matrix of the \(k\)-th Gaussian component.

The GMM Challenge: A Latent Variable Problem

Estimating the parameters of a GMM (\(π_k, μ_k, Σ_k\)) is much more complex than for a single Gaussian.

This is because we don’t know which Gaussian component each data point \(x_n\) actually belongs to. This “membership” information is a latent variable.

This is a “chicken-and-egg” problem:

• If we knew which cluster each point belonged to, we could easily estimate each cluster’s parameters.
• If we knew each cluster’s parameters, we could calculate the probability of each point belonging to each cluster.

The Solution: The Expectation-Maximization (EM) Algorithm

The Expectation-Maximization (EM) algorithm is an iterative algorithm designed specifically to solve parameter estimation problems with latent variables.

It elegantly solves the “chicken-and-egg” dilemma by alternating between two steps until convergence.

The EM Algorithm Flow

After the optional topic: after GMM/EM, continue to the final summary without replaying completed Core material.

Part 5: The Naive Bayes Classifier

The ‘Naive’ Assumption: Conditional Independence of Features

The Naïve Bayes classifier makes a very bold (but often effective) simplifying assumption:

Given the class y, all features \(x_i\) are mutually independent.

This means: once we know a customer is in the ‘Default’ class, their ‘income’ level and their ‘age’ are considered independent pieces of information.

The Power of the Independence Assumption

This “naive” assumption makes calculating the joint probability incredibly simple:

The originally complex joint probability:

\[ \large{p(\mathbf{x}|y=c) = p(x_1, x_2, \ldots, x_d | y=c)} \]

Under the independence assumption, it decomposes into the product of the individual class-conditional probabilities for each feature:

\[ \large{p(\mathbf{x}|y=c) = \prod_{i=1}^{d} p(x_i | y=c)} \]

Now we only need to estimate a one-dimensional \(p(x_i|y=c)\) for each feature separately, which is much easier than estimating a high-dimensional joint distribution!

The Graphical Model of Naive Bayes

This assumption can be represented by a simple graphical model.

Another Practical Problem: The Zero-Probability Issue

Consider a text classification task (e.g., spam detection), where the features are discrete words.

We use MLE to estimate the class-conditional probability p(word="offer" | class="spam") by calculating the frequency of “offer” in spam emails from the training set.

Problem: What if the word “stock” never appeared in any spam emails in our training data? Then:

\[ \large{p(\text{word}="\text{stock}" \mid y=\text{spam}) = 0} \]

The Catastrophic Consequence of Zero Probability

If p("stock" | spam) = 0, then for any new email containing the word “stock”, the posterior probability of it being spam will be zero!

\[ \large{P(\text{spam} \mid \text{email}) \propto \ldots \times \overbrace{p(\text{"stock"} \mid \text{spam})}^{=0} \times \ldots = 0} \]

A single word unseen in the training set completely rules out a class. This makes the model fragile and unreasonable. It has “seen” too little and is too certain.

The Solution: Laplace Smoothing

Laplace Smoothing, also known as Additive Smoothing, is a simple and effective method for handling the zero-probability problem.

The core idea is: When calculating probabilities, artificially add a small constant λ (usually 1) to the count of every possible event.

This is like giving every possible outcome a “head start,” ensuring that even if an event was never seen in the sample, its estimated probability will not be zero.

The Effect of Smoothing: An Example

Assume a feature has two values {Yes, No}, and we have 10 samples in class c.

• Observations: 10 ‘Yes’, 0 ‘No’.
• \(S_i = 2\) (two possible values)

MLE Estimate:

• P('Yes'|c) = 10/10 = 1
• P('No'|c) = 0/10 = 0 (Dangerous!)

Laplace Smoothing (λ=1):

• P('Yes'|c) = (10+1)/(10+1*2) = 11/12 ≈ 0.917
• P('No'|c) = (0+1)/(10+1*2) = 1/12 ≈ 0.083 (Problem solved!)

Chapter Summary: The Core Ideas of Bayesian Classifiers

Optional topic: after Core, optionally enter GMM/EM, then continue to the final summary without replaying zero probability, smoothing, or the main case.

Preview of Next Lecture

Today, we explored a classic Generative Model—the Bayesian classifier.

We learned how to model the class-conditional density p(x|y) and the prior P(y).

Next, we will study another major class of models: Discriminative Models, such as Logistic Regression. These models bypass modeling p(x|y) and model the posterior probability P(y|x) directly.

Thank You!

Q & A