Comparison of MLE and MAP - Beta Prior

 

Experiment

Title

Comparison of Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP)

---

🎯 Objective

• To understand and implement MLE and MAP estimation
• To compare their behavior under different sample sizes and priors
• To analyze how prior knowledge affects parameter estimation

---

Background Theory

🔹 Maximum Likelihood Estimation (MLE)

MLE estimates parameters by maximizing the likelihood of observed data:

$$\theta_{MLE} = \arg\max_\theta P(D|\theta)$$

• Depends only on data
• Sensitive to small datasets

---

🔹 Maximum A Posteriori (MAP)

MAP incorporates prior knowledge:

$$\theta_{MAP} = \arg\max_\theta P(\theta|D) = \arg\max_\theta P(D|\theta)P(\theta)$$

• Combines data + prior belief
• More robust with limited data

---

Problem Statement

Estimate the probability of a coin landing Heads (θ) using:

• MLE
• MAP (with Beta prior)

---

Sample Dataset

We simulate coin toss outcomes:

D = [1, 0, 1, 1, 0, 1, 0, 1, 1, 1]

Where:

• 1 → Heads
• 0 → Tails

---

Mathematical Formulation

MLE Estimate

$$\theta_{MLE} = \frac{\text{Number of Heads}}{\text{Total Tosses}}$$

---

MAP Estimate (Beta Prior)

Assume prior:

$$\theta \sim Beta(\alpha, \beta)$$

Then:

$$\theta_{MAP} = \frac{H + \alpha - 1}{N + \alpha + \beta - 2}$$

Where:

• $H$ = number of heads
• $N$ = total observations

---

💻 Algorithm

Step 1: Input dataset
Step 2: Count heads (H) and total (N)
Step 3: Compute MLE
Step 4: Choose prior (α, β)
Step 5: Compute MAP
Step 6: Compare results

---

🧑‍💻 Python Implementation

import numpy as np

# Sample data
data = np.array([1, 0, 1, 1, 0, 1, 0, 1, 1, 1])

# Count heads and total
H = np.sum(data)
N = len(data)

# MLE
theta_mle = H / N

# MAP (Beta prior)
alpha = 2   # prior belief
beta = 2

theta_map = (H + alpha - 1) / (N + alpha + beta - 2)

print("Heads:", H)
print("Total:", N)
print("MLE Estimate:", theta_mle)
print("MAP Estimate:", theta_map)

---

📈 Output

Heads: 7  
Total: 10  
MLE Estimate: 0.7  
MAP Estimate: 0.6667

---

🔍 Observations

• MLE gives 0.7, purely data-driven
• MAP gives 0.667, slightly adjusted due to prior
• MAP is less extreme, especially useful with small datasets

---

Experiment Variations

1. Change Prior

Try:

• (α=1, β=1) → Uniform prior
• (α=10, β=10) → Strong belief in fairness

---

2. Reduce Dataset Size

Use:

D = [1, 0, 1]

Observe:

• MLE fluctuates significantly
• MAP remains stable

---

3. Increase Dataset Size

Use 100+ samples → MLE ≈ MAP

---

📊 Comparison Table

Aspect	MLE	MAP
Uses prior	❌ No	✅ Yes
Data dependency	High	Balanced
Small data	Unstable	Stable
Bias	Unbiased	Slightly biased

---

Result

• MLE works well with large datasets
• MAP is preferred when:
  • Data is limited
  • Prior knowledge is available
• MAP converges to MLE as dataset size increases