Comparison of MLE and MAP Estimation using Sample Data

 

Experiment

Comparison of MLE and MAP Estimation using Sample Data

---

🎯 Aim

To estimate parameters using MLE and MAP and compare  their behavior.

---

Objective

• Generate sample data
• Estimate mean using MLE
• Estimate mean using MAP (with prior)
• Compare results visually

---

📖 Theory 

What are MLE and MAP?

---

🔹 1. Maximum Likelihood Estimation (MLE)

📌 Idea:

Find parameters that maximize the likelihood of observed data

$$\theta_{MLE} = \arg\max P(D|\theta)$$

👉 “Choose parameters that make the data most likely”

---

🔹 2. Maximum A Posteriori (MAP)

📌 Idea:

Includes prior knowledge about parameters

$$\theta_{MAP} = \arg\max P(D|\theta) \cdot P(\theta)$$

👉 “Choose parameters that fit data + prior belief”

---

🔑 Key Difference

Method	Uses Data	Uses Prior
MLE	✅	❌
MAP	✅	✅

---

🧠 Intuition

• MLE → trusts only data
• MAP → balances data + prior belief

Assume:

• Data follows Normal Distribution
• Known variance

🔹 MLE Estimate of Mean

MLE only uses the observed data.

Formula:

$$\mu_{MLE} = \frac{1}{n}\sum x_i$$

​ This is simply:

👉 sample mean

​​​

---

🔹 MAP ( Maximum Aposteriori ) Estimate of Mean

---

💻 Program

import numpy as np
import matplotlib.pyplot as plt
# -----------------------------
# Generate Sample Data
# -----------------------------
np.random.seed(42)
true_mean = 5
data = np.random.normal(loc=true_mean, scale=2, size=50)
# -----------------------------
# MLE Estimation
# -----------------------------
mle_mean = np.mean(data)
# -----------------------------
# MAP Estimation
# -----------------------------
# Prior (assume we believe mean is around 0)
prior_mean = 5
prior_variance = 1
# Data variance
data_variance = 4  # since std=2 → variance=4
n = len(data)
# MAP formula
map_mean = (n * mle_mean + (data_variance / prior_variance) * prior_mean) / \
           (n + (data_variance / prior_variance))
# -----------------------------
# Print Results
# -----------------------------
print("True Mean:", true_mean)
print("MLE Estimate:", round(mle_mean, 4))
print("MAP Estimate:", round(map_mean, 4))
# -----------------------------
# Visualization
# -----------------------------
plt.hist(data, bins=15, alpha=0.5, label="Data")
plt.axvline(true_mean, color='black', linestyle='--', label="True Mean")
plt.axvline(mle_mean, color='blue', label="MLE Mean")
plt.axvline(map_mean, color='red', label="MAP Mean")
plt.title("MLE vs MAP Estimation")
plt.legend()
plt.show()

---

📊  Output

True Mean: 5
MLE Estimate: 4.5491
MAP Estimate: 4.5825



vary the sample size and observe the output

---

📈 Interpretation

🔹 MLE

• Close to sample data
• Ignores prior

---

🔹 MAP

• Pulled toward prior (5)
• Balances data + belief

---

Key Observation

Scenario	Behavior
Large data	MLE ≈ MAP
Small data	MAP influenced by prior

---

📉 Graph Explanation

• Histogram → data distribution
• Black line → true mean
• Blue line → MLE
• Red line → MAP

👉 MAP shifts toward prior

---

Result

MLE and MAP estimation were implemented and compared. MAP incorporates prior knowledge, whereas MLE relies solely on observed data.

• MLE is purely data-driven
• MAP incorporates prior belief
• MAP is useful when:
  • Data is limited
  • Prior knowledge exists