Comparison of Gaussian Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) - Gaussian Prior

 

Experiment

Comparison of Gaussian Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) - Gaussian Prior

---

🎯 Objective

• To estimate parameters (mean) of a Gaussian distribution using:
  • MLE
  • MAP
• To observe the effect of prior knowledge on parameter estimation
• To compare performance under small and large datasets

---

 Theory

🔹 Gaussian Distribution

A Gaussian (Normal) distribution is defined as:

$p(x\mathrm{\mid }\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$

Where:

• $\mu$ = mean
• $\sigma^2$ = variance

---

🔹 MLE for Gaussian Mean

Assuming variance is known, MLE estimate of mean:

$$\mu_{MLE} = \frac{1}{N} \sum_{i=1}^{N} x_i$$

• Simple average of data
• No prior knowledge used

---

🔹 MAP for Gaussian Mean

Assume prior:

$$\mu \sim \mathcal{N}(\mu_0, \tau^2)$$

Then MAP estimate:

$$\mu_{MAP} = \frac{\frac{N}{\sigma^2}\bar{x} + \frac{1}{\tau^2}\mu_0}{\frac{N}{\sigma^2} + \frac{1}{\tau^2}}$$

• Weighted combination of:
  • Data mean
  • Prior mean

---

🧩 Problem Statement

Estimate the mean height of students using:

• MLE
• MAP (with prior belief about mean height)

---

📊 Sample Dataset

Heights (in cm):

D = [168, 170, 169, 171, 172]

Assume:

• Known variance: $\sigma^2 = 4$
  
• Prior: $\mu_0 = 165$, $\tau^2 = 9$
  

---

🧮 Manual Calculation

Step 1: Compute Sample Mean

$$\bar{x} = \frac{168 + 170 + 169 + 171 + 172}{5} = 170$$

---

Step 2: MLE Estimate

$$\mu_{MLE} = 170$$

---

Step 3: MAP Estimate

$$\mu_{MAP} = \frac{\frac{5}{4}(170) + \frac{1}{9}(165)}{\frac{5}{4} + \frac{1}{9}}$$ $$\mu_{MAP} \approx 169.3$$

---

💻 Python Implementation

import numpy as np

# Data
data = np.array([168, 170, 169, 171, 172])

# Known variance
sigma2 = 4

# Prior parameters
mu0 = 165
tau2 = 9

# Sample statistics
N = len(data)
x_bar = np.mean(data)

# MLE
mu_mle = x_bar

# MAP
mu_map = ((N/sigma2)*x_bar + (1/tau2)*mu0) / ((N/sigma2) + (1/tau2))

print("Sample Mean:", x_bar)
print("MLE Estimate:", mu_mle)
print("MAP Estimate:", mu_map)

---

📈 Output

Sample Mean: 170  
MLE Estimate: 170  
MAP Estimate: 169.59183673469389

---

🔍 Observations

• MLE = 170 (purely data-driven)
• MAP shifts toward prior (165) → gives ~169.3
• MAP acts like a regularized estimate

---

Experiment Variations

1. Change Prior Strength

• Strong prior: ${\tau }^2=\mathrm{1\to \; MAP\; closer\; to\; 165}$
• Weak prior: $\tau^2 = 100$ → MAP ≈ MLE

---

2. Small Dataset Case

D = [172, 174]

• MLE becomes unstable
• MAP remains controlled

---

3. Large Dataset Case

Use 100+ samples → MAP ≈ MLE

---

📊 Comparison Table

Aspect	MLE	MAP
Uses prior	❌ No	✅ Yes
Formula	Mean of data	Weighted mean
Small data	Unstable	Stable
Interpretation	Data-only estimate	Bayesian estimate

---

Results

• MLE is optimal with large data
• MAP is useful when:
  • Data is limited
  • Prior knowledge exists
• MAP = Regularized MLE
• As $N \to \infty$, MAP → MLE