Multinomial Parameter Estimation using MLE and MAP (Dirichlet Prior) on Text Data

 

Experiment

Title

Multinomial Parameter Estimation using MLE  and MAP (Dirichlet Prior) on Text Data

---

🎯 Objective

• To estimate word probabilities using:
  • MLE (Maximum Likelihood Estimation)
  • MAP (Dirichlet prior)
• To analyze how priors affect probability estimates
• To understand smoothing in text models (Naive Bayes context)

---

📚 Background Theory

🔹 Multinomial Distribution

Used for discrete counts (e.g., word frequencies in documents).

$P(x|\theta)=\prod_{i=1}^{V}\theta_i^{x_i}$

---

🔹 MLE Estimate

$$\theta_i^{MLE} = \frac{n_i}{\sum_j n_j}$$

• $n_i$: count of word $i$
  
• No smoothing → zero probabilities possible

---

🔹 MAP Estimate (Dirichlet Prior)

$$\theta_i^{MAP} = \frac{n_i + \alpha_i - 1}{\sum_j (n_j + \alpha_j - 1)}$$

• $\alpha$: prior parameters
• Acts as smoothing

---

🧩 Dataset

Use 20 Newsgroups dataset:

• Text classification dataset
• Each document → bag-of-words

---

💻 Combined Simple Program

# Multinomial MLE vs MAP (Dirichlet Prior)
# 20 Newsgroups Dataset

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer

# -------------------------------
# 1. Load Dataset
# -------------------------------
categories = ['sci.space', 'rec.sport.baseball']  # binary subset for simplicity

data = fetch_20newsgroups(subset='train',
                         categories=categories,
                         remove=('headers', 'footers', 'quotes'))

texts = data.data

# -------------------------------
# 2. Text Vectorization
# -------------------------------
vectorizer = CountVectorizer(max_features=5000, stop_words='english')
X = vectorizer.fit_transform(texts)

# Convert to numpy array (word counts)
X = X.toarray()

# -------------------------------
# 3. Aggregate Word Counts
# -------------------------------
word_counts = np.sum(X, axis=0)  # total counts per word
total_count = np.sum(word_counts)

# -------------------------------
# 4. MLE Estimation
# -------------------------------
theta_mle = word_counts / total_count

# -------------------------------
# 5. MAP Estimation (Dirichlet Prior)
# -------------------------------

def compute_map(alpha):
    alpha_vec = np.ones_like(word_counts) * alpha
    theta_map = (word_counts + alpha_vec - 1) / np.sum(word_counts + alpha_vec - 1)
    return theta_map

# Different priors
theta_map_1 = compute_map(alpha=1)   # Uniform prior
theta_map_2 = compute_map(alpha=2)   # Mild smoothing
theta_map_10 = compute_map(alpha=10) # Strong smoothing

# -------------------------------
# 6. Compare Top Words
# -------------------------------
feature_names = np.array(vectorizer.get_feature_names_out())

def top_words(theta, name):
    top_idx = np.argsort(theta)[-10:]
    print(f"\nTop words ({name}):")
    print(feature_names[top_idx])

top_words(theta_mle, "MLE")
top_words(theta_map_1, "MAP alpha=1")
top_words(theta_map_2, "MAP alpha=2")
top_words(theta_map_10, "MAP alpha=10")

# -------------------------------
# 7. Compare Probability Spread
# -------------------------------
print("\n=== Probability Statistics ===")
print("MLE min prob:", np.min(theta_mle))
print("MAP (alpha=1) min prob:", np.min(theta_map_1))
print("MAP (alpha=10) min prob:", np.min(theta_map_10))

# -------------------------------
# 8. Visualization
# -------------------------------
import matplotlib.pyplot as plt

plt.figure()

# Sort probabilities for visualization
plt.plot(np.sort(theta_mle), label="MLE")
plt.plot(np.sort(theta_map_2), label="MAP alpha=2")
plt.plot(np.sort(theta_map_10), label="MAP alpha=10")

plt.xlabel("Word Index (sorted)")
plt.ylabel("Probability")
plt.title("MLE vs MAP (Dirichlet Prior)")
plt.legend()
plt.grid()

plt.show()

---

🔍 Observations

🔹 MLE

• Assigns zero probability to unseen words
• Highly skewed distribution
• Dominated by frequent words

---

🔹 MAP (Dirichlet Prior)

Prior (α)	Effect
α = 1	Same as MLE (no smoothing)
α = 2	Slight smoothing
α = 10	Strong smoothing

---

🔹 Key Insights

• MAP prevents zero probabilities
• Higher α → more uniform distribution
• Acts like Laplace smoothing

---

📊 Comparison Table

Aspect	MLE	MAP (Dirichlet)
Zero probabilities	Yes	No
Robustness	Low	High
Prior knowledge	Not used	Used
Smoothing	None	Yes

---

Result

• MLE is sensitive to sparse data
• MAP (Dirichlet) provides stable probability estimates
• Essential in:
  • Text classification
  • Naive Bayes
• Larger priors → smoother distributions