Parameter Estimation in Logistic Regression using MLE and MAP

 

Experiment

Parameter Estimation in Logistic Regression using MLE and MAP

---

🎯 Objective

• To implement Logistic Regression
• To estimate model parameters using:
  • Maximum Likelihood Estimation (MLE)
  • Maximum A Posteriori (MAP)
• To compare the effect of regularization (prior)

---

Theory

🔹 Logistic Regression Model

$P(y=1|x)=\sigma(wx+b)=\frac{1}{1+e^{-(wx+b)}}$

Where:

• $w$ = weight
• $b$ = bias
• $\sigma$= sigmoid function

---

🔹 Likelihood Function (MLE)

For dataset $(x_i,y_i):$

$$L(w, b) = \prod_{i=1}^{N} P(y_i|x_i)$$

Log-likelihood:

$$\ell(w,b) = \sum_{i=1}^{N} \left[ y_i \log p_i + (1-y_i)\log(1-p_i) \right]$$

---

🔹 MAP Estimation

Assume Gaussian prior:

$$w \sim \mathcal{N}(0, \sigma^2)$$

MAP objective:

$$\ell_{MAP} = \ell(w,b) - \frac{\lambda}{2} ||w||^2$$

👉 Equivalent to L2 regularization (Ridge)

---

🧩 Problem Statement

Classify whether a student passes (1) or fails (0) based on study hours.

---

📊 Sample Dataset

Study Hours (x)	Result (y)
1	0
2	0
3	0
4	1
5	1
6	1

---

⚙️ Algorithm

MLE:

1. Initialize $w, b$
   
2. Compute predictions using sigmoid
3. Compute log-likelihood
4. Update using gradient ascent

---

MAP:

1. Same as MLE
2. Add L2 penalty term
3. Update gradients with regularization

---

💻 Python Implementation

import numpy as np

# Dataset
X = np.array([1, 2, 3, 4, 5, 6]).reshape(-1,1)
y = np.array([0, 0, 0, 1, 1, 1])

# Add bias term
X = np.hstack((X, np.ones((X.shape[0],1))))

# Sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Training function
def train(X, y, lr=0.01, epochs=1000, lam=0):
    w = np.zeros(X.shape[1])

    for _ in range(epochs):
        z = X @ w
        p = sigmoid(z)

        # Gradient
        gradient = X.T @ (y - p)

        # MAP: add regularization
        gradient -= lam * w

        w += lr * gradient

    return w

# Train MLE (lambda = 0)
w_mle = train(X, y, lam=0)

# Train MAP (lambda > 0)
w_map = train(X, y, lam=1)

print("MLE Weights:", w_mle)
print("MAP Weights:", w_map)

---

📈 Output 

MLE Weights:[ 1.35787282 -4.42963961] [large magnitude values]
MAP Weights:[ 0.37945791 -0.75882761] [smaller, regularized values]

---

📊 Visualization Code

import matplotlib.pyplot as plt

# Plot data
plt.figure()
plt.scatter(X[:,0], y)

# Generate smooth curve
x_vals = np.linspace(0, 7, 100)
X_plot = np.hstack((x_vals.reshape(-1,1), np.ones((100,1))))

# Predictions
y_mle = sigmoid(X_plot @ w_mle)
y_map = sigmoid(X_plot @ w_map)

# Plot curves
plt.plot(x_vals, y_mle, label="MLE", linestyle="--")
plt.plot(x_vals, y_map, label="MAP")

plt.xlabel("Study Hours")
plt.ylabel("Probability of Passing")
plt.title("Logistic Regression: MLE vs MAP")
plt.legend()
plt.grid()

plt.show()

---

🔍 Observations

• MLE:
  • Fits data aggressively
  • Can produce large weights
  • May overfit
• MAP:
  • Produces smoother curve
  • Penalizes large weights
  • Better generalization

---

 Experiment Variations

1. Change Regularization Strength

lam = 0.1, 1, 10

👉 Observe curve smoothing

---

2. Add Noise

(2,1), (3,1)  # noisy labels

👉 MLE overfits, MAP resists noise

---

3. Increase Features

Add:

• Study hours
• Sleep hours

---

📊 Comparison Table

Aspect	MLE	MAP
Prior used	❌ No	✅ Yes
Regularization	❌ None	✅ L2
Overfitting	High	Reduced
Stability	Low (small data)	High

---

Results

• Logistic regression MLE = maximize likelihood
• MAP = MLE + regularization
• MAP improves:
  • Stability
  • Generalization
• Widely used in real ML systems