Logistic Regression using MLE and MAP (L1 & L2 Regularization) on Breast Cancer Dataset

 

Experiment

Title

Logistic Regression using MLE and MAP (L1 & L2 Regularization) on Breast Cancer Dataset

---

🎯 Objective

• To implement Logistic Regression using:
  • MLE (no regularization)
  • MAP with L2 prior (Ridge)
  • MAP with L1 prior (Lasso)
• To compare:
  • Model performance
  • Parameter estimates
  • Effect of regularization

---

📚 Background Theory

🔹 Logistic Regression

$P(y=1|x)=\frac{1}{1+e^{-(w^Tx+b)}}$

---

🔹 MLE Objective

$$\ell(w) = \sum_{i=1}^{N} \left[y_i \log p_i + (1-y_i)\log(1-p_i)\right]$$

---

🔹 MAP Estimation

• L2 Prior (Gaussian):

$$\ell_{MAP} = \ell(w) - \frac{\lambda}{2} ||w||^2$$

• L1 Prior (Laplace):

$$\ell_{MAP} = \ell(w) - \lambda ||w||$$

---

🧩 Dataset

Use the Breast Cancer Wisconsin dataset (available in sklearn):

• Features: Tumor characteristics
• Target:
  • 0 → Malignant
  • 1 → Benign

---

💻 Python Implementation

# Logistic Regression: MLE vs MAP (L1 & L2)

# Breast Cancer Wisconsin Dataset

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

from sklearn.datasets import load_breast_cancer

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LogisticRegression

from sklearn.metrics import accuracy_score, classification_report

# -------------------------------

# 1. Load Dataset

# -------------------------------

data = load_breast_cancer()

X = data.data

y = data.target

# -------------------------------

# 2. Train-Test Split

# -------------------------------

X_train, X_test, y_train, y_test = train_test_split(

    X, y, test_size=0.2, random_state=42

)

# -------------------------------

# 3. Feature Scaling

# -------------------------------

scaler = StandardScaler()

X_train = scaler.fit_transform(X_train)

X_test = scaler.transform(X_test)

# -------------------------------

# 4. Train Models

# -------------------------------

# MLE (No Regularization)

mle_model = LogisticRegression(penalty=None, max_iter=5000)

mle_model.fit(X_train, y_train)

# MAP - L2 Regularization

l2_model = LogisticRegression(penalty='l2', C=1.0, max_iter=5000)

l2_model.fit(X_train, y_train)

# MAP - L1 Regularization

l1_model = LogisticRegression(penalty='l1', solver='liblinear', C=1.0)

l1_model.fit(X_train, y_train)

# -------------------------------

# 5. Predictions

# -------------------------------

y_pred_mle = mle_model.predict(X_test)

y_pred_l2 = l2_model.predict(X_test)

y_pred_l1 = l1_model.predict(X_test)

# -------------------------------

# 6. Accuracy Comparison

# -------------------------------

print("\n=== Accuracy ===")

print("MLE Accuracy:", accuracy_score(y_test, y_pred_mle))

print("L2 MAP Accuracy:", accuracy_score(y_test, y_pred_l2))

print("L1 MAP Accuracy:", accuracy_score(y_test, y_pred_l1))

# -------------------------------

# 7. Classification Reports

# -------------------------------

print("\n=== Classification Report: MLE ===")

print(classification_report(y_test, y_pred_mle))

print("\n=== Classification Report: L2 MAP ===")

print(classification_report(y_test, y_pred_l2))

print("\n=== Classification Report: L1 MAP ===")

print(classification_report(y_test, y_pred_l1))

# -------------------------------

# 8. Coefficient Comparison

# -------------------------------

coeffs = pd.DataFrame({

    "Feature": data.feature_names,

    "MLE": mle_model.coef_[0],

    "L2": l2_model.coef_[0],

    "L1": l1_model.coef_[0]

})

print("\n=== Sample Coefficients ===")

print(coeffs.head(10))

# -------------------------------

# 9. Visualization

# -------------------------------

plt.figure()

plt.plot(np.abs(mle_model.coef_[0]), label="MLE")

plt.plot(np.abs(l2_model.coef_[0]), label="L2 (MAP)")

plt.plot(np.abs(l1_model.coef_[0]), label="L1 (MAP)")

plt.xlabel("Feature Index")

plt.ylabel("Coefficient Magnitude")

plt.title("Coefficient Comparison: MLE vs MAP")

plt.legend()

plt.grid()

plt.show()

# -------------------------------

# 10. Sparsity Check (L1)

# -------------------------------

num_zero = np.sum(l1_model.coef_[0] == 0)

print("\nNumber of zero coefficients in L1 (feature selection):", num_zero)

📈 Output 

=== Accuracy === MLE Accuracy: 0.9385964912280702 L2 MAP Accuracy: 0.9736842105263158 L1 MAP Accuracy: 0.9736842105263158 === Classification Report: MLE === precision recall f1-score support 0 0.88 0.98 0.92 43 1 0.98 0.92 0.95 71 accuracy 0.94 114 macro avg 0.93 0.95 0.94 114 weighted avg 0.94 0.94 0.94 114 === Classification Report: L2 MAP === precision recall f1-score support 0 0.98 0.95 0.96 43 1 0.97 0.99 0.98 71 accuracy 0.97 114 macro avg 0.97 0.97 0.97 114 weighted avg 0.97 0.97 0.97 114 === Classification Report: L1 MAP === precision recall f1-score support 0 0.95 0.98 0.97 43 1 0.99 0.97 0.98 71 accuracy 0.97 114 macro avg 0.97 0.97 0.97 114 weighted avg 0.97 0.97 0.97 114 === Sample Coefficients === Feature MLE L2 L1 0 mean radius 9.433096 -0.431904 0.000000 1 mean texture -16.986388 -0.387326 0.000000 2 mean perimeter 40.020926 -0.393432 0.000000 3 mean area 10.890150 -0.465210 0.000000 4 mean smoothness 5.033443 -0.071667 0.000000 5 mean compactness 274.940098 0.540164 0.000000 6 mean concavity -134.838158 -0.801458 0.000000 7 mean concave points -266.184736 -1.119804 -2.412006 8 mean symmetry 40.996505 0.236119 0.017561 9 mean fractal dimension -168.017946 0.075921 0.000000

📊 Expected Observations

🔹 Accuracy

• All models perform well (~95–99%)
• L2 often slightly better generalization

---

🔹 Coefficients

Model	Behavior
MLE	Large coefficients
L2 (MAP)	Shrinks coefficients smoothly
L1 (MAP)	Many coefficients become zero

---

🔹 Sparsity

• L1 → feature selection
• Some weights become exactly 0

---

🔍 Key Analysis

🔹 Effect of Regularization

• MLE:
  • No constraint
  • Risk of overfitting
• L2 (MAP):
  • Penalizes large weights
  • Smooth shrinkage
• L1 (MAP):
  • Sparse solution
  • Performs feature selection

---

📊 Summary Table

Aspect	MLE	MAP (L2)	MAP (L1)
Regularization	None	Smooth shrinkage	Sparse
Overfitting	Higher	Lower	Lower
Feature selection	No	No	Yes

 Result

• MLE gives best fit to training data but may overfit
• MAP improves generalization via priors
• L1 is useful when:
  • Feature selection is needed
• L2 is preferred when:
  • All features contribute

---