Logistic Regression with Evaluation Metrics and Decision Boundary Visualization

 

Experiment

Logistic Regression with Evaluation Metrics and Decision Boundary Visualization

---

🎯 Aim

To implement Logistic Regression on a sample dataset and evaluate its performance using confusion matrix, accuracy, precision, recall, and F1-score, along with visualization of the decision boundary.

---

Objectives

• Generate a classification dataset
• Train Logistic Regression model
• Evaluate model using multiple metrics
• Understand confusion matrix interpretation
• Visualize decision boundary

---

🛠️ Tools Required

• Python
• NumPy
• Matplotlib
• Scikit-learn

---

📖 Theory

🔹 Logistic Regression

Used for binary classification problems.

$$P(y=1) = \frac{1}{1 + e^{-(\theta_0 + \theta_1 x_1 + \theta_2 x_2)}}$$

• Uses sigmoid function
• Output range: 0 to 1 (probability)
• Decision rule:
  • If probability ≥ 0.5 → Class 1
  • Else → Class 0

---

🔹 Confusion Matrix

	Predicted 0	Predicted 1
Actual 0	TN	FP
Actual 1	FN	TP

---

🔹 Evaluation Metrics

Accuracy

$$Accuracy = \frac{TP + TN}{Total}$$

Precision

$$Precision = \frac{TP}{TP + FP}$$

Recall

$$Recall = \frac{TP}{TP + FN}$$

F1 Score

$$F1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall}$$

Support

👉 Number of actual samples for each class​

---

📋 Procedure

1. Generate sample dataset
2. Split into training and testing sets
3. Train Logistic Regression model
4. Predict test results
5. Compute confusion matrix
6. Calculate evaluation metrics
7. Visualize decision boundary

---

💻 Program 

import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# -----------------------------
# Step 1: Generate Dataset
# -----------------------------
X, y = make_classification(
    n_samples=200,
    n_features=2,
    n_redundant=0,
    n_informative=2,
    n_clusters_per_class=1,
    random_state=42
)

# -----------------------------
# Step 2: Train-Test Split
# -----------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# -----------------------------
# Step 3: Train Model
# -----------------------------
model = LogisticRegression()
model.fit(X_train, y_train)

# -----------------------------
# Step 4: Predictions
# -----------------------------
y_pred = model.predict(X_test)

# -----------------------------
# Step 5: Evaluation Metrics
# -----------------------------
accuracy = accuracy_score(y_test, y_pred)
cm = confusion_matrix(y_test, y_pred)

print("Accuracy:", round(accuracy, 4))
print("\nConfusion Matrix:\n", cm)

# Extract values for explanation
TN, FP, FN, TP = cm.ravel()

print("\nDetailed Interpretation:")
print(f"True Positives (TP): {TP}")
print(f"True Negatives (TN): {TN}")
print(f"False Positives (FP): {FP}")
print(f"False Negatives (FN): {FN}")

# Classification report
print("\nClassification Report:\n")
print(classification_report(y_test, y_pred))

# -----------------------------
# Step 6: Visualization
# -----------------------------
plt.figure(figsize=(8,6))

# Plot data points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='bwr', edgecolors='k')

# Create grid
x_min, x_max = X[:, 0].min()-1, X[:, 0].max()+1
y_min, y_max = X[:, 1].min()-1, X[:, 1].max()+1

xx, yy = np.meshgrid(
    np.linspace(x_min, x_max, 200),
    np.linspace(y_min, y_max, 200)
)

# Predict on grid
Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

# Plot decision boundary
plt.contourf(xx, yy, Z, alpha=0.3)
plt.contour(xx, yy, Z)

plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.title("Logistic Regression Decision Boundary")

plt.show()

---

📊 Output

✔ Confusion Matrix Example

[[18  5]
 [ 0 17]]

---

📈 Interpretation

• TN = 18 → correctly predicted class 0
• TP = 17 → correctly predicted class 1
• FP = 5→ false alarms
• FN = 0→ missed detections

---

✔ Accuracy

$$= \frac{TP + TN}{Total} = \frac{35}{40} = 0.88$$

---

📉 Visualization Insight

• Colored regions → predicted classes
• Boundary line → separation rule
• Points → actual data

Accuracy: 0.875 Confusion Matrix: [[18 5] [ 0 17]] Detailed Interpretation: True Positives (TP): 17 True Negatives (TN): 18 False Positives (FP): 5 False Negatives (FN): 0 Classification Report: precision recall f1-score support 0 1.00 0.78 0.88 23 1 0.77 1.00 0.87 17 accuracy 0.88 40 macro avg 0.89 0.89 0.87 40 weighted avg 0.90 0.88 0.88 40

---

📊 Key Learning

Concept	Insight
Decision Boundary	Separates classes
Confusion Matrix	Shows errors
Precision	Avoids false positives
Recall	Avoids false negatives

---

Result

Logistic Regression model was successfully implemented and evaluated using confusion matrix and classification metrics, and its decision boundary was visualized.

📌 Conclusion

• Logistic regression is effective for binary classification
• Evaluation metrics provide deeper insights than accuracy
• Visualization improves understanding of model behavior