Logistic Regression using gradient descent ( sample data)

 

Experiment: 

Simple Logistic Regression

🎯 Aim

To implement Logistic Regression from scratch using NumPy and apply it to a sample dataset for binary classification.

---

📘 Theory

🔹 What is Logistic Regression?

Logistic Regression is a supervised learning algorithm used for binary classification problems. Unlike linear regression, it predicts probabilities using a sigmoid function.

---

🔹 Hypothesis Function

We model the probability as:

$h_\theta(x) = \frac{1}{1 + e^{-\theta^T x}}$

• Output lies between 0 and 1
• Interpreted as probability of class 1

---

🔹 Cost Function (Log Loss)

$J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[y^{(i)} \log(h_\theta(x^{(i)})) + (1 - y^{(i)}) \log(1 - h_\theta(x^{(i)}))\right]$

• Penalizes wrong predictions heavily
• Ensures convex optimization

---

🔹 Gradient Descent Update Rule

$\theta = \theta - \alpha \cdot \frac{1}{m} X^T (h_\theta(x) - y)$

Where:

• $\alpha$ = learning rate
• $m$= number of samples

Look at ONE training example

Instead of the full sum, take just one data point:

$$J = -\left[y \log(h) + (1-y)\log(1-h)\right]$$

where:

• $h = \sigma(z)$
  
• $z={\theta }^{T}x$

---

Think of it like layers (very important)

We don’t differentiate everything at once.

We go step by step:

θ → z → h → J

So we use chain rule:

$$\frac{dJ}{d\theta} = \frac{dJ}{dh} \cdot \frac{dh}{dz} \cdot \frac{dz}{d\theta}$$

---

Compute each part (simple pieces)

✅ 1. Derivative of cost w.r.t h

$$\frac{dJ}{dh} = \frac{h - y}{h(1-h)}$$

---

✅ 2. Derivative of sigmoid

$$\frac{dh}{dz} = h(1-h)$$

---

✅ 3. Derivative of z

$$\frac{dz}{d\theta} = x$$

 Multiply them

Now multiply all three:

$$\frac{dJ}{d\theta} = \left(\frac{h - y}{h(1-h)}\right) \times (h(1-h)) \times x$$

---

Simplification

The $h(1-h)$ cancels:

$$\frac{dJ}{d\theta} = (h - y) \cdot x$$

---

For all data points

Average over all samples:

$$\frac{\partial J}{\partial \theta} = \frac{1}{m} X^T (h - y)$$

---

Final Gradient Descent Rule

$\theta =\theta -\alpha \cdot \frac{1}{m}{X}^T(h-y)$

---

🧾 Algorithm

1. Initialize weights and bias
2. Compute linear combination:
   $z=X\theta$
3. Apply sigmoid function
4. Compute cost
5. Update weights using gradient descent
6. Repeat until convergence
7. Predict using threshold (0.5)

---

📊 Sample Dataset

We use a simple dataset for binary classification:

Feature 1	Feature 2	Label
1	2	0
2	3	0
3	3	0
5	6	1
6	7	1

 7                   8                    1

💻 Program  

import numpy as np
import matplotlib.pyplot as plt
# -----------------------------
# Step 1: Sample Data
# -----------------------------
X = np.array([
    [1, 2],
    [2, 3],
    [3, 3],
    [5, 6],
    [6, 7],
    [7, 8]
])
y = np.array([0, 0, 0, 1, 1, 1])
# -----------------------------
# Step 2: Add Bias Term
# -----------------------------
X = np.c_[np.ones(X.shape[0]), X]  # add column of 1s
# -----------------------------
# Step 3: Initialize Parameters
# -----------------------------
theta = np.zeros(X.shape[1])
learning_rate = 0.1
epochs = 1000
# -----------------------------
# Step 4: Sigmoid Function
# -----------------------------
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
# -----------------------------
# Step 5: Training (Gradient Descent)
# -----------------------------
for i in range(epochs):
    
    z = np.dot(X, theta)
    y_pred = sigmoid(z)
    
    # Gradient
    gradient = np.dot(X.T, (y_pred - y)) / len(y)
    
    # Update
    theta = theta - learning_rate * gradient
# -----------------------------
# Step 6: Predictions
# -----------------------------
y_pred_final = sigmoid(np.dot(X, theta))
y_pred_class = (y_pred_final >= 0.5).astype(int)
# -----------------------------
# Step 7: Accuracy
# -----------------------------
accuracy = np.mean(y_pred_class == y)
print("Final Theta:", theta)
print("Predictions:", y_pred_class)
print("Accuracy:", accuracy)
# -----------------------------
# Step 8: Visualization
# -----------------------------
plt.scatter(X[:,1], X[:,2], c=y, cmap='bwr')
# Decision boundary
x_values = np.linspace(0, 8, 100)
y_values = -(theta[0] + theta[1]*x_values) / theta[2]
plt.plot(x_values, y_values)
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.title("Logistic Regression (Manual)")
plt.show()

---

📈 Output (Sample)

Final Theta: [-5.9238476   0.92143227  0.5681818 ]
Predictions: [0 0 0 1 1 1]
Accuracy: 1.0

---

Result

• Logistic Regression model was successfully implemented using NumPy.
• Gradient descent optimizes parameters
• Sigmoid function enables classification
• The model correctly classified the dataset with high accuracy.