Simple Linear Regression using Gradient Descent

 

 Experiment 

Simple Linear Regression using Gradient Descent

---

Aim

To implement Simple Linear Regression using Gradient Descent on sample data and evaluate the model using MSE and R².

---

Objectives

• Understand Gradient Descent optimization

• Iteratively compute regression coefficients

• Compare with analytical solution

• Compute MSE and R² manually

• Visualize regression line

---

🛠️ Tools Required

• Python

• NumPy

• Matplotlib

---

📖 Theory

🔹 Linear Regression Model

$$y = \theta_0 + \theta_1 x$$

---

🔹 Cost Function (MSE)

$$J(\theta) = \frac{1}{n} \sum (y_i - \hat{y}_i)^2$$

---

🔹 Gradient Descent

Gradient Descent minimizes the cost function iteratively:

$$\theta_0 := \theta_0 - \alpha \cdot \frac{\partial J}{\partial \theta_0}$$ $$\theta_1 := \theta_1 - \alpha \cdot \frac{\partial J}{\partial \theta_1}$$

$$\boxed{ \frac{\partial J}{\partial \theta_0} = -\frac{2}{n} \sum (y_i - \hat{y}_i) }$$

​​​

$$\boxed{ \frac{\partial J}{\partial \theta_1} = -\frac{2}{n} \sum (y_i - \hat{y}_i)x_i }$$

Where:

• $\alpha$ = learning rate

• Updates continue until convergence

---

📋 Procedure

1. Initialize parameters $\theta_0, \theta_1$

2. Choose learning rate and number of iterations

3. Compute predictions

4. Calculate gradients

5. Update parameters iteratively

6. Compute MSE and R²

7. Plot regression line

---

💻 Program

import numpy as np
import matplotlib.pyplot as plt

# Sample data
x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
y = np.array([1, 3, 2, 5, 7, 8, 8, 9, 10, 12])

# Scatter plot
plt.scatter(x, y, color="m", marker="o", s=30)

# -----------------------------
# Gradient Descent
# -----------------------------
theta0 = 0
theta1 = 0

learning_rate = 0.01
epochs = 1000
n = len(x)

for _ in range(epochs):
    y_pred = theta0 + theta1 * x
    
    # Compute gradients
    d_theta0 = (-2/n) * np.sum(y - y_pred)
    d_theta1 = (-2/n) * np.sum((y - y_pred) * x)
    
    # Update parameters
    theta0 -= learning_rate * d_theta0
    theta1 -= learning_rate * d_theta1

print("Theta0 (Intercept):", theta0)
print("Theta1 (Slope):", theta1)

# Final predictions
y_pred = theta0 + theta1 * x

# -----------------------------
# Manual MSE
# -----------------------------
mse = np.sum((y - y_pred) ** 2) / n
print("MSE:", mse)

# -----------------------------
# Manual R²
# -----------------------------
y_mean = np.mean(y)

SS_res = np.sum((y - y_pred) ** 2)
SS_tot = np.sum((y - y_mean) ** 2)

r2 = 1 - (SS_res / SS_tot)
print("R²:", r2)

# -----------------------------
# Plot regression line
# -----------------------------
plt.plot(x, y_pred, color="g", label="Gradient Descent Line")

plt.xlabel("X")
plt.ylabel("Y")
plt.title("Linear Regression using Gradient Descent")
plt.legend()

plt.show()

---

Output

• Learned parameters:

  • $\theta_0$ (intercept)

  • $\theta_1$ (slope)

• Mean Squared Error (MSE)

• R² score

• Regression line plot

---

 Result

The regression model was successfully implemented using Gradient Descent, and model parameters were learned iteratively.

• Gradient Descent finds optimal parameters iteratively

• Results are similar to analytical method

• Learning rate affects convergence speed and accuracy