Simple Linear Regression using Gradient Descent on California Housing Dataset

 

Experiment: 

Simple Linear Regression using Gradient Descent on California Housing Dataset

---

📌 Aim

To implement Simple Linear Regression using Gradient Descent to predict the Median House Value using the feature Total Rooms from the California Housing Dataset.

---

🎯 Objectives

• Load and preprocess the housing dataset
• Implement Simple Linear Regression manually
• Apply Gradient Descent optimization
• Compute regression parameters
• Evaluate model using:
  • Mean Squared Error (MSE)
  • R² Score
• Visualize regression line and cost convergence

---

📖 Theory

---

🔹 Simple Linear Regression

Simple Linear Regression models the relationship between:

• One independent variable $X$
  
• One dependent variable $Y$
  

---

Mathematical Equation

$$\hat{y} = \theta_0 + \theta_1 x$$

Where:

• $\hat{y}$→ Predicted value
• $\theta_0$ → Intercept
• $\theta_1$ → Slope
• $x$→ Input feature

---

🔹 Cost Function (Mean Squared Error)

The objective is to minimize the error between actual and predicted values.

$$J(\theta_0,\theta_1)=\frac{1}{n}\sum(y-\hat{y})^2$$

---

🔹 Gradient Descent

Gradient Descent updates parameters iteratively to minimize cost.

---

Gradient Equations

For $\theta_0$

$$\frac{\partial J}{\partial \theta_0} = -\frac{2}{n}\sum(y-\hat{y})$$

---

For $\theta_1$

$$\frac{\partial J}{\partial \theta_1} = -\frac{2}{n}\sum(y-\hat{y})x$$

---

Update Rules

$$\theta_0 = \theta_0 - \alpha \frac{\partial J}{\partial \theta_0}$$ $$\theta_1 = \theta_1 - \alpha \frac{\partial J}{\partial \theta_1}$$

Where:

• $\alpha$→ Learning rate

---

📊 Dataset Description

The California Housing dataset contains housing-related information collected from the California census.

---

Selected Features

Feature	Description
total_rooms	Total number of rooms
median_house_value	Median house price

---

📋 Algorithm

1. Import libraries
2. Load housing dataset
3. Select predictor and target variable
4. Remove missing values
5. Normalize feature values
6. Initialize parameters
7. Apply gradient descent
8. Compute predictions
9. Calculate MSE and R²
10. Plot regression line
11. Plot cost convergence

---

💻 Program

# ============================================
# SIMPLE LINEAR REGRESSION USING GRADIENT DESCENT
# California Housing Dataset
# ============================================

# --------------------------------------------
# Step 1: Import Libraries
# --------------------------------------------

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# --------------------------------------------
# Step 2: Load Dataset
# --------------------------------------------

df = pd.read_csv("housing.csv")

# Display first 5 rows
print("\nFIRST 5 ROWS OF DATASET\n")
print(df.head())

# --------------------------------------------
# Step 3: Select Features
# --------------------------------------------

# Predictor Variable
X = df['total_rooms'].values

# Target Variable
y = df['median_house_value'].values

# --------------------------------------------
# Step 4: Remove Missing Values
# --------------------------------------------

mask = ~np.isnan(X) & ~np.isnan(y)

X = X[mask]
y = y[mask]

# --------------------------------------------
# Step 5: Normalize Data
# --------------------------------------------

X_mean = np.mean(X)
X_std = np.std(X)

X = (X - X_mean) / X_std

# --------------------------------------------
# Step 6: Initialize Parameters
# --------------------------------------------

theta0 = 0
theta1 = 0

learning_rate = 0.01
iterations = 1000

n = len(X)

# Store cost values
cost_history = []

# --------------------------------------------
# Step 7: Gradient Descent
# --------------------------------------------

for i in range(iterations):

    # Predicted values
    y_pred = theta0 + theta1 * X

    # Error
    error = y - y_pred

    # Gradients
    d_theta0 = (-2/n) * np.sum(error)

    d_theta1 = (-2/n) * np.sum(error * X)

    # Update parameters
    theta0 = theta0 - learning_rate * d_theta0

    theta1 = theta1 - learning_rate * d_theta1

    # Cost function
    cost = (1/n) * np.sum(error**2)

    cost_history.append(cost)

# --------------------------------------------
# Step 8: Final Predictions
# --------------------------------------------

y_pred_final = theta0 + theta1 * X

# --------------------------------------------
# Step 9: Model Parameters
# --------------------------------------------

print("\nMODEL PARAMETERS\n")

print("Theta0 (Intercept):", theta0)

print("Theta1 (Slope):", theta1)

# --------------------------------------------
# Step 10: Evaluation Metrics
# --------------------------------------------

# Mean Squared Error
mse = np.mean((y - y_pred_final)**2)

# R² Score
SS_res = np.sum((y - y_pred_final)**2)

SS_tot = np.sum((y - np.mean(y))**2)

r2 = 1 - (SS_res / SS_tot)

print("\nEVALUATION METRICS\n")

print("Mean Squared Error (MSE):", round(mse, 4))

print("R² Score:", round(r2, 4))

# --------------------------------------------
# Step 11: Plot Regression Line
# --------------------------------------------

plt.figure(figsize=(10,6))

# Scatter plot
plt.scatter(X,
            y,
            color='blue',
            alpha=0.5,
            label='Actual Data')

# Regression line
plt.plot(X,
         y_pred_final,
         color='red',
         linewidth=2,
         label='Regression Line')

plt.title("Simple Linear Regression using Gradient Descent")

plt.xlabel("Normalized Total Rooms")

plt.ylabel("Median House Value")

plt.legend()

plt.grid(True)

plt.show()

# --------------------------------------------
# Step 12: Plot Cost Convergence
# --------------------------------------------

plt.figure(figsize=(8,5))

plt.plot(range(iterations),
         cost_history,
         color='green')

plt.title("Cost Function Convergence")

plt.xlabel("Iterations")

plt.ylabel("Cost")

plt.grid(True)

plt.show()

# --------------------------------------------
# Step 13: Predict New Value
# --------------------------------------------

new_total_rooms = 3000

# Normalize input
new_total_rooms_norm = (new_total_rooms - X_mean) / X_std

predicted_value = theta0 + theta1 * new_total_rooms_norm

print("\nPREDICTION\n")

print("Predicted House Value for",
      new_total_rooms,
      "rooms =",
      round(predicted_value, 2))

# ============================================
# END OF PROGRAM
# ============================================

---

📊 Sample Output

FIRST 5 ROWS OF DATASET

   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \
0    -122.23     37.88                41.0        880.0           129.0   
1    -122.22     37.86                21.0       7099.0          1106.0   
2    -122.24     37.85                52.0       1467.0           190.0   
3    -122.25     37.85                52.0       1274.0           235.0   
4    -122.25     37.85                52.0       1627.0           280.0   

   population  households  median_income  median_house_value ocean_proximity  
0       322.0       126.0         8.3252            452600.0        NEAR BAY  
1      2401.0      1138.0         8.3014            358500.0        NEAR BAY  
2       496.0       177.0         7.2574            352100.0        NEAR BAY  
3       558.0       219.0         5.6431            341300.0        NEAR BAY  
4       565.0       259.0         3.8462            342200.0        NEAR BAY  

MODEL PARAMETERS

Theta0 (Intercept): 206855.81656078316
Theta1 (Slope): 15480.306142081694

EVALUATION METRICS

Mean Squared Error (MSE): 13075863121.7589
R² Score: 0.018



PREDICTION

Predicted House Value for 3000 rooms = 209440.43

---

📈 Graphs

1. Regression Line

• Blue points → Actual data
• Red line → Regression line

---

2. Cost Function Convergence

• X-axis → Iterations
• Y-axis → Cost
• Decreasing curve indicates learning

---

🔍 Interpretation

Observation	Meaning
Low cost over iterations	Model converges
Low R²	Weak linear relationship
Regression line	Best fit line

---

📉 Why R² May Be Low?

Using only:

total_rooms

may not sufficiently explain house price.

House value depends on multiple variables like:

• income
• location
• population
• bedrooms
• ocean proximity

---

✅ Result

Simple Linear Regression using Gradient Descent was successfully implemented on the California Housing dataset.

The experiment demonstrated:

• Manual implementation of gradient descent
• Optimization of regression parameters
• Prediction of house values using a single feature
• Gradient Descent works effectively on real datasets
• Feature scaling improves convergence
• Using only one feature limits model accuracy