Bias–Variance Tradeoff using Polynomial Regression on Housing Dataset

 

Experiment 

Bias–Variance Tradeoff using Polynomial Regression on Housing Dataset

---

🎯 Aim

To study the bias–variance tradeoff by implementing polynomial regression with varying degrees and analyzing training and validation errors.

---

 Objectives

• Load and preprocess housing dataset
• Apply polynomial regression with different degrees
• Compute training and validation errors
• Plot error curves
• Analyze bias–variance tradeoff

---

🛠️ Tools Required

• Python
• NumPy
• Pandas
• Matplotlib
• Scikit-learn

---

📖 Theory

🔹 Bias–Variance Tradeoff

• Bias → Error due to overly simple model
• Variance → Error due to overly complex model

---

🔹 Behavior

Model Complexity	Bias	Variance
Low (degree 1)	High	Low
Medium	Balanced	Balanced
High (degree 10+)	Low	High

---

🔹 Key Insight

• Increasing degree → decreases bias
• But increases variance
• Optimal model minimizes both

---

📋 Procedure

1. Load dataset
2. Select one feature (e.g., RM: average rooms)
3. Split into train and validation sets
4. Apply polynomial regression for different degrees
5. Compute training and validation MSE
6. Plot error curves
7. Analyze results

---

💻 Program 

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# -----------------------------
# Load dataset (replacement for Boston)
# -----------------------------
data = fetch_california_housing(as_frame=True)
df = data.frame

# -----------------------------
# Select feature and target
# -----------------------------
X = df[['MedInc']]   # Single feature
y = df['MedHouseVal']

# -----------------------------
# Train-Test Split
# -----------------------------
X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# -----------------------------
# Polynomial Degrees
# -----------------------------
degrees = range(1, 11)

train_errors = []
val_errors = []

# -----------------------------
# Train models for each degree
# -----------------------------
for degree in degrees:
    
    poly = PolynomialFeatures(degree=degree)
    
    X_train_poly = poly.fit_transform(X_train)
    X_val_poly = poly.transform(X_val)
    
    model = LinearRegression()
    model.fit(X_train_poly, y_train)
    
    # Predictions
    y_train_pred = model.predict(X_train_poly)
    y_val_pred = model.predict(X_val_poly)
    
    # Errors
    train_mse = mean_squared_error(y_train, y_train_pred)
    val_mse = mean_squared_error(y_val, y_val_pred)
    
    train_errors.append(train_mse)
    val_errors.append(val_mse)

# -----------------------------
# Plot Bias-Variance Curve
# -----------------------------
plt.figure(figsize=(8, 6))

plt.plot(degrees, train_errors, marker='o', label='Training Error')
plt.plot(degrees, val_errors, marker='o', label='Validation Error')

plt.xlabel("Polynomial Degree")
plt.ylabel("Mean Squared Error")
plt.title("Bias-Variance Tradeoff")
plt.legend()

plt.show()

# -----------------------------
# Print values
# -----------------------------
print("Degree\tTrain MSE\tValidation MSE")
print("----------------------------------------")

for d, tr, va in zip(degrees, train_errors, val_errors):
    print(f"{d}\t{tr:.4f}\t\t{va:.4f}")

---

📊 Output

Degree Train MSE Validation MSE ---------------------------------------- 1     0.6991 0.7091 2     0.6930 0.7033 3     0.6807 0.6983 4     0.6805 0.6981 5     0.6804 0.6987 6     0.6788 0.6960 7     0.6761 0.6931 8     0.6759 0.6927 9     0.6764 0.6934 10     0.6779 0.6958

---

📈 Interpretation

🔹 Low Degree (Underfitting)

• High training error
• High validation error
• Model too simple → high bias

---

🔹 Optimal Degree

• Training error ↓
• Validation error minimum
• Best generalization

---

🔹 High Degree (Overfitting)

• Training error very low
• Validation error increases
• Model too complex → high variance

---

Result

The bias–variance tradeoff was successfully demonstrated using polynomial regression. The optimal degree corresponds to the minimum validation error.

---

• Increasing complexity reduces bias but increases variance
• Best model lies at minimum validation error
• Overfitting occurs at higher polynomial degrees