Polynomial Regression using Grid Search for Optimal Degree Selection

 

Experiment

Polynomial Regression using Grid Search for Optimal Degree Selection

📌 Aim

To implement Polynomial Regression on sample data and determine the optimal polynomial degree using Grid Search with Cross Validation.

---

🎯 Objectives

• Understand Polynomial Regression
• Generate sample nonlinear data
• Train polynomial regression models of different degrees
• Use GridSearchCV to find the best polynomial degree
• Evaluate model performance using:
  • MSE
  • R² Score
• Visualize polynomial fitting

---

📖 Theory

---

🔹 Linear Regression

Linear regression models data using a straight-line equation:

$$y = \theta_0 + \theta_1 x$$

This works well only when data is approximately linear.

---

🔹 Polynomial Regression

Polynomial regression extends linear regression by adding higher-order terms:

$$y = \theta_0 + \theta_1 x + \theta_2 x^2 + \theta_3 x^3 + \cdots$$

This allows the model to fit curved relationships.

---

🔹 Polynomial Degree

The degree determines complexity:

Degree	Model Shape
1	Straight line
2	Quadratic curve
3	Cubic curve
Higher	More flexible

---

🔹 Underfitting and Overfitting

Condition	Meaning
Underfitting	Model too simple
Overfitting	Model too complex
Good Fit	Balanced complexity

---

🔹 Grid Search

Grid Search systematically tests multiple polynomial degrees and selects the one with best performance.

---

🔹 Cross Validation

Cross Validation divides the training data into folds and evaluates model performance multiple times.

Example:

$$cv = 5$$

means:

5-fold cross validation.

---

🔹 Evaluation Metrics

---

Mean Squared Error (MSE)

Measures average squared prediction error.

$$MSE = \frac{1}{n} \sum (y_i - \hat y_i)^2$$

Lower MSE is better.

---

🔹 R² Score

Measures goodness of fit.

$$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$$

Range:

Value	Meaning
1	Perfect fit
0	Poor fit

Higher R² is better.

---

📊 Sample Dataset

A nonlinear dataset is generated using:

$$y = x^2 + noise$$

This makes polynomial regression appropriate.

---

📋 Algorithm

1. Import libraries
2. Generate nonlinear sample data
3. Split dataset
4. Create polynomial regression pipeline
5. Define polynomial degrees
6. Apply Grid Search
7. Train models
8. Find optimal degree
9. Evaluate performance
10. Plot polynomial fit

---

💻 Program

# ============================================
# POLYNOMIAL REGRESSION USING GRID SEARCH
# ============================================

# --------------------------------------------
# Step 1: Import Libraries
# --------------------------------------------

import numpy as np
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV

from sklearn.pipeline import Pipeline

from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import StandardScaler

from sklearn.linear_model import LinearRegression

from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score

# --------------------------------------------
# Step 2: Generate Sample Data
# --------------------------------------------

np.random.seed(42)

# Input feature
X = np.linspace(-5, 5, 100).reshape(-1,1)

# Nonlinear target
y = X**2 + np.random.randn(100,1) * 2

# Flatten target
y = y.ravel()

# --------------------------------------------
# Step 3: Split Dataset
# --------------------------------------------

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# --------------------------------------------
# Step 4: Create Pipeline
# --------------------------------------------

pipeline = Pipeline([
    ('poly', PolynomialFeatures()),
    ('scaler', StandardScaler()),
    ('model', LinearRegression())
])

# --------------------------------------------
# Step 5: Define Parameter Grid
# --------------------------------------------

param_grid = {
    'poly__degree': [1, 2, 3, 4, 5, 6]
}

# --------------------------------------------
# Step 6: Apply Grid Search
# --------------------------------------------

grid = GridSearchCV(
    estimator=pipeline,
    param_grid=param_grid,
    cv=5,
    scoring='r2'
)

# Train model
grid.fit(X_train, y_train)

# --------------------------------------------
# Step 7: Best Polynomial Degree
# --------------------------------------------

best_degree = grid.best_params_['poly__degree']

print("\nBEST POLYNOMIAL DEGREE\n")

print("Best Degree:", best_degree)

print("Best Cross Validation R²:",
      round(grid.best_score_, 4))

# --------------------------------------------
# Step 8: Predictions
# --------------------------------------------

best_model = grid.best_estimator_

y_pred = best_model.predict(X_test)

# --------------------------------------------
# Step 9: Evaluation Metrics
# --------------------------------------------

mse = mean_squared_error(y_test, y_pred)

r2 = r2_score(y_test, y_pred)

print("\nMODEL EVALUATION\n")

print("MSE :", round(mse, 4))

print("R²  :", round(r2, 4))

# --------------------------------------------
# Step 10: Visualization
# --------------------------------------------

# Sort values for smooth plotting
X_plot = np.linspace(X.min(), X.max(), 500).reshape(-1,1)

y_plot = best_model.predict(X_plot)

plt.figure(figsize=(10,6))

# Original data
plt.scatter(X, y, color='blue', label='Actual Data')

# Polynomial curve
plt.plot(X_plot,
         y_plot,
         color='red',
         linewidth=3,
         label='Polynomial Fit')

plt.xlabel("X")

plt.ylabel("y")

plt.title(f"Polynomial Regression (Degree = {best_degree})")

plt.legend()

plt.grid(True)

plt.show()

# --------------------------------------------
# Step 11: Plot Degree vs Cross Validation Score
# --------------------------------------------

degrees = param_grid['poly__degree']

scores = grid.cv_results_['mean_test_score']

plt.figure(figsize=(8,5))

plt.plot(degrees,
         scores,
         marker='o')

plt.xlabel("Polynomial Degree")

plt.ylabel("Cross Validation R²")

plt.title("Degree vs Model Performance")

plt.grid(True)

plt.show()

# ============================================
# END OF PROGRAM
# ============================================

---

📊 Sample Output

BEST POLYNOMIAL DEGREE

Best Degree: 3
Best Cross Validation R²: 0.9447

MODEL EVALUATION

MSE : 2.409
R²  : 0.9538

---

📈 Graph Interpretation

---

🔹 Polynomial Fit Graph

• Blue points → actual data
• Red curve → fitted polynomial regression model

---

🔹 Degree vs R² Graph

Shows performance for different polynomial degrees.

---

Small Degree

• underfitting
• poor curve fitting

---

Large Degree

• overfitting
• unstable model

---

Optimal Degree

Best balance between bias and variance.

---

🔍 Interpretation

Degree	Behavior
1	Linear underfit
2	Good fit
5+	Possible overfitting

---

📌 Advantages of Polynomial Regression

✅ Captures nonlinear relationships
✅ Flexible model
✅ Better fit for curved data

---

📌 Limitations

❌ High-degree polynomials may overfit
❌ Computational complexity increases

---

✅ Result

Polynomial Regression was successfully implemented and the optimal polynomial degree was determined using Grid Search with Cross Validation.

Grid Search effectively selects the best polynomial degree and improves model generalization performance.