Comparison of Linear, Ridge, and Lasso Regression with Model Evaluation

 

Experiment 

Comparison of Linear, Ridge, and Lasso Regression with Model Evaluation

---

🎯 Aim

To compare Ordinary Linear Regression, Ridge Regression, and Lasso Regression using evaluation metrics (MSE and R²) and visualize their performance.

---

Objectives

• Generate sample dataset
• Apply polynomial transformation
• Train Linear, Ridge, and Lasso models
• Evaluate using MSE and R²
• Compare coefficients
• Visualize model behavior

---

🛠️ Tools Required

• Python
• NumPy
• Matplotlib
• Scikit-learn

---

📖 Theory

In Machine Learning, Linear Regression may perform poorly when:

• the dataset has many features,
• features are highly correlated,
• the model overfits training data.

To overcome this problem, Regularization techniques are used.

Two important regularization methods are:

1. Ridge Regression (L2 Regularization)
2. Lasso Regression (L1 Regularization)

These techniques reduce overfitting by adding a penalty term to the cost function.

🔹 Regularization

Regularization is a technique used to:

✅ reduce model complexity
✅ prevent overfitting
✅ improve generalization

It works by penalizing large coefficient values.

Regularization prevents overfitting by adding a penalty term:

🔵 Ridge Regression (L2 Regularization)

Ridge Regression adds the squared magnitude of coefficients as penalty.

---

📌 Ridge Cost Function

$$J(\theta) = \frac{1}{n} \sum(y-\hat{y})^2 + \lambda \sum \theta_j^2$$

---

📌 Penalty Term

$$\lambda \sum \theta_j^2$$

This is called:

$$L2 \text{ Regularization}$$

---

📌 Meaning of λ (Lambda)

controls regularization strength.

λ Value	Effect
Small λ	behaves like normal regression
Large λ	stronger regularization

---

📌 Effect of Ridge Regression

• Shrinks coefficients toward zero
• Reduces variance
• Keeps all features
• Handles multicollinearity well

---

📌 Geometric Interpretation

Ridge creates a circular constraint region.

📌 Advantages of Ridge Regression

✅ Reduces overfitting
✅ Works well with correlated features
✅ Stable model coefficients

🟢 Lasso Regression (L1 Regularization)

---

📌 Definition

Lasso Regression adds the absolute value of coefficients as penalty.

---

📌 Lasso Cost Function

$$J(\theta) = \frac{1}{n} \sum(y-\hat{y})^2 + \lambda \sum |\theta_j|$$

---

📌 Penalty Term

$$\lambda \sum |\theta_j|$$

This is called:

$$L1 \text{ Regularization}$$

---

📌 Effect of Lasso Regression

• Shrinks coefficients
• Some coefficients become exactly zero
• Performs feature selection automatically

---

📌 Geometric Interpretation

Lasso creates a diamond-shaped constraint.

Corners increase chance of coefficients becoming zero.

---

📌 Advantages of Lasso

✅ Performs feature selection
✅ Reduces overfitting
✅ Produces sparse models

• Ridge (L2) → shrinks coefficients
• Lasso (L1) → shrinks + eliminates coefficients

---

🔹 Evaluation Metrics

Mean Squared Error (MSE)

$$MSE = \frac{1}{n} \sum (y - \hat{y})^2$$

R-squared (R²)

$$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$$

---

📋 Procedure

1. Generate noisy dataset
2. Apply high-degree polynomial features
3. Split into train and test sets
4. Train:
   • Linear Regression
   • Ridge Regression
   • Lasso Regression
5. Evaluate using MSE and R²
6. Plot regression curves
7. Compare coefficients

---

💻 Program 

import numpy as np
import matplotlib.pyplot as plt

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# -----------------------------
# Generate Sample Data
# -----------------------------
np.random.seed(42)

X = np.linspace(-3, 3, 100).reshape(-1, 1)
y = 0.5 * X**2 + X + 2 + np.random.randn(100, 1) * 2

# -----------------------------
# Polynomial Features
# -----------------------------
poly = PolynomialFeatures(degree=10)
X_poly = poly.fit_transform(X)

# -----------------------------
# Train-Test Split
# -----------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X_poly, y, test_size=0.2, random_state=42
)

# -----------------------------
# Models
# -----------------------------
models = {
    "Linear": LinearRegression(),
    "Ridge": Ridge(alpha=1),
    "Lasso": Lasso(alpha=0.1, max_iter=10000)
}

results = {}

# -----------------------------
# Training and Evaluation
# -----------------------------
for name, model in models.items():
    
    model.fit(X_train, y_train)
    
    y_pred = model.predict(X_test)
    
    mse = mean_squared_error(y_test, y_pred)
    r2 = r2_score(y_test, y_pred)
    
    results[name] = (mse, r2)
    
    print(f"\n{name} Regression:")
    print("MSE:", round(mse, 4))
    print("R²:", round(r2, 4))

# -----------------------------
# Visualization
# -----------------------------
X_range = np.linspace(-3, 3, 200).reshape(-1, 1)
X_range_poly = poly.transform(X_range)

plt.figure(figsize=(8, 6))

plt.scatter(X, y, s=20, label="Data")

for name, model in models.items():
    y_range = model.predict(X_range_poly)
    plt.plot(X_range, y_range, label=name)

plt.xlabel("X")
plt.ylabel("y")
plt.title("Linear vs Ridge vs Lasso (with Evaluation)")
plt.legend()

plt.show()

# -----------------------------
# Coefficient Comparison
# -----------------------------
print("\nNumber of Non-Zero Coefficients:")

for name, model in models.items():
    count = np.sum(model.coef_ != 0)
    print(f"{name}: {count}")

---

📊 Sample Output 

Linear Regression: MSE: 3.2691 R²: 0.4823 Ridge Regression: MSE: 3.0232 R²: 0.5212 Lasso Regression: MSE: 2.9161 R²: 0.5382

Number of Non-Zero Coefficients: Linear: 10 Ridge: 10 Lasso: 8

---

📈 Interpretation

• Linear Regression
  • May overfit (high variance)
  • Higher MSE
• Ridge Regression
  • Best balance
  • Lower MSE, higher R²
• Lasso Regression
  • Slightly higher error than Ridge
  • Simpler model (fewer features)

---

📊 Observations

Model	MSE	R²	Behavior
Linear	High	Lower	Overfitting
Ridge	Lowest	Highest	Best generalization
Lasso	Moderate	Good	Feature selection

---

 Result

Ridge and Lasso regression improved model performance compared to ordinary linear regression by reducing overfitting and improving generalization.

• Regularization improves model efficiency
• Ridge is better when all features are important
• Lasso is useful for feature selection
• Evaluation metrics help choose the best model