Implementation of Linear SVM from Scratch using sample data

 

Experiment 

 Implementation of Linear SVM from Scratch (Without Built-in Functions)

---

🎯 Objective

To implement a Support Vector Machine (SVM) classifier manually using Python without using built-in SVM functions.

---

📚 Theory

A linear SVM tries to find a hyperplane:

$$w \cdot x + b = 0$$

such that:

• Correct classification is ensured
• Margin is maximized

The optimization problem is:

$$\min \frac{1}{2} ||w||^2$$

Subject to:

$y_i (w \cdot x_i + b) \geq 1$

---

🔁 Loss Function (Hinge Loss)

Hinge Loss

$$\boxed{ L=\max(0,1-y(w^Tx+b)) }$$

It penalizes:

• incorrectly classified points
• correctly classified points inside the margin

---

SVM Objective

$$\boxed{ J(w,b)= \frac12\|w\|^2+ C\sum_i\max(0,1-y_i(w^Tx_i+b)) }$$

​

---

📊 Dataset

We use a small dataset:

x₁	x₂	y
2	2	0
4	5	1
7	4	1

---

🔬 Algorithm / Procedure

1. Initialize weights $w$ and bias $b$
2. Convert labels to (-1, +1)
3. For each epoch:
   • Check condition $y(w\cdot x+b)\ge 1$
   • Update weights:
     • If correct → only regularization update
     • If incorrect → update using hinge loss
4. Repeat until convergence
5. Use final $w, b$ for prediction
6. Compute hing loss
7. Plot the graph

---

💻 Program (SVM from Scratch)

import numpy as np

# ---------------------------------------------
# 1. Training Data
# ---------------------------------------------

X = np.array([
    [2, 2],
    [4, 5],
    [7, 4]
])

# Original class labels
y_original = np.array([0, 1, 1])

# Convert labels:
# 0 -> -1
# 1 -> +1
y = np.where(y_original == 0, -1, 1)


# ---------------------------------------------
# 2. Initialize Parameters
# ---------------------------------------------

# Number of features
n_features = X.shape[1]

# Initialize weights and bias
w = np.zeros(n_features)
b = 0.0

# Learning rate
learning_rate = 0.01

# Regularization parameter
C = 1.0

# Number of training iterations
epochs = 1000


# ---------------------------------------------
# 3. Training using Gradient Descent
# ---------------------------------------------

for epoch in range(epochs):

    for i in range(len(X)):

        # Get one training example
        xi = X[i]
        yi = y[i]

        # Calculate decision function
        decision_value = np.dot(w, xi) + b

        # Calculate margin
        margin = yi * decision_value

        # Check whether hinge loss is active
        if margin >= 1:

            # Correctly classified and outside margin
            # Only regularization gradient
            dw = w
            db = 0

        else:

            # Inside margin or incorrectly classified
            # Regularization + hinge loss gradient
            dw = w - C * yi * xi
            db = -C * yi

        # Update weights
        w = w - learning_rate * dw

        # Update bias
        b = b - learning_rate * db


# ---------------------------------------------
# 4. Display Learned Parameters
# ---------------------------------------------

print("Weights:", w)
print("Bias:", b)


# ---------------------------------------------
# 5. Decision Boundary
# ---------------------------------------------

print("\nDecision Boundary:")

print(f"{w[0]:.4f} * X1 + {w[1]:.4f} * X2 + {b:.4f} = 0")


# ---------------------------------------------
# 6. Prediction Function
# ---------------------------------------------

def predict(X_new):

    decision_values = np.dot(X_new, w) + b

    # Convert to class labels -1 and +1
    predictions = np.where(decision_values >= 0, 1, -1)

    # Convert back:
    # -1 -> 0
    # +1 -> 1
    predictions = np.where(predictions == -1, 0, 1)

    return predictions


# ---------------------------------------------
# 7. Test on Training Data
# ---------------------------------------------

predictions = predict(X)

print("\nPredictions:")

for i in range(len(X)):
    print(
        "Point:", X[i],
        "Actual:", y_original[i],
        "Predicted:", predictions[i]
    )


# ---------------------------------------------
# 8. Calculate Hinge Loss
# ---------------------------------------------

hinge_losses = []

for i in range(len(X)):

    decision_value = np.dot(w, X[i]) + b

    hinge_loss = max(
        0,
        1 - y[i] * decision_value
    )

    hinge_losses.append(hinge_loss)


print("\nHinge Loss for each point:")

for i, loss in enumerate(hinge_losses):
    print(f"Point {X[i]}: {loss:.4f}")


print("\nTotal Hinge Loss:", sum(hinge_losses))
import matplotlib.pyplot as plt

# Separate the two classes
X_class0 = X[y_original == 0]
X_class1 = X[y_original == 1]

# Plot Class 0
plt.scatter(
    X_class0[:, 0],
    X_class0[:, 1],
    label="Class 0"
)

# Plot Class 1
plt.scatter(
    X_class1[:, 0],
    X_class1[:, 1],
    label="Class 1"
)


# Create values for X1
x1_values = np.linspace(0, 9, 100)


# Calculate X2 values for decision boundary
#
# w1*x1 + w2*x2 + b = 0
#
# Therefore:
#
# x2 = -(w1*x1 + b) / w2

if abs(w[1]) > 1e-10:

    x2_values = -(w[0] * x1_values + b) / w[1]

    plt.plot(
        x1_values,
        x2_values,
        label="Decision Boundary"
    )


# Labels
plt.xlabel("X1")
plt.ylabel("X2")

plt.title("Linear SVM using Hinge Loss and Gradient Descent")

plt.legend()
plt.grid()

plt.show()

---

📊 Output

Weights: [0.3032517  0.45804657]
Bias: -2.569999999999989

Decision Boundary:
0.3033 * X1 + 0.4580 * X2 + -2.5700 = 0

Predictions:
Point: [2 2] Actual: 0 Predicted: 0
Point: [4 5] Actual: 1 Predicted: 1
Point: [7 4] Actual: 1 Predicted: 1

Hinge Loss for each point:
Point [2 2]: 0.0000
Point [4 5]: 0.0668
Point [7 4]: 0.0000

Total Hinge Loss: 0.06676037892843523

---

✅ Result

The SVM classifier was successfully implemented from scratch using gradient descent and hinge loss, achieving correct classification of the dataset.