Multi Layer Neural Network for XOR Classification with Visualization

 

Experiment

Multi Layer Neural Network for XOR Classification with Visualization 

---

🎯 Objective

• Implement a simple neural network from scratch
• Train it on XOR logic
• Visualize:
• Loss curve
• Decision boundary

Theory

🔹 1. Neural Network Overview

A Neural Network consists of layers of neurons:

• Input Layer → receives data
• Hidden Layer → processes data
• Output Layer → produces prediction

Each connection has:

• Weight (W) → importance of input
• Bias (b) → adjusts output

---

🔹 2. Forward Pass

Forward pass is the process of computing the output from input.

Hidden Layer:

$Z_1 = XW_1 + b_1$

$$A_1 = \sigma(Z_1)$$

Output Layer:

$Z_2 = A_1 W_2 + b_2$

$$A_2 = \sigma(Z_2)$$

👉 $A_2$ is the predicted output.

---

🔹 3. Sigmoid Activation Function

The sigmoid function converts values into range (0,1):

$\sigma (x)=\frac{1}{1+e^{-\mathrm{x}}}$

Properties:

• Smooth and differentiable
• Used for binary classification

Derivative:

$$\sigma'(x) = x(1 - x)$$

---

🔹 4. Loss Function

We use Mean Squared Error (MSE):

$$Loss = \frac{1}{n} \sum (y - \hat{y})^2$$

---

🔹 5. Backpropagation

Backpropagation is used to update weights by propagating error backward.

Output Layer Error:

$$\delta_2 = (y - A_2) \cdot A_2(1 - A_2)$$

Hidden Layer Error:

$$\delta_1 = (\delta_2 \cdot W_2^T) \cdot A_1(1 - A_1)$$

---

🔹 6. Weight Update Rule

Weights are updated using gradient descent:

Output Weights:

$$W_2 = W_2 + A_1^T \cdot \delta_2 \cdot \eta$$

Hidden Weights:

$$W_1 = W_1 + X^T \cdot \delta_1 \cdot \eta$$

👉 $\eta$ = learning rate

---

📊 Dataset (XOR Problem)

x1	x2	y
0	0	0
0	1	1
1	0	1
1	1	0

---

🧠 Concept

• 2 input neurons
• 1 hidden layer (2 neurons)
• 1 output neuron
• Sigmoid activation

---

💻 Program 

import numpy as np

# Sigmoid activation
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Derivative of sigmoid
def sigmoid_derivative(x):
    return x * (1 - x)

# XOR dataset
X = np.array([[0,0],
              [0,1],
              [1,0],
              [1,1]])

y = np.array([[0],
              [1],
              [1],
              [0]])

np.random.seed(0)

# Network architecture
input_neurons = 2
hidden_neurons = 2
output_neurons = 1

# Initialize weights and biases
W1 = np.random.rand(input_neurons, hidden_neurons)
b1 = np.random.rand(1, hidden_neurons)

W2 = np.random.rand(hidden_neurons, output_neurons)
b2 = np.random.rand(1, output_neurons)

learning_rate = 0.1
epochs = 8000
losses=[]
# Training loop
for epoch in range(epochs):

    # -------- Forward Pass --------
    Z1 = np.dot(X, W1) + b1
    A1 = sigmoid(Z1)

    Z2 = np.dot(A1, W2) + b2
    y_pred = sigmoid(Z2)

    # -------- Loss --------
    loss = np.mean((y - y_pred)**2)
    losses.append(loss)
    # -------- Backpropagation --------
    error = y - y_pred

    d_output = error * sigmoid_derivative(y_pred)

    error_hidden = np.dot(d_output, W2.T)
    d_hidden = error_hidden * sigmoid_derivative(A1)

    # -------- Gradients --------
    dW2 = np.dot(A1.T, d_output)
    db2 = np.sum(d_output, axis=0, keepdims=True)

    dW1 = np.dot(X.T, d_hidden)
    db1 = np.sum(d_hidden, axis=0, keepdims=True)

    # -------- Update --------
    W2 += learning_rate * dW2
    b2 += learning_rate * db2

    W1 += learning_rate * dW1
    b1 += learning_rate * db1

# -------- Output --------
print("Final Predictions:\n", y_pred)

Final Predictions:
 [[0.07248634]
 [0.93166221]
 [0.93162608]
 [0.07484762]]

---

1. Loss Visualization

plt.plot(losses)
plt.title("Loss vs Epochs")
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.show()

---

2. Decision Boundary Visualization 

import numpy as np
import matplotlib.pyplot as plt

# Prediction function
def predict(X):
    A1 = 1 / (1 + np.exp(-(np.dot(X, W1) + b1)))
    A2 = 1 / (1 + np.exp(-(np.dot(A1, W2) + b2)))
    return A2

# Grid
x_min, x_max = -0.5, 1.5
y_min, y_max = -0.5, 1.5

xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))

grid = np.c_[xx.ravel(), yy.ravel()]
Z = predict(grid)

# Convert probabilities to classes
Z = (Z > 0.5).astype(int)
Z = Z.reshape(xx.shape)

#  Decision boundary colors
plt.contourf(xx, yy, Z, alpha=0.4, cmap=plt.cm.coolwarm)

#  Data points with clear colors
for i in range(len(X)):
    if y[i] == 0:
        plt.scatter(X[i,0], X[i,1], color='blue', edgecolors='black', s=100)
    else:
        plt.scatter(X[i,0], X[i,1], color='red', edgecolors='black', s=100)

plt.title("Decision Boundary (Colored)")
plt.xlabel("x1")
plt.ylabel("x2")

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)

plt.show()

---

Result

• Neural network correctly learns XOR logic
• Loss decreases over epochs
• Decision boundary clearly separates classes