Single Layer Neural Network for OR Gate Classification with Visualization

 

Experiment

Title:

Single Layer Neural Network for OR Gate Classification with Visualization

---

🎯 Objective

• To implement a single-layer neural network (logistic model)
• To understand:
  • Forward propagation
  • Sigmoid activation
  • Backpropagation
• To visualize the decision boundary (straight line)

---

Theory

---

🔹 1. Neural Network Model (Single Layer)

This is the simplest neural network:

• No hidden layer
• Direct mapping: input → output

Mathematically, it behaves like logistic regression.

---

🔹 2. Forward Propagation

The model computes:

$Z = XW + b$

$$\hat{y} = \sigma(Z)$$

Where:

• $X$ → input matrix
• $W$ → weight vector
• $b$ → bias
• $\hat{y}$ → predicted output

---

🔹 3. Sigmoid Activation Function

$\sigma(x) = \frac{1}{1 + e^{-x}}$

Properties:

• Output between 0 and 1
• Suitable for binary classification

Derivative:

$$\sigma'(x) = x(1 - x)$$

---

🔹 4. Loss Function (Mean Squared Error)

$$L = \frac{1}{n} \sum (\hat{y}-y)^2$$

---

🔹 5. Backpropagation (Derivation)

We compute gradient of loss w.r.t weights.

Using chain rule:

$$\frac{\partial L}{\partial W} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial Z} \cdot \frac{\partial Z}{\partial W}$$

---

Step-by-step:

• $\frac{\partial L}{\partial \hat{y}} = ( \hat{y} - y )$
  
• $\frac{\partial \hat{y}}{\partial Z} = \hat{y}(1 - \hat{y})$
  
• $\frac{\partial Z}{\partial W} = X$
  

---

✅ Final Gradient:

$$\delta = (y - \hat{y}) \cdot \hat{y}(1 - \hat{y})$$

---

🔹 6. Update Rules

Weights:

$$W = W + \eta \cdot X^T \cdot \delta$$

Bias:

$$b = b + \eta \cdot \sum \delta$$

👉 $\eta$ = learning rate

---

📊 Dataset (OR Gate)

x1	x2	y
0	0	0
0	1	1
1	0	1
1	1	1

---

💻 Python Program

import numpy as np
import matplotlib.pyplot as plt
# Sigmoid function
def sigmoid(x):
    return 1/(1+np.exp(-x))
def sigmoid_derivative(x):
    return x*(1-x)
# Inputs
x1 = np.array([0,0,1,1])
x2 = np.array([0,1,0,1])
# Target output (OR)
y = np.array([0,1,1,1])
np.random.seed(0)
# Initialize weights and bias
w1 = np.random.rand()
w2 = np.random.rand()
b = np.random.rand()
lr = 0.1
epochs = 2000
losses=[]
# Training
for epoch in range(epochs):
    # Forward pass
    z = x1*w1 + x2*w2 + b
    output = sigmoid(z)
    # Loss
    loss = np.mean((output-y)**2)
    losses.append(loss)
    # Backpropagation
    d_output = (output-y)*sigmoid_derivative(output)
    # Weight updates
    w1 -= lr*np.sum(x1*d_output)
    w2 -= lr*np.sum(x2*d_output)
    # Bias update
    b -= lr*np.sum(d_output)
print("Final weights:")
print("w1 =",w1)
print("w2 =",w2)
print("b =",b)
print("\nPredictions:")
print(np.round(output,3))
# Loss curve
plt.plot(losses)
plt.title("Loss vs Epochs")
plt.xlabel("Epochs")
plt.ylabel("Loss")
plt.show()

# Decision boundary
plt.scatter(x1,x2,c=y,s=100)
# Boundary:
# w1*x1 + w2*x2 + b = 0
x_vals=np.linspace(-0.5,1.5,100)
y_vals=-(w1*x_vals+b)/w2
plt.plot(x_vals,y_vals)
plt.title("Decision Boundary for OR Problem")
plt.xlabel("x1")
plt.ylabel("x2")
plt.xlim(-0.5,1.5)
plt.ylim(-0.5,1.5)
plt.show()

---

📈 Outputs

Final weights:
w1 = 4.242662208780825
w2 = 4.243605197016924
b = -1.8451162473914662

Predictions:
[0.136 0.917 0.917 0.999]

---

🔍 Observations

• Loss decreases over epochs
• Model predicts correctly
• Decision boundary is a straight line

---

Result

• The neural network successfully classifies OR data
• A linear boundary separates the classes