Implementation of XOR Problem using a Small Neural Network in TensorFlow/Keras and Visualization

 

Experiment 

Implementation of XOR Problem using a Small Neural Network in TensorFlow/Keras

---

Aim

To implement and train a small neural network using TensorFlow/Keras for solving the XOR problem and visualize:

• Neural Network architecture
• Loss curve
• Accuracy curve
• Decision boundary
• Final predictions

---

Objectives

• To understand the XOR classification problem
• To build a feed-forward neural network using Keras
• To train the model using backpropagation
• To evaluate model performance
• To visualize learning behavior

---

🧠 Theory

XOR Problem

XOR (Exclusive OR) gives output 1 only when the inputs are different.

Truth table:

x₁	x₂	Output
0	0	0
0	1	1
1	0	1
1	1	0

---

Why XOR is Important

XOR is a non-linearly separable problem.

A single straight line cannot separate the classes.

Example:

(0,1) and (1,0) → Class 1

(0,0) and (1,1) → Class 0

Thus:

• Single-layer neural networks fail
• Hidden layers are required

---

Neural Network Architecture

The neural network used:

• Input layer → 2 neurons
• Hidden layer → 2 neurons
• Output layer → 1 neuron

Architecture:

Input Layer          Hidden Layer          Output Layer

x1 --------------> h1 ------------\
                                    \
                                     > Output
                                    /
x2 --------------> h2 ------------/

---

Forward Propagation

Each neuron computes:

$$z=XW+b$$

Activation:

$$A=\sigma (z)$$

where sigmoid function is:

Output values lie between:

$$0\le y\le 1$$

---

Backpropagation

During training:

1. Forward propagation computes prediction
2. Error is computed
3. Chain rule calculates gradients
4. Weights are updated

Weight update rule:

$w_{new}=w_{old}-\eta\frac{\partial E}{\partial w}$

where:

• $\eta$= learning rate
• $\frac{\partial E}{\partial w}$= gradient

---

Loss Function

Binary Crossentropy:

$L=-[y\log (\widehat{y})+(1-y)\log (1-\widehat{y})]$

---

Algorithm

1. Load XOR dataset
2. Create neural network architecture
3. Compile model
4. Train using backpropagation
5. Evaluate predictions
6. Plot loss and accuracy
7. Visualize decision boundary

---

💻 Program

Step 1: Import Libraries

import numpy as np
import matplotlib.pyplot as plt

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.utils import plot_model

---

Step 2: Create XOR Dataset

X = np.array([[0,0],
              [0,1],
              [1,0],
              [1,1]])

y = np.array([[0],
              [1],
              [1],
              [0]])

---

Step 3: Build Neural Network

model = Sequential([

    Input(shape=(2,)),

    Dense(2, activation='sigmoid'),

    Dense(1, activation='sigmoid')

])

---

Step 4: Compile Model

model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

---

Step 5: Train Model

history=model.fit(
    X,
    y,
    epochs=5000,
    verbose=0
)

---

Step 6: Display Model Summary

model.summary()

---

Step 7: Predictions

pred=model.predict(X)

print("Predicted values:")
print(np.round(pred,3))

print("\nPredicted classes:")

print((pred>0.5).astype(int))

---

Step 8: Plot Loss Curve

plt.plot(history.history['loss'])

plt.title("Loss vs Epochs")

plt.xlabel("Epochs")
plt.ylabel("Loss")

plt.show()

---

Step 9: Plot Accuracy Curve

plt.plot(history.history['accuracy'])

plt.title("Accuracy vs Epochs")

plt.xlabel("Epochs")
plt.ylabel("Accuracy")

plt.show()

---

Step 10: Decision Boundary Visualization

xx,yy=np.meshgrid(
    np.linspace(-0.5,1.5,200),
    np.linspace(-0.5,1.5,200)
)

grid=np.c_[xx.ravel(),yy.ravel()]

Z=model.predict(grid)

Z=Z.reshape(xx.shape)

plt.contourf(
    xx,
    yy,
    Z,
    levels=50,
    cmap=plt.cm.coolwarm
)

plt.scatter(
    X[:,0],
    X[:,1],
    c=y.flatten(),
    s=100,
    edgecolors='black'
)

plt.xlabel("x1")
plt.ylabel("x2")

plt.title(
    "Decision Boundary for XOR"
)

plt.show()

---

Step 11: Neural Network Visualization

plot_model(
    model,
    show_shapes=True,
    show_layer_names=True,
    to_file='model.png'
)

If needed:

!pip install pydot
!apt install graphviz -y

Display model:

from IPython.display import Image
Image("model.png")

---

Sample Output

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 2)              │             6 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 1)              │             3 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 29 (120.00 B)

 Trainable params: 9 (36.00 B)

 Non-trainable params: 0 (0.00 B)

 Optimizer params: 20 (84.00 B)

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 63ms/step
Predicted values:
[[0.075]
 [0.919]
 [0.92 ]
 [0.082]]

Predicted classes:
[[0]
 [1]
 [1] 

 [0]]

📊 Expected Observations

• Loss gradually decreases
• Accuracy gradually increases
• Final accuracy approaches 100%
• Non-linear decision boundary appears

---

Result

A small neural network was successfully implemented using TensorFlow/Keras to solve the XOR problem. The model correctly learned the non-linear relationship between inputs and achieved accurate predictions.