KNN for Multi-Class Classification with Visualization and Distance-Weighted Voting

 

Experiment

Title

KNN for Multi-Class Classification with Visualization and Distance-Weighted Voting

---

🎯 Objective

• To implement KNN for multi-class classification
• To visualize classification
• To handle tie cases using distance-weighted KNN

---

📊 Sample Dataset

We classify fruits into 3 classes:

• 🍎 Apple (0)
• 🍊 Orange (1)
• 🍇 Grapes (2)

Weight	Size	Class
150	7.0	Apple
160	7.2	Apple
170	7.5	Apple
130	6.5	Orange
120	6.2	Orange
110	6.0	Orange
100	5.5	Grapes
90	5.0	Grapes
95	5.2	Grapes

---

📚 Key Idea

• Find K nearest neighbors
• Use distance-weighted voting to avoid ties
• In weighted KNN, closer neighbors are given higher importance (weights) than farther neighbors.

$$\text{Weight} = \frac{1}{\text{distance}}$$

​Multi-Class Procedure

Algorithm:

1. Compute distance from test point to all training points
2. Select $k$ nearest neighbors
3. Compute weights based on distances
4. Sum weights class-wise
5. Choose class with highest total weight 

---

💻 Python Program

import numpy as np

import matplotlib.pyplot as plt

from collections import defaultdict

# -------------------------------

# 1. Dataset

# -------------------------------

X = np.array([

    [150, 7.0], [160, 7.2], [170, 7.5],   # Apple

    [130, 6.5], [120, 6.2], [110, 6.0],   # Orange

    [100, 5.5], [90, 5.0], [95, 5.2]      # Grapes

])

y = np.array([0,0,0, 1,1,1, 2,2,2])

classes = {0: "Apple", 1: "Orange", 2: "Grapes"}

# -------------------------------

# 2. Distance Function

# -------------------------------

def euclidean_distance(a, b):

    return float(np.sqrt(np.sum((a - b)**2)))

# -------------------------------

# 3. Distance-Weighted KNN

# -------------------------------

def weighted_knn(X, y, test_point, k):

    distances = []

   

    # Compute distances

    for i in range(len(X)):

        dist = euclidean_distance(X[i], test_point)

        distances.append((dist, float(y[i])))

   

    # Sort

    distances.sort(key=lambda x: x[0])

   

    # Select K neighbors

    neighbors = distances[:k]

   

    # Weighted voting

    weights = defaultdict(float)

   

    for dist, label in neighbors:

        weights[label] += 1 / (dist + 1e-5)

   

    # Choose class with max weight

    prediction = max(weights, key=weights.get)

   

    return prediction, neighbors, weights

# -------------------------------

# 4. Test Point

# -------------------------------

test_point = np.array([140, 6.4])

prediction, neighbors, weights = weighted_knn(X, y, test_point, k=3)

# -------------------------------

# 5. Output

# -------------------------------

print("Test Point:", test_point)

print("\nNeighbors (distance, class):")

for n in neighbors:

    print(n)

print("\nWeights:", dict(weights))

print("\nPredicted Class:", classes[prediction])

# -------------------------------

# 6. Visualization

# -------------------------------

colors = ['red', 'orange', 'purple']

plt.figure()

# Plot training data

for i in range(3):

    plt.scatter(X[y==i][:,0], X[y==i][:,1],

                color=colors[i], label=classes[i])

# Plot test point

plt.scatter(test_point[0], test_point[1],

            color='black', marker='x', s=120, label='Test Point')

# Draw lines to neighbors

for dist, label in neighbors:

    idx = np.where(y == label)[0][0]

    plt.plot([test_point[0], X[idx][0]],

             [test_point[1], X[idx][1]],

             linestyle='--', color='gray')

plt.xlabel("Weight")

plt.ylabel("Size")

plt.title("KNN Multi-Class with Distance-Weighted Voting")

plt.legend()

plt.grid()

plt.show()

📈 Expected Output

Test Point: [140. 6.4] Neighbors (distance, class): (10.000499987500625, 1.0) (10.017983829094554, 0.0) (20.00099997500125, 1.0) Weights: {1.0: 0.14999237557506437, 0.0: 0.09982038490538599} Predicted Class: Orange

📊 Key Insights

Method	Behavior
Normal KNN	May cause tie
Weighted KNN	Resolves tie
Small K	Sensitive
Large K	Smooth

---

Result

• KNN works naturally for multi-class problems
• Distance-weighted KNN:
  • Improves accuracy
  • Resolves ties
• Visualization helps understand decision