Implementation of Decision Tree using ID3 Algorithm with Visualization (scikit-learn)

 

Experiment

Title

Implementation of Decision Tree using ID3 Algorithm with Visualization

---

🎯 Objective

• To implement ID3 Decision Tree
• To compute:
  • Entropy
  • Information Gain
• To visualize the decision tree

---

📊 Sample Dataset (Play Golf)

Outlook	Temperature	Humidity	Windy	Play
Rainy	Hot	High	False	Yes
Rainy	Hot	High	True	No
Overcast	Hot	High	False	Yes
Sunny	Mild	High	False	No
Sunny	Cool	Normal	False	Yes
Sunny	Cool	Normal	True	No
Overcast	Cool	Normal	True	Yes
Rainy	Mild	High	False	No
Rainy	Cool	Normal	False	Yes
Sunny	Mild	Normal	False	Yes
Rainy	Mild	Normal	True	Yes
Overcast	Mild	High	True	Yes
Overcast	Hot	Normal	False	Yes
Sunny	Mild	High	True	No

---

📚 Theory

🔹 ID3 Algorithm

• Uses Entropy & Information Gain
• Builds tree top-down
• Greedy approach

---

💻 Complete Python Program

# -------------------------------
# 1. Import Libraries
# -------------------------------
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

# -------------------------------
# 2. Dataset
# -------------------------------
data = {
    'Outlook': ['Rainy','Rainy','Overcast','Sunny','Sunny','Sunny','Overcast',
                'Rainy','Rainy','Sunny','Rainy','Overcast','Overcast','Sunny'],
    'Temperature': ['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool',
                    'Mild','Mild','Mild','Hot','Mild'],
    'Humidity': ['High','High','High','High','Normal','Normal','Normal','High',
                 'Normal','Normal','Normal','High','Normal','High'],
    'Windy': ['False','True','False','False','False','True','True','False',
              'False','False','True','True','False','True'],
    'Play': ['Yes','No','Yes','No','Yes','No','Yes','No','Yes','Yes','Yes','Yes','Yes','No']
}

df = pd.DataFrame(data)

# -------------------------------
# 3. Encode Categorical Data
# -------------------------------
le = LabelEncoder()

for col in df.columns:
    df[col] = le.fit_transform(df[col])

# Split features and target
X = df.drop('Play', axis=1)
y = df['Play']

# -------------------------------
# 4. Train Decision Tree (ID3)
# -------------------------------
# entropy criterion = ID3
model = DecisionTreeClassifier(criterion='entropy')
model.fit(X, y)

# -------------------------------
# 5. Visualization
# -------------------------------
plt.figure(figsize=(10,8))

plot_tree(model,
          feature_names=X.columns,
          class_names=['No','Yes'],
          filled=True)

plt.title("Decision Tree using ID3")
plt.show()

---

📊 Output

---

🔍 Observations

• Tree matches manual ID3 calculation
• Entropy-based splitting
• Pure nodes become leaves

---

🧪 Lab Tasks

Task 1

Change criterion:

criterion = 'gini'

---

Task 2

Limit depth:

max_depth = 2

---

Task 3

Test prediction:

model.predict([[...]])

---

📊 Key Insights

Concept	Meaning
Entropy	Impurity
Gain	Feature selection
Leaf node	Final decision

---

Result

• ID3 builds tree using:
  • Entropy
  • Information Gain
• Easy to interpret
• Works well for categorical data