Implementation of Categorical Naive Bayes Classifier using Scikit-Learn

 

Experiment

Implementation of Categorical Naive Bayes Classifier using Scikit-Learn

Aim

To implement a Naive Bayes Classifier using Scikit-Learn on a categorical weather dataset and predict whether a game will be played based on weather conditions.

---

Theory

Naive Bayes Classifier

Naive Bayes is a supervised machine learning algorithm based on Bayes' Theorem. It is primarily used for classification tasks.

Bayes' theorem is given by:

$$P(C|X) = \frac{P(X|C)P(C)}{P(X)}$$

Where:

Term	Meaning
$C$	Class label
$X$	Feature vector
P(C|X)	Posterior
P(X|C)	Likelyhood
$P(C)$	Prior probability
$P(X)$	Evidence

The classifier predicts the class having the highest posterior probability.

Naive Assumption

Naive Bayes assumes that all features are conditionally independent given the class.

$$P(X|C)=P(x_1|C)P(x_2|C)\cdots P(x_n|C)$$

This assumption simplifies computation and makes the algorithm efficient for classification tasks.

---

Dataset Description

The dataset contains weather-related attributes:

Attribute	Possible Values
Outlook	Sunny, Rainy, Overcast
Temperature	Hot, Mild, Cool
Humidity	High, Normal
Windy	True, False
Play	Yes, No

Target Variable:

• Play = Yes
• Play = No

---

Algorithm

1. Create the dataset.
2. Separate features and target variable.
3. Convert categorical values into numerical values using Label Encoding.
4. Train a Naive Bayes classifier.
5. Test the classifier using a sample record.
6. Display the prediction and class probabilities.

---

Program

import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.naive_bayes import CategoricalNB

# -----------------------------------
# 1. Create Dataset
# -----------------------------------
data = {
    'Outlook': ['Rainy','Rainy','Overcast','Sunny','Sunny','Sunny','Overcast',
                'Rainy','Rainy','Sunny','Rainy','Overcast','Overcast','Sunny'],
    'Temperature': ['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool',
                    'Mild','Mild','Mild','Hot','Mild'],
    'Humidity': ['High','High','High','High','Normal','Normal','Normal','High',
                 'Normal','Normal','Normal','High','Normal','High'],
    'Windy': ['False','True','False','False','False','True','True','False',
              'False','False','True','True','False','True'],
    'Play': ['Yes','No','Yes','No','Yes','No','Yes','No','Yes','Yes','Yes',
             'Yes','Yes','No']
}

df = pd.DataFrame(data)

print("Dataset:")
print(df)

# -----------------------------------
# 2. Encode Categorical Features
# -----------------------------------
encoders = {}

for col in df.columns:
    le = LabelEncoder()
    df[col] = le.fit_transform(df[col])
    encoders[col] = le

print("\nEncoded Dataset:")
print(df)

# -----------------------------------
# 3. Split Features and Target
# -----------------------------------
X = df.drop('Play', axis=1)
y = df['Play']

# -----------------------------------
# 4. Train Naive Bayes Model
# -----------------------------------
model = CategoricalNB()

model.fit(X, y)

# -----------------------------------
# 5. Test Sample
# -----------------------------------
test_sample = pd.DataFrame({
    'Outlook':['Sunny'],
    'Temperature':['Cool'],
    'Humidity':['High'],
    'Windy':['True']
})

# Encode test sample
for col in test_sample.columns:
    test_sample[col] = encoders[col].transform(test_sample[col])

# -----------------------------------
# 6. Prediction
# -----------------------------------
prediction = model.predict(test_sample)

predicted_class = encoders['Play'].inverse_transform(prediction)

print("\nPrediction:")
print(predicted_class[0])

# -----------------------------------
# 7. Class Probabilities
# -----------------------------------
probabilities = model.predict_proba(test_sample)

print("\nClass Probabilities:")
print(probabilities)

classes = encoders['Play'].inverse_transform(model.classes_)

for cls, prob in zip(classes, probabilities[0]):
    print(f"{cls}: {prob:.4f}")

---

Output

Dataset:
     Outlook Temperature Humidity  Windy Play
0      Rainy         Hot     High  False  Yes
1      Rainy         Hot     High   True   No
2   Overcast         Hot     High  False  Yes
3      Sunny        Mild     High  False   No
4      Sunny        Cool   Normal  False  Yes
5      Sunny        Cool   Normal   True   No
6   Overcast        Cool   Normal   True  Yes
7      Rainy        Mild     High  False   No
8      Rainy        Cool   Normal  False  Yes
9      Sunny        Mild   Normal  False  Yes
10     Rainy        Mild   Normal   True  Yes
11  Overcast        Mild     High   True  Yes
12  Overcast         Hot   Normal  False  Yes
13     Sunny        Mild     High   True   No

Encoded Dataset:
    Outlook  Temperature  Humidity  Windy  Play
0         1            1         0      0     1
1         1            1         0      1     0
2         0            1         0      0     1
3         2            2         0      0     0
4         2            0         1      0     1
5         2            0         1      1     0
6         0            0         1      1     1
7         1            2         0      0     0
8         1            0         1      0     1
9         2            2         1      0     1
10        1            2         1      1     1
11        0            2         0      1     1
12        0            1         1      0     1
13        2            2         0      1     0

Prediction:
No

Class Probabilities:
[[0.72006665 0.27993335]]
No: 0.7201
Yes: 0.2799

---

Result

Thus, a Naive Bayes classifier was successfully implemented using the Scikit-Learn library. The model was trained using weather-related categorical attributes and used to predict whether a game would be played under given weather conditions.

• Naive Bayes is a probabilistic classification algorithm based on Bayes' theorem.
• It assumes conditional independence among features.
• CategoricalNB is suitable for categorical datasets such as the Play Tennis dataset.
• The classifier predicts the class with the highest posterior probability.
• Naive Bayes is simple, computationally efficient, and effective for many classification problems