Implementation of Bernoulli Naive Bayes Classifier Using Scikit-Learn

 

Experiment

Implementation of Bernoulli Naive Bayes Classifier Using Scikit-Learn

Aim

To implement a Bernoulli Naive Bayes classifier using Scikit-Learn for classifying text messages as Spam or Not Spam.

---

Theory

Bernoulli Naive Bayes

Bernoulli Naive Bayes is a variant of the Naive Bayes algorithm designed for binary-valued features.

It assumes that each feature can take only two values:

$$0 \quad \text{or} \quad 1$$

where:

• 1 → Feature is present
• 0 → Feature is absent

The classifier uses Bayes' theorem:

$$P(C|X)=\frac{P(X|C)P(C)}{P(X)}$$

where:

• $C$ = Class label
• $X$ = Feature vector
• $P(C|X)$= Posterior probability
• $P(X|C)$= Likelihood
• $P(C)$ = Prior probability
• $P(X)$= Evidence

Bernoulli Naive Bayes considers only the presence or absence of features, not their frequency.

---

Dataset Description

We use a small SMS dataset.

Message	Class
free money now	Spam
win prize now	Spam
limited time offer	Spam
project meeting tomorrow	Not Spam
schedule the meeting	Not Spam
discuss project details	Not Spam

Target Classes:

• Spam
• Not Spam

---

Algorithm

1. Create the text dataset.
2. Convert text into binary feature vectors.
3. Train the Bernoulli Naive Bayes classifier.
4. Test the model with a new message.
5. Predict the class label.
6. Display class probabilities.

---

Program

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import BernoulliNB

# -----------------------------------
# 1. Training Dataset
# -----------------------------------
texts = [
    "free money now",
    "win prize now",
    "limited time offer",
    "project meeting tomorrow",
    "schedule the meeting",
    "discuss project details"
]

labels = [
    "Spam",
    "Spam",
    "Spam",
    "Not Spam",
    "Not Spam",
    "Not Spam"
]

# -----------------------------------
# 2. Convert Text to Binary Features
# -----------------------------------
vectorizer = CountVectorizer(binary=True)

X = vectorizer.fit_transform(texts)

print("Vocabulary:")
print(vectorizer.get_feature_names_out())

print("\nBinary Feature Matrix:")
print(X.toarray())

# -----------------------------------
# 3. Train BernoulliNB
# -----------------------------------
model = BernoulliNB()

model.fit(X, labels)

# -----------------------------------
# 4. Test Message
# -----------------------------------
test_message = ["free prize"]

X_test = vectorizer.transform(test_message)

# -----------------------------------
# 5. Prediction
# -----------------------------------
prediction = model.predict(X_test)

print("\nTest Message:")
print(test_message[0])

print("\nPredicted Class:")
print(prediction[0])

# -----------------------------------
# 6. Class Probabilities
# -----------------------------------
probabilities = model.predict_proba(X_test)

print("\nClass Probabilities:")
print(probabilities)

for cls, prob in zip(model.classes_, probabilities[0]):
    print(f"{cls}: {prob:.4f}")

---

Output

Vocabulary:
['details' 'discuss' 'free' 'limited' 'meeting' 'money' 'now' 'offer'
 'prize' 'project' 'schedule' 'the' 'time' 'tomorrow' 'win']

Binary Feature Matrix:
[[0 0 1 0 0 1 1 0 0 0 0 0 0 0 0]
 [0 0 0 0 0 0 1 0 1 0 0 0 0 0 1]
 [0 0 0 1 0 0 0 1 0 0 0 0 1 0 0]
 [0 0 0 0 1 0 0 0 0 1 0 0 0 1 0]
 [0 0 0 0 1 0 0 0 0 0 1 1 0 0 0]
 [1 1 0 0 0 0 0 0 0 1 0 0 0 0 0]]

Test Message:
free prize

Predicted Class:
Spam

Class Probabilities:
[[0.11111111 0.88888889]]
Not Spam: 0.1111
Spam: 0.8889

---

Explanation of binary=True

vectorizer = CountVectorizer(binary=True)

Normally:

"free free prize"

would become:

[2,1]

(counts)

With binary=True:

[1,1]

Only presence is recorded.

This matches the Bernoulli Naive Bayes assumption that features are binary.

---

Result

Thus, the Bernoulli Naive Bayes classifier was successfully implemented using Scikit-Learn. The model classified the test message "free prize" as Spam based on the presence of spam-related words.

• Bernoulli Naive Bayes works with binary features (0/1).
• It considers only whether a word is present or absent.
• CountVectorizer(binary=True) is commonly used to prepare data for BernoulliNB.
• The classifier is suitable for spam filtering and other binary text classification tasks.
• It is simple, efficient, and performs well on text data with binary feature representations.