Prediction of Playing Golf using Naive Bayes Classifier ( Toy example)

 

Experiment

Title

Prediction of Playing Golf using Naive Bayes Classifier

---

🎯 Objective

• To implement a Naive Bayes classifier
• To compute probabilities using:
  • Prior probabilities
  • Conditional probabilities
• To predict whether Play Golf = Yes/No

---

📊 Dataset

Outlook	Temp	Humidity	Windy	Play
Rainy	Hot	High	False	Yes
Rainy	Hot	High	True	No
Overcast	Hot	High	False	Yes
Sunny	Mild	High	False	No
Sunny	Cool	Normal	False	Yes
Sunny	Cool	Normal	True	No
Overcast	Cool	Normal	True	Yes
Rainy	Mild	High	False	No
Rainy	Cool	Normal	False	Yes
Sunny	Mild	Normal	False	Yes
Rainy	Mild	Normal	True	Yes
Overcast	Mild	High	True	Yes
Overcast	Hot	Normal	False	Yes
Sunny	Mild	High	True	No

This creates a small dataset with:

Feature	Meaning
Outlook	Weather condition
Temperature	Hot/Mild/Cool
Humidity	High/Normal
Windy	True/False
Play	Target class

Play is what we want to predict.

Example:

Outlook	Temp	Humidity	Windy	Play
Sunny	Cool	High	True	No

---

📚 Theory

🔹 Naive Bayes

Bayes theorem is used in machine learning to calculate the probability that a data sample belongs to a particular class based on its features.

The formula is:

$$P(C|X) = \frac{P(X|C)\,P(C)}{P(X)}$$

Where:

Term	Meaning
$C$	Class label (e.g., Yes/No, Spam/Not Spam)
$X$	Feature vector (input data)
$P(C)$	Prior Probability - initial probability of class $C$ before seeing data
$P(X)$	Evidence - total probability of observing features $X$

P(X∣C)Prior Probability -Probability of observing features X when the class is already known to be C.

P(C∣X)Posterior Probability-Probability of class $C$ given the observed features $X$.It is the final probability after seeing the input data.

In classification problems, the goal is to compute the posterior probability for each class and choose the class with the highest probability.

In Naive Bayes classification, the conditional independence assumption simplifies the likelihood term:

$$P(X|C) = \prod_{i=1}^{n} P(x_i|C)$$

Thus, the classifier becomes:

$$P(C|X) \propto P(C)\prod_{i=1}^{n} P(x_i|C)$$

Meaning:

• Start with the prior probability of a class
• Multiply by probabilities of each feature given that class

        (Assume features are independent - Naive ) 

• Compute this for all classes
• Select the class with maximum posterior probability

The algorithm assumes:

$$P(x_1,x_2,x_3,x_4|C) = P(x_1|C)P(x_2|C)P(x_3|C)P(x_4|C)$$

Meaning:

• Outlook independent of Temperature
• Temperature independent of Humidity
• etc.

Laplace Smoothing

return (count + 1) / (total + unique_vals)

This is:

$P(x_i\mid C)=\frac{\text{count}(x_i,C)+1}{\text{count}(C)+k}$

where:

• $k$= number of unique feature values

---

Why smoothing?

Without smoothing:

If a feature value never appeared in a class:

$$P(x_i|C)=0$$

Then the entire posterior becomes zero because probabilities are multiplied.

Laplace smoothing avoids this by adding 1 to all counts.

---

⚙️ Steps

1. Compute prior probabilities
2. Compute conditional probabilities
3. Apply Naive Bayes formula
4. Predict class

---

💻 Simple Python Program

import pandas as pd

# -------------------------------

# 1. Create Dataset

# -------------------------------

data = {

    'Outlook': ['Rainy','Rainy','Overcast','Sunny','Sunny','Sunny','Overcast',

                'Rainy','Rainy','Sunny','Rainy','Overcast','Overcast','Sunny'],

    'Temperature': ['Hot','Hot','Hot','Mild','Cool','Cool','Cool','Mild','Cool',

                    'Mild','Mild','Mild','Hot','Mild'],

    'Humidity': ['High','High','High','High','Normal','Normal','Normal','High',

                 'Normal','Normal','Normal','High','Normal','High'],

    'Windy': ['False','True','False','False','False','True','True','False',

              'False','False','True','True','False','True'],

    'Play': ['Yes','No','Yes','No','Yes','No','Yes','No','Yes','Yes','Yes','Yes','Yes','No']

}

df = pd.DataFrame(data)

# -------------------------------

# 2. Priors

# -------------------------------

P_yes = len(df[df['Play']=='Yes']) / len(df)

P_no = len(df[df['Play']=='No']) / len(df)

# -------------------------------

# 3. Conditional Probability Function (Laplace smoothing)

# -------------------------------

def cond_prob(feature, value, label):

    subset = df[df['Play'] == label]

    count = len(subset[subset[feature] == value])

    total = len(subset)

    print(subset[subset[feature]==value],count,"/",total,"=",count/total)

    unique_vals = df[feature].nunique()

# Laplace smoothing return (count + 1) / (total + unique_vals)

   

# -------------------------------

# 4. Prediction Function

# -------------------------------

def predict(sample):

    # Initialize with priors

    prob_yes = P_yes

    prob_no = P_no

    for feature in sample:

        prob_yes *= cond_prob(feature, sample[feature], 'Yes')

        prob_no *= cond_prob(feature, sample[feature], 'No')

        print("probalities Yes No")

        print(prob_yes,prob_no)

    return "Yes" if prob_yes > prob_no else "No"

# -------------------------------

# 5. Test Example

# -------------------------------

test_sample = {

    'Outlook': 'Sunny',

    'Temperature': 'Cool',

    'Humidity': 'High',

    'Windy': 'True'

}

print(P_yes,P_no)

print("Prediction:", predict(test_sample))

📈 Output

0.6428571428571429 0.35714285714285715

Outlook Temperature Humidity Windy Play 4 Sunny Cool Normal False Yes 9 Sunny Mild Normal False Yes 2 / 9 = 0.2222222222222222 Outlook Temperature Humidity Windy Play 3 Sunny Mild High False No 5 Sunny Cool Normal True No 13 Sunny Mild High True No 3 / 5 = 0.6

probalities Yes No 0.14285714285714285 0.21428571428571427

Outlook Temperature Humidity Windy Play 4 Sunny Cool Normal False Yes 6 Overcast Cool Normal True Yes 8 Rainy Cool Normal False Yes 3 / 9 = 0.3333333333333333 Outlook Temperature Humidity Windy Play 5 Sunny Cool Normal True No 1 / 5 = 0.2

probalities Yes No 0.047619047619047616 0.04285714285714286

Outlook Temperature Humidity Windy Play 0 Rainy Hot High False Yes 2 Overcast Hot High False Yes 11 Overcast Mild High True Yes 3 / 9 = 0.3333333333333333 Outlook Temperature Humidity Windy Play 1 Rainy Hot High True No 3 Sunny Mild High False No 7 Rainy Mild High False No 13 Sunny Mild High True No 4 / 5 = 0.8

probalities Yes No 0.015873015873015872 0.03428571428571429

Outlook Temperature Humidity Windy Play 6 Overcast Cool Normal True Yes 10 Rainy Mild Normal True Yes 11 Overcast Mild High True Yes 3 / 9 = 0.3333333333333333 Outlook Temperature Humidity Windy Play 1 Rainy Hot High True No 5 Sunny Cool Normal True No 13 Sunny Mild High True No 3 / 5 = 0.6

probalities Yes No 0.005291005291005291 0.02057142857142857

Prediction: No

📊 Key Insight

• Each feature contributes independently
• Final decision = product of probabilities

---

Result

• Naive Bayes is:
  • Simple
  • Fast
  • Effective for categorical data
• Works well even with small datasets
• Smoothing is essential