Lab Assignment-5

 

Assignment-5 week 1 Aug

Learning Objective: Compare MLE and MAP

1.Problem Statement

An instructor conducted an online quiz for a small group of students. Each student's result is recorded as:

• 1 → Pass
• 0 → Fail

The observed quiz results are

1, 1, 0, 1, 1, 0, 1, 0, 1, 1

Assume that the quiz outcomes follow a Bernoulli distribution, where

• θ = Probability of passing the quiz

The instructor believes that, based on previous years, most students are likely to pass. This prior belief is represented by a Beta distribution with parameters

• α = 5
• β = 2

Using the given data,

Part A

Estimate the probability of passing using

1. Maximum Likelihood Estimation (MLE)

Part B

Estimate the probability of passing using

2. Maximum A Posteriori (MAP)

Part C

Compare the MLE and MAP estimates.

Part D

Plot the

• Prior Distribution
• Likelihood Function
• Posterior Distribution

on the same graph.

Part E

Comment on the following:

1. Which estimate is affected by prior knowledge?
2. Why are MLE and MAP estimates different?
3. Which estimate is more reliable when the dataset is very small?

---

Dataset

Student	Result
1	1
2	1
3	0
4	1
5	1
6	0
7	1
8	0
9	1
10	1

---

Expected Formulae

MLE

$$\hat{\theta}_{MLE}=\frac{\text{Number of Passes}}{\text{Total Observations}}$$

MAP

For a Beta prior,

$$\hat{\theta}_{MAP}=\frac{x+\alpha-1}{n+\alpha+\beta-2}$$

where

• x = Number of successes
• n = Total observations

---

Expected Output

Number of Passes = 7

Number of Fails = 3

MLE Estimate = 0.70

MAP Estimate = 0.75

Students should explain why the MAP estimate is higher due to the optimistic prior (α = 5, β = 2).

---

Additional Tasks

1. Change the prior to Beta(2,2). Compare the MAP estimate.
2. Change the prior to Beta(10,1). Observe the effect.
3. Increase the dataset to 100 observations. Compare MLE and MAP.
4. Explain what happens when the number of observations becomes very large.

2.Problem Statement

A company has developed a machine that fills bottles with fruit juice. The nominal fill volume is expected to be around 50 ml, but due to manufacturing variations, the actual fill volume differs slightly.

An engineer randomly selects 10 bottles and measures their fill volumes (in ml).

Assume that:

• The fill volumes follow a Gaussian (Normal) distribution.
• The variance of the measurements is known to be σ² = 4 ml².
• Based on historical production data, the engineer believes that the average fill volume is approximately 50 ml.

This prior belief is represented by a Gaussian prior with

• Prior Mean (μ₀) = 50 ml
• Prior Variance (σ₀²) = 9 ml²

The observed fill volumes are

Bottle	Fill Volume (ml)
1	52
2	49
3	51
4	53
5	48
6	50
7	54
8	51
9	49
10	52

---

Tasks

Part A

Compute the sample mean.

---

Part B

Estimate the population mean using

• Maximum Likelihood Estimation (MLE)

---

Part C

Estimate the population mean using

• Maximum A Posteriori (MAP)

using the given Gaussian prior.

---

Part D

Compare the estimates obtained using

• Sample Mean
• MLE
• MAP

---

Part E

Plot

• Gaussian Prior
• Gaussian Likelihood
• Gaussian Posterior

on the same graph.

---

Part F

Discuss

1. Which estimate is influenced by prior knowledge?
2. Why is the MAP estimate closer to the prior mean?
3. What happens if more observations are collected?

---

Given Information

Observed Data

52 49 51 53 48 50 54 51 49 52

Known Variance

σ² = 4

Prior Mean

μ₀ = 50

Prior Variance

σ₀² = 9

---

Formulae

Maximum Likelihood Estimate

$$\mu_{MLE}=\frac{1}{n}\sum_{i=1}^{n}x_i$$

---

MAP Estimate

For a Gaussian likelihood with Gaussian prior,

$$\mu_{MAP} = \frac{\frac{n}{\sigma^2}\bar{x}+\frac{1}{\sigma_0^2}\mu_0} {\frac{n}{\sigma^2}+\frac{1}{\sigma_0^2}}$$

where

• n = number of observations
• σ² = known variance
• μ₀ = prior mean
• σ₀² = prior variance

---

Expected Output

Sample Mean

50.9 ml

MLE Estimate

50.9 ml

MAP Estimate

50.86 ml (approximately)

Students should observe that the MAP estimate is slightly pulled toward the prior mean (50 ml).

---

Additional Exercises

Experiment 1

Repeat the experiment with

Prior Mean = 48

Compare the MAP estimate.

---

Experiment 2

Use

Prior Variance = 1

What happens?

---

Experiment 3

Use

Prior Variance = 25

Compare the result.

---

Experiment 4

Increase the number of observations from 10 to 100 by generating additional synthetic samples.

Discuss

• Does MAP become similar to MLE?

3.Problem Statement

A university wants to predict whether a student will pass or fail an entrance examination based on

• Study Hours
• Attendance Percentage

The target variable is

• 1 → Pass
• 0 → Fail

Train two Logistic Regression models:

1. Logistic Regression using Maximum Likelihood Estimation (MLE).
2. Logistic Regression using Maximum A Posteriori (MAP) with a Gaussian prior (L2 Regularization).

Compare both models using

• Accuracy
• Precision
• Recall
• F1-score
• Confusion Matrix

Finally compare the learned model coefficients.

---

Dataset

Student	Study Hours	Attendance (%)	Pass
1	2	55	0
2	3	60	0
3	4	58	0
4	4	65	0
5	5	70	1
6	5	75	1
7	6	72	1
8	6	80	1
9	7	82	1
10	7	85	1
11	8	88	1
12	8	90	1
13	3	68	0
14	5	65	1
15	6	78	1
16	2	62	0
17	7	75	1
18	4	72	0
19	9	92	1
20	8	95	1

Students may save this as

student_exam.csv

---

Tasks

Part A

Load the dataset using Pandas.

Display

• First five rows
• Shape
• Data types
• Summary statistics

---

Part B

Split the dataset into

• Training (80%)
• Testing (20%)

Standardize the features.

---

Part C – Logistic Regression using MLE

Train a Logistic Regression model without regularization (or with a very large value of C, e.g., C=1e6) so that regularization is negligible.

Compute

• Accuracy
• Precision
• Recall
• F1-score
• Confusion Matrix

Display the learned coefficients.

---

Part D – Logistic Regression using MAP

Assume that the model parameters follow a Gaussian prior.

Train another Logistic Regression model using L2 regularization, for example:

• penalty='l2'
• C=1.0

This corresponds to MAP estimation.

Again compute

• Accuracy
• Precision
• Recall
• F1-score
• Confusion Matrix

Display the coefficients.

---

Part E

Compare

• MLE coefficients
• MAP coefficients

Comment on

• Which coefficients are larger?
• Which model is less likely to overfit?
• How does the Gaussian prior influence the parameters?

---

Part F

Predict whether the following student will pass.

Study Hours	Attendance
6	76

Predict using

• MLE model
• MAP model

Compare the predicted probabilities.

---

Additional Exercises

Experiment 1

Repeat the MAP estimation with

C = 0.1

Observe the coefficients.

---

Experiment 2

Repeat with

C = 10

Compare the results.

---

Experiment 3

Add a few noisy samples to the dataset.

Compare

• MLE
• MAP

Which model is more robust?

---

Experiment 4

Generate a larger synthetic dataset (100 observations).

Compare the coefficients obtained using MLE and MAP.

---

4.Estimate the parameters of a logistic regression model using MLE and MAP on the Breast Cancer Wisconsin dataset. Compare the results and discuss the effects of regularization.
Tasks:

● Load and preprocess the dataset.

● Implement logistic regression with MLE.

● Apply MAP estimation with different regularization priors (L1 and L2 regularization).

● Compare the performance and parameter estimates with MLE and MAP.

5.Use MLE and MAP to estimate the parameters of a multinomial distribution on the 20 Newsgroups dataset. Explore the impact of different priors on the estimation.
Tasks:

● Load and preprocess the dataset.

● Implement MLE for multinomial distribution parameter estimation.

● Apply MAP estimation with various priors (e.g., Dirichlet priors).

● Compare results and evaluate the effect of different priors.