Introduction to Bayesian Inference

Osamu Maruyama

1

Bayesian inference

• Method of statistical inference

• Bayes’ theorem is used

• The probability of a hypothesis is updated when a data set is given.

2

Contents

• What is probability?

• Conditional probability （条件付き確率）

• Multiplication theorem （乗法定理）

• Bayes’ theorem

• Prior and posterior probability （事前と事後の確率）

3

𝑓 is just a function satisfying

0 ≤ 𝑓 𝑥 ≤ 1

𝑥

𝑓 𝑥 = 1

Could you make an example?

4

What’s the definition of a probability?

The probability that today the dinner is mapo dofu（麻婆豆腐） is 0.05.

The probability that you win a bet is 0.0001.

• 𝑥 is an event happening by chance• 𝑝 𝑥 : probability of 𝑥• 0 ≤ 𝑝 𝑥 ≤ 1• σ𝑥 𝑝 𝑥 = 1

What’s the meaning of a probability?

The probability that today the dinner is mapo dofu（麻婆豆腐） is 0.05.

Degree of belief or uncertainty

The probability that you win a bet is 0.0001.

Frequency

Conditional probability（条件付確率）

𝑃 𝐵 𝐴 = 𝑃𝐴 𝐵 =𝑃 𝐴 ∩ 𝐵

𝑃 𝐴

7

Conditional probabilityExample• A die is thrown twice.

• What’s the probability that the sum is greater than or equal to 9?

• P(sum is greater than or equal to 9)=5/18.(3,6)

(4,5), (4,6)

(5,4), (5,5), (5,6)

(6,3), (6,4), (6,5), (6,6)

8

Conditional probabilityExample• A die is thrown twice.• What’s the probability that the sum is greater than or

equal to 9 when the first number is 4. • P(sum is greater than or equal to 9 when the first

number is 4)

=𝑃 𝑠𝑢𝑚 ≥ 9 ∩ 𝑓𝑖𝑟𝑠𝑡 𝑛𝑢𝑚 = 4

𝑃 𝑓𝑖𝑟𝑠𝑡 𝑛𝑢𝑚 = 4=

23616

=2

6=

1

3.

(3,6)(4,5), (4,6)(5,4), (5,5), (5,6)(6,3), (6,4), (6,5), (6,6)

9

Conditional probabilityExercise• A die is thrown twice.

• Q1: What’s the probability that the sum is greater than or equal to 8?

• Q2: What’s the probability that the sum is greater than or equal to 8 when the first number is 5.

10

Independent （独立）

Events A and B are independent if

𝑃 𝐵 𝐴 = 𝑃 𝐵 .

12

Mutually exclusive （互いに排反）

Events A and B are mutually exclusive

⇔

When A (B, resp.) happens, B (A, resp.) never happen.

⇔𝑃 𝐵 𝐴 = 0.

13

Multiplication theorem （乗法定理）

Recall

𝑃 𝐵 𝐴 =𝑃 𝐴 ∩ 𝐵

𝑃 𝐴.

Multiplication theorem:𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴

Note𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐵 ∩ 𝐴 = 𝑃 𝐴 𝐵 𝑃 𝐵

14

Multiplication theorem:Example

15

当当当 当

当当当 当

What’s the probability that the dog and mammoth get winning tickets?

𝑃 𝐷 =3

10: prob. that the dog gets a winning ticket.

𝑃 𝑀 =3

10: prob. that the mammoth gets a winning ticket.

𝑃 𝐷 ∩ 𝑀 = 𝑃 𝐷 × 𝑃 𝑀 =3

10×

3

10=

9

100.

𝐷 and 𝑀 are independent!

Multiplication theorem:Example

16

当当当 当

当当

当

What’s the probability that the dog and mammoth get winning tickets?

𝑃 𝐷 =3

10: prob. that the dog gets a winning ticket.

𝑃 𝑀 𝐷 =2

9: prob. that the mammoth gets a winning ticket after the dog gets a

winning ticket.

𝑃 𝐷 ∩ 𝑀 = 𝑃 𝐷 × 𝑃 𝑀 𝐷 =3

10×

2

9=

1

15.

Multiplication theorem:Homework 1

17

当当当

当当

当

What’s the probability that the dog fails to get a winning ticket and the mammoth gets a winning ticket?𝑃(ഥ𝐷)= : prob. that the dog fails to get a winning ticket.

𝑃 𝑀 ഥ𝐷 = : prob. that the mammoth gets a winning ticket after the dog fails to get a winning ticket.

𝑃 ഥ𝐷 ∩ 𝑀 = ⋯

当

Bayes’ theorem

Recall multiplication theorem:𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴 ,

𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐵 ∩ 𝐴 = 𝑃 𝐴 𝐵 𝑃 𝐵 .

Combine them:𝑃 𝐴 ∩ 𝐵 = 𝑃 𝐴 𝐵 𝑃 𝐵 = 𝑃 𝐵 𝐴 𝑃 𝐴 .

∴ 𝑃 𝐵 𝐴 =𝑃 𝐴 𝐵 𝑃 𝐵

𝑃 𝐴.

Key concepts of Bayes’ theorem

Bayes’ theorem

𝑃 𝜃 𝐷 =𝑃 𝐷 𝜃 𝑃 𝜃

𝑃 𝐷𝜃 : parameter, hypothesis（ cause, 原因）

: You want to estimate

𝐷 : data （effect, 結果）

: Observed data

𝜃

𝑃 𝜃 𝐷

Bayes’ theorem

𝑃 𝜃 𝐷 =𝑃 𝐷 𝜃 𝑃 𝜃

𝑃 𝐷𝜃 : parameter, hypothesis（ cause, 原因）𝐷 : data （effect, 結果）

• 𝑃 𝜃 𝐷• posterior probability （事後確率）．• interpretation: probability of cause 𝜃 when event 𝐷 happens.

• 𝑃 𝐷 𝜃• likelihood function（尤度関数）• interpretation: likelihood of 𝐷 under 𝜃.

• 𝑃 𝜃• prior probability （事前確率）．• interpretation: a general degree of belief in 𝜃.

Usefulness of Bayes’ theorem

𝜃

𝑃 𝜃 𝐷

𝑃 𝜃 𝐷 =𝑃 𝐷 𝜃 𝑃 𝜃

𝑃 𝐷𝐷 is observed

𝑃 𝜃

Current knowledge: Updated knowledge:

𝜃

𝑃 𝜃

Compute the posterior probability of ℎ𝑖

𝐻 = ℎ1, … , ℎ𝑛 are assumed mutually exclusive.

Bayes’ theorem:

𝑃 𝐻 = ℎ𝑖 𝐷 =𝑃 𝐷|H = ℎ𝑖 𝑃 H = ℎ𝑖

𝑃 𝐷

Here we have𝑃 𝐷 = 𝑃 𝐷, H = ℎ1 + ⋯ + 𝑃 𝐷, H = ℎ𝑛

= 𝑃 𝐷|H = ℎ1 𝑃 H = ℎ1 + ⋯ + 𝑃 𝐷|H = ℎ𝑛 𝑃 H = ℎ𝑛

=

𝑗=1

𝑛

𝑃 𝐷|H = ℎ𝑗 𝑃 H = ℎ𝑗

∴ 𝑃 𝐻 = ℎ𝑖 𝐷 =𝑃 𝐷|H = ℎ𝑖 𝑃 H = ℎ𝑖

σ𝑗=1𝑛 𝑃 𝐷|H = ℎ𝑗 𝑃 H = ℎ𝑗

𝑃 𝐻 = ℎ𝑖 𝐷 =𝑃 𝐷|H = ℎ𝑖 𝑃 H = ℎ𝑖

σ𝑗=1𝑛 𝑃 𝐷|H = ℎ𝑗 𝑃 H = ℎ𝑗

ℎ1 ℎ2 … ℎ𝑖 … ℎ𝑛

𝐷

Bayes’ theorem:Example

25

bag A bag B bag C

A bag is randomly chosen with equal probabilities.A ball was randomly chosen from that bag which we do not know.The color was red. What’s the probability that the chosen bag is B?

𝑃 𝐴 = 𝑃 𝐵 = 𝑃 𝐶 =1

3

𝑃 𝑅 𝐴 =3

6=

1

2

𝑃 𝑅 𝐵 =5

7

𝑃 𝑅 𝐶 =3

9=

1

3

𝑃 𝐵 𝑅

=𝑃 𝑅|𝐵 𝑃 𝐵

𝑃 𝑅|𝐴 𝑃 𝐴 + 𝑃 𝑅|𝐵 𝑃 𝐵 + 𝑃 𝑅|𝐶 𝑃 𝐶

=

57

12 +

57 +

13

=30

41

We can estimate the cause from data!

Bayes’ theorem:Example

26

negative 10%

positive 90%

negative 80%

positive 20%

Affected罹患

0.001%

Unaffected非罹患

99.999%

You are diagnosed positive. What’s the probability that you are affected?

𝑃 𝐴 = 0.00001𝑃 𝑈 = 0.99999

𝑃 𝑃 𝐴 = 0.9𝑃 𝑁 𝐴 = 0.1𝑃 𝑃 𝑈 = 0.2𝑃 𝑁 𝑈 = 0.8

𝑃 𝐴 𝑃 = 4.5 × 10−5

No one never take this examination.

Bayes’ theorem:Homework 2

27

negative 20%

positive 80%

negative 90%

positive 10%

Affected罹患

1%

Unaffected非罹患

99%

You are diagnosed positive. What’s the probability that you are affected?

𝑃 𝐴 =𝑃 𝑈 =

𝑃 𝑃 𝐴 =𝑃 𝑁 𝐴 =𝑃 𝑃 𝑈 =𝑃 𝑁 𝑈 =

𝑃 𝐴 𝑃 =

What’s your advice?

Monty Hall problem: “Let’s Make a Deal,” hosted by Monty Hall

Suppose you were a player in the show

You're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say A.

The host, who knows what's behind the doors, opens another door except your choice.

The host open a door, say C, which has a goat.

He then says to you, "Do you want to pick the remaining door B?"

Question: Is it to your advantage to switch your choice?

Solve the problem by Bayes’ theorem• A: A has a car.

• B: B has a car.

• C: C has a car.

• EC: Monty opens the door C.

Compare the following two posteriorprobabilities：

𝑃 𝐴|𝐸𝐶 =𝑃 𝐸𝐶|𝐴 𝑃 𝐴

𝑃 𝐸𝐶

=𝑃 𝐸𝐶|𝐴 𝑃 𝐴

𝑃 𝐸𝐶,𝐴 +𝑃 𝐸𝐶,𝐵 +𝑃 𝐸𝐶,𝐶

=𝑃 𝐸𝐶|𝐴 𝑃 𝐴

𝑃 𝐸𝐶|𝐴 𝑃 𝐴 +𝑃 𝐸𝐶|𝐵 𝑃 𝐵 +𝑃 𝐸𝐶|𝐶 𝑃 𝐶

𝑃 𝐵|𝐸𝐶 =𝑃 𝐸𝐶|𝐵 𝑃 𝐵

𝑃 𝐸𝐶𝑃 𝐸𝐶|𝐵 𝑃 𝐵

𝑃 𝐸𝐶|𝐴 𝑃 𝐴 +𝑃 𝐸𝐶|𝐵 𝑃 𝐵 +𝑃 𝐸𝐶|𝐶 𝑃 𝐶

𝑃 𝐴 = 𝑃 𝐵 = 𝑃 𝐶 =1

3

𝑃 𝐸𝐶|𝐴 =1

2𝑃 𝐸𝐶|𝐵 = 1

𝑃 𝐸𝐶|𝐶 = 0

𝑃 𝐴|𝐸𝐶 =

𝟏𝟐

12

+ 1 + 0=

1

3

𝑃 𝐵|𝐸𝐶 =𝟏

12

+ 1 + 0=

2

3

Intuitive correct understanding

Before C is opened After C is opened

General Monty Hall problemWe have 100 doors!

⋯

Homework 3: General Monty Hall problemSuppose you were a player in the show.

You're given the choice of 100 doors: Behind one door is a car; behind the others, goats.

You pick a door, and the host, who knows what's behind the doors, opens other y doors, which have goats.

He then says to you, "Do you want to pick the remaining door"

Is it to your advantage to switch your choice?

Consider the following case and then consider the difference.

1. y=98

2. y=1

Case of y=98Do you want to change your door?

⋯

Your doorA1

A2 A3