Lecture 30 Confidence Versus Credibility

This is the R session for week 10. In lectures this week we have introduced priors, posteriors and credible intervals. In this session we compute a confidence interval and a credible interval for the same data, and we look at how a prior turns into a posterior as observations are added. As before, appendix B contains a written reference for the R commands used.

30.1 The Same Data, Two Intervals

We take the data of example 26.9, $T=37$ successes in $n=100$ Bernoulli trials, and compute two intervals for the success probability $\theta$ . The first is the Wald interval $\bar{x}\pm 1.96\sqrt{\bar{x}(1-\bar{x})/n}$ of that example. For the second we use the Bayesian model of lecture 28 with the flat prior: the posterior is $\text{Beta}(1+T,\,1+n-T)$ by example 28.5, and the interval between its $2.5\%$ and $97.5\%$ quantiles is the equal-tailed credible interval with credibility $0.95$ from definition 29.1. The command qbeta can be used to find these quantiles. Before you run the code, predict whether the credible interval will be wider or narrower than the Wald interval, and whether it will sit to the left or to the right of it.

n <- 100

total <- 37

xbar <- total / n

se <- sqrt(xbar * (1 - xbar) / n)

xbar + c(-1, 1) * qnorm(0.975) * se

a <- 1 + total

b <- 1 + n - total

qbeta(c(0.025, 0.975), a, b)

## [1] 0.2753721 0.4646279

## [1] 0.2817646 0.4680688

The two intervals, $[0.275,0.465]$ and $[0.282,0.468]$ , agree to within $0.01$ . The credible interval is slightly shifted towards $1/2$ , since the flat prior adds one success and one failure to the data. Although the numbers are close, the two statements are different. The Wald interval comes with the guarantee that the recipe used to compute it covers the truth in approximately $95\%$ of samples, and it makes no statement about this particular interval. The credible interval means that, given the observations and the flat prior, the probability that $\theta$ lies between $0.282$ and $0.468$ is $0.95$ . Thus the sentence which was wrong for the confidence intervals of section 27.1 is the correct interpretation here, since $\theta$ is a random variable in the Bayesian model. A different prior would give a different interval, and the numerical agreement is a property of the flat prior and of the sample size.

30.2 From Prior to Posterior

We now look at the update from example 28.5 one observation at a time. In the script R/S30-posterior.R we simulate $100$ Bernoulli observations with $\theta=0.3$ . After $k$ observations with $s$ successes, the prior $\text{Beta}(a,b)$ has turned into the posterior $\text{Beta}(a+s,\,b+k-s)$ , and we plot this posterior density for three priors: the flat prior $\text{Beta}(1,1)$ , the Jeffreys prior $\text{Beta}(1/2,1/2)$ from example 28.10 and the misplaced prior $\text{Beta}(12,3)$ , with mean $0.8$ and equivalent sample size $15$ . In the simulated sequence, the first five observations are all failures, there are $2$ successes among the first $20$ and $28$ among all $100$ . Before you look at figure 30.1, sketch the panel for $k=5$ : which curves are steep, where are their peaks and how wide are the curves?

Four panels of Beta densities on the unit
interval, each with three curves and a grey vertical line at 0.3.
For k=0 the flat prior is a horizontal line, the Jeffreys prior is a
shallow U and the informative prior is a hump with a maximum
around 0.85. For k=5, all failures, two curves start steep at
the left edge and the hump has moved to 0.6. For k=20, two
successes, the two narrow humps are around 0.1 and the third is
around 0.4. For k=100, 28 successes, two narrow peaks at 0.28
are on the grey line, with a third peak at 0.34 overlapping the
other two. — Figure 30.1: Posterior densities for the probability of success in a Bernoulli trial after $k=0$ , $5$ , $20$ and $100$ observations of one simulated sequence with $\theta=0.3$ (grey vertical line), for the priors $\text{Beta}(1,1)$ (solid), $\text{Beta}(1/2,1/2)$ (dashed) and $\text{Beta}(12,3)$ (dotted).

At $k=0$ the three curves in figure 30.1 are the priors, and we can see that the Jeffreys prior is not flat. At $k=5$ the two weak posteriors take their largest values at $0$ , since no successes have been observed so far, while the misplaced posterior has only moved from $0.8$ to around $0.6$ . At $k=20$ all three posteriors are humps, narrower than before. The two weak posteriors sit well below the truth after an unlucky start, and the misplaced one is closest to $0.3$ ; this shows how much the prior still matters after twenty observations, not that the misplaced prior is correct. At $k=100$ the two weak posteriors coincide and the misplaced posterior overlaps them: as the weight of the prior decreases, as discussed in example 28.5, every posterior concentrates near the truth.

30.3 A Task to Take Home

For the data of section 30.1, compute the equal-tailed credible interval with credibility $0.95$ under the prior $\text{Beta}(12,3)$ from section 30.2, and compare it with the two intervals found there. Then compute all three intervals for $T=4$ successes in $n=10$ trials. Write a short paragraph about how the influence of the prior depends on the sample size.

After the session, you should type the commands used in this session, and should check that you get the same results as the text.

Summary.

•

The written reference for the commands used in this session is appendix B.
•

For the same Bernoulli data, the Wald interval and the credible interval with flat prior agree to within $0.01$ , but they are different statements: the credible interval is a probability statement about $\theta$ , conditioned on the data and on the prior, and thus depends on the prior.
•

The Beta posterior becomes sharper as observations are added. After $20$ observations the prior is still noticeable, but after $100$ its influence is small.