Lecture 28 Credible Intervals and Bayesian Testing

A confidence interval, as given in definition 25.1, is a random interval which covers the fixed parameter with a given probability. In contrast, in a Bayesian model the data are fixed after observation, the parameter is random and an interval estimate is a set which contains most of the posterior probability.

The probability in the definition is a probability about $\theta$, for the given data set: the sentence “$\theta$ is in $[l,u]$ with probability $0.95$” (which we had to forbid in lecture 25) is the exact statement a $95\%$ credible interval makes. The trade-off is that this probability depends on the prior distribution.

Any interval which has posterior probability $1-\alpha$ works, and the equal-tailed interval is the usual choice, because the edges of this interval are quantiles. The alternative would be to find the shortest such set.

The set $H_{c}$ is the smallest set with posterior probability $1-\alpha$. Any time we want to exchange part of this set with a region where the density is below $c$, we need to include more length in the interval to make up for the lost probability. For a unimodal posterior distribution, $H_{c}$ forms an interval, called the HPD interval, where the two boundaries have the same posterior density. For a symmetric, unimodal posterior distribution, this interval will be an equal-tailed interval (see exercise 28.1).

For a skewed posterior the HPD interval is shifted towards the mode, slightly shorter than the equal-tailed interval, and the numerical determination of this interval is the topic of exercise 28.2, which shows the difference between the two intervals for the skewed posterior $\text{Beta}(9,5)$ in example 26.5. As $n$ increases, the difference between the intervals decreases, because the posterior distribution approaches a symmetric normal distribution and the equal-tailed interval is then appropriate.

In the Bayesian model a hypothesis $H_{0}\colon\theta\in\Theta_{0}$, in the sense of definition 20.1, is an event concerning the random variable $\theta$ and thus has a probability before and after the data are seen. Testing does not require new theory.

The threshold plays the role of the level $\alpha$ of definition 20.5, but the two numbers are not comparable: $\alpha$ bounds the probability of a type I error over repeated samples, whereas the threshold bounds the posterior probability of $H_{0}$ for the data in hand. Any asymmetry between the hypotheses can be expressed by the prior.

A simple hypothesis $\theta=\theta_{0}$ has probability zero under every prior with a density, and thus for two simple hypotheses the prior must be a two-point distribution. The posterior in this case is then simple.

Using the result of Bayes’ theorem, proposition A.20, we find $\mathbb{P}(H_{j}\mskip 1.0mu|\mskip 1.0mux)=p_{j}\,f(x\mskip 1.0mu|\mskip 1.0mu\theta_{j})/m(x)$ for $j=0,1$ and dividing these two expressions we find that the marginal density $m(x)$ cancels out. This gives

i.e. the posterior odds are the prior odds times the likelihood ratio. In this context, the likelihood ratio is also called the Bayes factor in favour of $H_{0}$. This is the inverse of the ratio considered in the Neyman–Pearson lemma, theorem 22.2. From the posterior odds $r$ we can derive the posterior probabilities as $\mathbb{P}(H_{0}\mskip 1.0mu|\mskip 1.0mux)=r/(1+r)$.

Now we have two different sets of answers, for the same data, for the normal mean. We conclude by comparing the two sets.

The first difference is in the nature of the statements. A $95\%$ confidence interval is a statement about a procedure: it says that for repeated samples, the interval covers the fixed truth in $95\%$ of cases. A $95\%$ credible interval is a statement about the parameter, given the data and the prior. Similarly, a $p$-value is computed assuming $H_{0}$ is true, whereas $\mathbb{P}(H_{0}\mskip 1.0mu|\mskip 1.0mux)$ is the probability of the hypothesis given the data.

The second observation is that for a location parameter with a flat prior, both numbers coincide. Example 28.3 shows the two intervals coinciding, and the tests coincide as well: for $H_{0}\colon\theta\leq\theta_{0}$ the flat-prior posterior probability $\Phi\bigl{(}\sqrt{n}\,(\theta_{0}-\bar{x})/\sigma\bigr{)}$ equals, by the symmetry of $\Phi$, the $p$-value $1-\Phi\bigl{(}\sqrt{n}\,(\bar{x}-\theta_{0})/\sigma\bigr{)}$ of the one-sided $z$-test from example 20.6; for the running data set both values equal $0.274$. This is the case because the flat-prior posterior density in $\theta$ has the same shape as the sampling density of $\bar{X}$ in $\bar{x}$. This is the reason why the misreadings in lectures 20 and 25 were so tempting: the forbidden sentence about the $p$-value is, for the simplest model, a true sentence about the flat-prior posterior, and only the interpretation needs to be changed.

The coincidence is only true for special models and priors. A genuine prior will move the Bayesian answer closer to the prior mean and will concentrate the answer around this value, as in the case of $N(12,1)$ above. If the prior is correct, this is an advantage, but if the prior is wrong this effect will work against us. The frequentist guarantee from definition 25.1 applies for all $\theta$ and thus cannot be violated in this way; exercise 28.5, however, shows that the frequentist coverage of a credible interval can be worse than $95\%$ when $\theta$ is far from the prior mean.

As the sample size increases, the influence of the prior decreases. For the normal mean, equation (28.1) shows that the weight of the prior mean in $m_{n}$ goes to zero and that $\tau_{n}^{2}\leq\sigma^{2}/n$ so that both the posterior mean and the posterior standard deviation converge to the values they would have with flat prior. The same effect holds for all regular models. The general version of this result is the Bayesian analogue of theorem 17.1: under the regularity conditions from definition 16.1, and for any continuous, positive prior density at the true value, the posterior distribution is approximately

where $\hat{\theta}_{n}$ is the maximum likelihood estimator. Without proof, we state that this result holds. The proof is similar to the one for theorem 17.1: the log-posterior is $\ell(\theta)+\log\pi(\theta)$ up to a constant, the log-likelihood is approximately $\ell(\hat{\theta}_{n})-\tfrac{1}{2}\mathcal{I}_{n}(\hat{\theta}_{n})(\theta-\hat{\theta}_{n})^{2}$ at its maximum, and this quadratic term grows like $n$ while $\log\pi(\theta)$ does not. From (28.4) we know that the equal-tailed credible interval is approximately the Wald interval $\hat{\theta}_{n}\pm z_{1-\alpha/2}\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ of proposition 25.7, independently of the prior: for large sample size both approaches will determine the same interval with approximately frequentist coverage $1-\alpha$.

The two approaches differ most when testing for a point null hypothesis $H_{0}\colon\theta=\theta_{0}$ against a two-sided alternative. In this case a Bayesian test requires a prior which puts a lump of probability on $\theta_{0}$ but which spreads the remaining probability over the alternative. The exact distribution of the remaining probability over the alternative matters: if the prior is diffuse on the alternative, $H_{0}$ can have posterior probability of more than one half for a data set with $p$-value $0.01$. We will not consider this case in detail, but we can conclude that small $p$-value and small posterior probability of $H_{0}$ are different quantities and that in this case the difference can be substantial.