Lecture 29 Credible Intervals and Bayesian Testing

A confidence interval, as given in definition 26.1, is a random interval which covers the fixed parameter with a given probability. In contrast, in a Bayesian model the data are fixed after observation, the parameter is random and an interval estimate is a set which contains most of the posterior probability.

The probability in the definition is a probability about $\theta$, for the given data set: the sentence “$\theta$ is in $[l,u]$ with probability $0.95$” (which we had to forbid in lecture 26) is the statement that a $95\%$ credible interval makes. The trade-off is that this probability depends on the prior distribution.

Any interval which has posterior probability $1-\alpha$ works, and the equal-tailed interval is the usual choice, because the edges of this interval are quantiles. The alternative would be to find the shortest such set.

The set $H_{c}$ is the smallest set with posterior probability $1-\alpha$. Any time we want to exchange part of this set with a region where the density is below $c$, we need to include more length in the interval to make up for the lost probability. For a unimodal posterior distribution, $H_{c}$ forms an interval, called the HPD interval, where the two boundaries have the same posterior density. For a symmetric, unimodal posterior distribution, this interval will be an equal-tailed interval (see exercise 29.1).

For a skewed posterior the HPD interval is shifted towards the mode, slightly shorter than the equal-tailed interval, and the numerical determination of this interval is the topic of exercise 29.2, which shows the difference between the two intervals for the skewed posterior $\text{Beta}(9,5)$ in example 28.5. As $n$ increases, the difference between the intervals decreases, because the posterior distribution approaches a symmetric normal distribution and the equal-tailed interval is then appropriate.

In the Bayesian model a hypothesis $H_{0}\colon\theta\in\Theta_{0}$, in the sense of definition 22.1, is an event concerning the random variable $\theta$ and thus has a probability before and after the data are seen. Testing does not require new theory.

The threshold plays the role of the level $\alpha$ of definition 22.5, but the two numbers are not comparable: $\alpha$ bounds the probability of a type I error over repeated samples, whereas the threshold bounds the posterior probability of $H_{0}$ for the data in hand. Any asymmetry between the hypotheses can be expressed by the prior.

A simple hypothesis $\theta=\theta_{0}$ has probability zero under every prior with a density, and thus for two simple hypotheses the prior must be a two-point distribution. The posterior in this case is then simple.

Using the result of Bayes’ theorem, proposition A.20, we find $\mathbb{P}(H_{j}\mskip 1.0mu|\mskip 1.0mux)=p_{j}\,f(x\mskip 1.0mu|\mskip 1.0mu\theta_{j})/m(x)$ for $j=0,1$ and dividing these two expressions we find that the marginal density $m(x)$ cancels out. This gives

i.e. the posterior odds are the prior odds times the likelihood ratio. In this context, the likelihood ratio is also called the Bayes factor in favour of $H_{0}$. This is the inverse of the ratio considered in the Neyman–Pearson lemma, theorem 23.2. From the posterior odds $r$ we can derive the posterior probabilities as $\mathbb{P}(H_{0}\mskip 1.0mu|\mskip 1.0mux)=r/(1+r)$.

Now we have two different sets of answers, for the same data, for the normal mean. We conclude by comparing the two sets.

The first difference is in the nature of the statements. A $95\%$ confidence interval is a statement about a procedure: it says that for repeated samples, the interval covers the fixed truth in $95\%$ of cases. A $95\%$ credible interval is a statement about the parameter, given the data and the prior. Similarly, a $p$-value is computed assuming $H_{0}$ is true, whereas $\mathbb{P}(H_{0}\mskip 1.0mu|\mskip 1.0mux)$ is the probability of the hypothesis given the data.

The second observation is that for the normal mean with known variance and a flat prior, both numbers coincide. Example 29.3 shows the two intervals coinciding, and the tests coincide as well: for $H_{0}\colon\theta\leq\theta_{0}$ the flat-prior posterior probability $\Phi\bigl{(}\sqrt{n}\,(\theta_{0}-\bar{x})/\sigma\bigr{)}$ equals, by the symmetry of $\Phi$, the $p$-value $1-\Phi\bigl{(}\sqrt{n}\,(\bar{x}-\theta_{0})/\sigma\bigr{)}$ of the one-sided $z$-test from example 22.6. That example tests the simple null $\theta=\theta_{0}$, but since $\mathbb{P}_{\theta}(\bar{X}\geq\bar{x})$ increases with $\theta$, like the power function of the test, the supremum in definition 22.8 for the composite null $\theta\leq\theta_{0}$ is attained at $\theta_{0}$ and gives the same $p$-value. For the running data set both values equal $0.274$. This is the case because the flat-prior posterior density in $\theta$ has the same shape as the sampling density of $\bar{X}$ in $\bar{x}$. This is the reason why the misreadings in lectures 22 and 26 were so tempting: the forbidden sentence about the $p$-value is, for the simplest model, a true sentence about the flat-prior posterior, and only the interpretation needs to be changed.

The coincidence is only true for special models and priors. A genuine prior will move the Bayesian answer closer to the prior mean and will concentrate the answer around this value, as in the case of $N(12,1)$ above. If the prior is correct, this is an advantage, but if the prior is wrong this effect will work against us. The frequentist guarantee from definition 26.1 applies for all $\theta$ and thus cannot be violated in this way. Exercise 29.5, however, shows that the frequentist coverage of a credible interval can be worse than $95\%$ when $\theta$ is far from the prior mean.

As the sample size increases, the influence of the prior decreases. For the normal mean, equation (29.1) shows that the weight of the prior mean in $m_{n}$ goes to zero and that $\tau_{n}^{2}\leq\sigma^{2}/n$ so that both the posterior mean and the posterior standard deviation converge to the values they would have with flat prior. The same effect holds for all regular models. The general version of this result is the Bayesian analogue of theorem 17.1: under the regularity conditions from definition 16.1, and for any continuous, positive prior density at the true value, the posterior distribution is approximately

where $\hat{\theta}_{n}$ is the maximum likelihood estimator. We state this result without proof. The idea is that the log-posterior is $\ell(\theta)+\log\pi(\theta)$ up to a constant, and that the log-likelihood grows like $n$ while $\log\pi(\theta)$ does not. From (29.4) we know that the equal-tailed credible interval is approximately the Wald interval $\hat{\theta}_{n}\pm z_{1-\alpha/2}\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ of proposition 26.8, independently of the prior: for large sample size both approaches will determine the same interval with approximately frequentist coverage $1-\alpha$.

For a point null hypothesis against a two-sided alternative the two approaches can differ substantially. We do not pursue this here.