Lecture 25 Confidence Intervals

Here, $X=(X_{1},\dots,X_{n})$ is a sample from a parametric model with parameter $\theta\in\Theta\subseteq\mathbb{R}$, and $\mathbb{P}_{\theta}$ is the distribution of the sample under $\theta$. For a fixed small $\alpha\in(0,1)$, typically $\alpha=0.05$, the aim is to find an interval which misses the truth with probability at most $\alpha$.

The endpoints of the interval are statistics, i.e. functions of the data, while the parameter $\theta$ is an unknown constant which does not change between samples. For continuous data we can usually achieve equality in the definition, but for discrete data the coverage probability of the interval can jump as $\theta$ changes. For this reason, the definition only requires the left-hand side to satisfy the inequality. More generally, any random subset $C(X)\subseteq\Theta$ of the parameter space such that $\mathbb{P}_{\theta}\bigl{(}\theta\in C(X)\bigr{)}\geq 1-\alpha$ for all $\theta$ is called a confidence set.

Some care is needed when interpreting a confidence interval. For example, if the data lead to the $95\%$ interval $[9.32,11.28]$, both endpoints are now fixed numbers, and so is the unknown $\theta$; the statement $9.32\leq\theta\leq 11.28$ is either true or false. The $95\%$ describes the procedure: $95\%$ of the computed intervals would cover the truth in repeated samples, but we do not know whether the interval in front of us is one of these. The statement “$\theta$ is in $[9.32,11.28]$ with probability $0.95$” is wrong, since it assumes that the parameter is random (this only holds in the Bayesian model from lecture 28). The statement “$95\%$ of the observations are in the interval” is wrong, since this would be a confusion of parameter with data. The correct interpretation of such an interval can be illustrated by simulating many intervals and seeing how many of these cover the truth. This is done in lecture 27.

The most direct approach to constructing a confidence interval is to construct an interval based on a quantity whose distribution is known exactly, regardless of the parameter value. Such a quantity is called a pivotal quantity.

A pivot is not a statistic, because it contains the unknown $\theta$, but we know the quantiles of the pivot distribution. To get a confidence interval, we can choose constants $a<b$ such that $\mathbb{P}_{\theta}\bigl{(}a\leq Q(X,\theta)\leq b\bigr{)}=1-\alpha$ and then solve the inequalities $a\leq Q(X,\theta)\leq b$ for $\theta$. If $Q$ is monotone in $\theta$, the solution of these inequalities forms an interval which, with probability exactly $1-\alpha$, contains the unknown parameter value. Using the quantile notation $z_{p}$ from lecture 20, we have $z_{0.975}=1.96$ and, using symmetry, $z_{1-p}=-z_{p}$. Similarly, we write $t_{k,p}$ and $\chi^{2}_{k,p}$ for the $p$-quantiles of the $t_{k}$ and the $\chi^{2}_{k}$ distribution, respectively. These quantiles are all tabulated.

If $\sigma^{2}$ is unknown, this pivot is not available. If we replace $\sigma$ by the sample standard deviation $S$, we get a new pivot and we have to consider the $t$ distribution from appendix A.

If we take the square root of both endpoints, the interval $[1.55,3.25]$ for $\sigma$ has the same coverage probability: applying a strictly increasing function to both endpoints does not change the coverage probability, whereas applying a decreasing function swaps the endpoints.

A level-$\alpha$ test of $H_{0}\colon\theta=\theta_{0}$ decides whether a single value $\theta_{0}$ is compatible with the data; a confidence interval collects all the values which are. The following theorem, using the notions of critical region and level from definitions 20.2 and 20.5, makes this correspondence exact, in both directions.

The theorem is a change of viewpoint, rather than a new result: the event “$\theta_{0}$ is not rejected by $X$” can be read both as a test (with $\theta_{0}$ fixed and $X$ random) and as a confidence set (with $X$ fixed and $\theta_{0}$ varying). As an illustration, the size-$\alpha$ two-sided test from example 20.7 for the mean of an i.i.d. $N(\theta,\sigma^{2})$ sample with known $\sigma^{2}$ rejects $H_{0}\colon\theta=\theta_{0}$ if $|\bar{X}-\theta_{0}|>z_{1-\alpha/2}\,\sigma/\sqrt{n}$ and the values not rejected by the test form the interval from example 25.3. Similarly, by inverting the one-sided test from example 20.6, we find a one-sided confidence interval. We will consider this type of interval again in section 25.5. Finally, in exercise 25.4, the same approach is repeated with unknown $\sigma$. In practice, the duality is most often used in the opposite direction: a value $\theta_{0}$ is rejected at level $\alpha$ if it does not belong to the $1-\alpha$ interval.

Exact pivots only exist for special models. For every regular model, however, corollary 17.3 provides an approximate pivot: the standardised error $(\hat{\theta}_{n}-\theta)/\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ of the maximum likelihood estimator, with $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=1/\sqrt{\mathcal{I}_{n}(\hat{\theta}_{n})}$, converges in distribution to $N(0,1)$ whatever the value of $\theta$.

The Wald interval, for $95\%$, is the interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$. This is the interval introduced in lectures 12 and 18. Using the delta method, theorem A.18, the same approach can be used to derive the interval $g(\hat{\theta}_{n})\pm z_{1-\alpha/2}\,|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ for any smooth function $g(\theta)$ (see exercise 25.3). However, the coverage of this interval is only guaranteed in the limit, and section 17.5 lists the cases to be aware of.

Sometimes it is only important to consider one direction of a confidence interval, for example when we want to show that a failure rate is less than or equal to some value. A lower confidence bound for $\theta$ with level $1-\alpha$ is a statistic $L(X)$ such that $\mathbb{P}_{\theta}\bigl{(}L(X)\leq\theta\bigr{)}\geq 1-\alpha$ for all $\theta$. The corresponding one-sided confidence interval is then $[L(X),\infty)$. An upper confidence bound is defined symmetrically. In the pivot method, a one-sided interval can be obtained by considering the single inequality $Q(X,\theta)\leq b$, where $b$ is the $(1-\alpha)$-quantile of the pivot. For example, for the normal distribution with known variance, this gives the lower bound $\bar{X}-z_{1-\alpha}\,\sigma/\sqrt{n}$ as promised in section 25.3, where $z_{1-\alpha}$ is used instead of $z_{1-\alpha/2}$ since all of the error probability is allocated to one side. In the context of the pivot method, any pair of values $a<b$ with $\mathbb{P}(a\leq Q\leq b)=1-\alpha$ defines a valid two-sided interval. Here we use the equal-tailed choice, i.e. $a$ and $b$ are the $(\alpha/2)$- and $(1-\alpha/2)$-quantiles, respectively. This choice is the shortest interval for a symmetric, unimodal pivot density, but may not be shortest for skewed densities like the chi-squared distribution; the interval boundaries can be found in the tables. Exercise 25.2 shows an example where the shortest interval is one-sided. In lecture 28 we will see the same choice of interval again, when we consider credible intervals.