Lecture 23 Likelihood-Ratio Tests

As in lecture 22, we write $L(\theta)$ for the likelihood of the data $X=(X_{1},\dots,X_{n})$ and $\ell(\theta)=\log L(\theta)$ for the log-likelihood, where the parameter space $\Theta$ is now allowed to be a subset of $\mathbb{R}^{k}$. We consider the following testing problem:

for a subset $\Theta_{0}\subseteq\Theta$, where both hypotheses can be composite. Since the alternative is the complement of the null hypothesis within $\Theta$, the null hypothesis is a restriction of the full model and we have nested hypotheses.

If the hypothesis is composite, the likelihood principle suggests representing it in the Neyman–Pearson comparison of the two likelihoods by the parameter value which gives the best fit to the data, i.e. by the maximum of the likelihood over the hypothesis.

The denominator of $\Lambda$ is the likelihood at the maximum likelihood estimator $\hat{\theta}$ from lecture 5, the numerator is the likelihood at the restricted maximum likelihood estimator $\hat{\theta}_{0}$, which is the best value within $H_{0}$; if $H_{0}$ is simple, we have $\Theta_{0}=\{\theta_{0}\}$ and the numerator is just $L(\theta_{0})$. Since $\Theta_{0}\subseteq\Theta$ we always have $0\leq\Lambda\leq 1$: values close to one mean that the restriction to $\Theta_{0}$ costs little in terms of fit, values close to zero mean that $H_{0}$ is a much worse fit for the data than the full model is, and this is the evidence on which we base the rejection. The factor $-2$ in $W$ is the scaling which, as we will see in section 23.3, gives the limiting distribution of $W$ its standard form. In terms of the log-likelihood we have

which is twice the amount the maximised log-likelihood decreases when the null hypothesis is imposed.

The test is an extension of the Neyman–Pearson test, not a replacement. For example, for a two-point parameter space $\Theta=\{\theta_{0},\theta_{1}\}$ where $\Theta_{0}=\{\theta_{0}\}$, and using the notation $R=L(\theta_{1})/L(\theta_{0})$ for the likelihood ratio from lecture 22, we have

and for $c<1$ the event $\Lambda\leq c$ is the same as the event $R\geq 1/c$; thus, for this case, the test is the most powerful test from theorem 22.2 with $k=1/c$. For composite hypotheses no optimality is guaranteed and in fact proposition 22.5 shows that for the two-sided normal problem no such test can exist.

Two more properties follow directly from the definition. First, if $T$ is a sufficient statistic, then we can use the factorisation theorem 8.2 to write $L(\theta)=g\bigl{(}T(x);\theta\bigr{)}h(x)$, where the factor $h(x)$ can be cancelled between numerator and denominator. This shows that $\Lambda$ can be written as a function of the data which depends on the data only through $T$. Secondly, if $\psi=g(\theta)$ is a one-to-one reparametrisation, the suprema in the definition of $\Lambda$ do not change and thus $\Lambda$ and $W$ do not depend on the choice of parametrisation of the model. This is the same invariance as in theorem 5.6, and exercise 23.3 verifies it in an example.

We will now consider the question which was left open at the end of lecture 22, i.e. we will consider the two-sided alternative for the mean of a normal sample with known variance. This example will carry over as an example for all the distributions considered later, since for this case the distribution of $W$ under $H_{0}$ can be found exactly.

A familiar test has been recovered from the general recipe, and the next section shows that the chi-squared distribution of $W$ is the general case for large samples, even in models where normality is not exactly satisfied.

In example 23.2 the distribution of $W$ under $H_{0}$ was known, because $\bar{X}$ is exactly normal; in most models this will not be the case and Wilks’ theorem provides the large-sample approximation which allows the test to be used practically.

The theorem is stated for hypotheses which fix some coordinates of the parameter. Let $\Theta\subseteq\mathbb{R}^{k}$ be open, write $\theta=(\theta_{1},\dots,\theta_{k})$, and let the null hypothesis fix the first $r$ coordinates at given values,

where $1\leq r\leq k$. For $r=k$ the null hypothesis is simple, and for $r<k$ the free coordinates $\theta_{r+1},\dots,\theta_{k}$ are nuisance parameters, unknown parameters about which the hypothesis says nothing. The number $r$ of constraints is the difference between the dimensions of $\Theta$ and $\Theta_{0}$.

Whatever the model, the only feature of the problem which enters the limit is the number $r$ of constraints. In example 23.2 we have $k=r=1$ and there the statement holds exactly for every $n$. We do not prove the theorem: together with the vector case of theorem 17.1, on which its general form rests, it is one of the few results of this module which we state without proof. The following argument shows where the chi-squared distribution comes from.

Consider the case $k=r=1$ of a one-dimensional parameter and a simple null hypothesis $H_{0}\colon\theta=\theta_{0}$, so that $W_{n}=2\bigl{(}\ell(\hat{\theta}_{n})-\ell(\theta_{0})\bigr{)}$. We will expand the log-likelihood around the maximum likelihood estimator (MLE) instead of around $\theta_{0}$ as in lecture 17: using Taylor’s theorem with the Lagrange form of the remainder we find

for some $\theta_{n}^{*}$ between $\theta_{0}$ and $\hat{\theta}_{n}$. Since $\hat{\theta}_{n}$ solves the likelihood equation, the first derivative at this point is zero and rearranging this gives

Now assume that $H_{0}$ is true. Then, by theorem 17.1, the second term is the square of a quantity which converges in distribution to $N\bigl{(}0,1/\mathcal{I}(\theta_{0})\bigr{)}$, and by the continuous mapping theorem, theorem A.17, this shows that the second term converges in distribution to $Y^{2}/\mathcal{I}(\theta_{0})$ where $Y$ is standard normally distributed. The first term is the term $B_{n}$ from the proof of theorem 17.1 with $\theta_{n}^{*}$ instead of $\theta_{0}$ and since $\theta_{n}^{*}$ is between $\theta_{0}$ and the consistent estimator $\hat{\theta}_{n}$, the bound (R4) on the third derivative shows that this term still converges in probability to $\mathcal{I}(\theta_{0})$. Finally, by Slutsky’s theorem, theorem A.16, we find

by definition A.2. Since the Fisher information cancels, the limit is independent of the details of the model.

For a simple null hypothesis in one dimension, this argument is close to a complete proof. The proof for the general case is more challenging: the expansion will be in $k$ variables and will use the vector version of theorem 17.1, and the quadratic form which replaces $Y^{2}$ will consist of $r$ independent, squared standard normal variables, one for each constraint. This is the origin of the number $r$ of degrees of freedom in the argument.

Wilks’ theorem turns the generalised likelihood-ratio test into a recipe: compute the statistic $W=2\bigl{(}\ell(\hat{\theta})-\ell(\hat{\theta}_{0})\bigr{)}$ from the two maximised log-likelihoods, found numerically if need be as in lecture 18; count the degrees of freedom $r=\dim\Theta-\dim\Theta_{0}$; reject $H_{0}$ at level $\alpha$, if $W\geq\chi^{2}_{r,1-\alpha}$; and report the approximate $p$-value $\mathbb{P}(\chi^{2}_{r}\geq w)$ for the observed value $w$, which for $r=1$ equals $2\bigl{(}1-\Phi(\sqrt{w})\bigr{)}$. Only the counting of degrees of freedom requires some care. For example, in the case of the mean of a normal sample with unknown variance, and $H_{0}\colon\mu=\mu_{0}$, we have $k=2$ and $r=1$, since the variance is unrestricted under both hypotheses. From exercise 23.2 we know that $r=m-1$ for a multinomial distribution with $m$ categories and a fully specified null hypothesis.

The theorem is a limit statement and there are three cases where problems can occur when applying the theorem to a fixed $n$. First, the approximation may be poor for small sample size: for the mean of a normal sample with unknown variance, exercise 23.1 shows that the level $0.05$ test based on $\chi^{2}_{1,0.95}$ rejects too often at $n=10$. Lecture 24 watches the approximation improve as $n$ increases, by simulation. Whenever the exact distribution of $W$ or of an equivalent test statistic is known, as in the situation of the exercise, this distribution should be used. Lecture 29 takes this approach, using the exact distribution, for the standard tests. Secondly, if the regularity conditions are violated, for example if the support of the distribution depends on the parameter or if the null value coincides with the boundary of the parameter space, the expansion (23.4) will break down and the limit will not be $\chi^{2}_{r}$. Finally, the theorem only describes $W$ under $H_{0}$. The theorem gives the size of the test, but the power of the test must be determined separately.

Software output often lists two relatives of the likelihood-ratio test: the Wald test, using $n\mathcal{I}(\hat{\theta})(\hat{\theta}-\theta_{0})^{2}$ as in corollary 17.3, and the score test, using the squared score $\ell^{\prime}(\theta_{0})^{2}/\bigl{(}n\mathcal{I}(\theta_{0})\bigr{)}$. Both tests have the same chi-squared limit under $H_{0}$ as $W$, but here we use the likelihood-ratio test throughout.