Lecture 22 The Neyman–Pearson Lemma

In lecture 20 it was clear that for the mean of a normal sample we could not do better than to reject for large values of $\bar{X}$ . This lecture proves that this is indeed correct: for the most basic testing problem, where one distribution is being tested against a different distribution, the Neyman–Pearson lemma allows us to identify the most powerful test. This is in analogy to the role of the Cramer–Rao bound from lecture 13 for estimators. The same test is optimal for all one-sided alternatives, but there is no single test which is optimal for a two-sided alternative. In fact, in lecture 23 we will see the required tool for this case.

22.1 Simple Hypotheses and Most Powerful Tests

We consider the problem of testing a simple null hypothesis $H_{0}\colon\theta=\theta_{0}$ against a simple alternative $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{0}\neq\theta_{1}$ are two fixed points of the parameter space, for data $X=(X_{1},\dots,X_{n})$ with joint density or probability weights $f(x;\theta)$ as in lecture 8. Under either hypothesis the distribution of the data is completely specified; section 22.4 shows that the solution of this problem also settles more realistic ones.

We will compare different tests, so we write $\beta_{C}(\theta)=\mathbb{P}_{\theta}(X\in C)$ for the power function of definition 20.4 of the test with critical region $C$ . For simple hypotheses, the size from definition 20.5 and the power are the single numbers $\beta_{C}(\theta_{0})$ and $\beta_{C}(\theta_{1})$ . Among all tests with level $\alpha$ , i.e. with $\beta_{C}(\theta_{0})\leq\alpha$ , we want to find the test which makes type II errors least often.

Definition 22.1.

Let $0<\alpha<1$ . A test with critical region $C$ is said to be most powerful at level $\alpha$ for testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ , if $\beta_{C}(\theta_{0})\leq\alpha$ and

\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})

for all critical regions $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ .

As with the uniformly minimum variance unbiased estimator from definition 10.3, the definition does not give a method to find such a test or even whether such a test exists; we will answer both of these questions in the next section.

22.2 The Lemma

If $f(x;\theta_{1})$ is much larger than $f(x;\theta_{0})$ , the data favour the alternative, and we should reject. The natural test statistic is the likelihood ratio

R(x)=\frac{f(x;\theta_{1})}{f(x;\theta_{0})}=\frac{L(\theta_{1})}{L(\theta_{0}% )},

and the natural test rejects if $R(X)$ exceeds some threshold $k$ . The lemma states that no test with the same level can have more power. We write the inequality $f(x;\theta_{1})>k\,f(x;\theta_{0})$ instead of $R(x)>k$ so that points with $f(x;\theta_{0})=0$ do not require separate treatment and so that the boundary of the critical region can include any points where equality holds, as required for discrete models and in exercise 22.3.

Theorem 22.2 (Neyman–Pearson lemma).

Let $k\geq 0$ and let $C$ be a critical region with

\bigl{\{}x\mathrel{\big{|}}f(x;\theta_{1})>k\,f(x;\theta_{0})\bigr{\}}% \subseteq C\subseteq\bigl{\{}x\mathrel{\big{|}}f(x;\theta_{1})\geq k\,f(x;% \theta_{0})\bigr{\}},

where $\alpha=\beta_{C}(\theta_{0})$ is the size of the critical region. Then the following statements hold.

1.

The test with critical region $C$ is the most powerful test at level $\alpha$ for the hypothesis $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ .
2.

If $C^{\prime}$ is another critical region, which is most powerful at level $\alpha$ for the same problem, then $C$ and $C^{\prime}$ differ at most on the boundary set $\{x\mid f(x;\theta_{1})=k\,f(x;\theta_{0})\}$ , except for a set of observations with probability zero for every $\theta\in\Theta$ .

Proof.

Let $C^{\prime}$ be a critical region with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ and let $\mathbf{1}_{C}$ and $\mathbf{1}_{C^{\prime}}$ be the indicator functions for the two regions. Then the proof is based on the observation that

equation (22.1) (22.1)

h(x)=\bigl{(}\mathbf{1}_{C}(x)-\mathbf{1}_{C^{\prime}}(x)\bigr{)}\bigl{(}f(x;% \theta_{1})-k\,f(x;\theta_{0})\bigr{)}\geq 0\qquad\text{for all }x.

If $x\in C$ , the first factor is either $0$ or $1$ and the second factor is non-negative by the choice of $C$ . If $x\notin C$ , the first factor is either $0$ or $-1$ and the second factor is zero or negative by the choice of $C$ .

Integrating (22.1) over all $x$ , with a sum in place of the integral for a discrete model, and using $\int\mathbf{1}_{A}(x)\,f(x;\theta)\,\mathrm{d}x=\mathbb{P}_{\theta}(X\in A)=% \beta_{A}(\theta)$ for each of the four terms, we find

equation (22.2) (22.2)

0\leq\int h(x)\,\mathrm{d}x=\bigl{(}\beta_{C}(\theta_{1})-\beta_{C^{\prime}}(% \theta_{1})\bigr{)}-k\bigl{(}\beta_{C}(\theta_{0})-\beta_{C^{\prime}}(\theta_{% 0})\bigr{)}.

Since $\beta_{C}(\theta_{0})=\alpha\geq\beta_{C^{\prime}}(\theta_{0})$ and $k\geq 0$ , the subtracted term is non-negative and thus we have

\beta_{C}(\theta_{1})-\beta_{C^{\prime}}(\theta_{1})\geq k\bigl{(}\alpha-\beta% _{C^{\prime}}(\theta_{0})\bigr{)}\geq 0,

and this is the first statement.

For the second statement assume that $C^{\prime}$ is also most powerful at level $\alpha$ . Then $\beta_{C^{\prime}}(\theta_{1})\geq\beta_{C}(\theta_{1})$ by definition 22.1, and $\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})$ by the first statement, so that the two powers are equal and the first bracket in (22.2) vanishes. The right-hand side of (22.2) is then $-k\bigl{(}\alpha-\beta_{C^{\prime}}(\theta_{0})\bigr{)}\leq 0$ , and together with the inequality on the left this gives $\int h(x)\,\mathrm{d}x=0$ . Now $h(x)>0$ holds exactly for those $x$ which lie in one of the regions $C$ and $C^{\prime}$ but not in the other and at which $f(x;\theta_{1})\neq k\,f(x;\theta_{0})$ , since at such points both factors of $h$ are non-zero and, by the first paragraph, of the same sign. For a discrete model a sum of non-negative terms is zero only if every term is zero, and thus this set is empty. For a model with densities, a non-negative function with integral zero vanishes outside a set of measure zero, and a set of measure zero has probability zero under every density $f(x;\theta)$ . This completes the proof. ∎

Two comments help in using the lemma. First, the constant $k$ is rarely computed: whenever $R$ is a strictly monotone function of a simpler statistic $T$ , we rewrite $R(x)>k$ as $T(x)>c$ or $T(x)<c$ and choose $c$ from the distribution of $T(X)$ under $\theta_{0}$ to give size $\alpha$ . Second, the lemma shows that the test is most powerful at the level equal to its own size, whatever value that size takes. For models with densities, $R(X)$ usually has a continuous distribution, and thus in theory any size can be achieved and the boundary set in the second statement has probability zero. Together these results show that the second statement implies that the most powerful test is essentially unique. In addition, it can be shown (for example in exercise 22.4) that the most powerful test can be chosen to depend on the data only via a sufficient statistic.

Remark.

For discrete models the size $\mathbb{P}_{\theta_{0}}\bigl{(}R(X)>k\bigr{)}$ of tests can only take certain values as $k$ changes and it is in general not possible to obtain a test with a given size $\alpha$ . The usual solution in textbooks, to randomise the test on the boundary set to achieve the required size, is not taken here. Instead, we consider the test with the largest possible size which is less than or equal to $\alpha$ . This test, by the result of exercise 22.1, will be the most powerful test for the smaller level.

22.3 The Normal Mean

We now apply the lemma to the normal-mean example from lecture 20: the test constructed there using common sense is the most powerful one.

Example 22.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\theta,\sigma^{2})$ where $\sigma^{2}$ is known, and consider the hypothesis $H_{0}\colon\theta=\theta_{0}$ against the alternative $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{1}>\theta_{0}$ . The joint density of the observations is $f(x;\theta)=(2\pi\sigma^{2})^{-n/2}\exp\bigl{(}-\sum_{i}(x_{i}-\theta)^{2}/(2% \sigma^{2})\bigr{)}$ . In the ratio of two densities, the constant factors cancel. After expanding the squares in the definition of the test statistic, we find

\sum_{i=1}^{n}(x_{i}-\theta_{1})^{2}-\sum_{i=1}^{n}(x_{i}-\theta_{0})^{2}=-2(% \theta_{1}-\theta_{0})\sum_{i=1}^{n}x_{i}+n(\theta_{1}^{2}-\theta_{0}^{2}),

and thus

\log R(x)=\frac{n(\theta_{1}-\theta_{0})}{\sigma^{2}}\,\bar{x}-\frac{n(\theta_% {1}^{2}-\theta_{0}^{2})}{2\sigma^{2}}.

Since $\theta_{1}-\theta_{0}>0$ , the right-hand side is a strictly increasing function of $\bar{x}$ and the inequality $R(x)>k$ is equivalent to $\bar{x}>c$ where $c=\sigma^{2}\log k/\bigl{(}n(\theta_{1}-\theta_{0})\bigr{)}+(\theta_{1}+\theta% _{0})/2$ . Thus, the critical region is of the form $\{\bar{x}>c\}$ and we can choose the value of $c$ by considering the size of the test: under $H_{0}$ we have $\bar{X}\sim N(\theta_{0},\sigma^{2}/n)$ by proposition A.3 and thus

\mathbb{P}_{\theta_{0}}(\bar{X}>c)=1-\Phi\Bigl{(}\frac{\sqrt{n}\,(c-\theta_{0}% )}{\sigma}\Bigr{)}=\alpha

if and only if

c=c_{\alpha}=\theta_{0}+z_{1-\alpha}\frac{\sigma}{\sqrt{n}},

where $z_{1-\alpha}$ is the $(1-\alpha)$ -quantile of the standard normal distribution. By theorem 22.2 the test which rejects for $\bar{X}>c_{\alpha}$ is most powerful at level $\alpha$ and, since the boundary set $\{\bar{x}=c_{\alpha}\}$ has probability zero, this is the only test up to sets of probability zero. This is the test from example 20.6. The power of this test for the alternative is

\beta(\theta_{1})=\mathbb{P}_{\theta_{1}}(\bar{X}>c_{\alpha})=1-\Phi\Bigl{(}z_% {1-\alpha}-\frac{\sqrt{n}\,(\theta_{1}-\theta_{0})}{\sigma}\Bigr{)}.

For $n=25$ , $\sigma=2$ , $\theta_{0}=10$ and $\alpha=0.05$ we have $z_{0.95}=1.645$ and thus $c_{\alpha}=10+1.645\cdot 2/5=10.658$ , and the power against $\theta_{1}=11$ is $1-\Phi(1.645-2.5)=\Phi(0.855)=0.804$ , the value we found in lecture 20. The lemma allows us to turn this number into a statement about all possible procedures: no test of level $0.05$ , based on the given $25$ observations, can be better.

22.4 One-Sided Alternatives

The threshold $c_{\alpha}=\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}$ in example 22.3 does not involve $\theta_{1}$ : the alternative entered the computation only through the sign of $\theta_{1}-\theta_{0}$ , and thus every $\theta_{1}>\theta_{0}$ leads to the same critical region. This has a consequence for the composite problem of testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta>\theta_{0}$ , the problem of interest in lecture 20.

Proposition 22.4.

Consider testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta>\theta_{0}$ . Assume that $C$ is a critical region with $\beta_{C}(\theta_{0})=\alpha$ which, for every $\theta_{1}>\theta_{0}$ , satisfies the condition from theorem 22.2 for testing $\theta_{0}$ against $\theta_{1}$ , for some constant $k=k(\theta_{1})$ which may depend on $\theta_{1}$ . Then, for every $\theta_{1}>\theta_{0}$ and every critical region $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ , we have $\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})$ .

Proof.

Let $\theta_{1}>\theta_{0}$ and let $C^{\prime}$ be a critical region with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ . Then, since the null hypothesis is simple, $C^{\prime}$ is also a test of level $\alpha$ for the simple hypothesis $\theta_{0}$ against $\theta_{1}$ and by assumption $C$ satisfies the condition from theorem 22.2 for this problem. Thus, by the first statement of the theorem, we have $\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})$ and since $\theta_{1}$ was arbitrary this completes the proof. ∎

A test which has this property is most powerful against every alternative in $H_{1}$ simultaneously and is said to be uniformly most powerful at level $\alpha$ . We only use this name as a description and do not consider the general theory of such tests.

For the normal mean, example 22.3 shows that the critical region $\{\bar{x}>\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}\}$ is the Neyman–Pearson region for every $\theta_{1}>\theta_{0}$ , and thus the one-sided test of example 20.6 is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ , with power function

\beta_{+}(\theta)=1-\Phi\Bigl{(}z_{1-\alpha}-\frac{\sqrt{n}\,(\theta-\theta_{0% })}{\sigma}\Bigr{)}.

For alternatives $\theta_{1}<\theta_{0}$ the same computation with the inequality reversed gives the region $\{\bar{x}<\theta_{0}-z_{1-\alpha}\sigma/\sqrt{n}\}$ , uniformly most powerful against $H_{1}\colon\theta<\theta_{0}$ , with power function

\beta_{-}(\theta)=1-\Phi\Bigl{(}z_{1-\alpha}+\frac{\sqrt{n}\,(\theta-\theta_{0% })}{\sigma}\Bigr{)}.

22.5 Two-Sided Alternatives

We now consider the alternative $H_{1}\colon\theta\neq\theta_{0}$ . The two one-sided tests disagree: one rejects for large values of $\bar{X}$ , the other for small values, and each one is poor on the other side of $\theta_{0}$ , since for $\theta<\theta_{0}$ we have $\beta_{+}(\theta)<\alpha$ . Thus, no single test can agree with both tests simultaneously, and the uniqueness statement in the lemma implies that this argument constitutes a proof.

Proposition 22.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\theta,\sigma^{2})$ with known $\sigma^{2}$ and let $0<\alpha<1$ . Then there is no critical region $C^{*}$ with $\beta_{C^{*}}(\theta_{0})\leq\alpha$ such that $\beta_{C^{*}}(\theta)\geq\beta_{C^{\prime}}(\theta)$ for all $\theta\neq\theta_{0}$ and any critical region $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ .

Proof.

Suppose that $C^{*}$ were such a region, and fix some $\theta_{1}>\theta_{0}$ . Then $C^{*}$ is most powerful at level $\alpha$ for testing $\theta_{0}$ against $\theta_{1}$ , and so is the region $C_{+}=\{\bar{x}>c_{\alpha}\}$ of example 22.3. The boundary set $\{\bar{x}=c_{\alpha}\}$ has probability zero, and thus by the second statement of theorem 22.2 the regions $C^{*}$ and $C_{+}$ differ only on a set which has probability zero for every $\theta$ . Consequently the two tests have the same power function, $\beta_{C^{*}}(\theta)=\beta_{+}(\theta)$ for all $\theta$ . Now take any $\theta_{2}<\theta_{0}$ . From the formulas of section 22.4 we get

\beta_{C^{*}}(\theta_{2})=\beta_{+}(\theta_{2})<1-\Phi(z_{1-\alpha})=\alpha<% \beta_{-}(\theta_{2}),

since $\Phi$ is strictly increasing and $\sqrt{n}\,(\theta_{2}-\theta_{0})/\sigma<0$ . The left-sided test $C_{-}=\{\bar{x}<\theta_{0}-z_{1-\alpha}\sigma/\sqrt{n}\}$ has size $\alpha$ and thus is an admissible competitor $C^{\prime}$ , and it has strictly larger power at $\theta_{2}$ than $C^{*}$ . This contradicts the assumed property of $C^{*}$ , which completes the proof. ∎

The argument is not particular to the normal distribution: whenever the one-sided most powerful regions disagree, a uniformly most powerful two-sided test would have to coincide with both, which is impossible, and so we have to settle for a compromise. The two-sided test in example 20.7 is such a compromise: the power of this test is less than $\beta_{+}(\theta)$ for $\theta>\theta_{0}$ , but it is strictly larger than $\alpha$ for all $\theta\neq\theta_{0}$ . In lecture 23 we will see that this test is the result of applying the likelihood principle to a composite alternative. The generalised likelihood-ratio test obtained in this way is a general-purpose tool which can be used when no uniformly most powerful test exists.

Summary.

•

A test is most powerful at level $\alpha$ for a simple null hypothesis against a simple alternative, if it has level $\alpha$ and no other test of level $\alpha$ has larger power.
•

The Neyman–Pearson lemma: The test which rejects when $f(x;\theta_{1})>k\,f(x;\theta_{0})$ , where $k$ is chosen so that the size is $\alpha$ , is most powerful at level $\alpha$ . Any other most powerful test coincides with this test outside the boundary set. The proof involves integration of $(\mathbf{1}_{C}-\mathbf{1}_{C^{\prime}})(f(x;\theta_{1})-k\,f(x;\theta_{0}))\geq 0$ .
•

In practice, $R(x)>k$ can be rewritten as $T(x)>c$ for a simple statistic $T$ and $c$ can be found by considering the distribution of $T$ under $H_{0}$ ; for the normal mean this gives $\bar{X}>\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}$ .
•

If the region of acceptance does not depend on the alternative $\theta_{1}$ , the same test will be most powerful against all $\theta_{1}>\theta_{0}$ : the test is then uniformly most powerful for the one-sided problem.
•

For the two-sided alternative, no uniformly most powerful test can exist, since the two one-sided tests disagree; instead, the likelihood-ratio tests from lecture 23 can be used.

Exercise 22.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with mean $\theta>0$ and let $S=\sum_{i=1}^{n}X_{i}$ , Poisson with mean $n\theta$ . We test $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ where $\theta_{1}>\theta_{0}$ .

1.

Show that the likelihood ratio $R(x)$ is a strictly increasing function of $s=\sum_{i}x_{i}$ and that the most powerful test rejects when $S>c$ . Why does $c$ not depend on $\theta_{1}$ ? What are the consequences for testing against the composite alternative $H_{1}\colon\theta>\theta_{0}$ ?
2.

Let $n=5$ , $\theta_{0}=1$ and $\alpha=0.05$ . Determine the smallest integer $c$ such that the test has size at most $0.05$ . Compute the actual size of the test for this value. You may use $\mathbb{P}(S\leq 8)=0.9319$ and $\mathbb{P}(S\leq 9)=0.9682$ for a Poisson random variable $S$ with mean $5$ .
3.

Determine the power of the test at $\theta_{1}=2$ , using $\mathbb{P}(S\leq 9)=0.4579$ for a Poisson random variable with mean $10$ .
4.

Why cannot a test of the form $\{S>c\}$ have size exactly $0.05$ ? At which level is the test from part (b) most powerful?

Exercise 22.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $p\in(0,1)$ , and let $S=\sum_{i=1}^{n}X_{i}$ . In exercise 20.2 we tested $H_{0}\colon p=1/2$ against $H_{1}\colon p>1/2$ by rejecting for large values of $S$ .

1.

For a fixed alternative $p_{1}>1/2$ , show that the likelihood ratio is given by

$R(x)=\Bigl{(}\frac{p_{1}}{1-p_{1}}\Bigr{)}^{s}\bigl{(}2(1-p_{1})\bigr{)}^{n},% \qquad s=\sum_{i=1}^{n}x_{i},$

and that it is strictly increasing in $s$ . Deduce that the most powerful test rejects when $S>c$ , and that the test of exercise 20.2 is uniformly most powerful against $H_{1}\colon p>1/2$ .
2.

Let $n=20$ . Using $\mathbb{P}_{1/2}(S\geq 14)=0.0577$ and $\mathbb{P}_{1/2}(S\geq 15)=0.0207$ , find the critical region of the largest size not exceeding $0.05$ , and compute its power at $p_{1}=0.7$ using $\mathbb{P}_{0.7}(S\leq 14)=0.5836$ . Compare the size and the power with those found for $n=10$ in exercise 20.2.
3.

For an alternative $p_{1}<1/2$ , find the most powerful critical region, and explain why the two one-sided problems lead to different tests.

Exercise 22.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the set $(0,\theta)$ , with joint density $f(x;\theta)=\theta^{-n}\mathbf{1}_{\{m\leq\theta\}}$ for $x_{1},\dots,x_{n}\geq 0$ , where $m=\max_{i}x_{i}$ , and let $M=\max_{i}X_{i}$ . From example 8.10 we know that $\mathbb{P}_{\theta}(M\leq y)=(y/\theta)^{n}$ for $0\leq y\leq\theta$ . We want to test $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{1}>\theta_{0}$ , at significance level $\alpha\in(0,1)$ . In exercise 20.3 we have considered tests based on $M$ for this model, and in the following we will see that these tests are optimal.

1.

Let $k=(\theta_{0}/\theta_{1})^{n}$ . Determine the sets $\{x\mid f(x;\theta_{1})>k\,f(x;\theta_{0})\}$ and $\{x\mid f(x;\theta_{1})=k\,f(x;\theta_{0})\}$ in terms of $m$ .
2.

Show that the critical region $C=\{m>c\}$ for $c=\theta_{0}(1-\alpha)^{1/n}$ has size $\alpha$ and satisfies the condition from theorem 22.2 for this $k$ . Conclude that this is the most powerful test at level $\alpha$ . Compute the power of the test and explain why the test is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ .
3.

Show that the region $C^{\prime\prime}=\{m>\theta_{0}\}\cup\{m\leq\theta_{0}\alpha^{1/n}\}$ also has size $\alpha$ and the same power as $C$ . Explain, using the second statement of theorem 22.2, why this does not contradict the theorem, even though the model has densities.

Exercise 22.4.

Let $T$ be a sufficient statistic for the model $f(x;\theta)$ and let $\theta_{0}\neq\theta_{1}$ be two parameter values. Using the factorisation theorem 8.2, show that the likelihood ratio $R(x)=f(x;\theta_{1})/f(x;\theta_{0})$ depends on the data only through $T(x)$ , for all $x$ where the denominator is positive.