Lecture 23 The Neyman–Pearson Lemma

In lecture 22 it seemed clear that for the mean of a normal sample we could not do better than to reject for large values of $\bar{X}$ . This lecture proves that this is correct: for the most basic testing problem, where one distribution is tested against another, the Neyman–Pearson lemma identifies the most powerful test, in analogy to the role of the Cramér–Rao bound from lecture 13 for estimators. We then see how far this optimality extends to composite alternatives.

23.1 Simple Hypotheses and Most Powerful Tests

We consider the problem of testing a simple null hypothesis $H_{0}\colon\theta=\theta_{0}$ against a simple alternative $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{0}\neq\theta_{1}$ are two fixed points of the parameter space, for data $X=(X_{1},\dots,X_{n})$ with joint density or probability weights $f(x;\theta)$ as in lecture 8. Under either hypothesis the distribution of the data is completely specified; section 23.4 shows that the solution of this problem also settles more realistic ones.

We will compare different tests, so we write $\beta_{C}(\theta)=\mathbb{P}_{\theta}(X\in C)$ for the power function of definition 22.4 of the test with critical region $C$ . For simple hypotheses, the size from definition 22.5 and the power are the single numbers $\beta_{C}(\theta_{0})$ and $\beta_{C}(\theta_{1})$ . Among all tests with level $\alpha$ , i.e. with $\beta_{C}(\theta_{0})\leq\alpha$ , we want to find the test which makes type II errors least often.

Definition 23.1.

Let $0<\alpha<1$ . A test with critical region $C$ is said to be most powerful at level $\alpha$ for testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ , if $\beta_{C}(\theta_{0})\leq\alpha$ and

\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})

for all critical regions $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ .

As with the uniformly minimum variance unbiased estimator from definition 10.4, the definition does not say how to find such a test, or whether one exists; the next section answers both questions.

23.2 The Lemma

If $f(x;\theta_{1})$ is much larger than $f(x;\theta_{0})$ , the data favour the alternative, and we should reject. The natural test statistic is the likelihood ratio

R(x)=\frac{f(x;\theta_{1})}{f(x;\theta_{0})}=\frac{L(\theta_{1})}{L(\theta_{0}% )},

and the natural test rejects if $R(X)$ exceeds some threshold $k$ . The lemma states that no test with the same level can have more power. We write the inequality $f(x;\theta_{1})>k\,f(x;\theta_{0})$ instead of $R(x)>k$ so that points with $f(x;\theta_{0})=0$ do not require separate treatment and so that the boundary of the critical region can include any points where equality holds, as required for discrete models and in exercise 23.3.

Theorem 23.2 (Neyman–Pearson lemma).

Let $k\geq 0$ and let $C$ be a critical region with

\bigl{\{}x\mathrel{\big{|}}f(x;\theta_{1})>k\,f(x;\theta_{0})\bigr{\}}% \subseteq C\subseteq\bigl{\{}x\mathrel{\big{|}}f(x;\theta_{1})\geq k\,f(x;% \theta_{0})\bigr{\}},

where the size $\alpha=\beta_{C}(\theta_{0})$ satisfies $0<\alpha<1$ . Then the following statements hold.

1.

The test with critical region $C$ is the most powerful test at level $\alpha$ for the hypothesis $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ .
2.

If $C^{\prime}$ is another critical region, which is most powerful at level $\alpha$ for the same problem, then $C$ and $C^{\prime}$ differ at most on the boundary set $\{x\mid f(x;\theta_{1})=k\,f(x;\theta_{0})\}$ , except for a set of observations with probability zero for every $\theta\in\Theta$ .

Proof.

Let $C^{\prime}$ be a critical region with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ and let $\mathbf{1}_{C}$ and $\mathbf{1}_{C^{\prime}}$ be the indicator functions for the two regions. Then the proof is based on the observation that

equation (23.1) (23.1)

h(x)=\bigl{(}\mathbf{1}_{C}(x)-\mathbf{1}_{C^{\prime}}(x)\bigr{)}\bigl{(}f(x;% \theta_{1})-k\,f(x;\theta_{0})\bigr{)}\geq 0\qquad\text{for all }x.

If $x\in C$ , the first factor is either $0$ or $1$ and the second factor is non-negative by the choice of $C$ . If $x\notin C$ , the first factor is either $0$ or $-1$ and the second factor is zero or negative by the choice of $C$ .

Integrating (23.1) over all $x$ , with a sum in place of the integral for a discrete model, and using $\int\mathbf{1}_{A}(x)\,f(x;\theta)\,\mathrm{d}x=\mathbb{P}_{\theta}(X\in A)=% \beta_{A}(\theta)$ for each of the four terms, we find

equation (23.2) (23.2)

0\leq\int h(x)\,\mathrm{d}x=\bigl{(}\beta_{C}(\theta_{1})-\beta_{C^{\prime}}(% \theta_{1})\bigr{)}-k\bigl{(}\beta_{C}(\theta_{0})-\beta_{C^{\prime}}(\theta_{% 0})\bigr{)}.

Since $\beta_{C}(\theta_{0})=\alpha\geq\beta_{C^{\prime}}(\theta_{0})$ and $k\geq 0$ , the subtracted term is non-negative and thus we have

\beta_{C}(\theta_{1})-\beta_{C^{\prime}}(\theta_{1})\geq k\bigl{(}\alpha-\beta% _{C^{\prime}}(\theta_{0})\bigr{)}\geq 0,

and this is the first statement.

For the second statement assume that $C^{\prime}$ is also most powerful at level $\alpha$ . Then the two powers at $\theta_{1}$ are equal, and thus the right-hand side of (23.2) equals $-k\bigl{(}\alpha-\beta_{C^{\prime}}(\theta_{0})\bigr{)}\leq 0$ . Together with the inequality on the left we find $\int h(x)\,\mathrm{d}x=0$ . By the first paragraph, $h(x)$ is strictly positive exactly at those $x$ outside the boundary set which lie in one of the regions $C$ and $C^{\prime}$ but not in the other. For a discrete model a sum of non-negative terms is zero only if every term is zero, and for a model with densities a non-negative function with integral zero vanishes outside a set of measure zero, which has probability zero under every density $f(x;\theta)$ . This completes the proof. ∎

Two comments help in using the lemma. First, the constant $k$ is rarely computed: whenever $R$ is a strictly monotone function of a simpler statistic $T$ , we rewrite $R(x)>k$ as $T(x)>c$ or $T(x)<c$ and choose $c$ from the distribution of $T(X)$ under $\theta_{0}$ to give size $\alpha$ . Second, the lemma shows that the test is most powerful at the level equal to its own size. For models with densities, $R(X)$ usually has a continuous distribution, and thus any size can be achieved and the boundary set in the second statement has probability zero. In this case the most powerful test is essentially unique. For discrete models only certain sizes are attainable, and a test of size $\alpha^{\prime}<\alpha$ is guaranteed to be most powerful only at level $\alpha^{\prime}$ ; at level $\alpha$ another critical region can have more power.

23.3 The Normal Mean

We now apply the lemma to the normal-mean example from lecture 22: the test constructed there using common sense is the most powerful one.

Example 23.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\theta,\sigma^{2})$ where $\sigma^{2}$ is known, and consider the hypothesis $H_{0}\colon\theta=\theta_{0}$ against the alternative $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{1}>\theta_{0}$ . The joint density of the observations is $f(x;\theta)=(2\pi\sigma^{2})^{-n/2}\exp\bigl{(}-\sum_{i}(x_{i}-\theta)^{2}/(2% \sigma^{2})\bigr{)}$ . In the ratio of two densities, the constant factors cancel. After expanding the squares in the exponent, we find

\sum_{i=1}^{n}(x_{i}-\theta_{1})^{2}-\sum_{i=1}^{n}(x_{i}-\theta_{0})^{2}=-2(% \theta_{1}-\theta_{0})\sum_{i=1}^{n}x_{i}+n(\theta_{1}^{2}-\theta_{0}^{2}),

and thus

\log R(x)=\frac{n(\theta_{1}-\theta_{0})}{\sigma^{2}}\,\bar{x}-\frac{n(\theta_% {1}^{2}-\theta_{0}^{2})}{2\sigma^{2}}.

Since $\theta_{1}-\theta_{0}>0$ , the right-hand side is a strictly increasing function of $\bar{x}$ and the inequality $R(x)>k$ is equivalent to $\bar{x}>c$ for a constant $c$ which depends on $k$ . We choose $c$ by considering the size of the test: under $H_{0}$ we have $\bar{X}\sim N(\theta_{0},\sigma^{2}/n)$ by proposition A.4 and thus

\mathbb{P}_{\theta_{0}}(\bar{X}>c)=1-\Phi\Bigl{(}\frac{\sqrt{n}\,(c-\theta_{0}% )}{\sigma}\Bigr{)}=\alpha

if and only if

c=c_{\alpha}=\theta_{0}+z_{1-\alpha}\frac{\sigma}{\sqrt{n}},

where $z_{1-\alpha}$ is the $(1-\alpha)$ -quantile of the standard normal distribution. By theorem 23.2 the test which rejects for $\bar{X}>c_{\alpha}$ is most powerful at level $\alpha$ and, since the boundary set $\{\bar{x}=c_{\alpha}\}$ has probability zero, this is the only test up to sets of probability zero. This is the test from example 22.6. The power of this test for the alternative is

\beta(\theta_{1})=\mathbb{P}_{\theta_{1}}(\bar{X}>c_{\alpha})=\Phi\Bigl{(}% \frac{\sqrt{n}\,(\theta_{1}-\theta_{0})}{\sigma}-z_{1-\alpha}\Bigr{)}.

For $n=25$ , $\sigma=2$ , $\theta_{0}=10$ and $\alpha=0.05$ we have $z_{0.95}=1.645$ and thus $c_{\alpha}=10+1.645\cdot 2/5=10.658$ , and the power against $\theta_{1}=11$ is $\Phi(2.5-1.645)=\Phi(0.855)=0.804$ , the value we found in lecture 22. The lemma allows us to turn this number into a statement about all possible procedures: no test of level $0.05$ , based on the given $25$ observations, can be better.

23.4 One-Sided Alternatives

The threshold $c_{\alpha}=\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}$ in example 23.3 does not involve $\theta_{1}$ : the alternative entered the computation only through the sign of $\theta_{1}-\theta_{0}$ , and thus every $\theta_{1}>\theta_{0}$ leads to the same critical region. This has a consequence for the composite problem of testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta>\theta_{0}$ , the problem of interest in lecture 22.

Proposition 23.4.

Consider testing $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta>\theta_{0}$ . Assume that $C$ is a critical region with $\beta_{C}(\theta_{0})=\alpha$ which, for every $\theta_{1}>\theta_{0}$ , satisfies the condition from theorem 23.2 for testing $\theta_{0}$ against $\theta_{1}$ , for some constant $k=k(\theta_{1})$ which may depend on $\theta_{1}$ . Then, for every $\theta_{1}>\theta_{0}$ and every critical region $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ , we have $\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})$ .

Proof.

Let $\theta_{1}>\theta_{0}$ and let $C^{\prime}$ be a critical region with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ . Then, since the null hypothesis is simple, $C^{\prime}$ is also a test of level $\alpha$ for the simple hypothesis $\theta_{0}$ against $\theta_{1}$ and by assumption $C$ satisfies the condition from theorem 23.2 for this problem. Thus, by the first statement of the theorem, we have $\beta_{C}(\theta_{1})\geq\beta_{C^{\prime}}(\theta_{1})$ and since $\theta_{1}$ was arbitrary this completes the proof. ∎

A test which has this property is most powerful against every alternative in $H_{1}$ simultaneously and is said to be uniformly most powerful at level $\alpha$ . We only use this name as a description and do not consider the general theory of such tests.

For the normal mean, example 23.3 shows that the critical region $\{\bar{x}>\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}\}$ is the Neyman–Pearson region for every $\theta_{1}>\theta_{0}$ , and thus the one-sided test of example 22.6 is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ , with power function

\beta_{+}(\theta)=\Phi\Bigl{(}\frac{\sqrt{n}\,(\theta-\theta_{0})}{\sigma}-z_{% 1-\alpha}\Bigr{)}.

For alternatives $\theta_{1}<\theta_{0}$ the same computation with the inequality reversed gives the region $\{\bar{x}<\theta_{0}-z_{1-\alpha}\sigma/\sqrt{n}\}$ , uniformly most powerful against $H_{1}\colon\theta<\theta_{0}$ , with power function

\beta_{-}(\theta)=\Phi\Bigl{(}-\frac{\sqrt{n}\,(\theta-\theta_{0})}{\sigma}-z_% {1-\alpha}\Bigr{)}.

23.5 Two-Sided Alternatives

We now consider the alternative $H_{1}\colon\theta\neq\theta_{0}$ . The two one-sided tests disagree: one rejects for large values of $\bar{X}$ , the other for small values, and each one is poor on the other side of $\theta_{0}$ , since for $\theta<\theta_{0}$ we have $\beta_{+}(\theta)<\alpha$ . Thus, no single test can agree with both tests simultaneously, and the uniqueness statement in the lemma turns this observation into the following proposition.

Proposition 23.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\theta,\sigma^{2})$ with known $\sigma^{2}$ and let $0<\alpha<1$ . Then there is no critical region $C^{*}$ with $\beta_{C^{*}}(\theta_{0})\leq\alpha$ such that $\beta_{C^{*}}(\theta)\geq\beta_{C^{\prime}}(\theta)$ for all $\theta\neq\theta_{0}$ and any critical region $C^{\prime}$ with $\beta_{C^{\prime}}(\theta_{0})\leq\alpha$ .

Proof.

Suppose that $C^{*}$ were such a region, and fix some $\theta_{1}>\theta_{0}$ . Then $C^{*}$ is most powerful at level $\alpha$ for testing $\theta_{0}$ against $\theta_{1}$ , and so is the region $C_{+}=\{\bar{x}>c_{\alpha}\}$ of example 23.3. The boundary set $\{\bar{x}=c_{\alpha}\}$ has probability zero, and thus by the second statement of theorem 23.2 the regions $C^{*}$ and $C_{+}$ differ only on a set which has probability zero for every $\theta$ . Consequently the two tests have the same power function, $\beta_{C^{*}}(\theta)=\beta_{+}(\theta)$ for all $\theta$ . Now take any $\theta_{2}<\theta_{0}$ . From the formulas of section 23.4 we get

\beta_{C^{*}}(\theta_{2})=\beta_{+}(\theta_{2})<\Phi(-z_{1-\alpha})=\alpha<% \beta_{-}(\theta_{2}),

since $\Phi$ is strictly increasing and $\sqrt{n}\,(\theta_{2}-\theta_{0})/\sigma<0$ . The left-sided test $C_{-}=\{\bar{x}<\theta_{0}-z_{1-\alpha}\sigma/\sqrt{n}\}$ has size $\alpha$ and thus is an admissible competitor $C^{\prime}$ , and it has strictly larger power at $\theta_{2}$ than $C^{*}$ . This contradicts the assumed property of $C^{*}$ , which completes the proof. ∎

The argument is not particular to the normal distribution: whenever the one-sided most powerful regions disagree, no uniformly most powerful two-sided test exists, and we have to settle for a compromise such as the two-sided test of example 22.7. In lecture 25 we will see that this test arises from applying the likelihood-ratio idea to the composite alternative, which gives a general-purpose tool for such problems.

Summary.

•

A test is most powerful at level $\alpha$ for a simple null hypothesis against a simple alternative, if it has level $\alpha$ and no other test of level $\alpha$ has larger power.
•

The Neyman–Pearson lemma states that the test which rejects when $f(x;\theta_{1})>k\,f(x;\theta_{0})$ , where $k$ is chosen so that the size is $\alpha$ , is most powerful at level $\alpha$ . Any other most powerful test coincides with this test outside the boundary set, up to a set of probability zero. The proof involves integration of $(\mathbf{1}_{C}-\mathbf{1}_{C^{\prime}})(f(x;\theta_{1})-k\,f(x;\theta_{0}))\geq 0$ .
•

In practice, $R(x)>k$ can be rewritten as $T(x)>c$ for a simple statistic $T$ and $c$ can be found by considering the distribution of $T$ under $H_{0}$ ; for the normal mean this gives $\bar{X}>\theta_{0}+z_{1-\alpha}\sigma/\sqrt{n}$ .
•

If the critical region does not depend on the alternative $\theta_{1}$ , the same test will be most powerful against all $\theta_{1}>\theta_{0}$ : the test is then uniformly most powerful for the one-sided problem.
•

For the two-sided alternative, no uniformly most powerful test can exist, since the two one-sided tests disagree; instead, the likelihood-ratio tests from lecture 25 can be used.

Exercise 23.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with mean $\theta>0$ and let $S=\sum_{i=1}^{n}X_{i}$ , Poisson with mean $n\theta$ . We test $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ where $\theta_{1}>\theta_{0}$ .

1.

Show that the likelihood ratio $R(x)$ is a strictly increasing function of $s=\sum_{i}x_{i}$ and that the most powerful test rejects when $S>c$ . Why does $c$ not depend on $\theta_{1}$ ? What are the consequences for testing against the composite alternative $H_{1}\colon\theta>\theta_{0}$ ?
2.

Let $n=5$ , $\theta_{0}=1$ and $\alpha=0.05$ . Determine the smallest integer $c$ such that the test has size at most $0.05$ . Compute the actual size of the test for this value. You may use $\mathbb{P}(S\leq 8)=0.9319$ and $\mathbb{P}(S\leq 9)=0.9682$ for a Poisson random variable $S$ with mean $5$ .
3.

Determine the power of the test at $\theta_{1}=2$ , using $\mathbb{P}(S\leq 9)=0.4579$ for a Poisson random variable with mean $10$ .
4.

Why cannot a test of the form $\{S>c\}$ have size exactly $0.05$ ? At which level is the test from part (b) most powerful?

Exercise 23.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $p\in(0,1)$ , and let $S=\sum_{i=1}^{n}X_{i}$ . In exercise 22.3 we tested $H_{0}\colon p=1/2$ against $H_{1}\colon p>1/2$ by rejecting for large values of $S$ .

1.

For a fixed alternative $p_{1}>1/2$ , show that the likelihood ratio is given by

$R(x)=\Bigl{(}\frac{p_{1}}{1-p_{1}}\Bigr{)}^{s}\bigl{(}2(1-p_{1})\bigr{)}^{n},% \qquad s=\sum_{i=1}^{n}x_{i},$

and that it is strictly increasing in $s$ . Deduce that the most powerful test rejects when $S>c$ , and that the test of exercise 22.3 is uniformly most powerful against $H_{1}\colon p>1/2$ .
2.

Let $n=20$ . Using $\mathbb{P}_{1/2}(S\geq 14)=0.0577$ and $\mathbb{P}_{1/2}(S\geq 15)=0.0207$ , find the critical region of the largest size not exceeding $0.05$ , and compute its power at $p_{1}=0.7$ using $\mathbb{P}_{0.7}(S\leq 14)=0.5836$ . Compare the size and the power with those found for $n=10$ in exercise 22.3.
3.

For an alternative $p_{1}<1/2$ , find the most powerful critical region, and explain why the two one-sided problems lead to different tests.

Exercise 23.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the interval $(0,\theta)$ , with joint density $f(x;\theta)=\theta^{-n}\mathbf{1}_{\{m\leq\theta\}}$ for $x_{1},\dots,x_{n}\geq 0$ , where $m=\max_{i}x_{i}$ , and let $M=\max_{i}X_{i}$ . From example 8.9 we know that $\mathbb{P}_{\theta}(M\leq y)=(y/\theta)^{n}$ for $0\leq y\leq\theta$ . We want to test $H_{0}\colon\theta=\theta_{0}$ against $H_{1}\colon\theta=\theta_{1}$ , where $\theta_{1}>\theta_{0}$ , at significance level $\alpha\in(0,1)$ . In exercise 22.4 we have considered tests based on $M$ for this model, and in the following we will see that these tests are optimal.

1.

Let $k=(\theta_{0}/\theta_{1})^{n}$ . Determine the sets $\{x\mid f(x;\theta_{1})>k\,f(x;\theta_{0})\}$ and $\{x\mid f(x;\theta_{1})=k\,f(x;\theta_{0})\}$ in terms of $m$ .
2.

Show that the critical region $C=\{m>c\}$ for $c=\theta_{0}(1-\alpha)^{1/n}$ has size $\alpha$ and satisfies the condition from theorem 23.2 for this $k$ . Conclude that this is the most powerful test at level $\alpha$ . Compute the power of the test and explain why the test is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ .
3.

Show that the region $C^{\prime\prime}=\{m>\theta_{0}\}\cup\{m\leq\theta_{0}\alpha^{1/n}\}$ also has size $\alpha$ and the same power as $C$ . Explain, using the second statement of theorem 23.2, why this does not contradict the theorem, even though the model has densities.

Exercise 23.4.

Let $T$ be a sufficient statistic for the model $f(x;\theta)$ and let $\theta_{0}\neq\theta_{1}$ be two parameter values. Using the factorisation theorem (proposition 8.3), show that the likelihood ratio $R(x)=f(x;\theta_{1})/f(x;\theta_{0})$ depends on the data only through $T(x)$ , for all $x$ where the denominator is positive.