Lecture 24 Power and Size by Simulation

The test we consider today is the one-sided $z$-test from example 22.6, in the numerical setting of the example: we have $n=25$ observations from $N(\theta,\sigma^{2})$ with known $\sigma=2$ and we want to test $H_{0}\colon\theta=10$ against $H_{1}\colon\theta>10$ at level $\alpha=0.05$, i.e. we reject if and only if $\bar{X}>10.658$. By definition 22.4, the power function $\beta(\theta)=\mathbb{P}_{\theta}(\bar{X}>10.658)$ is a probability, and by proposition A.14 we can approximate this probability by the proportion of samples where the test rejects. As in lecture 12, we write a function which, for one run of the test, returns TRUE if the test rejects and FALSE otherwise. The command mean can then be used to get the proportion of rejections. Before you run the code, predict the value it prints. Will it be exactly $0.05$? How far away from $0.05$ would it have to be before you would suspect a mistake in the code?

The rejection rate under $H_{0}$ is $0.0523$. This is the estimated size of the test in the sense of definition 22.5. The value is not exactly $0.05$, and it should not be: as in lecture 12, the estimate has standard deviation $\sqrt{0.05\cdot 0.95/10000}=0.0022$. Thus, a value between $0.046$ and $0.054$ is what we expect for a correct test, while a value of $0.07$ would point to an error in the code. Calling the same function with a value of $\theta$ above $\theta_{0}$ gives the power at this point; for $\theta=10.5$ and $\theta=11$ the exact values are $0.346$ and $0.804$, from example 22.6.

A single number is a poor summary of a test, so we now consider the power function over a grid of alternatives. Figure 24.1 shows the rejection rates on the grid of values from $9$ to $12$ in steps of $0.1$ for three level $0.05$ tests: the one-sided $z$-test from above, the two-sided $z$-test from example 22.7 and the one-sided $z$-test applied to the first $12$ observations only. The last test is wasteful, but it is a legal level $0.05$ test. The script R/S24-power.R can be used to reproduce the figure. Before you look at the figure, predict how the two-sided test compares with the one-sided test. Is its rejection rate above or below that of the one-sided test for $\theta<10$? Is it above or below for $\theta>10$?

The line in the figure is the exact power function (22.1) of the one-sided test, and the filled circles lie on it. All three curves pass through $0.05$ at $\theta_{0}=10$, as required for level $0.05$ tests, and the tests differ only in power. The two-sided test has lower power than the one-sided test for $\theta>10$ and much higher power for $\theta<10$. Thus, neither curve lies above the other everywhere, and this illustrates proposition 23.5. From example 23.3 and proposition 23.4 we know that no level $0.05$ test based on these $25$ observations can reject at $\theta=11$ more than $80\%$ of the time. The crosses show the cost of having discarded thirteen of the observations: the power at $\theta=11$ decreases from $0.80$ to $0.53$.

The recipe of the first section gives the size of every procedure which ends in a decision, and we now apply it to a procedure whose size is not what it seems to be. Assume that the observations arrive one at a time, up to $100$ in total, and that we perform the two-sided $z$-test from example 22.7 after every ten observations. We stop and reject $H_{0}$ as soon as one of the $p$-values is below $0.05$: this is the case of the researcher who “collects a few more data points and looks again”. The following function simulates one run of this procedure under $H_{0}$, using the $p$-value $2\bigl{(}1-\Phi(|z|)\bigr{)}=2\,\Phi(-|z|)$ from lecture 22, and returns TRUE if the procedure rejects. Before you run the code, predict the rejection rate. Each of the ten tests has level $0.05$. Will the rejection rate be $0.05$? Could it be as large as $0.5$?

The procedure rejects a true null hypothesis in $20\%$ of all runs, four times as often as the level of the single tests suggests. By lemma 22.9, each of the ten $p$-values is uniformly distributed under $H_{0}$, and thus each single look rejects with probability $0.05$. The procedure, however, rejects if at least one of the ten looks rejects. Since the probability of a union is at most the sum of the probabilities, the rejection rate cannot exceed $10\cdot 0.05=0.5$, and ten independent tests would give $1-0.95^{10}=0.40$. Here the looks share observations: a sample which is far from rejection at one look is likely to be far from rejection at the next look as well, and for this reason the rate $0.20$ is smaller than both values. The level is a property of the whole procedure, including the rule which determines when to stop, and for this reason the sample size must be fixed before the data are seen.

Set the seed of the random number generator to your student ID and consider the setting from section 24.2, but with sample size $n=10$ instead of $n=25$. Estimate the size of the one-sided $z$-test for $10000$ samples and state whether the estimated size is consistent with $0.05$, within simulation error. Next, estimate the power of the test on the grid seq(9, 13, by = 0.1) and create a plot of the estimated power curve, together with the exact power function from lecture 22; the script R/S24-power.R shows how the command sapply can be used to apply the rejection-rate recipe to every point of a grid. From the plot, determine the smallest $\theta$ at which the power is greater than $0.9$. Finally, modify the function peek from section 24.3 to test after every five observations, estimate the rejection rate under $H_{0}$ from $10000$ runs and compare it with $0.199$.

Before the next session, please type the commands of this session into R, and check that what you get is what the text says you should get.