Lecture 22 Hypothesis Testing

Throughout, $X_{1},\dots,X_{n}$ follow a parametric model in the sense of definition 1.1. Questions such as whether a coin is fair or whether a treatment has an effect ask whether the true value of $\theta$ belongs to a subset of the parameter space $\Theta$.

From now on we write $\theta_{0}$ for the value of $\theta$ named by a null hypothesis. In lectures 16 and 17 the same symbol denoted the true value of the parameter; the context makes clear which meaning is intended.

Often we have $\Theta_{1}=\Theta\setminus\Theta_{0}$, and then exactly one of the two hypotheses is true. For example, for the mean $\theta$ of a normal sample, the hypothesis $H_{0}\colon\theta=\theta_{0}$ is simple, whereas the one-sided alternative $H_{1}\colon\theta>\theta_{0}$, the two-sided alternative $H_{1}\colon\theta\neq\theta_{0}$ and the one-sided null hypothesis $H_{0}\colon\theta\leq\theta_{0}$ are all composite.

The two hypotheses have different roles: the null hypothesis is the default position, and the alternative is the claim which requires evidence before we can act on it. Section 22.3 makes this distinction precise.

A test decides between the two hypotheses, and can be described by the set of data values where it rejects $H_{0}$.

In practice, the critical region usually has the form $C=\{x\mid T(x)>c\}$ or $C=\{x\mid T(x)\geq c\}$ for a statistic $T$ given by definition 1.3, called the test statistic, and a constant $c$ called the critical value. The two forms give tests with different error probabilities only if $T(X)=c$ has positive probability, as can happen for discrete distributions (exercise 22.3).

A test can decide wrongly in two different ways.

Table 22.1 summarises the four possible outcomes.

As an example, in a criminal trial the null hypothesis is the innocence of the defendant, and the alternative is guilt. Convicting an innocent person is a type I error, and acquitting a guilty person is a type II error. Under the standard of proof “beyond reasonable doubt”, many acquittals of the guilty are accepted to avoid wrongful convictions, because a type I error is more serious. Since an acquittal does not prove innocence, the table says “do not reject $H_{0}$” instead of “accept $H_{0}$”.

Both error probabilities depend on the true parameter value, and can be read off a single function of $\theta$.

For $\theta\in\Theta_{0}$ the value $\beta(\theta)$ is the probability of a type I error, and for $\theta\in\Theta_{1}$ the value $1-\beta(\theta)$ is the probability of a type II error (some books write $\beta$ for the latter). Enlarging the critical region can only increase $\beta(\theta)$, which reduces type II errors at the cost of type I errors, and thus we have to choose which probability to control. Since a type I error abandons the default position without good reason, we restrict its probability first and only then consider the power of the test.

For a simple null hypothesis $H_{0}\colon\theta=\theta_{0}$, the size of the test equals $\beta(\theta_{0})$. The number $\alpha$ is called the significance level and the values $0.05$ and $0.01$ are only conventions. Among all level $\alpha$ tests, we prefer the test with the largest power at the alternatives. Lecture 23 shows how to find the most powerful test when both hypotheses are simple.

We now construct tests for $X_{1},\dots,X_{n}$ i.i.d. $N(\theta,\sigma^{2})$ with known variance $\sigma^{2}$. We write $\Phi$ for the standard normal distribution function and $z_{q}$ for the $q$-quantile of the standard normal distribution, i.e. we have $\Phi(z_{q})=q$. The most commonly used quantiles are $z_{0.95}=1.645$ and $z_{0.975}=1.96$.

The values $\theta<\theta_{0}$ belong to neither hypothesis of this example. Since the power function (22.1) is increasing, the test also has size $\sup_{\theta\leq\theta_{0}}\beta(\theta)=\beta(\theta_{0})=\alpha$ for the composite null hypothesis $H_{0}\colon\theta\leq\theta_{0}$, which covers these values. This is why definition 22.5 takes a supremum.

At $\theta=11$ the one-sided test has power $0.804$ against $0.705$, since it places all of its type I error probability in the upper tail, but it is useless for alternatives below $\theta_{0}$. Lecture 23 shows that the one-sided test is most powerful against every alternative above $\theta_{0}$, and that no single test is best for both directions. Thus, before we consider the data, we must decide which departures from $H_{0}$ are of interest.

A test at a fixed level only states whether $H_{0}$ is rejected. The $p$-value also records how extreme the observed value of the test statistic is under $H_{0}$.

For a simple null hypothesis the supremum reduces to $\mathbb{P}_{\theta_{0}}$. Since $p(x)$ is the size of the test with critical region $\{T\geq T(x)\}$, it is the smallest level at which a test of the form $\{T\geq c\}$ rejects $x$, and thus $H_{0}$ is rejected at level $\alpha$ if and only if $p(x)\leq\alpha$. In this case we call the data significant at level $\alpha$. For the one-sided $z$-test we find $p(x)=1-\Phi(z)$, where $z=\sqrt{n}\,(\bar{x}-\theta_{0})/\sigma$ is the observed value of $Z$, and for the two-sided test we find $p(x)=2\bigl{(}1-\Phi(|z|)\bigr{)}$. In the situation of example 22.6, the observed mean $\bar{x}=10.9$ gives $z=2.25$ and thus $p=1-\Phi(2.25)=0.0122$ for the one-sided test and $p=0.0244$ for the two-sided test.

For a simple null hypothesis and a continuous test statistic, the distribution of the $p$-value under $H_{0}$ does not depend on the model.

For a discrete test statistic, e.g. the number of successes in a Bernoulli sample, the $p$-value takes only countably many values and is not uniformly distributed. For an integer-valued $T$, the test which rejects if $p(x)\leq\alpha$ is the test $\{T\geq c\}$ with the smallest integer $c$ of level $\alpha$, and thus it still has size at most $\alpha$. A test whose size is smaller than its level in this way is called conservative.

Since the $p$-value is a probability, it is tempting to misread it. It is not the probability that $H_{0}$ is true: the parameter $\theta$ is a fixed, unknown number, and the probability in (22.2) refers to the data. Nor does a large $p$-value establish $H_{0}$, since data compatible with $H_{0}$ may be compatible with many alternatives as well: the test from example 22.6 has power only $0.346$ at $\theta=10.5$.