Lecture 20 Hypothesis Testing

Throughout, $X_{1},\dots,X_{n}$ follow a parametric model in the sense of definition 1.1. Questions like the one about fairness of a coin or about the effect of a treatment ask whether the true parameter value $\theta_{0}$ belongs to a subset of the parameter space $\Theta$.

As an abbreviation, from this lecture on we write $\theta_{0}$ for the value chosen to test a null hypothesis. In lectures 16 and 17, the same symbol was used to denote the unknown, true value of the parameter. The context should make it clear which meaning is intended.

In most problems we have $\Theta_{1}=\Theta\setminus\Theta_{0}$, i.e. we can assume that exactly one of the two hypotheses is true. For example, for the mean $\theta$ of a normal sample, the hypothesis $H_{0}\colon\theta=\theta_{0}$ is simple, whereas the one-sided alternative $H_{1}\colon\theta>\theta_{0}$, the two-sided alternative $H_{1}\colon\theta\neq\theta_{0}$ and the one-sided null hypothesis $H_{0}\colon\theta\leq\theta_{0}$ are all composite.

The two hypotheses have different roles: the null hypothesis is the default position, and the alternative is the claim which requires evidence before we can act on it. Section 20.3 will make this distinction precise.

A test decides between the two hypotheses under consideration, and can be described by the set of data values where it rejects $H_{0}$.

In practice, the critical region is almost always of the form $C=\{x\mid T(x)>c\}$ or $C=\{x\mid T(x)\geq c\}$ for a statistic $T$ given by definition 1.3, called the test statistic, and a constant $c$ called the critical value. The two forms of the critical region differ if $T(x)=c$ has positive probability (exercise 20.2 shows an example of this case, for discrete distributions).

A test applied to random data will sometimes wrongly decide. This can happen in two different ways.

Table 20.1 summarises the possible outcomes of statistical tests, in relation to the truth of the hypothesis being tested.

To understand the interpretation of this table, we can use the following example: in a criminal trial, the null hypothesis is the presumption of innocence of the defendant, and the alternative is guilt. If an innocent person is convicted, this is a type I error. If a guilty person is acquitted, this is a type II error. The standard of proof “beyond reasonable doubt” is such that many acquittals of the guilty are acceptable in order to avoid too many wrongful convictions. The reason for this is that the two types of error have different consequences: the consequences of type I errors are more serious. Furthermore, since an acquittal does not prove innocence, we write “do not reject $H_{0}$” instead of “accept $H_{0}$” in the table.

The probabilities of the two errors depend on the true parameter value, and both can be found using a single function of $\theta$.

For $\theta\in\Theta_{0}$ the value $\beta(\theta)$ is the probability of a type I error, and for $\theta\in\Theta_{1}$ the value $1-\beta(\theta)$ is the probability of a type II error. The two probabilities cannot both be reduced by making the critical region larger or smaller. If the size of the critical region is increased, the value $\beta(\theta)$ increases for all $\theta$, i.e. type I errors become more likely and type II errors become less likely. If the size of the critical region is decreased, the opposite happens. For this reason, we have to choose one of the two types of error to control and it transpires that we can decide which type of error to control by considering the asymmetry between the two types of hypotheses: type I errors are errors where the default position is abandoned without good reason, and thus we should restrict the probability of type I errors first and only then should we consider the power of the test.

For a simple null hypothesis $H_{0}\colon\theta=\theta_{0}$, the size of the test equals $\beta(\theta_{0})$. The number $\alpha$ is called the significance level and the values $0.05$ and $0.01$ are only conventions. Among all level $\alpha$ tests, we prefer the test with the largest power at the alternatives. In lecture 22 we will see how to find the test with optimal power, when both the null and the alternative hypothesis are simple.

We now construct a test from first principles for $X_{1},\dots,X_{n}$ i.i.d. $N(\theta,\sigma^{2})$, where the variance $\sigma^{2}$ is known. Here we write $\Phi$ for the standard normal distribution function and $z_{q}$ for the $q$-quantile of the standard normal distribution, i.e. we have $\Phi(z_{q})=q$. The most commonly used values of this function are $z_{0.95}=1.645$ and $z_{0.975}=1.96$.

Since the power function (20.2) is increasing, the same test also has size $\sup_{\theta\leq\theta_{0}}\beta(\theta)=\beta(\theta_{0})=\alpha$ for the composite null hypothesis $H_{0}\colon\theta\leq\theta_{0}$, attained at the boundary point; this is why definition 20.5 takes a supremum.

At $\theta=11$ the one-sided test has power $0.804$ against $0.705$, since it rejects all values with probability $\alpha$ from the upper tail. Thus, the one-sided test is superior to the two-sided test for all alternatives above $\theta_{0}$, but is useless for alternatives below. Before we consider the data, we need to decide which departures from $H_{0}$ are of interest. In lecture 22 we will see that there is no single test which is best for both directions, and in lecture 24 we will consider how size and power behave in simulation.

A test at a fixed level only states whether $H_{0}$ is rejected or not. The $p$-value, introduced in this section, allows us to quantify the strength of the evidence in a more detailed way.

The tests from section 20.4 are nested in this way, since $z_{1-\alpha}$ decreases as $\alpha$ increases. For a test which rejects for large values of a statistic $T$ and which has a simple null hypothesis, the smallest level at which the data can lead to rejection is

This is the probability under $H_{0}$ of the test statistic taking a value at least as extreme as the observed value. For the one-sided $z$-test this probability is $p(x)=1-\Phi(z)$, for the observed value $z=\sqrt{n}\,(\bar{x}-\theta_{0})/\sigma$. For the two-sided test we get $p(x)=2\bigl{(}1-\Phi(|z|)\bigr{)}$. For the situation of example 20.6, the observed mean $\bar{x}=10.9$ leads to $z=2.25$ and thus to $p=1-\Phi(2.25)=0.0122$ for the one-sided test and $p=0.0244$ for the two-sided test. We can see that we can reject $H_{0}$ at level $\alpha$ if and only if $p(x)\leq\alpha$, and thus the $p$-value is a compact summary of a test. Under $H_{0}$, the $p$-value of a continuous test statistic is uniformly distributed on $(0,1)$ (see exercise 20.4). This result is used in lecture 30.

Both statements assume that the test statistic is continuous. For discrete test statistics, e.g. the number of successes in a Bernoulli sample, the tail probability (20.5) can only take the countably many values $\mathbb{P}_{\theta_{0}}(T(X)\geq t)$ and the $p$-value is discontinuous. Most levels $\alpha$ are not attained by any critical region. We can still perform the test for $p(x)\leq\alpha$ and the resulting test is conservative. The size of the test is the largest attainable value which does not exceed the level $\alpha$. The test will reject a true $H_{0}$ less often than the nominal level suggests and the test will lose some power. The $p$-value is no longer uniformly distributed either, but still satisfies $\mathbb{P}_{\theta_{0}}(p(X)\leq\alpha)\leq\alpha$. This is what allows the level to still be valid.

Since the $p$-value is a probability, it is tempting to interpret the results in wrong ways; the following statements about a $p$-value of $0.0122$ are all wrong.

• •
  ​

  “The probability that $H_{0}$ is true is $0.0122$.” Since $\theta$ is a fixed, unknown number, $H_{0}$ is either true or false; the probability in (20.5) refers to the data. To get a probability for a hypothesis, we have to introduce a prior distribution for $\theta$, i.e. we have to follow the Bayesian approach from lecture 28.

• •
  ​

  “The probability that the observed data could have been produced by chance is $0.0122$.” The $p$-value corresponds to extreme values of $T$, not to the probability of the data.

• •
  ​

  “If we had observed a $p$-value of $0.4$, we would have known that $H_{0}$ is true.” Data which is compatible with $H_{0}$ may be compatible with many different alternatives: the test from example 20.6 has power only $0.346$ at $\theta=10.5$.

• •
  ​

  “The effect is important, because $p$ is small.” If $n$ is large enough, any difference between $\theta$ and $\theta_{0}$ will result in a tiny $p$-value. Reports should include both the estimate and the standard error.

None of these mistakes invalidates the $p$-value as a tool, but they do show that the $p$-value answers a very specific question, and that the same value should not be used to answer different questions.