Lecture 31 Standard Tests Revisited

Every test in this lecture is based on the following principle: the test statistic is known to follow a certain distribution under the null hypothesis, and the test rejects if the statistic exceeds a quantile of this distribution, taken from a table. Using the language of lecture 26, the test statistic is a pivot $Q(X,\theta)$ for the hypothesised value $\theta=\theta_{0}$ and the test is the one which, when inverted by lemma 26.6, gives the confidence interval based on this pivot.

We use the notation $z_{p}$, $t_{k,p}$ and $\chi^{2}_{k,p}$ for the quantiles from lecture 26, so that $\mathbb{P}(T\leq t_{k,p})=p$ for $T\sim t_{k}$. A table shows these quantiles for a few values of $p$. For a one-sided test at level $\alpha$, we need the $(1-\alpha)$-quantile, and for a two-sided test at the same level we need the $(1-\alpha/2)$-quantile. A $p$-value in the sense of definition 22.8 cannot be found exactly in a table, but can be bracketed between two tabulated levels; we will see an example of this approach in example 31.2, below.

The most basic example is the $z$-test from lecture 22: if $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ are i.i.d. with known $\sigma^{2}$, then $Z=\sqrt{n}\,(\bar{X}-\mu_{0})/\sigma$ is standard normally distributed under $H_{0}\colon\mu=\mu_{0}$ and in examples 22.6 and 22.7 we can reject for $Z>z_{1-\alpha}$ or $|Z|>z_{1-\alpha/2}$ for the one-sided and two-sided alternative, respectively. The drawback of this test is that the variance of the data must be known. The rest of the lecture is concerned with how to replace $Z$ when $\sigma^{2}$ is unknown.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\mu,\sigma^{2})$, where both parameters are unknown, and let $\bar{X}$ and $S^{2}$ be the sample mean and sample variance, respectively. If we replace $\sigma$ in the definition of $Z$ by the estimate $S$, we get the test statistic

In example 26.4 we have seen that for $\mu=\mu_{0}$ this is the ratio of the standard normal variable $\sqrt{n}\,(\bar{X}-\mu_{0})/\sigma$ to the square root of $V/(n-1)$, where $V=(n-1)S^{2}/\sigma^{2}\sim\chi^{2}_{n-1}$, and that the two values are independent of each other, both by proposition A.4. Consequently, $T\sim t_{n-1}$ by definition A.5. This leads to the following tests.

These are the one-sided and two-sided one-sample $t$-tests. Since $t_{n-1,p}>z_{p}$ for every $p>1/2$, the $t$-test requires a larger standardised deviation than the $z$-test before it can reject the hypothesis, and this is the price we pay for estimating $\sigma$. As $n$ increases, this price decreases as the $t_{n-1}$ distribution gets closer to the standard normal distribution. As in lecture 22, the one-sided tests have level $\alpha$ for the composite null hypotheses $\mu\leq\mu_{0}$ and $\mu\geq\mu_{0}$, since the probability of rejection increases as $\mu$ moves into the alternative. Unlike in lecture 22, where the power function was explicit, we state this monotonicity without proof.

By exercise 25.1, the two-sided $t$-test is the generalised likelihood-ratio test for this problem, with the exact $t_{n-1}$ quantile in place of the $\chi^{2}_{1}$ approximation of Wilks’ theorem, theorem 25.3.

The second question we can ask a normal sample, about the spread of the sample, is whether the variance equals a given value $\sigma_{0}^{2}$. In proposition A.4 we have seen that the pivot $(n-1)S^{2}/\sigma^{2}\sim\chi^{2}_{n-1}$ does not depend on the mean, and the test will follow the structure of the test from the previous section.

Unlike the $t$ distribution, the chi-squared distribution is not symmetric and so we need different values from the table for the two tails of a two-sided test. This is the inverse of the confidence interval from example 26.5. Similarly, the one-sided test against $\sigma^{2}>\sigma_{0}^{2}$ has level $\alpha$ for the composite null hypothesis $\sigma^{2}\leq\sigma_{0}^{2}$, since the probability of the test to reject the null hypothesis increases with $\sigma^{2}$. This is exercise 31.2.

The numerical carrying-out of the test is also part of that exercise, and it follows the pattern of example 31.2 with the chi-squared table in place of the $t$-table. Note that tests about a variance have low power for small samples, because $S^{2}$ is a much more variable estimator than $\bar{X}$.

The tests so far have considered a single sample. Here we will consider three more standard tests, which compare two sets of measurements each, and are again pivot tests.

The first of these three tests is the paired $t$-test. Assume that we have observed data in pairs $(X_{i},Y_{i})$, $i=1,\dots,n$, e.g. before and after measurements for the same subject. Since the two measurements for one subject are not independent, we can reduce the data for each pair to the difference $D_{i}=X_{i}-Y_{i}$, model the differences as i.i.d. $N(\mu_{D},\sigma_{D}^{2})$ and then apply proposition 31.1 with $\mu_{0}=0$: the test statistic is $T=\sqrt{n}\,\bar{D}/S_{D}$, constructed from the sample mean and variance of the differences, and is $t_{n-1}$ distributed under $H_{0}\colon\mu_{D}=0$. The model is only for the differences, as the example in exercise 31.3 shows.

The second test is the two-sample $t$-test. Here we assume that $X_{1},\dots,X_{m}$ are i.i.d. $N(\mu_{X},\sigma^{2})$ and that $Y_{1},\dots,Y_{n}$ are i.i.d. $N(\mu_{Y},\sigma^{2})$, where the two samples are independent of each other, and where we assume that the unknown variances of the two populations are the same. We consider $H_{0}\colon\mu_{X}=\mu_{Y}$. The estimator for the common variance is now

and the test statistic is

The distribution of this test statistic is similar to the distribution of the test statistic for the one-sample case: under $H_{0}$ the numerator is $N\bigl{(}0,\sigma^{2}(1/m+1/n)\bigr{)}$-distributed, and $(m+n-2)S_{p}^{2}/\sigma^{2}$ is the sum of two independent $\chi^{2}_{m-1}$ and $\chi^{2}_{n-1}$ variables and thus is $\chi^{2}_{m+n-2}$ by definition A.2, and numerator and denominator are independent of each other by proposition A.4, applied to each of the two samples, and the independence of the samples. Thus, the test can be used to test $H_{0}$ against $\mu_{X}\neq\mu_{Y}$, rejecting if $|T|>t_{m+n-2,1-\alpha/2}$, and the one-sided variants of the test use $t_{m+n-2,1-\alpha}$.

The third test is the $F$-test for two variances. In the same two-sample setting, but now with variances $\sigma_{X}^{2}$ and $\sigma_{Y}^{2}$ which are allowed to be different, the ratio $(S_{X}^{2}/\sigma_{X}^{2})\big{/}(S_{Y}^{2}/\sigma_{Y}^{2})$ is the ratio of two independent chi-squared variables, divided by the degrees of freedom. Thus, it is $F_{m-1,n-1}$ distributed by definition A.5. Under $H_{0}\colon\sigma_{X}^{2}=\sigma_{Y}^{2}$ the unknown variances cancel and the statistic $F=S_{X}^{2}/S_{Y}^{2}$ is $F_{m-1,n-1}$ distributed. The two-sided test rejects if $F$ does not fall into the interval $[F_{m-1,n-1,\alpha/2},\,F_{m-1,n-1,1-\alpha/2}]$, where $F_{k,l,p}$ is the $p$-quantile of $F_{k,l}$. Normally, tables only give the upper quantiles and the lower quantiles can be found by using $F_{k,l,p}=1/F_{l,k,1-p}$. This test is also used to check the assumption of equal variances in (31.3).

Table 31.1 summarises the six tests. The choice between them is determined by the following three questions: Is the parameter in question a mean or a variance? Is there one sample or two? If two samples, are the observations paired or independent? All six tests assume that the data are normally distributed.