Lecture 29 Standard Tests Revisited

Every test in this lecture is based on the following principle: the test statistic is known to follow a certain distribution under the null hypothesis, and the test rejects if the statistic exceeds a quantile of this distribution, taken from a table. Using the language of lecture 25, the test statistic is a pivot $Q(X,\theta)$ for the hypothesised value $\theta=\theta_{0}$ and the test is the one which, when inverted by theorem 25.6, gives the confidence interval based on this pivot.

We use the notation $z_{p}$, $t_{k,p}$ and $\chi^{2}_{k,p}$ for the quantiles from lecture 25, so that $\mathbb{P}(T\leq t_{k,p})=p$ for $T\sim t_{k}$. A table shows these quantiles for a few values of $p$. For a one-sided test at level $\alpha$, we need the $(1-\alpha)$-quantile, and for a two-sided test at the same level we need the $(1-\alpha/2)$-quantile. A $p$-value in the sense of definition 20.8 cannot be found exactly in a table, but can be bracketed between two tabulated levels; we will see an example of this approach in example 29.2, below.

The most basic example is the $z$-test from lecture 20: if $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ are i.i.d. with known $\sigma^{2}$, then $Z=\sqrt{n}\,(\bar{X}-\mu_{0})/\sigma$ is standard normally distributed under $H_{0}\colon\mu=\mu_{0}$ and in examples 20.6 and 20.7 we can reject for $Z>z_{1-\alpha}$ or $|Z|>z_{1-\alpha/2}$ for the one-sided and two-sided alternative, respectively. The simplicity of this test is countered by the fact that the variance of the data is known. The rest of the lecture is concerned with how to replace $Z$ when $\sigma^{2}$ is unknown.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\mu,\sigma^{2})$, where both parameters are unknown, and let $\bar{X}$ and $S^{2}$ be the sample mean and sample variance, respectively. If we replace $\sigma$ in the definition of $Z$ by the estimate $S$, we get the test statistic

In example 25.4 we have seen that for $\mu=\mu_{0}$ this is the ratio of the standard normal variable $\sqrt{n}\,(\bar{X}-\mu_{0})/\sigma$ to the square root of $V/(n-1)$, where $V=(n-1)S^{2}/\sigma^{2}\sim\chi^{2}_{n-1}$ by proposition A.3, and that the two values are independent of each other by corollary 10.10. Consequently, $T\sim t_{n-1}$ by definition A.4. This leads to the following tests.

These are the one-sided and two-sided one-sample $t$-tests. Since $t_{n-1,p}>z_{p}$ for every $p>1/2$, the $t$-test requires a larger standardised deviation than the $z$-test before it can reject the hypothesis, and this is the price we pay for estimating $\sigma$. As $n$ increases, this price decreases as the $t_{n-1}$ distribution gets closer to the standard normal distribution. As in lecture 20, the one-sided tests have level $\alpha$ for the composite null hypotheses $\mu\leq\mu_{0}$ and $\mu\geq\mu_{0}$, since the probability of rejection increases as $\mu$ moves into the alternative.

The $t$-test is the test obtained by applying the likelihood-ratio principle from lecture 23. In exercise 23.1 the generalised likelihood-ratio statistic for $H_{0}\colon\mu=\mu_{0}$, where $\sigma^{2}$ is unknown for both hypotheses, is found to be $-2\log\Lambda=n\log\bigl{(}1+T^{2}/(n-1)\bigr{)}$, which is a strictly increasing function of $T^{2}$. Thus, rejecting large values of $-2\log\Lambda$ corresponds to the two-sided $t$-test. Following the general method of lecture 23, we would use the critical value from the $\chi^{2}_{1}$ approximation of Wilks’ theorem, theorem 23.3; however, in this case the null distribution of $T$ is known exactly and the $t_{n-1}$ quantile is the exact critical value for every $n$. As $n$ gets large, the two approaches will be equal, since $n\log(1+t^{2}/(n-1))\to t^{2}$ and $t_{n-1,1-\alpha/2}\to z_{1-\alpha/2}$ and then $z_{1-\alpha/2}^{2}=\chi^{2}_{1,1-\alpha}$.

The second question we can ask a normal sample, about the spread of the sample, is whether the variance equals a given value $\sigma_{0}^{2}$. In proposition A.3 we have seen that the pivot $(n-1)S^{2}/\sigma^{2}\sim\chi^{2}_{n-1}$ does not depend on the mean, and the test will follow the structure of the test from the previous section.

Unlike the $t$ distribution, the chi-squared distribution is not symmetric and so we need different values from the table for the two tails of a two-sided test. This is the inverse of the confidence interval from example 25.5. Similarly, the one-sided test against $\sigma^{2}>\sigma_{0}^{2}$ has level $\alpha$ for the composite null hypothesis $\sigma^{2}\leq\sigma_{0}^{2}$, since the probability of the test to reject the null hypothesis increases with $\sigma^{2}$. This is exercise 29.2.

The numerical carrying-out of the test is also part of that exercise, and it follows the pattern of example 29.2 with the chi-squared table in place of the $t$-table. Note that tests about a variance have low power for small samples, because $S^{2}$ is a much more variable estimator than $\bar{X}$.

The tests so far have considered a single sample. Here we will consider three more standard tests, which compare two sets of measurements each, and are again pivot tests.

The first of these three tests is the paired $t$-test. Assume that we have observed data in pairs $(X_{i},Y_{i})$, $i=1,\dots,n$, e.g. before and after measurements for the same subject. Since the two measurements for one subject are not independent, we can reduce the data for each pair to the difference $D_{i}=X_{i}-Y_{i}$, model the differences as i.i.d. $N(\mu_{D},\sigma_{D}^{2})$ and then apply proposition 29.1 with $\mu_{0}=0$: the test statistic is $T=\sqrt{n}\,\bar{D}/S_{D}$, constructed from the sample mean and variance of the differences, and is $t_{n-1}$ distributed under $H_{0}\colon\mu_{D}=0$. The model is only for the differences, as the example in exercise 29.3 shows.

The second test is the two-sample $t$-test. Here we assume that $X_{1},\dots,X_{m}$ are i.i.d. $N(\mu_{X},\sigma^{2})$ and that $Y_{1},\dots,Y_{n}$ are i.i.d. $N(\mu_{Y},\sigma^{2})$, where the two samples are independent of each other, and where we assume that the unknown variances of the two populations are the same. We consider $H_{0}\colon\mu_{X}=\mu_{Y}$. The estimator for the common variance is now

and the test statistic is

The distribution of this test statistic is similar to the distribution of the test statistic for the one-sample case: under $H_{0}$ the numerator is $N\bigl{(}0,\sigma^{2}(1/m+1/n)\bigr{)}$, $(m+n-2)S_{p}^{2}/\sigma^{2}$ is the sum of two independent $\chi^{2}_{m-1}$ and $\chi^{2}_{n-1}$ variables and thus is $\chi^{2}_{m+n-2}$ by definition A.2, and numerator and denominator are independent of each other by corollary 10.10, applied to each of the two samples, and the independence of the samples. Thus, the test can be used to test $H_{0}$ against $\mu_{X}\neq\mu_{Y}$, rejecting if $|T|>t_{m+n-2,1-\alpha/2}$, and the one-sided variants of the test use $t_{m+n-2,1-\alpha}$.

The third test is the $F$-test for two variances. In the same two-sample setting, but now with variances $\sigma_{X}^{2}$ and $\sigma_{Y}^{2}$ which are allowed to be different, the ratio $(S_{X}^{2}/\sigma_{X}^{2})\big{/}(S_{Y}^{2}/\sigma_{Y}^{2})$ is the ratio of two independent chi-squared variables, divided by the degrees of freedom. Thus, it is $F_{m-1,n-1}$ distributed by definition A.4. Under $H_{0}\colon\sigma_{X}^{2}=\sigma_{Y}^{2}$ the unknown variances cancel and the statistic $F=S_{X}^{2}/S_{Y}^{2}$ is $F_{m-1,n-1}$ distributed. The two-sided test rejects if $F$ does not fall into the interval $[F_{m-1,n-1,\alpha/2},\,F_{m-1,n-1,1-\alpha/2}]$, where $F_{k,l,p}$ is the $p$-quantile of $F_{k,l}$. Normally, tables only give the upper quantiles and the lower quantiles can be found by using $F_{k,l,p}=1/F_{l,k,1-p}$. This test is also used to check the assumption of equal variances in (29.3).

Table 29.1 summarises the six tests. The choice between them is determined by the following three questions: Is the parameter in question a mean or a variance? Is there one sample or two? If two samples, are the observations paired or independent? All six tests assume that the data are normally distributed. In lecture 30 we will use simulation to study what happens when this assumption is violated.