Lecture 8 Sufficiency

A statistic $T=T(X_{1},\dots,X_{n})$ in the sense of definition 1.3 reduces the data to a smaller summary, and usually some information is lost in the process. The following definition expresses that no information about $\theta$ is lost.

If $T$ is sufficient, anyone who knows the value $t$ of $T$ can generate a new data set with the same distribution as the original data set, by sampling from the conditional distribution of $X$ given $T=t$. Since this distribution does not depend on $\theta$, all information about $\theta$ that we could obtain by studying the data set can now be obtained by studying $t$ alone.

For the coin tosses in the introduction, exercise 8.1 shows that the definition is correct: if the number of heads $T=\sum_{i=1}^{n}X_{i}$ is given as $t$, each of the $\binom{n}{t}$ possible arrangements of heads and tails has conditional probability $1/\binom{n}{t}$, independent of the value of $\theta$. Thus, the positions of the heads do not contain any information about the coin and $T$ is sufficient.

Two observations help to place the definition. First, the whole sample $T=(X_{1},\dots,X_{n})$ is always sufficient, since given the whole sample there is no variability left; sufficiency alone therefore says nothing about how much the data have been reduced, and we return to this in section 8.4. Secondly, a one-to-one function $\varphi(T)$ of a sufficient statistic carries the same information and is sufficient as well; for the coin tosses the sample mean $\bar{X}=T/n$ is thus sufficient too.

Directly verifying definition 8.1 requires knowledge of the distribution of $T$ and involves a conditional probability which, for most models, is tedious to compute. The following theorem replaces the computation with an inspection of the joint density.

The function $g$ may depend on the data only through $T(x)$, and the function $h$ may not depend on $\theta$ at all: the parameter and the data interact only through $T$. Nothing in the statement requires $T$ or $\theta$ to be one-dimensional. We give a complete proof for discrete data and then explain what changes in the continuous case.

For continuous data the event $\{T=t\}$ has probability zero and the conditional distribution of $X$ given $T=t$ cannot be described as a quotient of probabilities anymore. If the statistic can be extended to a change of variables, i.e. if we can find coordinates $U$ such that $x\mapsto\bigl{(}T(x),U(x)\bigr{)}$ is a bijection, then the argument presented above can be applied with densities instead of probabilities: The joint density of $(T,U)$ will factor in a similar way, and after integrating out $u$ and dividing we will obtain a conditional density of $U$ given $T=t$ which does not depend on $\theta$. In general, no such completion of the proof is possible, and the proof will have to replace the family of measures by one dominating probability measure, for example a weighted sum of countably many of the $\mathbb{P}_{\theta}$, such that the density of $\mathbb{P}_{\theta}$ is a function of $T$. We will not go into detail here and will use theorem 8.2 for continuous models in analogy to the proof for discrete models, with the understanding that the factorisation, similar to the density, only needs to hold up to sets of probability zero.

The factorisation has a consequence for maximum likelihood. For fixed data the likelihood is $L(\theta)=g\bigl{(}T(x);\theta\bigr{)}h(x)$, and the factor $h(x)$ does not affect where the maximum over $\theta$ is attained: the maximum likelihood estimator, whenever it is unique, is a function of any sufficient statistic. The examples of lecture 5 confirm this: the Bernoulli MLE is $T/n$, and the uniform MLE is the sample maximum, which exercise 8.2 shows to be sufficient.

For the exponential families from lecture 7, no work is needed to find a sufficient statistic, since the densities are already given in factorised form.

The same argument shows that for the $k$-parameter exponential families of lecture 7 the vector $\bigl{(}\sum_{i}T_{1}(X_{i}),\dots,\sum_{i}T_{k}(X_{i})\bigr{)}$ of summed natural statistics is sufficient for the parameter vector. It is instructive to see the factorisation directly in the standard models.

For the normal distribution $N(\mu,\sigma^{2})$, where both parameters are unknown, we can find the sufficient statistic by expanding the squares in the exponent of the density. It transpires that the joint density depends on the data only via the pair $T=\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$. Thus, $T$ is a sufficient statistic for $(\mu,\sigma^{2})$. Since $\sum_{i}(x_{i}-\bar{x})^{2}=\sum_{i}x_{i}^{2}-n\bar{x}^{2}$, the pair $(\bar{X},S^{2})$ maps to $T$ one-to-one and thus is also a sufficient statistic. This is the pair of statistics on which the maximum likelihood estimator from example 5.4 is based. If $\sigma^{2}$ is known, the factor $\exp\bigl{(}-\sum_{i}x_{i}^{2}/(2\sigma^{2})\bigr{)}$ can be moved into $h(x)$ and then $\bar{X}$ alone suffices as a sufficient statistic for $\mu$. The factorisation is worked out in exercise 8.4.

As we have seen in section 8.1, the whole sample is always sufficient. For an i.i.d. sample, the vector of ordered observations is also sufficient, since the joint density does not change when the $x_{i}$ are interchanged. The aim of this section is to characterise the statistic which allows us to reduce the data as much as possible, while still retaining all the information.

A minimal sufficient statistic can thus be computed from every other sufficient statistic. The definition is awkward to check directly, because it refers to all sufficient statistics at once; the following criterion instead compares the likelihood functions of two data sets.

The proof of this theorem is only sketched here. The statement of sufficiency follows from theorem 8.2: For every value $t$ of $T$ we can choose a fixed data vector $x_{t}$ such that $T(x_{t})=t$. Then the ratio $f(x;\theta)/f(x_{T(x)};\theta)$, by assumption, does not depend on $\theta$ and can be used as $h(x)$ while $g(t;\theta)=f(x_{t};\theta)$. To see minimality, let $S$ be any sufficient statistic. Then we can write $f$ as $f(x;\theta)=\tilde{g}(S(x);\theta)\,\tilde{h}(x)$. If $S(x)=S(y)$, then $f(x;\theta)/f(y;\theta)=\tilde{h}(x)/\tilde{h}(y)$ does not depend on $\theta$ and thus we have $T(x)=T(y)$ by assumption: the value of $T$ is already determined by the value of $S$ and $T$ is a function of $S$.

The criterion states that $T(x)$ contains the likelihood function of the data up to a constant factor. We apply the criterion once, to the Poisson sum. The two-parameter normal case is worked out in exercise 8.4.

For the results of lecture 10 we will need one more property of a statistic, concerning functions of the statistic which are unbiased estimators for zero.

This means that the only function of a complete statistic which has expectation zero for all parameter values is the zero function. The condition is violated for statistics which do not reduce the data enough: in exercise 8.6 we will see that the whole Bernoulli sample is sufficient, but not complete, whereas the total $\sum_{i}X_{i}$ is complete, as we will see in lecture 10 (proposition 10.5). The property is important because any two unbiased estimators for the same quantity, which are both functions of a complete statistic $T$, differ by a function of $T$ with expectation zero, and by completeness this function must be zero with probability one: for each quantity there is at most one unbiased estimator which is a function of a complete statistic. In lecture 10 we will use this uniqueness to turn the Rao–Blackwell improvement into an optimality result. In the lecture we will also discuss which of the standard statistics are complete.

So far we have considered sufficient statistics which are sums, and we have seen the distributions of these sums in appendix A. For the uniform distribution we will consider the sample maximum, and in order to determine the bias or the mean squared error of the maximum, we need to know the distribution of the maximum. The following result gives the distribution of the maximum and minimum of an i.i.d. sample.

The minimum plays the corresponding role for models where the density is largest at the left-hand end of the support. In exercise 8.5 we see that the minimum of an exponential sample is itself exponentially distributed, but with rate multiplied by $n$.