Lecture 8 Sufficiency

In example 4.2 we have observed ten coin tosses and got seven heads. Writing $\theta$ for the heads probability, the likelihood in this example is $L(\theta)=\theta^{7}(1-\theta)^{3}$ . This function depends on the data only via the number $7$ of heads, but not on the order of heads and tails, i.e. for learning about $\theta$ we can replace the full record of observations by this single number. This lecture makes the idea precise: a sufficient statistic contains all information about the parameter. The factorisation theorem helps to find sufficient statistics, and the concept of sufficiency forms the basis of improvements of estimators in lecture 10.

In this lecture we write $X=(X_{1},\dots,X_{n})$ for the data, $x=(x_{1},\dots,x_{n})$ for an observed value, and

f(x;\theta)=\prod_{i=1}^{n}f(x_{i};\theta)

for the joint density or joint probability weights of the sample; the vector argument distinguishes it from the density of a single observation. As a function of $\theta$ for fixed data, this is the likelihood $L(\theta)$ from lecture 4.

8.1 Sufficient Statistics

A statistic $T=T(X_{1},\dots,X_{n})$ in the sense of definition 1.3 reduces the data to a smaller summary, and usually some information is lost in the process. The following definition expresses that no information about $\theta$ is lost.

Definition 8.1.

A statistic $T=T(X_{1},\dots,X_{n})$ is sufficient for $\theta$ , if the conditional distribution of $(X_{1},\dots,X_{n})$ given the value $T=t$ does not depend on $\theta$ , for all values $t$ of $T$ .

If $T$ is sufficient, anyone who knows the value $t$ of $T$ can generate a new data set with the same distribution as the original data set, by sampling from the conditional distribution of $X$ given $T=t$ . Since this distribution does not depend on $\theta$ , all information about $\theta$ that we could obtain by studying the data set can now be obtained by studying $t$ alone.

Example 8.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $p$ and let $T=\sum_{i=1}^{n}X_{i}$ be the number of successes. We know that $T\sim\text{Binomial}(n,p)$ , as a sum of $n$ Bernoulli variables. Let $t\in\{0,1,\dots,n\}$ and let $x\in\{0,1\}^{n}$ be a data vector with $\sum_{i}x_{i}=t$ . Since the event $\{X=x\}$ is contained in the event $\{T=t\}$ , we have

\mathbb{P}_{p}(X=x\mskip 1.0mu|\mskip 1.0muT=t)=\frac{\mathbb{P}_{p}(X=x)}{% \mathbb{P}_{p}(T=t)}=\frac{p^{t}(1-p)^{n-t}}{\binom{n}{t}p^{t}(1-p)^{n-t}}=% \frac{1}{\binom{n}{t}}.

If $\sum_{i}x_{i}\neq t$ , the events $\{X=x\}$ and $\{T=t\}$ are disjoint and the conditional probability is zero. In both cases the value does not depend on $p$ , and thus $T$ is sufficient for $p$ . Given the number of successes, each of the $\binom{n}{t}$ possible arrangements of successes and failures is equally likely, whatever the value of $p$ : the positions of the successes do not contain any information about the coin.

Two simple observations follow from the definition. First, the whole sample $T=(X_{1},\dots,X_{n})$ is always sufficient, since given the whole sample there is no randomness left. Secondly, if $\varphi$ is a one-to-one function, the events $\{\varphi(T)=\varphi(t)\}$ and $\{T=t\}$ coincide, and thus $\varphi(T)$ is sufficient whenever $T$ is: a one-to-one function of a sufficient statistic is sufficient again. For the coin tosses the sample mean $\bar{X}=T/n$ is thus sufficient too.

8.2 The Factorisation Theorem

Directly verifying the definition of sufficiency requires knowledge of the distribution of $T$ and involves a conditional probability which, for most models, is tedious to compute. The following proposition replaces the computation with a condition on the density or weights.

Proposition 8.3 (Fisher–Neyman factorisation theorem).

The statistic $T$ is sufficient for $\theta$ , if and only if there are functions $h$ and $g$ such that

equation (8.1) (8.1)

f(x;\theta)=h(x)\,g\bigl{(}T(x);\theta\bigr{)}

for all data vectors $x$ and all $\theta\in\Theta$ .

The function $h$ may not depend on $\theta$ at all, and the function $g$ may depend on the data only through $T(x)$ : the parameter and the data interact only through $T$ . Nothing in the statement requires $T$ or $\theta$ to be one-dimensional. We give a proof for discrete data only.

Proof for discrete data.

Since we have discrete observations, the joint probability weight $f(x;\theta)=\mathbb{P}_{\theta}(X=x)$ is the probability of observing the data vector $x$ .

Assume first that $T$ is sufficient and let $x$ be a data vector with $t=T(x)$ . Since the event $\{X=x\}$ is contained in the event $\{T=t\}$ , we have

\mathbb{P}_{\theta}(X=x)=\mathbb{P}_{\theta}(X=x,\,T=t)=\mathbb{P}_{\theta}(X=% x\mskip 1.0mu|\mskip 1.0muT=t)\,\mathbb{P}_{\theta}(T=t)

whenever $\mathbb{P}_{\theta}(T=t)>0$ . Using the sufficiency of the statistic, the conditional probability does not depend on $\theta$ and thus is a function $h(x)$ of the data. The remaining factor is a function $g(t;\theta)$ of $t$ and $\theta$ . If $\mathbb{P}_{\theta}(T=t)=0$ for the given $\theta$ , both sides in (8.1) equal zero since $g(t;\theta)=0$ . For the data vectors $x$ with $\mathbb{P}_{\theta}(T=T(x))=0$ for every $\theta$ , which are never observed, we define $h(x)=0$ . Thus (8.1) holds for all $x$ and $\theta$ .

Conversely, assume that (8.1) holds and let $t$ be a value of $T$ and $\theta$ a parameter value with $\mathbb{P}_{\theta}(T=t)>0$ . Summing the factorisations for all data vectors $y$ with $T(y)=t$ , we find

\mathbb{P}_{\theta}(T=t)=\sum_{y\colon T(y)=t}f(y;\theta)=\sum_{y\colon T(y)=t% }h(y)\,g(t;\theta)=H(t)\,g(t;\theta),

where $H(t)$ is the sum of the $h(y)$ , which does not depend on $\theta$ . Since the left-hand side is positive, we have $g(t;\theta)\neq 0$ . For data $x$ with $T(x)=t$ we get

\mathbb{P}_{\theta}(X=x\mskip 1.0mu|\mskip 1.0muT=t)=\frac{\mathbb{P}_{\theta}% (X=x)}{\mathbb{P}_{\theta}(T=t)}=\frac{h(x)\,g(t;\theta)}{H(t)\,g(t;\theta)}=% \frac{h(x)}{H(t)},

and for $x$ with $T(x)\neq t$ the conditional probability is zero. Thus, in both cases the value does not depend on $\theta$ and $T$ is sufficient. This completes the proof. ∎

The proof for the continuous case is more technical, and we omit it here. As a first step, one needs to define what it means to condition the distribution of $X$ on $T=t$ , since we typically have $\mathbb{P}(T=t)=0$ in the continuous case. To avoid these complications, we will use (8.1) as the definition of sufficiency for the continuous case from now on. Once conditioning on $T=t$ has been given a meaning, it can be shown that the factorisation again implies that the conditional distribution of the sample given $T$ does not depend on $\theta$ , as in the discrete case. We will use this property without proof in lecture 10; exercise 10.3 verifies it for an exponential sample.

When the support depends on $\theta$ , the density has to be written down for all $x$ , including the values where it is zero, since this is where the parameter and the data may interact. To write such densities compactly we use the indicator function of a condition, $\mathbf{1}_{\{A\}}$ , which equals $1$ if the condition $A$ holds and $0$ otherwise. The uniform distribution is the standard example.

Example 8.4.

Let $X_{1},\dots,X_{n}\sim\text{Uniform}(0,\theta)$ be i.i.d. with $\theta>0$ , as in example 1.2. The density of a single observation, written for all $x_{i}\in\mathbb{R}$ , is $f(x_{i};\theta)=\theta^{-1}\mathbf{1}_{\{0\leq x_{i}\leq\theta\}}$ . A product of indicators is the indicator of the intersection of the conditions, and all $n$ conditions $0\leq x_{i}\leq\theta$ hold if and only if the smallest observation is at least $0$ and the largest is at most $\theta$ . Thus the joint density is given by

f(x;\theta)=\theta^{-n}\prod_{i=1}^{n}\mathbf{1}_{\{0\leq x_{i}\leq\theta\}}=% \mathbf{1}_{\{\min_{i}x_{i}\geq 0\}}\;\theta^{-n}\,\mathbf{1}_{\{\max_{i}x_{i}% \leq\theta\}}.

If we set $h(x)=\mathbf{1}_{\{\min_{i}x_{i}\geq 0\}}$ and $g(t;\theta)=\theta^{-n}\mathbf{1}_{\{t\leq\theta\}}$ we have

f(x;\theta)=h(x)g\bigl{(}\max_{i}x_{i};\,\theta\bigr{)}

and by proposition 8.3 the sample maximum $X_{(n)}=\max_{i}X_{i}$ is sufficient for $\theta$ . It is a common mistake to try to write the joint density simply as $\theta^{-n}$ and to conclude that no statistic is needed: the indicator is where the data and the parameter interact.

The factorisation has a consequence for maximum likelihood estimators. For fixed data the likelihood is $L(\theta)=h(x)\,g\bigl{(}T(x);\theta\bigr{)}$ , and the factor $h(x)$ does not affect where the maximum over $\theta$ is attained: the maximum likelihood estimator, whenever it is unique, is a function of any sufficient statistic. The examples of lecture 5 confirm this: the Bernoulli MLE is $T/n$ with the sufficient statistic $T$ of example 8.2, and the uniform MLE is the sample maximum, which example 8.4 shows to be sufficient.

8.3 Sufficiency in Exponential Families

For the exponential families from lecture 7, no work is needed to find a sufficient statistic, since the densities are already given in factorised form.

Corollary 8.5.

Let $X_{1},\dots,X_{n}$ be an i.i.d. sample from a one-parameter exponential family as given in definition 7.1 and let $T$ be the natural statistic. Then $T_{n}=\sum_{i=1}^{n}T(X_{i})$ is a sufficient statistic for $\theta$ .

For a $k$ -parameter exponential family as given in definition 7.6, the vector $\bigl{(}\sum_{i}T_{1}(X_{i}),\dots,\sum_{i}T_{k}(X_{i})\bigr{)}$ of summed natural statistics is sufficient for the parameter vector.

Proof.

From lemma 7.5 we know that the joint density of the sample is

f(x;\theta)=\Bigl{(}\prod_{i=1}^{n}h(x_{i})\Bigr{)}\exp\Bigl{(}\eta(\theta)% \sum_{i=1}^{n}T(x_{i})-nB(\theta)\Bigr{)}.

Thus, we can write the joint density in the form (8.1) with the product $\prod_{i}h(x_{i})$ in the role of $h(x)$ and with $g(t;\theta)=\exp\bigl{(}\eta(\theta)t-nB(\theta)\bigr{)}$ . Consequently, $T_{n}$ is sufficient by proposition 8.3. A very similar argument gives the proof for $k$ -parameter exponential families. ∎

Example 8.6.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta>0$ . The joint probability weights are given by

f(x;\theta)=\prod_{i=1}^{n}\frac{\theta^{x_{i}}}{x_{i}!}\,e^{-\theta}=\theta^{% \sum_{i}x_{i}}\,e^{-n\theta}\cdot\frac{1}{\prod_{i}x_{i}!},

and thus the sum $T=\sum_{i}X_{i}$ is sufficient for $\theta$ , with $g(t;\theta)=\theta^{t}e^{-n\theta}$ and $h(x)=1/\prod_{i}x_{i}!$ .

Example 8.7.

Let $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ be i.i.d. with both parameters unknown. Example 7.7 identifies the natural statistics as $x$ and $x^{2}$ , and thus by corollary 8.5 the pair $T=\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$ is sufficient for $(\mu,\sigma^{2})$ . Since $\sum_{i}(x_{i}-\bar{x})^{2}=\sum_{i}x_{i}^{2}-n\bar{x}^{2}$ , the pair $(\bar{X},S^{2})$ maps to $T$ one-to-one and thus is also sufficient. This is the pair of statistics on which the maximum likelihood estimator from example 5.4 is based; the factorisation is worked out in exercise 8.4.

8.4 The Sample Maximum and Minimum

Example 8.4 shows that a sufficient statistic need not be a sum. To find the bias or the mean squared error of the sample maximum we need its distribution, which the following result gives for the maximum and minimum of an i.i.d. sample.

Proposition 8.8.

Let $X_{1},\dots,X_{n}$ be i.i.d. with distribution function $F$ and density $f$ . Furthermore, let $X_{(n)}=\max_{i}X_{i}$ and $X_{(1)}=\min_{i}X_{i}$ . Then the random variables $X_{(n)}$ and $X_{(1)}$ have densities

f_{(n)}(y)=n\,F(y)^{n-1}f(y)\qquad\text{and}\qquad f_{(1)}(y)=n\,\bigl{(}1-F(y% )\bigr{)}^{n-1}f(y).

Proof.

The maximum is at most $y$ , if and only if all observations are at most $y$ . Using independence we get

\mathbb{P}(X_{(n)}\leq y)=\mathbb{P}(X_{1}\leq y,\dots,X_{n}\leq y)=\prod_{i=1% }^{n}\mathbb{P}(X_{i}\leq y)=F(y)^{n}.

Using the chain rule for differentiation with respect to $y$ , we find that the maximum has density $nF(y)^{n-1}f(y)$ . Similarly, the minimum is strictly greater than $y$ , if and only if all observations are strictly greater than $y$ . Thus we get

\mathbb{P}(X_{(1)}\leq y)=1-\mathbb{P}(X_{1}>y,\dots,X_{n}>y)=1-\bigl{(}1-F(y)% \bigr{)}^{n}.

Using the chain rule for differentiation we find $n\bigl{(}1-F(y)\bigr{)}^{n-1}f(y)$ . This completes the proof. ∎

Example 8.9.

Let $X_{1},\dots,X_{n}\sim\text{Uniform}(0,\theta)$ be i.i.d., as in example 8.4. The distribution function is $F(y)=y/\theta$ for $0\leq y\leq\theta$ , and the density is $f(y)=1/\theta$ on $[0,\theta]$ . By proposition 8.8 the sample maximum has density

f_{(n)}(y)=n\Bigl{(}\frac{y}{\theta}\Bigr{)}^{n-1}\frac{1}{\theta}=\frac{n\,y^% {n-1}}{\theta^{n}},\qquad 0\leq y\leq\theta,

and $f_{(n)}(y)=0$ outside this interval. This is the density supplied without proof in exercise 2.2, from which we computed $\mathbb{E}_{\theta}(X_{(n)})=n\theta/(n+1)$ there. The support of the maximum moves with $\theta$ , as the support of a single observation does.

8.5 Minimal Sufficiency and Completeness

As we have seen in section 8.1, the whole sample is always sufficient. For an i.i.d. sample, the vector of ordered observations is also sufficient, since the joint density does not change when the $x_{i}$ are interchanged. The aim of this section is to characterise the statistic which allows us to reduce the data as much as possible, while still retaining all the information.

Definition 8.10.

A sufficient statistic $T$ is minimal sufficient for $\theta$ , if for every sufficient statistic $S$ there is a function $\varphi$ such that $T=\varphi(S)$ .

A minimal sufficient statistic can thus be computed from every other sufficient statistic. The definition is awkward to check directly, because it refers to all sufficient statistics at once; the following criterion instead compares the likelihood functions of two data sets. In the criterion we write

\mathcal{X}=\bigl{\{}\,x\bigm{|}f(x;\theta)>0\text{ for some }\theta\in\Theta% \,\bigr{\}}

for the set of data vectors which can be observed under some parameter value, for the Poisson distribution the vectors of non-negative integers.

Proposition 8.11.

Let $T$ be a statistic such that for all $x,y\in\mathcal{X}$ the following two statements are equivalent:

1.

there is a constant $c(x,y)>0$ such that $f(x;\theta)=c(x,y)\,f(y;\theta)$ for all $\theta\in\Theta$ ;
2.

$T(x)=T(y)$ .

Then $T$ is minimal sufficient for $\theta$ .

When both $f(x;\theta)$ and $f(y;\theta)$ are positive for all $\theta$ , the first statement says that the ratio $f(x;\theta)/f(y;\theta)$ does not depend on $\theta$ . We only sketch the proof. For sufficiency, we choose for every value $t$ of $T$ a data vector $x_{t}\in\mathcal{X}$ with $T(x_{t})=t$ ; the first statement then gives the factorisation (8.1) with $g(t;\theta)=f(x_{t};\theta)$ and $h(x)=c(x,x_{t})$ . For minimality, let $S$ be sufficient and $x,y\in\mathcal{X}$ with $S(x)=S(y)$ . The factorisation for $S$ shows that $f(x;\theta)$ and $f(y;\theta)$ are proportional, and the criterion gives $T(x)=T(y)$ : the value of $T$ is determined by the value of $S$ .

The criterion states that $T(x)$ determines the likelihood function of the data up to a constant factor, and that two data sets with different values of $T$ have likelihood functions which are not proportional. We apply the criterion once, to the Poisson sum. The two-parameter normal case is worked out in exercise 8.4.

Example 8.12.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta>0$ , and let $T=\sum_{i}X_{i}$ . For two vectors $x$ and $y$ of non-negative integers, both probability weights are positive and the joint probability weights of example 8.6 give

\frac{f(x;\theta)}{f(y;\theta)}=\frac{\theta^{\sum_{i}x_{i}}e^{-n\theta}/\prod% _{i}x_{i}!}{\theta^{\sum_{i}y_{i}}e^{-n\theta}/\prod_{i}y_{i}!}=\theta^{\sum_{% i}x_{i}-\sum_{i}y_{i}}\,\frac{\prod_{i}y_{i}!}{\prod_{i}x_{i}!}.

If $\sum_{i}x_{i}=\sum_{i}y_{i}$ , the power of $\theta$ disappears and the ratio is a positive constant. If the sums differ by $k\neq 0$ , the ratio contains the factor $\theta^{k}$ , which is not constant as $\theta$ ranges over $(0,\infty)$ . Thus the ratio is free of $\theta$ if and only if $\sum_{i}x_{i}=\sum_{i}y_{i}$ , and $T$ is minimal sufficient by proposition 8.11.

Summary.

•

A statistic $T$ is sufficient for $\theta$ , if the conditional distribution of the data given $T$ does not depend on $\theta$ : the data can be reduced to $T$ without losing information about $\theta$ .
•

The factorisation theorem states that $T$ is sufficient if and only if $f(x;\theta)=h(x)\,g(T(x);\theta)$ . When the support depends on $\theta$ , its indicator is part of the density and belongs to $g$ . When the MLE is unique, it is a function of any sufficient statistic.
•

For exponential families, the summed natural statistic $\sum_{i}T(X_{i})$ is sufficient. This includes the Bernoulli total, the Poisson sum and the normal pair $(\sum_{i}X_{i},\sum_{i}X_{i}^{2})$ .
•

The densities of the sample maximum and minimum are $nF^{n-1}f$ and $n(1-F)^{n-1}f$ , obtained by differentiating $F^{n}$ and $1-(1-F)^{n}$ , respectively.
•

A minimal sufficient statistic is a function of any sufficient statistic. A statistic $T$ is minimal sufficient, if $T(x)=T(y)$ holds exactly when the likelihood functions of $x$ and $y$ are proportional.

Exercise 8.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , where $n\geq 2$ , and let $T=\sum_{i=1}^{n}X_{i}$ be the number of successes.

1.

Let $x\in\{0,1\}^{n}$ be a data vector. Compute the conditional probability $\mathbb{P}_{\theta}(X=x\mskip 1.0mu|\mskip 1.0muX_{1}=x_{1})$ and show that the first observation $X_{1}$ on its own is not sufficient for $\theta$ .
2.

Using the factorisation theorem (proposition 8.3), show that $T$ is sufficient for $\theta$ , confirming example 8.2.
3.

Is the statistic $(X_{1},T)$ sufficient for $\theta$ ? Justify your answer.

Exercise 8.2.

Let $X_{1},\dots,X_{n}\sim\text{Uniform}(-\theta,\theta)$ be i.i.d. with $\theta>0$ , so that a single observation has density $f(x_{i};\theta)=(2\theta)^{-1}\mathbf{1}_{\{-\theta\leq x_{i}\leq\theta\}}$ .

1.

Show that the joint density can be written as

$f(x;\theta)=(2\theta)^{-n}\,\mathbf{1}_{\{\max_{i}|x_{i}|\leq\theta\}}.$
2.

Deduce from proposition 8.3 that $M=\max_{i}|X_{i}|$ is sufficient for $\theta$ . Identify the functions $g$ and $h$ .
3.

Show that the pair $\bigl{(}X_{(1)},X_{(n)}\bigr{)}$ of the sample minimum and maximum is also sufficient for $\theta$ , and express $M$ as a function of this pair.

Exercise 8.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ , i.e. $f(x;\theta)=\theta e^{-\theta x}$ for $x\geq 0$ .

1.

Show that $T=\sum_{i}X_{i}$ is sufficient for $\theta$ .
2.

Using proposition 8.11, show that $T$ is minimal sufficient.
3.

In example 5.2 we have found the MLE $\hat{\theta}=1/\bar{X}$ . Write $\hat{\theta}$ as a function of $T$ and explain why the factorisation in the first part of the question guarantees that such a function exists.

Exercise 8.4.

Let $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ be i.i.d.

1.

Assume that $\mu$ is known and $\sigma^{2}$ is unknown. Show that $\sum_{i}(X_{i}-\mu)^{2}$ is sufficient for $\sigma^{2}$ .
2.

Assume that both parameters are unknown. By expanding the squares in the exponent of the joint density, show that the joint density depends on the data only via the pair $T=\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$ . Using proposition 8.3, show that $T$ is sufficient for $(\mu,\sigma^{2})$ .
3.

Using proposition 8.11, show that the pair $T$ from the previous part is minimal sufficient for $(\mu,\sigma^{2})$ .
4.

Is the sample mean $\bar{X}$ on its own sufficient for $(\mu,\sigma^{2})$ when both parameters are unknown? Justify your answer using the previous part.

Exercise 8.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ , and let $X_{(1)}=\min_{i}X_{i}$ .

1.

Show that $X_{(1)}$ is exponentially distributed with rate $n\theta$ , first by proposition 8.8 and then directly from $\mathbb{P}_{\theta}(X_{(1)}>y)$ .
2.

Show that $nX_{(1)}$ is an unbiased estimator for the mean $1/\theta$ , and compute its variance.
3.

Compare the variance of $nX_{(1)}$ with that of the sample mean $\bar{X}$ , which is also unbiased for $1/\theta$ , and comment on the behaviour of the two estimators as $n$ grows. Is $nX_{(1)}$ a function of the sufficient statistic $\sum_{i}X_{i}$ ?