Lecture 10 Improving Estimators: Rao–Blackwell and Lehmann–Scheffe

In lecture 8 we have seen that a sufficient statistic $T$ can be used to summarise all the information in the data about the parameter. In this lecture we will use this idea to construct improved versions of estimators. The Rao–Blackwell theorem states that conditioning an unbiased estimator on the value of $T$ never increases its variance. The Lehmann–Scheffe theorem states that, if $T$ is a complete sufficient statistic, this procedure gives the best unbiased estimator.

10.1 The Rao–Blackwell Theorem

Assume that we want to estimate a function $g(\theta)$ of the parameter, e.g. we could be interested in the probability $\mathbb{P}_{\theta}(X_{1}=0)$ of zero counts. As in lecture 2, an estimator $U$ with finite mean is unbiased for $g(\theta)$ , if $\mathbb{E}_{\theta}(U)=g(\theta)$ for all $\theta\in\Theta$ . Unbiased estimators are often easy to find, but a crude one, such as the indicator of the event $\{X_{1}=0\}$ , uses only a small part of the available sample. The Rao–Blackwell theorem uses a sufficient statistic $T$ to improve an existing estimator by replacing $U$ with the conditional expectation $V=\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)$ , i.e. the average of $U$ for all samples which have the same value of $T$ as the observed sample.

Proposition 10.1 (Rao–Blackwell).

Let $T$ be a sufficient statistic for $\theta$ and let $U$ be an unbiased estimator for $g(\theta)$ with $\mathop{\mathrm{Var}}\nolimits_{\theta}(U)<\infty$ for all $\theta$ . Then $V=\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)$ has the following properties.

1.

The random variable $V$ does not depend on $\theta$ , but is a function of $T$ alone and thus a statistic.
2.

The statistic $V$ is an unbiased estimator for $g(\theta)$ .
3.

For all $\theta\in\Theta$ we have $\mathop{\mathrm{Var}}\nolimits_{\theta}(V)\leq\mathop{\mathrm{Var}}\nolimits_{% \theta}(U)$ .

Proof.

Since $T$ is a sufficient statistic, the conditional distribution of the sample $(X_{1},\dots,X_{n})$ given $T$ does not depend on $\theta$ ; for discrete data this is definition 8.1, and for continuous data it is the property stated after proposition 8.3. Since $U$ is a function of the sample, the conditional distribution of $U$ given $T$ , and thus the conditional expectation $V$ , do not depend on $\theta$ either. Like every conditional expectation given $T$ , the random variable $V$ is a function of $T$ , as explained in appendix A, and since this function does not involve the parameter $\theta$ , the random variable $V$ is a statistic in the sense of definition 1.3.

For the second statement, using the tower property from proposition A.9, we find

\mathbb{E}_{\theta}(V)=\mathbb{E}_{\theta}\bigl{(}\mathbb{E}_{\theta}(U\mskip 1% .0mu|\mskip 1.0muT)\bigr{)}=\mathbb{E}_{\theta}(U)=g(\theta)

for all $\theta$ . This shows that $V$ is unbiased.

Finally, for the third statement, we can use the law of total variance from proposition A.11, with $U$ instead of $X$ and $T$ instead of $Y$ . Since $\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)=V$ we get

\mathop{\mathrm{Var}}\nolimits_{\theta}(U)=\mathbb{E}_{\theta}\bigl{(}\mathop{% \mathrm{Var}}\nolimits_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)\bigr{)}+\mathop{% \mathrm{Var}}\nolimits_{\theta}(V).

Since the conditional variance is non-negative, as explained in appendix A, we get $\mathop{\mathrm{Var}}\nolimits_{\theta}(V)\leq\mathop{\mathrm{Var}}\nolimits_{% \theta}(U)$ . This completes the proof. ∎

Since both estimators are unbiased, their mean squared errors equal their variances by lemma 2.7, and thus $V$ is at least as good as $U$ . The procedure of passing from $U$ to $V$ is called Rao–Blackwellisation. Sufficiency is needed here: conditioning on a statistic which is not sufficient will in general give a quantity which depends on $\theta$ and is thus not an estimator.

Example 10.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta$ and assume that we want to estimate $g(\theta)=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ . The indicator $U=\mathbf{1}_{\{X_{1}=0\}}$ is unbiased for $g(\theta)$ , since $\mathbb{E}_{\theta}(U)=\mathbb{P}_{\theta}(X_{1}=0)$ . By corollary 8.5 the sum $T=\sum_{i}X_{i}$ is sufficient. Given $T=t$ , the first observation is distributed as $\text{Binomial}(t,1/n)$ ; this is shown in exercise 10.1. Thus we have

V=\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)=\mathbb{P}_{\theta}(X_{1}=0% \mskip 1.0mu|\mskip 1.0muT)=\Bigl{(}1-\frac{1}{n}\Bigr{)}^{T}.

The Rao–Blackwellised estimator now uses all $n$ observations. Exercise 10.3 applies the same procedure to a continuous model.

10.2 Completeness and the Lehmann–Scheffe Theorem

The Rao–Blackwell theorem improves an existing estimator, but two different unbiased estimators could in principle be improved to two different functions of $T$ . To rule this out we need one more property of a statistic, concerning functions of the statistic which are unbiased estimators for zero.

Definition 10.3.

A statistic $T$ is complete, if for every function $w$ such that $w(T)$ has finite mean under $\mathbb{P}_{\theta}$ for all $\theta\in\Theta$ , the condition $\mathbb{E}_{\theta}\bigl{(}w(T)\bigr{)}=0$ for all $\theta\in\Theta$ implies $\mathbb{P}_{\theta}\bigl{(}w(T)=0\bigr{)}=1$ for all $\theta\in\Theta$ .

This means that a function of a complete statistic which has expectation zero for all parameter values is itself zero with probability one. The condition is violated for statistics which do not reduce the data enough: in exercise 10.2 we will see that the whole Bernoulli sample is sufficient, but not complete, whereas the total $\sum_{i}X_{i}$ is complete by proposition 10.6 below. Completeness is a separate property from minimal sufficiency: a minimal sufficient statistic need not be complete. If $T$ is complete, two unbiased estimators for the same quantity which are both functions of $T$ agree with probability one, and thus every Rao–Blackwellisation leads to the same estimator. Before we prove this, we name the target.

Definition 10.4.

An unbiased estimator $V$ for $g(\theta)$ with $\mathop{\mathrm{Var}}\nolimits_{\theta}(V)<\infty$ for all $\theta\in\Theta$ is a uniformly minimum variance unbiased estimator (UMVUE) for $g(\theta)$ , if $\mathop{\mathrm{Var}}\nolimits_{\theta}(V)\leq\mathop{\mathrm{Var}}\nolimits_{% \theta}(W)$ for all $\theta\in\Theta$ and for every unbiased estimator $W$ for $g(\theta)$ with finite variance.

The word “uniformly” refers to the parameter: the inequality has to hold for every $\theta$ at once, and there is no reason a priori why such an estimator should exist. The following theorem shows that, whenever a complete sufficient statistic exists, a UMVUE exists as well and is easy to identify.

Theorem 10.5 (Lehmann–Scheffe).

Let $T$ be a complete sufficient statistic for $\theta$ and let $V=\varphi(T)$ be a function of $T$ which is unbiased for $g(\theta)$ and has $\mathop{\mathrm{Var}}\nolimits_{\theta}(V)<\infty$ for all $\theta$ . Then $V$ is a UMVUE for $g(\theta)$ .

Proof.

We first show that $V$ is the only unbiased estimator for $g(\theta)$ among the functions of $T$ . Let $\psi(T)$ be another unbiased estimator for $g(\theta)$ which is a function of $T$ . Then we have $\mathbb{E}_{\theta}\bigl{(}\psi(T)\bigr{)}=g(\theta)=\mathbb{E}_{\theta}\bigl{% (}\varphi(T)\bigr{)}$ for all $\theta$ . The function $w=\psi-\varphi$ satisfies $\mathbb{E}_{\theta}\bigl{(}w(T)\bigr{)}=0$ for all $\theta$ and, since $T$ is complete, by definition 10.3, we have $\mathbb{P}_{\theta}\bigl{(}w(T)=0\bigr{)}=1$ for all $\theta$ , i.e. $\psi(T)=\varphi(T)$ with probability one.

Now let $W$ be any unbiased estimator for $g(\theta)$ with finite variance. Then, by the Rao–Blackwell theorem, proposition 10.1, the random variable $\mathbb{E}_{\theta}(W\mskip 1.0mu|\mskip 1.0muT)$ is a function of $T$ which is unbiased for $g(\theta)$ . Thus, by the first step of the proof, it equals $V$ with probability one. The Rao–Blackwell theorem then gives

\mathop{\mathrm{Var}}\nolimits_{\theta}(V)=\mathop{\mathrm{Var}}\nolimits_{% \theta}\bigl{(}\mathbb{E}_{\theta}(W\mskip 1.0mu|\mskip 1.0muT)\bigr{)}\leq% \mathop{\mathrm{Var}}\nolimits_{\theta}(W)

for all $\theta$ , i.e. $V$ is a UMVUE. This completes the proof. ∎

The theorem turns the search for a best unbiased estimator into a two-step recipe: find a complete sufficient statistic $T$ , then find any function of $T$ which is unbiased for $g(\theta)$ , by adjusting a constant or by Rao–Blackwellising a crude unbiased estimator. For the models of this module the following two facts settle the question of completeness.

Proposition 10.6.

1.

Let $X_{1},\dots,X_{n}$ be an i.i.d. sample from a one-parameter exponential family of full rank, as given in definitions 7.1 and 7.8. Then the natural statistic $T_{n}=\sum_{i=1}^{n}T(X_{i})$ of the sample is complete. For a $k$ -parameter exponential family of full rank, as given in definition 7.6, the vector

$\Bigl{(}\sum_{i=1}^{n}T_{1}(X_{i}),\dots,\sum_{i=1}^{n}T_{k}(X_{i})\Bigr{)}$

of summed natural statistics is complete.
2.

Let $X_{1},\dots,X_{n}$ be i.i.d. and uniformly distributed on the interval $[0,\theta]$ where $\theta>0$ . Then the sample maximum $\max_{i}X_{i}$ is complete.

We state this proposition without proof, and both statements may be used without proof in what follows. The first statement rests on the uniqueness theorem for Laplace transforms. The second statement rests on taking derivatives, with respect to $\theta$ , of the relation $\int_{0}^{\theta}w(t)\,t^{n-1}\,\mathrm{d}t=0$ , which is the condition $\mathbb{E}_{\theta}\bigl{(}w(\max_{i}X_{i})\bigr{)}=0$ up to a constant factor. Together with corollary 8.5 and example 8.4 they provide a complete sufficient statistic for every family in tables 1.1 and 1.2 (with $m$ known for the binomial). A one-to-one function $\varphi(T)$ of a complete sufficient statistic $T$ is again complete and sufficient: sufficiency was shown in lecture 8, and if $\mathbb{E}_{\theta}\bigl{(}w(\varphi(T))\bigr{)}=0$ for all $\theta$ , then $w\circ\varphi$ is a function of $T$ with mean zero and thus, since $T$ is complete, $w\bigl{(}\varphi(T)\bigr{)}=0$ with probability one. In particular, we can use the sample mean $\bar{X}$ instead of $\sum_{i}X_{i}$ if required.

Example 10.7.

For the Poisson sample from example 10.2, the natural parameter $\eta=\log\theta$ takes values on all of $\mathbb{R}$ , and thus the family is of full rank in the sense of definition 7.8. By corollary 8.5 and proposition 10.6 the sum $T=\sum_{i}X_{i}$ is a complete and sufficient statistic. Since $\bar{X}=T/n$ is a function of $T$ , is unbiased for $\theta$ and has finite variance $\theta/n$ , it is the UMVUE for $\theta$ by theorem 10.5. Similarly, the Rao–Blackwellised indicator $V=(1-1/n)^{T}$ takes values in $[0,1]$ and thus has finite variance, and it is the UMVUE for $e^{-\theta}$ .

Example 10.8.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\mu,\sigma^{2})$ with both parameters unknown. This is a two-parameter exponential family with natural statistic $T(x)=(x,x^{2})$ , and the pair $(\bar{X},S^{2})$ is sufficient by example 8.7. The family has full rank, and thus the pair is also complete by proposition 10.6; exercise 10.5 verifies the rank condition. Since $\bar{X}$ and $S^{2}$ are unbiased for $\mu$ and $\sigma^{2}$ by lemmas 2.2 and 2.3, they are the UMVUEs for these parameters.

The second example puts section 2.3 into perspective: there the biased estimator $\frac{1}{n+1}\sum_{i}(X_{i}-\bar{X})^{2}$ had smaller mean squared error than $S^{2}$ . This is no contradiction, since the Lehmann–Scheffe theorem compares $S^{2}$ to unbiased estimators only.

In lecture 13 we will see a second way to derive optimality, using the Cramer–Rao bound, which does not require conditioning.

Summary.

•

Rao–Blackwell: If an unbiased estimator is conditioned on a sufficient statistic $T$ , the conditioned estimator is unbiased and has no larger variance.
•

A UMVUE is an unbiased estimator which has minimal variance among all unbiased estimators, for all $\theta$ .
•

A statistic $T$ is complete, if every function of $T$ with expectation zero for all $\theta$ is zero with probability one.
•

Lehmann–Scheffe: If $T$ is complete and sufficient, then the only function of $T$ which is unbiased for $g(\theta)$ is the UMVUE for $g(\theta)$ .
•

The natural statistic for a full-rank exponential family and the maximum of a uniform sample are complete (stated without proof).

Exercise 10.1.

Let $n\geq 2$ , let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta>0$ , let $T=\sum_{i=1}^{n}X_{i}$ and let $U=\mathbf{1}_{\{X_{1}=0\}}$ , as in example 10.2. You may use the fact that a sum of independent Poisson random variables is again Poisson, with the parameters added.

1.

Show that $U$ is an unbiased estimator for $e^{-\theta}$ and determine the variance of this estimator.
2.

Show that for $t\in\mathbb{N}_{0}$ and $x\in\{0,1,\dots,t\}$ we have

$\mathbb{P}_{\theta}(X_{1}=x\mskip 1.0mu|\mskip 1.0muT=t)=\binom{t}{x}\Bigl{(}% \frac{1}{n}\Bigr{)}^{x}\Bigl{(}1-\frac{1}{n}\Bigr{)}^{t-x},$

i.e. that the conditional distribution of $X_{1}$ given $T=t$ is $\text{Binomial}(t,1/n)$ and that this distribution does not depend on $\theta$ .
3.

Deduce that $V=\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)=(1-1/n)^{T}$ .
4.

Show directly that $V$ is an unbiased estimator for $e^{-\theta}$ , using the probability generating function (7.3) of the Poisson distribution with parameter $n\theta$ , which is given by $\mathbb{E}_{\theta}(z^{T})=\exp\bigl{(}n\theta(z-1)\bigr{)}$ (exercise 7.6).
5.

Explain why $V$ is the UMVUE for $e^{-\theta}$ and discuss the differences between this UMVUE and the MLE $e^{-\bar{X}}$ , introduced in exercise 5.7.

Exercise 10.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , where $n\geq 2$ , and let $T=(X_{1},\dots,X_{n})$ be the whole sample.

1.

Show that $w(T)=X_{1}-X_{2}$ satisfies $\mathbb{E}_{\theta}\bigl{(}w(T)\bigr{)}=0$ for all $\theta$ and $\mathbb{P}_{\theta}\bigl{(}w(T)=0\bigr{)}<1$ .
2.

Using definition 10.3, show that $T$ is not complete, despite $T$ being sufficient.

Exercise 10.3.

Let $n\geq 2$ and let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ , i.e. $f(x;\theta)=\theta e^{-\theta x}$ for $x\geq 0$ and let $T=\sum_{i=1}^{n}X_{i}$ . You may use the fact that the sum of $k$ independent random variables of this type has density $f_{k}(t)=\theta^{k}t^{k-1}e^{-\theta t}/(k-1)!$ for $t\geq 0$ .

1.

Explain why $X_{1}$ is an unbiased estimator for the mean $1/\theta$ and why $T$ is sufficient for $\theta$ .
2.

Let $R=\sum_{i=2}^{n}X_{i}$ . Use the independence of $X_{1}$ and $R$ to find the joint density of $(X_{1},T)$ . Deduce the conditional density of $X_{1}$ given $T=s$ . The result is

$f(x\mskip 1.0mu|\mskip 1.0mus)=\frac{(n-1)(s-x)^{n-2}}{s^{n-1}}\qquad\text{for% }0<x<s,$

and zero elsewhere. Check that this density does not depend on $\theta$ .
3.

Show that $\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT=s)=s/n$ and conclude that Rao–Blackwellising $X_{1}$ leads to the sample mean $\bar{X}$ , again unbiased for $1/\theta$ .
4.

Find a second argument for $\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT)=T/n$ , based on the symmetry of the sample, without any integration.

Exercise 10.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the interval $[0,\theta]$ where $\theta>0$ , and let $M=\max_{i}X_{i}$ .

1.

Using the mean of $M$ from exercise 2.2, show that $\hat{\theta}=(n+1)M/n$ is an unbiased estimator for $\theta$ .
2.

Show that $\hat{\theta}$ is the UMVUE for $\theta$ .
3.

The estimator $\tilde{\theta}=2\bar{X}$ is also unbiased for $\theta$ . Determine the variances of $\hat{\theta}$ and of $\tilde{\theta}$ and verify that they are ordered as in theorem 10.5.
4.

Without computing any conditional expectations, find $\mathbb{E}_{\theta}(\tilde{\theta}\mskip 1.0mu|\mskip 1.0muM)$ .

Exercise 10.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\mu,\sigma^{2})$ , where both parameters are unknown. This is the situation of example 10.8.

1.

Write the density of an observation in the form from definition 7.6, confirm that the natural statistic is $T(x)=(x,x^{2})$ and determine the natural parameter $\eta(\mu,\sigma^{2})$ and the function $B(\mu,\sigma^{2})$ .
2.

Show that the family of densities has full rank, in the sense of definition 7.8, by determining the range of $\eta$ .
3.

Deduce that $\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$ is complete and sufficient and argue that the same holds for the pair $(\bar{X},S^{2})$ .