Lecture 13 The Cramer–Rao Inequality

Throughout this lecture, $X_{1},\dots,X_{n}$ are i.i.d. with density or probability weights $f(x;\theta)$, the parameter space $\Theta\subseteq\mathbb{R}$ is an open interval, and the conditions of section 11.1 hold. The score $\ell^{\prime}(\theta)$ and the Fisher information $\mathcal{I}_{n}(\theta)=\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell^{\prime}(\theta)\bigr{)}$ are as in lecture 11; we assume in addition that $0<\mathcal{I}_{n}(\theta)<\infty$ for all $\theta\in\Theta$, and we write $f(x;\theta)=\prod_{i=1}^{n}f(x_{i};\theta)$ for the joint density of a value $x=(x_{1},\dots,x_{n})$ of the sample.

The idea of the proof can be stated in advance: the covariance between an unbiased estimator $W$ and the score equals one, for any model, and then the Cauchy–Schwarz inequality implies that small Fisher information implies large variance. We state the result for the covariance as a separate lemma, for use later for functions of the parameter.

The assumption (13.1) is an additional exchange of differentiation and integration. For exponential families, this relation holds for all statistics with finite variance, by a similar argument from analysis as given in the proof of theorem 7.7, but we will not verify this relation for individual examples.

Three comments help to read the theorem. First, by proposition 11.4 the bound is $1/\bigl{(}n\mathcal{I}(\theta)\bigr{)}$: no unbiased estimator can have a standard deviation which decreases faster than the $1/\sqrt{n}$ of the sample mean (lemma 2.2). Second, the bound is independent of the estimator used, and only depends on the model: the bound states how much the data can reveal about $\theta$, regardless of the unbiased method used to estimate it. For the normal mean with known variance the bound is $\sigma^{2}/n=\mathop{\mathrm{Var}}\nolimits_{\mu}(\bar{X})$, as shown in example 11.7, and thus $\bar{X}$ achieves this bound. Third, by theorem 2.7, the bound also acts as a lower bound for the mean squared error, but only amongst unbiased estimators: in section 2.3 we have seen a biased estimator for the normal variance, with smaller mean squared error than $S^{2}$.

Often the quantity of interest is a function $g(\theta)$ of the parameter, for example the probability $e^{-\theta}$ of zero counts in example 10.2. If $W$ is an unbiased estimator for $g(\theta)$, the only change in the proof is the value of the covariance.

Theorem 13.2 is the case $g(\theta)=\theta$; it is the general form which is worth memorising: identify the function $g$ the estimator is unbiased for, differentiate, square and divide by the information. Both numerator and denominator depend on how the model is parametrised, but by proposition 11.10 the bound itself does not: it is a property of the estimator and the model, not of the name we give to the parameter. Exercise 13.6 verifies this.

We give a name to the estimators which attain the bound.

From theorem 13.3 we know that the efficiency is contained in the interval $(0,1]$, and since the bound is proportional to $1/n$, an efficiency of $1/2$ means that the estimator uses the information from $n$ observations as well as an efficient estimator would use the information from $n/2$. An efficient estimator is a UMVUE in the sense of definition 10.3, because no unbiased estimator can be better than the bound, but the converse statement is not true, since the bound is not always attainable: in exercise 13.4, the UMVUE for the rate of the exponential distribution has efficiency $(n-2)/n$. These two results complement each other: the Lehmann–Scheffe theorem allows us to identify the best unbiased estimator, whereas the Cramer–Rao bound provides an absolute scale for comparing estimators.

The proof tells us when the bound is attained: equality in the Cramer–Rao inequality is equality in the Cauchy–Schwarz step, and this only holds for random variables which are proportional to each other.

The condition (13.2) is restrictive. In an exponential family, an efficient estimator must be an affine function of the natural statistic $T_{n}$, and the coefficients of any such affine function cannot depend on $\theta$ (since $W$ is fixed while $\theta$ varies over $\Theta$). Thus, the only functions of $\theta$ which can lead to an efficient estimator are affine functions $a+b\,\mu(\theta)$ of the mean of the natural statistic, estimated by $a+bT_{n}/n$. For the Poisson distribution, an exponential family with $T(x)=x$ and $\mu(\theta)=\theta$, the sample mean is an efficient estimator for $\theta$. Exercise 13.3 shows that the attainment condition is violated in this case, by showing that $e^{-\theta}$ is not an affine function of $\theta$. Thus, the UMVUE $(1-1/n)^{T_{n}}$ from example 10.6 is the best one among the unbiased estimators, but does not achieve the bound.

As we have seen, the proof of the inequality in this lecture involves two instances of differentiating under the integral sign, once in lemma 11.1 and once in the assumption (13.1). If the support of the distribution moves as $\theta$ does, both of these exchanges will fail, since an additional term is added to the integral which is not included in the exchange.

There is no contradiction: the inequality has hypotheses, and the uniform model violates these hypotheses. Super-efficiency is a property of models where the parameter is an endpoint of the support, where a single observation near the endpoint can be used to estimate $\theta$ much more accurately than for a regular model. In lecture 16 we will prove the consistency of $M$ using a direct argument. In lecture 17 we will see that the limiting distribution of $M$ is not normal. In lecture 15 we will use simulation to demonstrate the $1/n^{2}$ decay of the variance.