Lecture 15 Cramér–Rao and Efficiency in R

In this section we consider $X_{1},\dots,X_{n}$ i.i.d. Poisson with mean $\theta=3$. From example 11.6 we know that the Cramér–Rao bound for an unbiased estimator for $\theta$ is $\theta/n$. The sample mean, $\bar{X}$, is an unbiased estimator for $\theta$ with $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})=\theta/n$ (see lemma 2.2) and thus, by definition 13.4, is an efficient estimator. We check this by simulation, with the loop over ns from lecture 9. The command var can be used to compute the sample variance of a vector of estimates, and dividing the bound by this value allows us to estimate the efficiency of the estimator. The full script for this section, with lines intentionally shuffled (except for the function body and the loop which are left as one piece), is given in the following box. Before you can run the script, you will need to arrange the lines yourself.

As in lectures 6 and 9, all names must be defined before they are used: the function, theta and ns come first, v needs length(ns), the seed must be set before the loop, and the table must be created last. Before you try the reconstructed script, make a prediction for the efficiency column. Will all entries be exactly $1$? Can an entry be larger than $1$, despite the fact that theorem 13.2 states that no unbiased estimator can have a variance smaller than the bound?

The empirical variances in the table follow the bound for all sample sizes, and the estimated efficiencies are clustered around $1$ with four of the six entries above this value. This is no contradiction: the theorem only bounds the true variance $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})$, whereas the table shows the sample variance of $N=10000$ simulated estimates, which is itself random. For a normal sample of size $N$, example 14.6 gives $\mathop{\mathrm{Var}}\nolimits(S^{2})=2\sigma^{4}/(N-1)$, so that the relative standard error of a sample variance is about $\sqrt{2/N}$, i.e. $1.4\%$ for $N=10000$. Estimates with a heavier tail than the normal distribution give a larger error. An estimated efficiency of $1.017$ thus means that, within the limits of error of the simulation, the estimator is efficient.

In this section we consider $X_{1},\dots,X_{n}$ i.i.d. exponential with rate $\theta>0$, and an unbiased estimator for $\theta$ which does not attain the bound. From exercise 13.4 we know that the bias-corrected maximum likelihood estimator $\tilde{\theta}=(n-1)/\sum_{i}X_{i}$ is unbiased and has variance $\theta^{2}/(n-2)$. On the other hand, the bound from example 11.8 is $\theta^{2}/n$ and thus the efficiency of this estimator is $(n-2)/n$. By the same exercise, $\tilde{\theta}$ is the UMVUE, and thus no unbiased estimator for the rate has efficiency larger than $(n-2)/n$.

We repeat the analysis of the previous section for $\tilde{\theta}$ with $\theta=2$. To do this, we replace est_pois by the function est_exp below, both in its definition and in the loop. We then set theta <- 2 and ns <- c(5, 10, 20, 50, 100), re-create v, reset the seed as before and replace the bound by theta^2 / ns. The resulting script is R/S15-efficiency.R. Before you run the script, write down $(n-2)/n$ for $n=5$, $10$, $20$, $50$ and $100$. Considering the simulation error we have observed, for which sample sizes could the simulation be expected to distinguish $\tilde{\theta}$ from an efficient estimator?

For $n=5$, $10$ and $20$ the simulated efficiencies are close to the exact values and clearly below $1$. The estimator $\tilde{\theta}$ has a heavy right tail, and repeating the simulation with other seeds shows that the estimated efficiency fluctuates by about $0.03$ for $n=5$ and by about $0.015$ for $n=50$ and $100$. For these two sample sizes, the exact loss of $4\%$ and $2\%$ is comparable to this error: the simulated $0.929$ suggests a larger loss than the true one, and the simulated $1.008$ suggests no loss at all. We would need considerably more than $10000$ repetitions to resolve these values. The loss of efficiency, $2/n$, goes to zero as $n$ increases: $\tilde{\theta}$ is asymptotically efficient in the sense of lecture 17, but not efficient.

To explain the difference between the two estimators, we return to the proof of theorem 13.2 in lecture 13, which uses the Cauchy–Schwarz inequality between the estimator $T$ and the score $\ell^{\prime}(\theta)$. Dividing both sides of this inequality by $\mathop{\mathrm{Var}}\nolimits_{\theta}(T)\,\mathcal{I}_{n}(\theta)$ shows that the efficiency of $T$ is given by the squared correlation between $T$ and the score. This equals $1$, if and only if $T-\theta$ is a multiple of the score. For the Poisson sample $\bar{X}-\theta$ equals $\theta/n$ times the score, and thus the correlation is $1$ without any calculation. For the exponential sample $\tilde{\theta}-\theta$ is a function of the score, but not a linear one. The following function computes the estimation error and the score for a given sample of size $n=5$. The command cor can be used to compute the correlation of two vectors. Predict the value of the squared correlation before you run the code.

The squared correlation $0.615$ coincides with the efficiency $3/5$ from the table, up to simulation error. To see where the remaining variance comes from, we plot the estimation error against the score and add the line $c(\theta)\,\ell^{\prime}(\theta)$ with $c(\theta)=\theta^{2}/n$ from proposition 13.5. Before you look at the plot, predict where the points will lie for an efficient estimator.

By proposition 13.5, every point lies on the dashed line for an efficient estimator. In figure 15.1 the points lie on a curve instead. The vertical distances between the points and the line are the part of the error which the score does not explain, and this part is the reason for the loss of efficiency.

The following task is in a style similar to the practical report. Consider the sample variance $S^{2}$, computed using var, as an estimator for $\sigma^{2}$ for a normal sample with $\mu=0$ and $\sigma^{2}=1$. The bound $2\sigma^{4}/n$ for this estimator is given in example 14.6, together with the variance $2\sigma^{4}/(n-1)$ of $S^{2}$. Repeat the study of section 15.1 for $S^{2}$, and create a table which shows the estimated and the exact efficiencies for the sample sizes in ns. Finally, in three sentences, discuss whether $S^{2}$ is efficient, how large its loss of efficiency is, and, using the relative error $\sqrt{2/N}$ from section 15.1, how many repetitions $N$ would be required to see the loss of efficiency of $S^{2}$ at $n=100$.

Before the next session, you should type the commands used in this session, and should check that you get the same results as the text.