Lecture 17 Asymptotic Normality and Efficiency of the MLE

In lecture 16 we have seen that under regularity conditions the maximum likelihood estimator converges to the true parameter value. In this lecture we will discuss the error $\hat{\theta}_{n}-\theta_{0}$ in more detail: It transpires that for large sample size the error is approximately normally distributed with mean zero and variance given by the Cramer–Rao bound from lecture 13. Thus, the MLE is unbiased in the limit, and more importantly, is as accurate as any unbiased estimator can be. The proof of this result is a challenging one, but it is based on tools we have already learned in this module. The argument is based on Taylor expansion of the score function, and the resulting theorem allows us to compute standard errors and to derive approximate confidence statements. We will see the resulting numbers in lecture 18 and will make the statements precise in lecture 25.

17.1 The Theorem

Using the notation from lecture 16, we write $\mathcal{I}(\theta)$ for the Fisher information of a single observation and $\mathcal{I}_{n}(\theta)=n\mathcal{I}(\theta)$ for the Fisher information of the sample, as given in definition 11.2 and proposition 11.4. The result is formulated in terms of the scaled error $\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})$ , as this is the only error term which has a non-degenerate limit: by consistency the error goes to zero and the factor $\sqrt{n}$ is required to balance this effect.

Theorem 17.1.

Suppose that the regularity conditions (R1) to (R4) from definition 16.1 are satisfied, that $\log\bigl{(}f(X_{1};\theta)/f(X_{1};\theta^{\prime})\bigr{)}$ has finite mean under $\mathbb{P}_{\theta^{\prime}}$ for all $\theta,\theta^{\prime}\in\Theta$ , and that for every sample size $n\geq 1$ and every sample where all coordinates are in the support of the model, the likelihood equation $\ell^{\prime}(\theta)=0$ has a unique solution $\hat{\theta}_{n}$ which coincides with the maximum likelihood estimator. Then

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{d}\ }N\Bigl{(}0,% \frac{1}{\mathcal{I}(\theta_{0})}\Bigr{)}\qquad(n\to\infty).

Before we give the proof, we unpack the statement of the theorem. The theorem informally states that, as $n$ gets large,

\hat{\theta}_{n}\approx N\Bigl{(}\theta_{0},\frac{1}{n\mathcal{I}(\theta_{0})}% \Bigr{)}=N\Bigl{(}\theta_{0},\frac{1}{\mathcal{I}_{n}(\theta_{0})}\Bigr{)}.

This statement contains three different aspects: (1) The centre of the approximating distribution equals the true value, i.e. the MLE is asymptotically unbiased (this does not mean that the MLE is unbiased for finite $n$ , only that it is asymptotically unbiased). (2) The spread of the distribution decreases as $1/\sqrt{n}$ , as for the sample mean. (3) The variance of the approximating distribution equals $1/\mathcal{I}_{n}(\theta_{0})$ . This is the Cramer–Rao lower bound from lecture 13: no unbiased estimator can have smaller variance for any $n$ and the MLE achieves this bound in the limit. The last property has a name.

Definition 17.2.

An estimator $\tilde{\theta}_{n}$ for $\theta$ , which does not have to be the maximum likelihood estimator, is asymptotically efficient, if $\sqrt{n}\,(\tilde{\theta}_{n}-\theta)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,1/% \mathcal{I}(\theta)\bigr{)}$ for all $\theta\in\Theta$ .

In this language theorem 17.1 states that under the regularity conditions the MLE is asymptotically efficient. In lecture 13 we have seen that an efficient estimator, one which achieves the Cramer–Rao bound exactly, only exists for special models and parametrisations, namely the natural statistic of an exponential family. The asymptotic statement is much wider in applicability, as it holds for every regular model and every parametrisation, since by theorem 5.6 the MLE transforms together with the parameter.

17.2 Proof of the Theorem

We will use a Taylor expansion of the score function $\ell^{\prime}$ around the true value. Since the MLE is a root of the score, we can use Taylor expansion of $\ell^{\prime}(\hat{\theta}_{n})$ around the point $\theta_{0}$ to write the error term $\hat{\theta}_{n}-\theta_{0}$ in terms of $\ell^{\prime}(\theta_{0})$ and $\ell^{\prime\prime}(\theta_{0})$ , which are both sums of i.i.d. terms and thus can be analysed using the limit theorems for sample means.

Proof of theorem 17.1.

Let the true value be $\theta_{0}$ . Then, by (R3), the function $\ell^{\prime}$ is two times continuously differentiable. Thus, by Taylor’s theorem with the Lagrange form of the remainder, for $\ell^{\prime}$ around the point $\theta_{0}$ we get

\ell^{\prime}(\hat{\theta}_{n})=\ell^{\prime}(\theta_{0})+\ell^{\prime\prime}(% \theta_{0})\,(\hat{\theta}_{n}-\theta_{0})+\tfrac{1}{2}\,\ell^{\prime\prime% \prime}(\theta_{n}^{*})\,(\hat{\theta}_{n}-\theta_{0})^{2},

where $\theta_{n}^{*}$ is a point between $\theta_{0}$ and $\hat{\theta}_{n}$ . Since $\hat{\theta}_{n}$ solves the likelihood equation, the left-hand side is zero. Solving for $\hat{\theta}_{n}-\theta_{0}$ and multiplying by $\sqrt{n}$ we can write the scaled error as a ratio:

equation (17.1) (17.1)

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})=\frac{A_{n}}{B_{n}-C_{n}},

where

A_{n}=\frac{1}{\sqrt{n}}\,\ell^{\prime}(\theta_{0}),\qquad B_{n}=-\frac{1}{n}% \,\ell^{\prime\prime}(\theta_{0}),\qquad C_{n}=\frac{1}{2n}\,\ell^{\prime% \prime\prime}(\theta_{n}^{*})\,(\hat{\theta}_{n}-\theta_{0}).

We now take the limit of each of the three terms separately.

The numerator $A_{n}$ is a scaled sum of i.i.d. terms: by definition of the score we have $\ell^{\prime}(\theta_{0})=\sum_{i}\ell_{i}^{\prime}(\theta_{0})$ and using the identities (16.1) we find that each term has mean zero and finite, positive variance $\mathcal{I}(\theta_{0})$ . Thus, by the central limit theorem, theorem A.15, for the sample mean of the $\ell_{i}^{\prime}(\theta_{0})$ we get

A_{n}=\sqrt{n}\,\Bigl{(}\frac{1}{n}\sum_{i=1}^{n}\ell_{i}^{\prime}(\theta_{0})% -0\Bigr{)}\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,\mathcal{I}(\theta_{0})\bigr{% )}.

The first term in the denominator is also a sample mean: $B_{n}=-\frac{1}{n}\sum_{i}\ell_{i}^{\prime\prime}(\theta_{0})$ and by the law of large numbers, theorem A.14, and the second identity in (16.1) we find

B_{n}\xrightarrow{\ \mathrm{p}\ }-\mathbb{E}_{\theta_{0}}\bigl{(}\ell_{1}^{% \prime\prime}(\theta_{0})\bigr{)}=\mathcal{I}(\theta_{0}).

The remainder term $C_{n}$ is where the consistency from lecture 16 and condition (R4) come into play. Let $\delta$ and $M$ be as in (R4). On the event $\{|\hat{\theta}_{n}-\theta_{0}|<\delta\}$ , the intermediate point $\theta_{n}^{*}$ also lies within $\delta$ of $\theta_{0}$ . Thus, writing $\ell^{\prime\prime\prime}(\theta_{n}^{*})=\sum_{i}\ell_{i}^{\prime\prime\prime% }(\theta_{n}^{*})$ as a sum of the observations and using (R4) for each term, we find

\Bigl{|}\frac{1}{n}\,\ell^{\prime\prime\prime}(\theta_{n}^{*})\Bigr{|}\leq% \frac{1}{n}\sum_{i=1}^{n}\bigl{|}\ell_{i}^{\prime\prime\prime}(\theta_{n}^{*})% \bigr{|}\leq\frac{1}{n}\sum_{i=1}^{n}M(X_{i})

on this event. By theorem 16.3, the probability of the event goes to one, and by the law of large numbers the right-hand side converges in probability to the finite constant $\mathbb{E}_{\theta_{0}}\bigl{(}M(X_{1})\bigr{)}$ . Thus, for all sufficiently large $K$ we have $\mathbb{P}_{\theta_{0}}\bigl{(}|\ell^{\prime\prime\prime}(\theta_{n}^{*})|/n>K% \bigr{)}\to 0$ . The other factor of $C_{n}$ is $\hat{\theta}_{n}-\theta_{0}$ , which converges in probability to zero by theorem 16.3. For any $\eta>0$ , the event $|C_{n}|>\eta$ is contained in the union of the events $|\ell^{\prime\prime\prime}(\theta_{n}^{*})|/n>K$ and $|\hat{\theta}_{n}-\theta_{0}|>2\eta/K$ , both of which have probability that goes to zero, and thus $C_{n}\xrightarrow{\ \mathrm{p}\ }0$ .

It remains to combine these three limits, and the appropriate tool for this is Slutsky’s theorem, theorem A.16. Since $B_{n}\xrightarrow{\ \mathrm{p}\ }\mathcal{I}(\theta_{0})$ and $C_{n}\xrightarrow{\ \mathrm{p}\ }0$ , the denominator satisfies $B_{n}-C_{n}\xrightarrow{\ \mathrm{p}\ }\mathcal{I}(\theta_{0})$ , where the value is constant and non-zero. By the quotient part of Slutsky’s theorem, for the representation (17.1), we have

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})=\frac{A_{n}}{B_{n}-C_{n}}\xrightarrow{% \ \mathrm{d}\ }\frac{N\bigl{(}0,\mathcal{I}(\theta_{0})\bigr{)}}{\mathcal{I}(% \theta_{0})}=N\Bigl{(}0,\frac{1}{\mathcal{I}(\theta_{0})}\Bigr{)},

where in the last step we used the fact that dividing a $N(0,\sigma^{2})$ random variable by the constant $c>0$ gives a $N(0,\sigma^{2}/c^{2})$ random variable, in this case with $\sigma^{2}=\mathcal{I}(\theta_{0})$ and $c=\mathcal{I}(\theta_{0})$ . This completes the proof. ∎

The structure of this argument is typical of most large-sample results in statistics: Taylor expansion is used to transform a question about an implicitly defined estimator into a question about sums of i.i.d. terms, the central limit theorem is used to analyse the leading term, the law of large numbers is used to analyse the coefficient of the leading term, the remainder is shown to be negligible using consistency of the estimator, and Slutsky’s theorem is used to combine the individual results. The only tricky step is the analysis of the remainder term, and condition (R4) is the reason that this term is straightforward to analyse.

17.3 Standard Errors

The variance $1/\mathcal{I}_{n}(\theta_{0})$ in theorem 17.1 depends on the unknown $\theta_{0}$ , and so cannot be reported as it stands. In practice we replace $\theta_{0}$ by its estimate, and the next result says that this does not disturb the limit.

Corollary 17.3.

With the same assumptions as in theorem 17.1, we have

\sqrt{n\mathcal{I}(\hat{\theta}_{n})}\,(\hat{\theta}_{n}-\theta_{0})% \xrightarrow{\ \mathrm{d}\ }N(0,1)\qquad(n\to\infty).

Proof.

By (R3) the function $\mathcal{I}$ is continuous and positive on $\Theta$ and thus the function $\theta\mapsto\sqrt{\mathcal{I}(\theta)/\mathcal{I}(\theta_{0})}$ is continuous at $\theta_{0}$ . By $\hat{\theta}_{n}\xrightarrow{\ \mathrm{p}\ }\theta_{0}$ from theorem 16.3, the continuous mapping theorem from theorem A.17 implies that $\sqrt{\mathcal{I}(\hat{\theta}_{n})/\mathcal{I}(\theta_{0})}\xrightarrow{\ % \mathrm{p}\ }1$ . By multiplying the result of theorem 17.1 by this factor and using the product part of Slutsky’s theorem, theorem A.16, we find

\sqrt{n\mathcal{I}(\hat{\theta}_{n})}\,(\hat{\theta}_{n}-\theta_{0})=\sqrt{% \frac{\mathcal{I}(\hat{\theta}_{n})}{\mathcal{I}(\theta_{0})}}\cdot\sqrt{n% \mathcal{I}(\theta_{0})}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{% d}\ }1\cdot N(0,1),

and the claim is proved. ∎

The quantity

\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=\frac{1}{\sqrt{\mathcal{I}_{n}% (\hat{\theta}_{n})}}=\frac{1}{\sqrt{n\mathcal{I}(\hat{\theta}_{n})}}

is called the standard error of the MLE. As a statistic, this quantity from the corollary is an approximation to the standard deviation of $\hat{\theta}_{n}$ for large $n$ . The corollary also allows us to justify approximate confidence statements: Since a standard normal variable takes values in $(-1.96,1.96)$ with probability $0.95$ , the random interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ covers $\theta_{0}$ with probability which tends to $0.95$ . Intervals of this form were used in an informal way in lecture 12 and we will make these intervals precise in lecture 25. In lecture 18 we will use simulation to check how close the real coverage of these intervals is to $0.95$ for moderate $n$ . If we do not have a closed formula for $\mathcal{I}$ , the denominator $\mathcal{I}_{n}(\hat{\theta}_{n})$ can be replaced by the observed curvature $-\ell^{\prime\prime}(\hat{\theta}_{n})$ of the log-likelihood at the maximum. This approach is taken by a numerical optimiser and we will use the resulting values in lecture 18.

17.4 Examples

For the MLEs which are sample means, the Bernoulli and Poisson MLEs from lecture 5, theorem 17.1 does not give any new results: the central limit theorem allows us to find the asymptotic normality of these estimators directly and from exercises 17.1 and 17.2 we know that the two variances are the same. For the normal mean with known variance, the MLE $\bar{X}$ is exactly normally distributed for every $n$ , with variance $\sigma^{2}/n=1/\mathcal{I}_{n}(\theta)$ . Thus, for this case the theorem gives an exact result. The first new case is an MLE which is not a sample mean.

Example 17.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. with exponential rate $\theta$ . Then the MLE for the parameter value is $\hat{\theta}_{n}=1/\bar{X}$ from example 5.2 and by exercise 16.1 the regularity conditions and the uniqueness of the root hold. From example 11.8 we know that the Fisher information is given by $\mathcal{I}(\theta)=1/\theta^{2}$ . Thus, by theorem 17.1,

\sqrt{n}\,\Bigl{(}\frac{1}{\bar{X}}-\theta\Bigr{)}\xrightarrow{\ \mathrm{d}\ }% N(0,\theta^{2}),

with standard error $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=\hat{\theta}_{n}/\sqrt{n}$ . We can compare the result to exact results as follows: In lecture 5 we found $\mathbb{E}_{\theta}(\hat{\theta}_{n})=n\theta/(n-1)$ , which is a bias of $\theta/(n-1)$ . In exercise 5.3 we found the variance $\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta}_{n})=n^{2}\theta^{2}/% \bigl{(}(n-1)^{2}(n-2)\bigr{)}$ . If we multiply this variance by $n$ , it converges to $\theta^{2}$ . If we multiply the bias by $\sqrt{n}$ , it converges to zero, as predicted by the theorem: for large $n$ , the bias is of a smaller order than the spread of the distribution, and thus disappears in the limit.

The theorem can also be applied to functions of the parameter. The mean of the exponential distribution is $\mu=g(\theta)=1/\theta$ and by theorem 5.6 the MLE for this parameter is $\hat{\mu}_{n}=1/\hat{\theta}_{n}=\bar{X}$ . Using the delta method, theorem A.18 with $g^{\prime}(\theta)=-1/\theta^{2}$ , we get

\sqrt{n}\,(\bar{X}-\mu)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,\ g^{\prime}(% \theta)^{2}\,\theta^{2}\bigr{)}=N\Bigl{(}0,\frac{1}{\theta^{2}}\Bigr{)}=N(0,% \mu^{2}).

This is the central limit theorem for $\bar{X}$ , since the exponential distribution has variance $1/\theta^{2}=\mu^{2}$ . The standard error of $\bar{X}$ , $\bar{X}/\sqrt{n}$ , is the standard error we would obtain by computing the standard error from first principles.

The example illustrates a general point: for any smooth function $g(\theta)$ of the parameter, theorem 5.6 gives the MLE $g(\hat{\theta}_{n})$ , and the delta method gives the asymptotic distribution and standard error $|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ , without any further maximisation.

17.5 Quality of the Approximation

Theorem 17.1 is a limit statement and does not give information about the required sample size for the normal approximation to be applicable. The required value of $n$ for the normal approximation to be usable depends on the model and there are three situations where care is needed. First, for small samples from a model with a skewed distribution: for the exponential rate, $\hat{\theta}_{n}=1/\bar{X}$ has a right-skewed distribution and a bias of $\theta/(n-1)$ which is not captured in the limit but which is visible for $n$ around $10$ . Secondly, when a parameter value is close to the boundary of the parameter space: for the Bernoulli distribution with $\theta_{0}$ close to zero or one, the standard error $\sqrt{\hat{\theta}_{n}(1-\hat{\theta}_{n})/n}$ is zero whenever all observations have the same value, and thus the interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ has zero width in this case. Finally, if the regularity conditions are violated: for the uniform distribution on $(0,\theta)$ , exercise 17.3 shows that the error of the MLE is of order $1/n$ instead of $1/\sqrt{n}$ , and that the limiting distribution is exponential instead of normal, as expected from the super-efficiency of the parameter estimates as shown in lecture 13. The first case is investigated by simulation in lecture 12, by checking the coverage of the interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ for moderate $n$ .

Finally, the theorem can be extended to a vector parameter $\theta=(\theta_{1},\dots,\theta_{k})$ : Under appropriate regularity conditions, the MLE is asymptotically normal with mean $\theta_{0}$ and covariance matrix $\mathcal{I}_{n}(\theta_{0})^{-1}$ , the inverse of the Fisher information matrix from lecture 14. Without proof, the argument is similar to the one above, but uses Taylor expansion for several variables. For the normal distribution with unknown parameters, where the information matrix is diagonal, this gives the asymptotic normality of $\bar{X}$ and of the variance estimator separately.

Summary.

•

Under regularity conditions and assuming the uniqueness of the likelihood root, we have $\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,% 1/\mathcal{I}(\theta_{0})\bigr{)}$ : the MLE is asymptotically unbiased, the error is of order $1/\sqrt{n}$ , and the asymptotic variance equals the Cramer–Rao bound, i.e. the MLE is asymptotically efficient.
•

The proof uses Taylor expansion of the score around $\theta_{0}$ , central limit theorem for the score, law of large numbers for the second derivative, consistency and (R4) for the remainder, and Slutsky’s theorem to combine these results.
•

Replacing $\theta_{0}$ by $\hat{\theta}_{n}$ in the expression for the variance does not change the limit and thus we get the standard error $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=1/\sqrt{\mathcal{I}_{n}(\hat{% \theta}_{n})}$ and the approximate $95\%$ confidence interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ for the parameter value.
•

Using the delta method, we can apply the result to any smooth function of the parameter. The standard error for such an application is $|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ .
•

The approximation can be poor for small $n$ in models with skewed distributions and at the boundary of the parameter space, and it breaks down completely when the regularity conditions are violated, e.g. for the uniform distribution.

Exercise 17.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , where the MLE $\hat{\theta}_{n}=\bar{X}$ and Fisher information $\mathcal{I}(\theta)=1/\bigl{(}\theta(1-\theta)\bigr{)}$ are given in example 11.5.

1.

State the result of theorem 17.1 for this model and verify that it coincides with the central limit theorem for $\bar{X}$ .
2.

The log-odds are given by $\psi=g(\theta)=\log\bigl{(}\theta/(1-\theta)\bigr{)}$ . Find the MLE for $\psi$ and use the delta method to show that $\sqrt{n}\,(\hat{\psi}_{n}-\psi)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,1/(% \theta(1-\theta))\bigr{)}$ .
3.

Determine the standard error of $\hat{\psi}_{n}$ and an approximate $95\%$ interval for $\psi$ , in terms of $\hat{\theta}_{n}$ and $n$ .

Exercise 17.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta>0$ , where MLE $\hat{\theta}_{n}=\bar{X}$ and Fisher information $\mathcal{I}(\theta)=1/\theta$ are given in example 11.6.

1.

State the result of theorem 17.1 for this model and verify that the result coincides with the central limit theorem.
2.

In exercise 5.7 we have found the MLE $\hat{p}_{0}=e^{-\bar{X}}$ for the probability $p_{0}=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ and we have seen that this MLE is biased. Using the delta method, determine the asymptotic distribution of $\sqrt{n}\,(\hat{p}_{0}-p_{0})$ . Why does the bias, found in the previous step, not contradict the result from this question?

Exercise 17.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the interval $(0,\theta)$ . The MLE for this model is $\hat{\theta}_{n}=\max_{i}X_{i}$ .

1.

For every $x\geq 0$ , show that $\mathbb{P}_{\theta}\bigl{(}n(\theta-\hat{\theta}_{n})>x\bigr{)}\to e^{-x/\theta}$ as $n\to\infty$ , i.e. that $n(\theta-\hat{\theta}_{n})$ converges in distribution to the exponential distribution with mean $\theta$ , i.e. with rate $1/\theta$ .
2.

Show that $\sqrt{n}\,(\hat{\theta}_{n}-\theta)\xrightarrow{\ \mathrm{p}\ }0$ . Why is the MLE not asymptotically normal in the sense of theorem 17.1? Why does this make sense in the context of the super-efficiency of the uniform model from lecture 13?