Lecture 17 Asymptotic Normality and Efficiency of the MLE ‣ MATH5703M Statistical Theory

17.1 The Theorem

As in lecture 11, we write $\mathcal{I}(\theta)$ and $\mathcal{I}_{n}(\theta)=n\mathcal{I}(\theta)$ for the Fisher information of one observation and of the sample, and as in lecture 16 we write $\theta_{0}$ for the true parameter value. Since the error $\hat{\theta}_{n}-\theta_{0}$ converges to zero, we need to magnify it to see the shape of its distribution. We use the factor $\sqrt{n}$ , which for the sample mean gives the central limit theorem, and thus we study the scaled error $\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})$ .

Theorem 17.1.

Suppose that the regularity conditions (R1) to (R4) from definition 16.1 are satisfied, that $\log\bigl{(}f(X_{1};\theta)/f(X_{1};\theta^{\prime})\bigr{)}$ has finite expectation under $\mathbb{P}_{\theta^{\prime}}$ for all $\theta,\theta^{\prime}\in\Theta$ and that for every $\theta_{0}\in\Theta$ the likelihood equation $\ell^{\prime}(\theta)=0$ has, with $\mathbb{P}_{\theta_{0}}$ -probability tending to one as $n\to\infty$ , a unique root $\hat{\theta}_{n}$ , which is the maximum likelihood estimator. Then for every $\theta_{0}\in\Theta$ we have, under $\mathbb{P}_{\theta_{0}}$ ,

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{d}\ }N\Bigl{(}0,% \frac{1}{\mathcal{I}(\theta_{0})}\Bigr{)}\qquad(n\to\infty).

The assumptions are those of theorem 16.3, in the weaker form of the remark after its proof, plus condition (R4). Informally, the theorem states that, as $n$ gets large,

\hat{\theta}_{n}\approx N\Bigl{(}\theta_{0},\frac{1}{n\mathcal{I}(\theta_{0})}% \Bigr{)}=N\Bigl{(}\theta_{0},\frac{1}{\mathcal{I}_{n}(\theta_{0})}\Bigr{)}.

This statement contains three different aspects: (1) the approximating distribution is centred at the true value, (2) the spread of the distribution decreases as $1/\sqrt{n}$ , as for the sample mean, and (3) the variance of the approximating distribution equals the Cramér–Rao lower bound $1/\mathcal{I}_{n}(\theta_{0})$ from lecture 13. The bound concerns unbiased estimators, while the MLE is often biased, but example 17.4 shows that its bias disappears in the limit. Property (3) is called asymptotic efficiency, and the definition applies to any estimator.

Definition 17.2.

An estimator $\tilde{\theta}_{n}$ for $\theta$ is asymptotically efficient, if $\sqrt{n}\,(\tilde{\theta}_{n}-\theta)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,1/% \mathcal{I}(\theta)\bigr{)}$ under $\mathbb{P}_{\theta}$ , for all $\theta\in\Theta$ .

In this language theorem 17.1 states that under its assumptions the MLE is asymptotically efficient.

17.2 Proof of the Theorem

The idea of the proof is easiest to see in a first-order approximation. Since the MLE is a root of the score, Taylor expansion of $\ell^{\prime}$ around the true value gives

0=\ell^{\prime}(\hat{\theta}_{n})\approx\ell^{\prime}(\theta_{0})+\ell^{\prime% \prime}(\theta_{0})\,(\hat{\theta}_{n}-\theta_{0}),

and solving for the scaled error we find

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\approx\frac{\ell^{\prime}(\theta_{0})/% \sqrt{n}}{-\ell^{\prime\prime}(\theta_{0})/n}.

The numerator is a scaled mean of the i.i.d. terms $\ell_{i}^{\prime}(\theta_{0})$ , which by (16.1) and (R3) have mean zero and finite, positive variance $\mathcal{I}(\theta_{0})$ . Thus the central limit theorem, theorem A.15, gives

equation (17.1) (17.1)

A_{n}=\frac{\ell^{\prime}(\theta_{0})}{\sqrt{n}}=\sqrt{n}\,\Bigl{(}\frac{1}{n}% \sum_{i=1}^{n}\ell_{i}^{\prime}(\theta_{0})-0\Bigr{)}\xrightarrow{\ \mathrm{d}% \ }N\bigl{(}0,\mathcal{I}(\theta_{0})\bigr{)}.

The denominator is the mean of the i.i.d. terms $-\ell_{i}^{\prime\prime}(\theta_{0})$ , which by (16.1) have finite mean $\mathcal{I}(\theta_{0})$ , and thus the law of large numbers, proposition A.14, gives

equation (17.2) (17.2)

B_{n}=-\frac{\ell^{\prime\prime}(\theta_{0})}{n}=\frac{1}{n}\sum_{i=1}^{n}% \bigl{(}-\ell_{i}^{\prime\prime}(\theta_{0})\bigr{)}\xrightarrow{\ \mathrm{p}% \ }\mathcal{I}(\theta_{0}).

The rigorous proof follows this outline, and most of the work goes into showing that the remainder of the expansion is negligible.

Proof of theorem 17.1.

Let the true value be $\theta_{0}$ , let $\delta>0$ be as in (R4), and let $E_{n}$ be the event that the likelihood equation has a unique root, that this root is the MLE $\hat{\theta}_{n}$ and that $|\hat{\theta}_{n}-\theta_{0}|<\delta$ . By the assumptions of the theorem, and since $\hat{\theta}_{n}$ is consistent by theorem 16.3 and the remark after its proof, we have $\mathbb{P}_{\theta_{0}}(E_{n})\to 1$ .

On the event $E_{n}$ we expand the score around $\theta_{0}$ . By (R3), the function $\ell^{\prime}$ is twice continuously differentiable. Thus, by Taylor’s theorem with the Lagrange form of the remainder, we get

0=\ell^{\prime}(\hat{\theta}_{n})=\ell^{\prime}(\theta_{0})+\ell^{\prime\prime% }(\theta_{0})\,(\hat{\theta}_{n}-\theta_{0})+\tfrac{1}{2}\,\ell^{\prime\prime% \prime}(\xi_{n})\,(\hat{\theta}_{n}-\theta_{0})^{2},

where $\xi_{n}$ is a point between $\theta_{0}$ and $\hat{\theta}_{n}$ . Outside the event $E_{n}$ we set $\xi_{n}=\theta_{0}$ . We define

C_{n}=\frac{1}{2n}\,\ell^{\prime\prime\prime}(\xi_{n})\,(\hat{\theta}_{n}-% \theta_{0}),

so that, like the terms $A_{n}$ and $B_{n}$ from (17.1) and (17.2), the term $C_{n}$ is defined for every sample. Dividing the expansion by $\sqrt{n}$ and rearranging, we find that on $E_{n}$ we have

A_{n}=\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\,(B_{n}-C_{n}).

To avoid division by zero, let $D_{n}=B_{n}-C_{n}$ if $B_{n}\neq C_{n}$ and $D_{n}=\mathcal{I}(\theta_{0})$ otherwise. Then, on the event $F_{n}$ that $E_{n}$ occurs and $B_{n}\neq C_{n}$ , we can write the scaled error as a ratio:

equation (17.3) (17.3)

\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})=\frac{A_{n}}{D_{n}}.

We have already seen the limits of $A_{n}$ and $B_{n}$ in (17.1) and (17.2), and it remains to control the remainder term $C_{n}$ , for which we use consistency and condition (R4). Let $M$ be the function from (R4). On the event $E_{n}$ , the point $\xi_{n}$ lies between $\theta_{0}$ and $\hat{\theta}_{n}$ , and thus within $\delta$ of $\theta_{0}$ . Writing $\ell^{\prime\prime\prime}(\xi_{n})=\sum_{i}\ell_{i}^{\prime\prime\prime}(\xi_{% n})$ and using (R4) for each term, we find

\Bigl{|}\frac{1}{n}\,\ell^{\prime\prime\prime}(\xi_{n})\Bigr{|}\leq\frac{1}{n}% \sum_{i=1}^{n}\bigl{|}\ell_{i}^{\prime\prime\prime}(\xi_{n})\bigr{|}\leq\frac{% 1}{n}\sum_{i=1}^{n}M(X_{i})=\bar{M}_{n}

on this event, where $\bar{M}_{n}$ denotes the mean of $M(X_{1}),\dots,M(X_{n})$ . Let $m=\mathbb{E}_{\theta_{0}}\bigl{(}M(X_{1})\bigr{)}<\infty$ and $K=m+1$ . If $|\ell^{\prime\prime\prime}(\xi_{n})|/n>K$ , then either $E_{n}$ does not occur, or $\bar{M}_{n}>m+1$ and thus $|\bar{M}_{n}-m|\geq 1$ . Since $\mathbb{P}_{\theta_{0}}(E_{n})\to 1$ and since $\bar{M}_{n}\xrightarrow{\ \mathrm{p}\ }m$ by the law of large numbers, we find

\mathbb{P}_{\theta_{0}}\Bigl{(}\frac{|\ell^{\prime\prime\prime}(\xi_{n})|}{n}>% K\Bigr{)}\leq\mathbb{P}_{\theta_{0}}(E_{n}^{\mathrm{c}})+\mathbb{P}_{\theta_{0% }}\bigl{(}|\bar{M}_{n}-m|\geq 1\bigr{)}\longrightarrow 0.

The other factor of $C_{n}$ is $\hat{\theta}_{n}-\theta_{0}$ , which converges in probability to zero, as we have seen at the start of the proof. Let $\eta>0$ . Since $|C_{n}|=|\ell^{\prime\prime\prime}(\xi_{n})|\,|\hat{\theta}_{n}-\theta_{0}|/(2n)$ , the event $|C_{n}|\geq\eta$ is contained in the union of the events $|\ell^{\prime\prime\prime}(\xi_{n})|/n>K$ and $|\hat{\theta}_{n}-\theta_{0}|\geq 2\eta/K$ . Both of these events have probability tending to zero, and thus

equation (17.4) (17.4)

C_{n}\xrightarrow{\ \mathrm{p}\ }0.

It remains to combine the three limits (17.1), (17.2) and (17.4) using Slutsky’s theorem, proposition A.16. By (17.2) and (17.4), the denominator satisfies $B_{n}-C_{n}\xrightarrow{\ \mathrm{p}\ }\mathcal{I}(\theta_{0})>0$ . Since $|B_{n}-C_{n}-\mathcal{I}(\theta_{0})|=\mathcal{I}(\theta_{0})$ on the event $B_{n}=C_{n}$ , this event has probability tending to zero, and since also $\mathbb{P}_{\theta_{0}}(E_{n})\to 1$ , we find $\mathbb{P}_{\theta_{0}}(F_{n})\to 1$ . Since $|D_{n}-\mathcal{I}(\theta_{0})|\leq|B_{n}-C_{n}-\mathcal{I}(\theta_{0})|$ for every sample, we have $D_{n}\xrightarrow{\ \mathrm{p}\ }\mathcal{I}(\theta_{0})$ . Let $Z\sim N\bigl{(}0,\mathcal{I}(\theta_{0})\bigr{)}$ . By the quotient part of Slutsky’s theorem we have

\frac{A_{n}}{D_{n}}\xrightarrow{\ \mathrm{d}\ }\frac{Z}{\mathcal{I}(\theta_{0}% )}\sim N\Bigl{(}0,\frac{1}{\mathcal{I}(\theta_{0})}\Bigr{)},

since $Z/\mathcal{I}(\theta_{0})$ has variance $\mathcal{I}(\theta_{0})/\mathcal{I}(\theta_{0})^{2}$ . Finally, by (17.3) the scaled error coincides with this ratio on the event $F_{n}$ . Thus, for every $\varepsilon>0$ , the event $\bigl{|}\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})-A_{n}/D_{n}\bigr{|}\geq\varepsilon$ is contained in $F_{n}^{\mathrm{c}}$ , and since $\mathbb{P}_{\theta_{0}}(F_{n})\to 1$ the difference converges in probability to zero. Slutsky’s theorem now shows that the scaled error has the same limit as $A_{n}/D_{n}$ . This completes the proof. ∎

Exercise 17.1 carries out these steps for the exponential distribution.

17.3 Standard Errors

The variance $1/\mathcal{I}(\theta_{0})$ in theorem 17.1 depends on the unknown $\theta_{0}$ . The next result shows that we may replace $\theta_{0}$ by its estimate.

Corollary 17.3.

Under the assumptions of theorem 17.1 we have

\sqrt{n\mathcal{I}(\hat{\theta}_{n})}\,(\hat{\theta}_{n}-\theta_{0})% \xrightarrow{\ \mathrm{d}\ }N(0,1)\qquad(n\to\infty).

Proof.

By (R3) the function $\mathcal{I}$ is continuous and positive on $\Theta$ , and thus the function $\theta\mapsto\sqrt{\mathcal{I}(\theta)/\mathcal{I}(\theta_{0})}$ is continuous at $\theta_{0}$ . Since $\hat{\theta}_{n}\xrightarrow{\ \mathrm{p}\ }\theta_{0}$ by theorem 16.3 and the remark after its proof, the continuous mapping theorem, proposition A.17, implies that $\sqrt{\mathcal{I}(\hat{\theta}_{n})/\mathcal{I}(\theta_{0})}\xrightarrow{\ % \mathrm{p}\ }1$ . Multiplying the result of theorem 17.1 by the constant $\sqrt{\mathcal{I}(\theta_{0})}$ , which is allowed by the continuous mapping theorem, gives $\sqrt{n\mathcal{I}(\theta_{0})}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ % \mathrm{d}\ }Z$ where $Z\sim N(0,1)$ . Using the product part of Slutsky’s theorem, proposition A.16, we find

\sqrt{n\mathcal{I}(\hat{\theta}_{n})}\,(\hat{\theta}_{n}-\theta_{0})=\sqrt{% \frac{\mathcal{I}(\hat{\theta}_{n})}{\mathcal{I}(\theta_{0})}}\cdot\sqrt{n% \mathcal{I}(\theta_{0})}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{% d}\ }1\cdot Z=Z.

This completes the proof. ∎

The standard error $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=1/\sqrt{\mathcal{I}_{n}(\hat{% \theta}_{n})}$ of lecture 12 is now justified by the corollary: for large $n$ , the distribution of $\hat{\theta}_{n}$ is approximately normal with mean $\theta_{0}$ and standard deviation $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ . This is a statement about distributions, not about moments: the MLE for the log-odds in exercise 17.2 does not exist when all observations are equal, an event of positive probability for every $n$ , and thus it has no variance.

The corollary also justifies the Wald interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ from lecture 12, whose coverage probability tends to $0.95$ , as we will show in lecture 26.

If we do not have a closed formula for $\mathcal{I}$ , the denominator $\mathcal{I}_{n}(\hat{\theta}_{n})$ can be replaced by the observed information $-\ell^{\prime\prime}(\hat{\theta}_{n})$ , the curvature of the log-likelihood at the maximum. Expanding $\ell^{\prime\prime}$ around $\theta_{0}$ gives the term $B_{n}$ from the proof plus a remainder which we can handle like $C_{n}$ , and we find

-\frac{1}{n}\,\ell^{\prime\prime}(\hat{\theta}_{n})\xrightarrow{\ \mathrm{p}\ % }\mathcal{I}(\theta_{0}).

By Slutsky’s theorem the corollary thus remains true with this replacement. This is the form we will use with a numerical optimiser in lecture 18.

17.4 Examples

The Bernoulli and Poisson MLEs from lecture 5 are sample means, and for them theorem 17.1 only restates the central limit theorem (exercises 17.2 and 17.3). The first new case is an MLE which is not a sample mean.

Example 17.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. with exponential rate $\theta$ . Then the MLE for $\theta$ is $\hat{\theta}_{n}=1/\bar{X}$ from example 5.2, and by exercise 16.1 all assumptions of theorem 17.1 hold for this model. From example 11.8 we know that the Fisher information is given by $\mathcal{I}(\theta)=1/\theta^{2}$ . Thus, by theorem 17.1,

\sqrt{n}\,\Bigl{(}\frac{1}{\bar{X}}-\theta\Bigr{)}\xrightarrow{\ \mathrm{d}\ }% N(0,\theta^{2}),

with standard error $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=\hat{\theta}_{n}/\sqrt{n}$ , and the approximate $95\%$ Wald interval for $\theta$ is $\hat{\theta}_{n}\pm 1.96\,\hat{\theta}_{n}/\sqrt{n}$ .

To compare with exact results, recall that in lecture 5 we found $\mathbb{E}_{\theta}(\hat{\theta}_{n})=n\theta/(n-1)$ , which is a bias of $\theta/(n-1)$ . In exercise 5.3 we found the variance $\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta}_{n})=n^{2}\theta^{2}/% \bigl{(}(n-1)^{2}(n-2)\bigr{)}$ . Multiplied by $n$ , this variance converges to $\theta^{2}$ , and multiplied by $\sqrt{n}$ , the bias converges to zero: the bias is of smaller order than the spread, and thus disappears in the limit.

We can also treat functions of the parameter. The mean of the exponential distribution is $\mu=g(\theta)=1/\theta$ , where $g$ is a bijection of $(0,\infty)$ , and by proposition 5.6 the MLE for this parameter is $\hat{\mu}_{n}=1/\hat{\theta}_{n}=\bar{X}$ . Using the delta method, proposition A.18 with $g^{\prime}(\theta)=-1/\theta^{2}$ , we get

\sqrt{n}\,(\bar{X}-\mu)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,\ g^{\prime}(% \theta)^{2}\,\theta^{2}\bigr{)}=N\Bigl{(}0,\frac{1}{\theta^{2}}\Bigr{)}=N(0,% \mu^{2}).

This is the central limit theorem for $\bar{X}$ , and it is also what theorem 17.1 gives in the parametrisation by $\mu$ , where the information is $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ by proposition 11.9. Replacing $\theta$ by $\hat{\theta}_{n}$ , we find that the standard error of $\bar{X}$ is

|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n}% )=\frac{1}{\hat{\theta}_{n}^{2}}\cdot\frac{\hat{\theta}_{n}}{\sqrt{n}}=\frac{% \bar{X}}{\sqrt{n}}.

The same works for every twice continuously differentiable bijection $g$ with $g^{\prime}(\theta)\neq 0$ : the MLE $g(\hat{\theta}_{n})$ has standard error $|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ , and since its asymptotic variance $g^{\prime}(\theta)^{2}/\mathcal{I}(\theta)$ is the reciprocal of the information for $g(\theta)$ , it is again asymptotically efficient. Thus asymptotic efficiency holds much more widely than the exact efficiency of lecture 13: by exercise 13.3 there is no efficient estimator for the probability $e^{-\theta}$ of observing zero in a Poisson sample, but by exercise 17.3 its MLE $e^{-\bar{X}}$ is asymptotically efficient. The conditions on $g$ matter: the Bernoulli variance $\theta(1-\theta)$ has derivative zero at $\theta=1/2$ , where the delta method does not apply.

17.5 Quality of the Approximation

Theorem 17.1 is a limit statement and does not say how large $n$ must be. For small samples the estimator can be skewed: for the exponential rate, $1/\bar{X}$ is right-skewed, and its bias $\theta/(n-1)$ is visible for $n$ around $10$ . Near the boundary of the parameter space the approximation is poor: for the Poisson distribution with $\theta_{0}$ close to zero, all observations are zero with appreciable probability, and then $\bar{X}=0$ lies outside $\Theta=(0,\infty)$ and the Wald interval has zero width. Finally, without the regularity conditions the limit need not be normal: for the uniform distribution, where (R2) fails, the error is of order $1/n$ , and exercise 17.5 shows that $n(\theta-\hat{\theta}_{n})$ converges in distribution to an exponential distribution.

17.6 Vector Parameters

We conclude with the extension of theorem 17.1 to a vector parameter $\theta=(\theta_{1},\dots,\theta_{k})$ , which we state without proof. Under appropriate regularity conditions, the MLE is approximately normal with mean $\theta_{0}$ and covariance matrix $\mathcal{I}_{n}(\theta_{0})^{-1}$ , the inverse of the information matrix from lecture 14. For the normal distribution with unknown mean and variance, the inverse matrix is $\mathrm{diag}(\sigma^{2}/n,\,2\sigma^{4}/n)$ , and thus $\bar{X}$ and the MLE $\hat{\sigma}^{2}$ are approximately normal with variances $\sigma^{2}/n$ and $2\sigma^{4}/n$ , and they are asymptotically independent.

Summary.

•

Under regularity conditions and assuming the uniqueness of the likelihood root, we have $\sqrt{n}\,(\hat{\theta}_{n}-\theta_{0})\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,% 1/\mathcal{I}(\theta_{0})\bigr{)}$ : the limit is centred at $\theta_{0}$ , the error is of order $1/\sqrt{n}$ and the asymptotic variance equals the Cramér–Rao bound, i.e. the MLE is asymptotically efficient.
•

The proof uses Taylor expansion of the score around $\theta_{0}$ , the central limit theorem for the score, the law of large numbers for the second derivative, consistency and (R4) for the remainder, and Slutsky’s theorem to combine these results. Events of vanishing probability do not affect the limit.
•

Replacing $\theta_{0}$ by $\hat{\theta}_{n}$ in the expression for the variance does not change the limit, and thus we get the standard error $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=1/\sqrt{\mathcal{I}_{n}(\hat{% \theta}_{n})}$ and the approximate $95\%$ Wald interval $\hat{\theta}_{n}\pm 1.96\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ for $\theta_{0}$ . The observed information $-\ell^{\prime\prime}(\hat{\theta}_{n})$ may be used in place of $\mathcal{I}_{n}(\hat{\theta}_{n})$ .
•

Using the delta method, we can apply the result to a twice continuously differentiable bijection $g$ of the parameter with $g^{\prime}(\theta)\neq 0$ . The standard error for such an application is $|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ .
•

The approximation can be poor for small $n$ , especially when the estimator is skewed or $\theta_{0}$ is near the boundary of the parameter space, and it breaks down completely when the regularity conditions are violated, e.g. for the uniform distribution.
•

For a vector parameter, the MLE is approximately normal with covariance matrix $\mathcal{I}_{n}(\theta_{0})^{-1}$ .

Exercise 17.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponentially distributed with rate $\theta_{0}>0$ , and let $\hat{\theta}_{n}=1/\bar{X}$ be the MLE. In this exercise we follow the proof of theorem 17.1 for this model.

1.

Find $\ell^{\prime}(\theta)$ , $\ell^{\prime\prime}(\theta)$ and $\ell^{\prime\prime\prime}(\theta)$ , and express the terms $A_{n}$ , $B_{n}$ and $C_{n}$ from the proof in terms of $\bar{X}$ , $\theta_{0}$ , $\hat{\theta}_{n}$ and the intermediate point $\xi_{n}$ .
2.

Use the central limit theorem to show directly that $A_{n}\xrightarrow{\ \mathrm{d}\ }N(0,1/\theta_{0}^{2})$ , and note that $B_{n}$ is not random.
3.

Without using the bound $M$ from condition (R4), show that $C_{n}\xrightarrow{\ \mathrm{p}\ }0$ . Hint: on the event $|\hat{\theta}_{n}-\theta_{0}|<\theta_{0}/2$ we have $\xi_{n}>\theta_{0}/2$ .

Exercise 17.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , where the MLE $\hat{\theta}_{n}=\bar{X}$ and the Fisher information $\mathcal{I}(\theta)=1/\bigl{(}\theta(1-\theta)\bigr{)}$ are given in examples 5.3 and 11.5.

1.

State the result of theorem 17.1 for this model and verify that it coincides with the central limit theorem for $\bar{X}$ .
2.

The log-odds are given by $\psi=g(\theta)=\log\bigl{(}\theta/(1-\theta)\bigr{)}$ . Find the MLE for $\psi$ on the event $0<\bar{X}<1$ , and use the delta method to show that $\sqrt{n}\,(\hat{\psi}_{n}-\psi)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,1/(% \theta(1-\theta))\bigr{)}$ . Hint: the event $0<\bar{X}<1$ has probability tending to one, and thus, as at the end of the proof of theorem 17.1, the value of $\hat{\psi}_{n}$ outside this event does not affect the limit.
3.

Determine the standard error of $\hat{\psi}_{n}$ and the approximate $95\%$ Wald interval for $\psi$ , in terms of $\hat{\theta}_{n}$ and $n$ .

Exercise 17.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with parameter $\theta>0$ , where the MLE $\hat{\theta}_{n}=\bar{X}$ and the Fisher information $\mathcal{I}(\theta)=1/\theta$ are given in exercise 5.7 and example 11.6.

1.

State the result of theorem 17.1 for this model and verify that the result coincides with the central limit theorem.
2.

In exercise 5.7 we have found the MLE $\hat{p}_{0}=e^{-\bar{X}}$ for the probability $p_{0}=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ and we have seen that this MLE is biased. Using the delta method, determine the asymptotic distribution of $\sqrt{n}\,(\hat{p}_{0}-p_{0})$ . Why does the bias found in exercise 5.7 not contradict the result from this part?
3.

Show that the asymptotic variance from the previous part equals $n$ times the Cramér–Rao bound for unbiased estimators for $e^{-\theta}$ . Compare this with exercise 13.3, where we have seen that no efficient estimator for $e^{-\theta}$ exists.

Exercise 17.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. from the geometric distribution with parameter $p\in(0,1)$ , where the MLE $\hat{p}_{n}=1/\bar{X}$ and the Fisher information $\mathcal{I}(p)=1/\bigl{(}p^{2}(1-p)\bigr{)}$ are given in exercises 5.1 and 11.4. You may use that the assumptions of theorem 17.1 hold for this model.

1.

State the result of theorem 17.1 for this model.
2.

Derive the same result from the central limit theorem for $\bar{X}$ and the delta method, using $\mathbb{E}_{p}(X_{1})=1/p$ and $\mathop{\mathrm{Var}}\nolimits_{p}(X_{1})=(1-p)/p^{2}$ .
3.

A sample of size $n=40$ has $\bar{x}=2.5$ . Compute the estimate $\hat{p}_{n}$ , its standard error and the approximate $95\%$ Wald interval for $p$ .

Exercise 17.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the interval $(0,\theta)$ . The MLE for this model is $\hat{\theta}_{n}=\max_{i}X_{i}$ .

1.

For every $x\geq 0$ , show that $\mathbb{P}_{\theta}\bigl{(}n(\theta-\hat{\theta}_{n})>x\bigr{)}\to e^{-x/\theta}$ as $n\to\infty$ . Deduce that $n(\theta-\hat{\theta}_{n})$ converges in distribution to the exponential distribution with rate $1/\theta$ , i.e. with mean $\theta$ .
2.

Show that $\sqrt{n}\,(\hat{\theta}_{n}-\theta)\xrightarrow{\ \mathrm{p}\ }0$ . Why is the MLE not asymptotically normal in the sense of theorem 17.1, and which of the regularity conditions fails? How does this relate to the super-efficiency of the uniform model from lecture 13?