Lecture 16 Consistency of the MLE

In lecture 5 we have seen that the maximum likelihood estimator can be biased, that it may not be unique and that it may not even exist. The following two lectures present the results which, despite these problems, allow us to justify the use of the MLE: for large sample size the maximum likelihood estimator converges to the true parameter value, and does so with minimal spread. In this lecture we will prove the first of these two results, the consistency of the MLE, from a set of assumptions which we will explicitly state. The proof of the asymptotic normality in lecture 17 builds on these same assumptions and on the consistency result. The numerical aspects, concerning estimates and standard errors, will be discussed in lecture 18.

16.1 The Regularity Conditions

Throughout this and the following lectures, $X_{1},\dots,X_{n}$ are i.i.d. with density or probability weights $f(x;\theta)$ , the parameter $\theta$ is one-dimensional and the parameter space $\Theta\subseteq\mathbb{R}$ is an open interval. As in lecture 11, the log-likelihood is the sum $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$ of the terms $\ell_{i}(\theta)=\log f(X_{i};\theta)$ , and for random data the $\ell_{i}(\theta)$ are i.i.d. random variables. We write $\theta_{0}$ for the true parameter value, and $\mathbb{P}_{\theta_{0}}$ and $\mathbb{E}_{\theta_{0}}$ for probabilities and expectations under this value.

The large-sample results are true for most models of interest, but not for all of them. The following assumptions are what the theorems require.

Definition 16.1.

The model $\bigl{(}f(x;\theta)\!\mathrel{\big{|}}\!\theta\in\Theta\bigr{)}$ satisfies the regularity conditions, if the following four statements hold.

(R1)

The parameter is identifiable in the sense of definition 4.10, i.e. different values of $\theta$ lead to different distributions.
(R2)

The support $\{x\mid f(x;\theta)>0\}$ does not depend on $\theta$ .
(R3)

The conditions of section 11.1 hold, for every $x$ the function $\theta\mapsto f(x;\theta)$ is three times continuously differentiable on $\Theta$ , and the Fisher information $\mathcal{I}(\theta)$ of a single observation is a continuous function of $\theta$ with $\mathcal{I}(\theta)>0$ for all $\theta\in\Theta$ .
(R4)

For every $\theta_{0}\in\Theta$ there are a $\delta>0$ and a function $M(x)$ such that $\mathbb{E}_{\theta_{0}}\bigl{(}M(X_{1})\bigr{)}<\infty$ and

$\Bigl{|}\frac{\partial^{3}}{\partial\theta^{3}}\log f(x;\theta)\Bigr{|}\leq M(% x)\qquad\text{for all $x$ and all $\theta$ with $|\theta-\theta_{0}|<\delta$.}$

The four conditions play different roles. Condition (R1) is needed for the question to make sense: if two parameter values give the same distribution, no estimator can converge to both. Condition (R2) excludes models like the uniform distribution from example 5.5, where the parameter moves the edge of the support and the likelihood jumps to zero. The arguments of this lecture fail for such models, as we will see in section 16.4. Condition (R3) is the smoothness needed to differentiate the log-likelihood. Since it includes the conditions of section 11.1, it also makes the two identities of lecture 11 available (lemma 11.1 and lemma 11.3),

equation (16.1) (16.1)

\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}=0\qquad\text{and}% \qquad\mathcal{I}(\theta)=\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_% {i}^{\prime}(\theta)\bigr{)}=-\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime% \prime}(\theta)\bigr{)}.

Condition (R4), and the parts of (R3) beyond the differentiability of the likelihood, are not needed in this lecture. Condition (R4) controls the remainder term in the Taylor expansion in the proof of asymptotic normality in lecture 17, and it is listed here so that both lectures work from the same definition. Although the condition looks technical, it is usually easy to verify, because the third derivative of the log-likelihood is an explicit function of $x$ and $\theta$ ; exercise 16.1 does this for the exponential distribution.

The one-parameter models from tables 1.1 and 1.2, and the two-parameter models with one parameter held fixed (with $m$ known for the binomial distribution), satisfy the regularity conditions, with the exception of the uniform distribution. The same is true for the $t$ location model from section 5.3, despite the fact that the likelihood equation may have more than one solution. We will return to this problem in section 16.4.

16.2 Comparing the True Value with a Wrong Value

If $\theta_{0}$ is the true value, it seems natural that for a large sample the likelihood $L(\theta_{0})$ should be larger than $L(\theta)$ for each fixed wrong value $\theta$ . This is the main step in the consistency proof, and thus we prove it first.

Lemma 16.2.

Suppose that (R1) and (R2) hold. Let $\theta\neq\theta_{0}$ and assume that $\log\bigl{(}f(X_{1};\theta)/f(X_{1};\theta_{0})\bigr{)}$ has finite expectation under $\mathbb{P}_{\theta_{0}}$ . Then

\lim_{n\to\infty}\mathbb{P}_{\theta_{0}}\bigl{(}L(\theta_{0})>L(\theta)\bigr{)% }=1.

Proof.

Let $Z_{i}=f(X_{i};\theta)/f(X_{i};\theta_{0})$ for $i=1,\dots,n$ , where, by (R2), the ratios are well defined and positive for all observations. Using logarithms and dividing by $n$ , we can rewrite the event $L(\theta_{0})>L(\theta)$ as

\frac{1}{n}\sum_{i=1}^{n}\log Z_{i}<0.

The left-hand side is the sample mean of the i.i.d. random variables $\log Z_{i}$ and by the law of large numbers, proposition A.14, converges in probability to $\mu=\mathbb{E}_{\theta_{0}}(\log Z_{1})$ . Therefore, we only need to show that $\mu$ is strictly negative: then on the event that the sample mean is non-negative, it differs from $\mu$ by at least $|\mu|$ , and by definition A.12 the probability of this event tends to zero.

To study the expectation $\mu$ , let $Z=Z_{1}$ , so that $\mu=\mathbb{E}_{\theta_{0}}(\log Z)$ . Writing the expectation of $Z$ as an integral over the common support, we can cancel the density $f(x;\theta_{0})$ and get

\mathbb{E}_{\theta_{0}}(Z)=\int\frac{f(x;\theta)}{f(x;\theta_{0})}\,f(x;\theta% _{0})\,\mathrm{d}x=\int f(x;\theta)\,\mathrm{d}x=1,

with a sum instead of an integral for a discrete model. Since $Z$ has finite mean and the function $g(z)=-\log z$ is strictly convex on $(0,\infty)$ , we can use Jensen’s inequality, lemma A.7, to get

-\mu=\mathbb{E}_{\theta_{0}}\bigl{(}-\log Z\bigr{)}\geq-\log\mathbb{E}_{\theta% _{0}}(Z)=-\log 1=0,

and the inequality is strict unless $Z$ is constant with probability one. If $Z$ were constant, we would have $f(x;\theta)=c\,f(x;\theta_{0})$ on the common support and, since both functions have integral one, this would imply $c=1$ , i.e. the two distributions would be the same, which (R1) excludes for $\theta\neq\theta_{0}$ . Thus the inequality is strict and we have $\mu<0$ . This completes the proof. ∎

The quantity $-\mathbb{E}_{\theta_{0}}(\log Z)=\mathbb{E}_{\theta_{0}}\bigl{(}\log(f(X_{1};% \theta_{0})/f(X_{1};\theta))\bigr{)}$ from the proof is called the Kullback–Leibler divergence. The proof shows that, on average, each observation increases the log-likelihood of the truth by this positive amount, compared to the log-likelihood of the competitor $\theta$ ; the amount depends on $\theta$ . Note that the lemma compares $\theta_{0}$ with one fixed value $\theta$ at a time. It does not state that $L(\theta_{0})$ eventually exceeds $L(\theta)$ for all $\theta\neq\theta_{0}$ at once, and it says nothing directly about the maximiser of the likelihood. In the next section we apply the lemma to the two fixed values $\theta_{0}\pm\varepsilon$ and turn the result into a statement about the MLE.

16.3 Consistency

From definition 2.9 we know that an estimator $\hat{\theta}_{n}$ is consistent for $\theta$ , if $\hat{\theta}_{n}\xrightarrow{\ \mathrm{p}\ }\theta$ for all possible true values. In lecture 2 we have already seen how to prove consistency using the mean squared error. For the maximum likelihood estimator this approach is not feasible in general, because there is no general formula for the mean squared error. Instead, we will use the likelihood equation from lecture 5 to show that with probability tending to one there is a root of the equation which is close to $\theta_{0}$ . If we assume that the root is unique and is the MLE, it follows that the MLE is close to $\theta_{0}$ .

Theorem 16.3.

Suppose that the regularity conditions (R1), (R2) and (R3) hold and that, in addition, the following two statements hold.

1.

For all $\theta,\theta^{\prime}\in\Theta$ the log-likelihood ratio $\log\bigl{(}f(X_{1};\theta)/f(X_{1};\theta^{\prime})\bigr{)}$ has finite expectation under $\mathbb{P}_{\theta^{\prime}}$ .
2.

For every sample size $n\geq 1$ and every sample whose observations lie in the common support from (R2), the likelihood equation $\ell^{\prime}(\theta)=0$ has a unique root, and this root is the maximum likelihood estimator $\hat{\theta}_{n}$ .

Then $\hat{\theta}_{n}$ is a consistent estimator for $\theta$ .

Proof.

Let the true value be $\theta_{0}$ and let $\varepsilon>0$ be small enough that the closed interval $[\theta_{0}-\varepsilon,\theta_{0}+\varepsilon]$ is contained in the open set $\Theta$ . Let $\theta_{1}=\theta_{0}-\varepsilon$ and $\theta_{2}=\theta_{0}+\varepsilon$ and consider the event

S_{\varepsilon}=\bigl{\{}L(\theta_{1})<L(\theta_{0})\text{ and }L(\theta_{2})<% L(\theta_{0})\bigr{\}}.

By the assumptions of the theorem, lemma 16.2 applies to both $\theta_{1}$ and $\theta_{2}$ , and thus the two events $L(\theta_{1})\geq L(\theta_{0})$ and $L(\theta_{2})\geq L(\theta_{0})$ have probability tending to zero. Since the complement of $S_{\varepsilon}$ is the union of these two events, we find

\mathbb{P}_{\theta_{0}}(S_{\varepsilon})\geq 1-\mathbb{P}_{\theta_{0}}\bigl{(}% L(\theta_{1})\geq L(\theta_{0})\bigr{)}-\mathbb{P}_{\theta_{0}}\bigl{(}L(% \theta_{2})\geq L(\theta_{0})\bigr{)}\longrightarrow 1

as $n\to\infty$ .

We now show that on the event $S_{\varepsilon}$ the MLE is within $\varepsilon$ of $\theta_{0}$ . Assume that the sample lies in $S_{\varepsilon}$ . By (R3) the likelihood $L$ is a continuous function of $\theta$ , and a continuous function on a closed, bounded interval attains its maximum. Thus there is a point $\theta^{*}\in[\theta_{1},\theta_{2}]$ with $L(\theta^{*})\geq L(\theta)$ for all $\theta\in[\theta_{1},\theta_{2}]$ . Since $L(\theta_{0})$ is larger than both $L(\theta_{1})$ and $L(\theta_{2})$ , the maximum is not attained at either end of the interval, i.e. we have $\theta^{*}\in(\theta_{1},\theta_{2})$ . The function $L$ is differentiable and has a maximum at the interior point $\theta^{*}$ , and thus $L^{\prime}(\theta^{*})=0$ . Since the observations lie in the common support, we have $L>0$ , and thus $\theta^{*}$ is also a root of the likelihood equation $\ell^{\prime}(\theta)=L^{\prime}(\theta)/L(\theta)=0$ . By assumption, this root is unique and equals the MLE, and thus $\hat{\theta}_{n}=\theta^{*}$ . Since $\theta^{*}$ is in $(\theta_{1},\theta_{2})$ , we have shown that

S_{\varepsilon}\subseteq\bigl{\{}|\hat{\theta}_{n}-\theta_{0}|<\varepsilon% \bigr{\}}.

Taking probabilities we find $\mathbb{P}_{\theta_{0}}\bigl{(}|\hat{\theta}_{n}-\theta_{0}|<\varepsilon\bigr{% )}\geq\mathbb{P}_{\theta_{0}}(S_{\varepsilon})\to 1$ , i.e. $\mathbb{P}_{\theta_{0}}\bigl{(}|\hat{\theta}_{n}-\theta_{0}|\geq\varepsilon% \bigr{)}\to 0$ as $n\to\infty$ . At the start of the proof we restricted $\varepsilon$ to small values, but this is no loss, since the probability $\mathbb{P}_{\theta_{0}}\bigl{(}|\hat{\theta}_{n}-\theta_{0}|\geq\varepsilon% \bigr{)}$ is non-increasing in $\varepsilon$ . Since $\theta_{0}$ was an arbitrary point of $\Theta$ , this is the statement $\hat{\theta}_{n}\xrightarrow{\ \mathrm{p}\ }\theta$ for every true value. This completes the proof. ∎

In the standard examples, the uniqueness of the root is typically proved by showing that the log-likelihood is strictly concave, as for the exponential distribution in example 5.2; exercise 16.1 carries out the argument for this model.

Remark.

The assumption that the likelihood equation has a unique root for every sample is slightly stronger than required: the proof is unchanged if the condition only holds on an event with probability tending to one, since we can intersect $S_{\varepsilon}$ with such an event. This covers the Bernoulli distribution of example 5.3, where the likelihood equation has no root when all observations are equal, an event of probability $\theta_{0}^{n}+(1-\theta_{0})^{n}\to 0$ . It also covers the Poisson distribution, where the same happens when all observations are zero, an event of probability $e^{-n\theta_{0}}\to 0$ .

16.4 When the Assumptions Fail

We now look at what happens when the main hypotheses of theorem 16.3 fail. These are the cases where more care is needed in practice.

If the support of the model depends on the parameter, i.e. if (R2) fails, the likelihood is not a smooth function of $\theta$ and the MLE is typically not a root of the likelihood equation. For example, for the uniform distribution on $(0,\theta)$ , the MLE is the sample maximum, as shown in example 5.5, and the theorem does not apply in this case. Nevertheless, the estimator is consistent: the mean squared error of the estimator goes to zero as the sample size increases, as shown in exercise 2.2, and thus lemma 2.11 applies; exercise 16.3 gives a direct proof of the same fact. We have seen in lecture 13 that for this model the Cramér–Rao bound can be violated, and in lecture 17 we will see that the limiting distribution of the MLE is not normal.

If the likelihood equation has several roots, the regularity conditions may all hold and the theorem still does not apply, because we cannot identify the root found in the proof with the MLE; the $t$ location model of section 5.3 is of this type. What survives is this: under the other assumptions of the theorem, for each fixed $\varepsilon$ , with probability tending to one there is a root within $\varepsilon$ of $\theta_{0}$ . What is lost is the guarantee that the root we find, or the global maximiser, is this one. The practical consequence of this is that numerical maximisation of the likelihood should be started with a simple consistent estimator, so that the iteration converges to the nearby root. This is a lesson we will take up in lecture 18.

Summary.

•

The regularity conditions (R1) to (R4) from definition 16.1 state that the parameter must be identifiable, the support of the model must not depend on $\theta$ , the log-likelihood must be three times differentiable with finite and positive Fisher information, and the third derivative of the log-likelihood must be bounded near each $\theta_{0}$ by a function with finite expectation.
•

For a fixed wrong value $\theta$ , the likelihood of the true value $\theta_{0}$ exceeds the likelihood of $\theta$ with probability tending to one. The proof uses the law of large numbers and Jensen’s inequality.
•

Under conditions (R1) to (R3), if the log-likelihood ratios have finite expectations and the likelihood equation has a unique root which is the MLE, then the MLE is consistent. The proof applies the lemma at $\theta_{0}\pm\varepsilon$ and finds a maximum of the likelihood, and thus a root of the likelihood equation, between these two points.
•

The theorem does not apply if the support of the model depends on the parameter (since then the MLE is typically not a root of the likelihood equation, as for the uniform distribution, even though it may still be consistent), or if the likelihood equation has multiple roots (since then we only know that, with probability tending to one, there is a root near $\theta_{0}$ ).

Exercise 16.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponentially distributed with rate $\theta>0$ , i.e. $f(x;\theta)=\theta e^{-\theta x}$ for all $x\geq 0$ .

1.

Show that the regularity conditions (R1) to (R4) hold for this model. Hint: For (R3) you can use the value $\mathcal{I}(\theta)=1/\theta^{2}$ from example 11.8 and the fact from lecture 11 that exponential families satisfy the conditions of section 11.1. For (R4) you should take derivatives of $\log f(x;\theta)$ with respect to $\theta$ and then find a bound $M(x)$ .
2.

Show that the likelihood equation has only one root for each sample with $\sum_{i}x_{i}>0$ and that this root is the MLE. Use theorem 16.3 and the remark after its proof to conclude that $\hat{\theta}_{n}=1/\bar{X}$ is consistent for $\theta$ .
3.

Find an alternative proof of the consistency of $1/\bar{X}$ , using the law of large numbers (proposition A.14), the continuous mapping theorem (proposition A.17), and any other arguments you need.

Exercise 16.2.

Let $X_{1}$ be the only observation of a Bernoulli random variable with success probability $\theta_{0}\in(0,1)$ , and let $\theta\in(0,1)$ be another parameter value.

1.

Compute the expectation $\mathbb{E}_{\theta_{0}}\bigl{(}\log\bigl{(}f(X_{1};\theta)/f(X_{1};\theta_{0})% \bigr{)}\bigr{)}$ from the proof of lemma 16.2, as an explicit function of $\theta$ and $\theta_{0}$ .
2.

Using the inequality $\log y\leq y-1$ for $y>0$ , with equality only for $y=1$ , show that this expectation is negative for all $\theta\neq\theta_{0}$ .

Exercise 16.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. uniformly distributed on the interval $(0,\theta)$ and let $\hat{\theta}_{n}=\max_{i}X_{i}$ be the MLE from example 5.5.

1.

Show that for $0<\varepsilon<\theta$ we have $\mathbb{P}_{\theta}\bigl{(}|\hat{\theta}_{n}-\theta|\geq\varepsilon\bigr{)}=(1% -\varepsilon/\theta)^{n}$ and thus $\hat{\theta}_{n}$ is consistent for $\theta$ .
2.

Which of the assumptions of theorem 16.3 are violated for this model? Why can the theorem not be applied in this case, despite the fact that the conclusion is still true?