Lecture 5 Maximum Likelihood Estimation

In the previous lecture we have seen that for the exponential model, the parameter value which maximises the likelihood is the one which ‘best’ explains the data. In this lecture we will make this idea into the main estimation method of the module: we define the maximum likelihood estimator, practise computing it for the families from tables 1.1 and 1.2, and will see problems where more thought is needed. We will see that the estimator is not unbiased in general, but is invariant under reparametrisation; the deeper justification of the method will be given in lectures 13, 16 and 17.

5.1 The Maximum Likelihood Estimator

Example 4.5 already contains the idea: for the exponential sample the log-likelihood increases up to $\theta=1/\bar{x}$ and then decreases, so this value fits the data best. We now formalise this principle.

Definition 5.1.

A maximum likelihood estimator (MLE) for $\theta$ is an estimator $\hat{\theta}=\hat{\theta}(X_{1},\dots,X_{n})$ , which maximises the likelihood function over the parameter space, i.e. which satisfies

L(\hat{\theta})\geq L(\theta)\qquad\text{for all $\theta\in\Theta$.}

Since the logarithm is strictly increasing, $\hat{\theta}$ maximises $L$ if and only if it maximises the log-likelihood $\ell$ and for the reasons given in lecture 4 we mostly work with $\ell$ . The definition is careful to avoid the problem of a maximum not existing or not being unique. For example, the likelihood could increase as the edge of the parameter space is approached, so that no maximum exists. Similarly, the maximum may not be unique. In such cases, any maximiser can be used. For the standard families of models and typical data, neither of these problems will occur and it will be safe to speak of “the” MLE.

If the set $\Theta$ is an interval and the log-likelihood function is differentiable, then calculus can be used to find the MLE: at a maximum inside the interval $\Theta$ , the derivative of $\ell$ equals zero and thus the MLE satisfies the likelihood equation

\ell^{\prime}(\theta)=0,

i.e. the MLE is a zero of the score function from definition 4.4. The general procedure is to find the log-likelihood function, to solve the likelihood equation and to verify that the solution is a maximum, for example by checking that $\ell^{\prime\prime}(\theta)<0$ or by checking that the score changes from positive to negative.

Warning.

The solution of the likelihood equation only gives interior critical points. These could be minima or inflection points and the checking step is required to avoid wrong answers. Also, the above procedure assumes that the maximum is interior and that $\ell$ is differentiable. We will see that these assumptions can be violated.

5.2 Worked Examples

We start our analysis by considering the MLE for commonly used families of probability distributions. For each of them we find the MLE using the recipe from the previous section, and then we check whether the resulting estimator is unbiased.

The Exponential Distribution

The exponential distribution is the simplest of the three cases considered here: the likelihood equation has a single root and the moments of the MLE can be computed exactly.

Example 5.2.

From examples 4.3 and 4.5 we know that the score function of an exponential sample $x_{1},\dots,x_{n}$ is given by

\ell^{\prime}(\theta)=\frac{n}{\theta}-\sum_{i=1}^{n}x_{i}

for all $\theta>0$ . Setting $\ell^{\prime}(\theta)=0$ gives $n/\theta=\sum_{i}x_{i}$ , and thus the likelihood equation has the unique solution

\theta=\frac{n}{\sum_{i=1}^{n}x_{i}}=\frac{1}{\bar{x}}.

To check that this root is a maximum, we differentiate once more and find

\ell^{\prime\prime}(\theta)=-\frac{n}{\theta^{2}}<0

for all $\theta>0$ . Thus, the function $\ell$ is strictly concave and the root $1/\bar{x}$ is the only global maximum. We conclude that the MLE for the rate of the exponential distribution is

\hat{\theta}=\frac{1}{\bar{X}}.

This coincides with the method of moments estimator from example 4.7.

To find the bias of this estimator, let $T=X_{1}+\dots+X_{n}$ , so that $\hat{\theta}=n/T$ . In exercise 2.3 we found, using the gamma density of $T$ , that $\mathbb{E}_{\theta}(\hat{\theta})=n\theta/(n-1)$ for $n\geq 2$ , while for $n=1$ the expectation is infinite. Thus, for $n\geq 2$ , the MLE is biased with bias

\mathop{\mathrm{bias}}\nolimits(\hat{\theta})=\mathbb{E}_{\theta}(\hat{\theta}% )-\theta=\frac{n\theta}{n-1}-\theta=\frac{\theta}{n-1}.

Thus, the MLE for the rate is biased but asymptotically unbiased in the sense of definition 2.4. Multiplying $\hat{\theta}$ by $(n-1)/n$ removes the bias: the estimator $\tilde{\theta}=(n-1)\hat{\theta}/n=(n-1)/T$ satisfies $\mathbb{E}_{\theta}(\tilde{\theta})=\theta$ .

The Bernoulli Distribution

For the Bernoulli distribution the recipe works whenever the data contain both successes and failures. The remaining two cases need separate treatment.

Example 5.3.

Let $X_{1},\dots,X_{n}$ be an i.i.d. Bernoulli sample with success probability $\theta$ , and let $s=\sum_{i}x_{i}$ be the observed number of successes. As in example 4.2 we have the likelihood $L(\theta)=\theta^{s}(1-\theta)^{n-s}$ and thus

\ell(\theta)=s\log\theta+(n-s)\log(1-\theta)

for all $\theta\in(0,1)$ . Assume $0<s<n$ . Then the likelihood equation is

0=\ell^{\prime}(\theta)=\frac{s}{\theta}-\frac{n-s}{1-\theta}.

This equation implies $s(1-\theta)=(n-s)\theta$ and the only solution to this equation is $\theta=s/n$ . Since $\ell^{\prime\prime}(\theta)=-s/\theta^{2}-(n-s)/(1-\theta)^{2}<0$ everywhere, this root is the global maximum and the MLE for the proportion of successes is

\hat{\theta}=\frac{s}{n}=\bar{X}.

Since $\mathbb{E}_{\theta}(\bar{X})=\theta$ , the MLE for the Bernoulli parameter is unbiased.

Remark.

The two cases $s=0$ and $s=n$ are special. For $s=0$ the function $\ell(\theta)=n\log(1-\theta)$ is strictly decreasing, and for $s=n$ the function $\ell(\theta)=n\log\theta$ is strictly increasing. In both cases the supremum is approached at the respective boundary point. We cannot find these points by setting $\ell^{\prime}(\theta)=0$ . Additionally, in this module we use the open interval $\Theta=(0,1)$ as the parameter space, so for us no $\theta\in\Theta$ maximises the log-likelihood and if we are pedantic the MLE does not exist.

The Normal Distribution

The normal distribution with both parameters unknown has a two-dimensional parameter vector, and thus the likelihood equation becomes a system of two equations.

Example 5.4.

Consider $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ , where both parameters are unknown. Then we have $\theta=(\mu,\sigma^{2})$ . Taking logarithms we get

\ell(\mu,\sigma^{2})=-\frac{n}{2}\log(2\pi\sigma^{2})-\frac{1}{2\sigma^{2}}% \sum_{i=1}^{n}(x_{i}-\mu)^{2}.

To find the maximum, we set both partial derivatives equal to zero. The partial derivatives are

\frac{\partial\ell}{\partial\mu}=\frac{1}{\sigma^{2}}\sum_{i=1}^{n}(x_{i}-\mu)

and

\frac{\partial\ell}{\partial\sigma^{2}}=-\frac{n}{2\sigma^{2}}+\frac{1}{2% \sigma^{4}}\sum_{i=1}^{n}(x_{i}-\mu)^{2}.

The first derivative is zero only for $\mu=\bar{x}$ . Substituting this value into the second derivative and setting it to zero gives $\sigma^{2}=(1/n)\sum_{i}(x_{i}-\bar{x})^{2}$ .

To check that this solution is the maximum, we note that for every fixed $\sigma^{2}$ the sum of squares $\sum_{i}(x_{i}-\mu)^{2}$ is smallest for $\mu=\bar{x}$ , and thus $\ell(\mu,\sigma^{2})\leq\ell(\bar{x},\sigma^{2})$ . At $\mu=\bar{x}$ the derivative with respect to $\sigma^{2}$ can be written as $\bigl{(}\sum_{i}(x_{i}-\bar{x})^{2}-n\sigma^{2}\bigr{)}/(2\sigma^{4})$ , which is positive to the left of the solution and negative to the right. Thus, the MLE is the pair

\hat{\mu}=\bar{X}\qquad\text{and}\qquad\hat{\sigma}^{2}=\frac{1}{n}\sum_{i=1}^% {n}(X_{i}-\bar{X})^{2}.

The MLE $\hat{\mu}=\bar{X}$ for the mean is unbiased. The MLE $\hat{\sigma}^{2}$ for the variance is the estimator from example 2.5, where we found the bias $-\sigma^{2}/n$ . Thus, $\hat{\sigma}^{2}$ is biased but asymptotically unbiased.

In all three examples considered in this section, the MLE is either unbiased or asymptotically unbiased. Note that there are other examples, where this property does not hold. The behaviour of the MLE for large samples in general is the subject of lectures 16 and 17.

5.3 When the Recipe Fails

The recipe for finding the MLE can fail in three different ways: the maximum may sit at a point where the log-likelihood is not differentiable, the likelihood equation may have several roots, and the likelihood equation may have no closed-form solution. We consider these three cases in turn.

A Maximum at the Boundary

Both assumptions in the recipe, i.e. an interior maximum and differentiable log-likelihood, break down for the uniform model. In this case we have to return to definition 5.1 and maximise the expression directly.

Example 5.5.

Let $X_{1},\dots,X_{n}\sim\text{Uniform}(0,\theta)$ be i.i.d., as in example 1.2. From the joint density found in exercise 1.3, the likelihood is

L(\theta)=\begin{cases}\theta^{-n}&\text{if $\theta\geq\max_{i}x_{i}$,}\\ 0&\text{otherwise.}\end{cases}

Calculus is useless here: on the region where $L$ is positive its derivative $-n\theta^{-n-1}$ is negative everywhere, so the likelihood equation has no solution, and at $\theta=\max_{i}x_{i}$ the function is not even continuous. But maximising directly is easy: the likelihood is zero for $\theta<\max_{i}x_{i}$ and strictly decreasing for $\theta\geq\max_{i}x_{i}$ , and thus largest at the left edge of the positive region. The MLE is the sample maximum,

\hat{\theta}=\max_{i}X_{i}.

This identifies the estimator we have been tracking since exercise 1.3, which by exercise 4.4 beats the method of moments estimator $2\bar{X}$ in mean squared error for every $n\geq 3$ . The likelihood also rules out impossible parameter values automatically: for the dataset $x_{1}=0.1$ , $x_{2}=0.2$ , $x_{3}=0.9$ of lecture 4, any $\theta<0.9$ has likelihood zero, so the contradiction the method of moments overlooked cannot arise.

Several Roots

The uniform model fails the recipe in an obvious way. A failure which is harder to spot is a likelihood equation with several roots. Consider the $t$ location model with $\nu\geq 1$ degrees of freedom, where $\nu$ is fixed and known and the density is given by

f(x;\theta)=c_{\nu}\Bigl{(}1+\frac{(x-\theta)^{2}}{\nu}\Bigr{)}^{-(\nu+1)/2},% \qquad x\in\mathbb{R},

with location $\theta\in\mathbb{R}$ and a normalising constant $c_{\nu}$ free of $\theta$ . The log-likelihood is differentiable everywhere, so the recipe seems to apply, but after clearing denominators the likelihood equation

\ell^{\prime}(\theta)=(\nu+1)\sum_{i=1}^{n}\frac{x_{i}-\theta}{\nu+(x_{i}-% \theta)^{2}}=0

turns into a polynomial equation of degree $2n-1$ in $\theta$ and for $n\geq 2$ this equation can have several roots. Exercise 5.4 shows that for $n=2$ there are three roots when the observations are more than $2\sqrt{\nu}$ apart: the mid-point of the interval between the observations, which is then a local minimum of the likelihood, and two global maxima of equal likelihood, one close to each of the observations. Thus, the model prefers to fit one data point and to consider the other as an outlier, and in this case the data cannot choose between the two maxima and the MLE will not be unique. The exercise also shows that a root of the likelihood equation can be a minimum and that the checking step in the recipe is not optional.

No Closed-Form Solution

For larger samples the likelihood equation of the $t$ location model cannot be solved in closed form, and similarly the likelihood equations for many other models, e.g. the two-parameter gamma distribution from example 4.9, cannot be solved in closed form. One solution to this problem is to maximise $\ell$ numerically. We will discuss this approach, together with its pitfalls, in lecture 18.

5.4 Invariance Under Reparametrisation

Often the quantity we want to estimate is not the parameter itself, but a function of the parameter, e.g. a probability, a mean or a median. The following proposition shows that under maximum likelihood such quantities transform together with the parameter.

Proposition 5.6 (Invariance of the MLE).

Let $g\colon\Theta\to\Psi$ be a bijection and let $\hat{\theta}$ be a maximum likelihood estimator for $\theta$ . Then $g(\hat{\theta})$ is a maximum likelihood estimator for the transformed parameter $\psi=g(\theta)$ .

The idea of the proof is that relabelling the parameter does not change the family of distributions: in terms of $\psi$ the likelihood is $\tilde{L}(\psi)=L\bigl{(}g^{-1}(\psi)\bigr{)}$ , so maximising $\tilde{L}$ over $\Psi$ is the same maximisation as before. Writing this down is exercise 5.5.

Remark.

If $g$ is not a bijection, then the likelihood of $\psi$ is defined to be the maximal likelihood of the values of $\theta$ which map to this value. While the statement $\hat{\psi}=g(\hat{\theta})$ is still true, we will not require the general case in this module.

As an application, we find the MLE for the median waiting time in the exponential model without any further maximisation.

Example 5.7.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ . The median waiting time is the value $m$ such that $\mathbb{P}_{\theta}(X_{1}\leq m)=1/2$ . Since $\mathbb{P}_{\theta}(X_{1}\leq m)=1-e^{-\theta m}$ , this condition is equivalent to $e^{-\theta m}=1/2$ , and taking logarithms gives

m=g(\theta)=\frac{\log 2}{\theta}.

The map $g$ is strictly decreasing on $(0,\infty)$ , with $g(\theta)\to\infty$ as $\theta\to 0$ and $g(\theta)\to 0$ as $\theta\to\infty$ , and thus it maps $(0,\infty)$ bijectively onto itself. Since the MLE for the rate is $\hat{\theta}=1/\bar{X}$ by example 5.2, proposition 5.6 shows that the MLE for the median is

\hat{m}=g(\hat{\theta})=\frac{\log 2}{\hat{\theta}}=(\log 2)\,\bar{X}.

Exercise 5.8 shows that this example contains a surprise: $\hat{m}$ is unbiased, despite the MLE for the rate being biased. In general, unbiasedness is lost under nonlinear reparametrisation, and thus there is no analogue of proposition 5.6 for unbiasedness.

Since we have seen in this lecture that the MLE can be biased, non-unique and even non-existent, we need to do more work to understand the method. The arguments for the validity of the MLE can be found in lectures 13, 16 and 17.

Summary.

•

The maximum likelihood estimator $\hat{\theta}$ is the parameter value which maximises the likelihood (or log-likelihood) of the data.
•

For smooth problems, where the maximum is an interior point of the parameter space, the MLE can be found by solving the likelihood equation $\ell^{\prime}(\theta)=0$ and verifying that the root is a maximum.
•

If this recipe fails, the definition can still be used: for the uniform distribution the maximum sits at the boundary, at $\hat{\theta}=\max_{i}X_{i}$ , and for the $t$ location model with two observations far apart the MLE is not unique.
•

The MLE can be biased.
•

The MLE is invariant under reparametrisation: the MLE for $g(\theta)$ is $g(\hat{\theta})$ . Unbiasedness is not invariant.

Exercise 5.1.

Let $X_{1},\dots,X_{n}$ be an i.i.d. sample from the geometric distribution with parameter $p\in(0,1)$ , i.e. $f(x;p)=(1-p)^{x-1}p$ for $x\in\mathbb{N}$ as in exercise 4.2.

1.

Show that the log-likelihood function is given by $\ell(p)=n\log p+\bigl{(}\sum_{i}x_{i}-n\bigr{)}\log(1-p)$ .
2.

Solve the likelihood equation and show that your solution is a maximum. Deduce that the MLE is $\hat{p}=1/\bar{X}$ .
3.

Discuss your results in the light of the method of moments estimator from exercise 4.2.

Exercise 5.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. with density $f(x;\theta)=\theta^{-2}x\,e^{-x/\theta}$ for $x\geq 0$ , where $\theta>0$ . This is the gamma distribution with known shape $2$ and scale $\theta$ , i.e. $\text{Gamma}(2,1/\theta)$ in the rate parametrisation from table A.2.

1.

Find the MLE $\hat{\theta}$ for $\theta$ .
2.

Show that your estimator $\hat{\theta}$ is unbiased.
3.

Determine $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})$ and use this to show that $\hat{\theta}$ is consistent.

Exercise 5.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta$ , let $\hat{\theta}=1/\bar{X}$ be the MLE and assume $n\geq 3$ .

1.

Using the gamma density of $T=X_{1}+\dots+X_{n}$ as in exercise 2.3, show that $\mathbb{E}_{\theta}(\hat{\theta}^{2})=n^{2}\theta^{2}/\bigl{(}(n-1)(n-2)\bigr{)}$ .
2.

Deduce that

$\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta})=\frac{n^{2}\theta^{2}}{(% n-1)^{2}(n-2)}$

and that the unbiased estimator $\tilde{\theta}=(n-1)\hat{\theta}/n$ from the text has variance $\mathop{\mathrm{Var}}\nolimits_{\theta}(\tilde{\theta})=\theta^{2}/(n-2)$ .

Exercise 5.4.

Let $x_{1}>x_{2}$ be any two observations from the $t$ location model given in section 5.3. Let $m=(x_{1}+x_{2})/2$ be the mid-point and $d=(x_{1}-x_{2})/2$ be half the distance between the two observations.

1.

Let $a=x_{1}-\theta$ and $b=x_{2}-\theta$ . Show that the likelihood equation $\ell^{\prime}(\theta)=0$ is equivalent to $(a+b)(\nu+ab)=0$ .
2.

Show that for $\theta=m+t$ the log-likelihood function is $\ell(m+t)=C-\frac{\nu+1}{2}\log P(t)$ , where $C$ does not depend on $t$ and

$P(t)=(\nu+d^{2})^{2}+2(\nu-d^{2})\,t^{2}+t^{4}.$
3.

Show that for $d^{2}<\nu$ the mid-point $m$ is the unique MLE, but for $d^{2}>\nu$ the likelihood equation has three roots $m$ , $m\pm\sqrt{d^{2}-\nu}$ , where $m$ is a local minimum of $\ell$ and the other two are maximum likelihood estimates with equal likelihood.

Exercise 5.5.

Prove the statement of proposition 5.6: Let $g\colon\Theta\to\Psi$ be a bijection and let the likelihood in the new parametrisation be given by $\tilde{L}(\psi)=L\bigl{(}g^{-1}(\psi)\bigr{)}$ . Show that $g(\hat{\theta})$ is the maximum of $\tilde{L}$ over $\Psi$ .

Exercise 5.6.

Let $X$ be an observation from the exponential distribution with rate $\theta>0$ , i.e. $\mathbb{P}_{\theta}(X>x)=e^{-\theta x}$ for $x\geq 0$ . The following properties of the exponential distribution will be needed later in the module.

1.

Show that the exponential distribution has no memory, i.e. that for all $s,t\geq 0$ we have

$\mathbb{P}_{\theta}(X>s+t\mskip 1.0mu|\mskip 1.0muX>s)=\mathbb{P}_{\theta}(X>t).$
2.

Show that for all $c\geq 0$ we have

$\mathbb{E}_{\theta}(X\mskip 1.0mu|\mskip 1.0muX>c)=c+\frac{1}{\theta}.$

Comment upon your result in the light of the first part of the question.

Exercise 5.7.

Let $X_{1},\dots,X_{n}$ be an i.i.d. sample from the Poisson distribution with parameter $\theta>0$ , and assume that $\sum_{i}x_{i}\geq 1$ .

1.

In exercise 4.1 we have seen that the score equals zero at $\bar{x}$ . Show that $\bar{X}$ is indeed the MLE for $\theta$ .
2.

Determine the MLE for the probability $p_{0}=\mathbb{P}_{\theta}(X_{1}=0)$ of observing zero.
3.

Using the relation $\mathbb{E}_{\theta}\bigl{(}z^{X_{1}}\bigr{)}=e^{\theta(z-1)}$ for $z>0$ , show that your estimator from the previous part is biased, but asymptotically unbiased. Comment on your result in the light of the fact that $\bar{X}$ is unbiased.

Exercise 5.8.

Let $\hat{m}=(\log 2)\,\bar{X}$ be the MLE for the median waiting time $m=(\log 2)/\theta$ of the exponential distribution from example 5.7. Show that $\hat{m}$ is an unbiased estimator for $m$ , and comment on why this does not contradict the bias of the MLE $\hat{\theta}=1/\bar{X}$ for the rate.