Lecture 11 Fisher Information

Throughout this lecture, $X_{1},\dots,X_{n}$ are i.i.d. with density or probability weights $f(x;\theta)$, where the parameter space $\Theta\subseteq\mathbb{R}$ is an open interval. As in lecture 4, the log-likelihood is the sum $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$ of the terms $\ell_{i}(\theta)=\log f(X_{i};\theta)$ and the score of definition 4.4 is its derivative $\ell^{\prime}(\theta)=\sum_{i}\ell_{i}^{\prime}(\theta)$. In this section we study the distribution of the score under $\mathbb{P}_{\theta}$, for a fixed value of $\theta$.

The results of this lecture are obtained by differentiating, with respect to $\theta$, the identity $\int f(x;\theta)\,\mathrm{d}x=1$, and we need this differentiation to be legitimate. We assume the following throughout. The support $\{x\mid f(x;\theta)>0\}$ of the distribution does not depend on $\theta$. For every $x$ the function $\theta\mapsto f(x;\theta)$ is twice continuously differentiable on $\Theta$. Finally, the integral of $f(x;\theta)$ over the support may be differentiated under the integral sign twice, so that

where the integrals on the right-hand side exist, with sums in place of integrals for a discrete model. In addition we assume that the score of a single observation has finite variance, $\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}<\infty$ for all $\theta\in\Theta$, so that the quantities defined below are finite. These four assumptions are referred to as the conditions of this section. The one-parameter models from tables 1.1 and 1.2, and the two-parameter models with one parameter held fixed, satisfy these conditions, with the exception of the uniform distribution (see section 11.7). The exponential families in the sense of definition 7.1 with twice continuously differentiable natural parameter $\eta(\theta)$ taking values in the interior of the natural parameter space also satisfy these conditions, as explained after lemma 7.10 in lecture 7. These conditions are included in the full list of regularity conditions in definition 16.1 of lecture 16.

The first identity is that, if $\theta$ is the correct parameter value, the data do not systematically deviate from this value: the score at the true parameter value is zero on average.

If the parameter value is wrong, the score is not mean zero: for the exponential model from example 4.5, the score $\ell^{\prime}(\theta)=n/\theta-\sum_{i}X_{i}$ has expectation $\mathbb{E}_{\theta_{0}}\bigl{(}\ell^{\prime}(\theta)\bigr{)}=n/\theta-n/\theta_{0}$ if the true parameter value is $\theta_{0}$, positive for $\theta<\theta_{0}$ and negative for $\theta>\theta_{0}$. The score on average points towards the correct parameter value from both sides. In lecture 16 we will use this behaviour to prove that the maximum likelihood estimator is consistent.

The next question we could ask is about the fluctuations of the score around zero. If the score is typically large in magnitude, this indicates that the data allow us to distinguish well between nearby parameter values. In contrast, if the score is typically close to zero, this indicates that the log-likelihood is nearly flat at the truth. The variance of the score is a measure for how informative the sample is.

The subscript in $\mathcal{I}_{n}$ is the sample size. Where we compare two different parametrisations of the same model, we write $\mathcal{I}_{\theta}$, $\mathcal{I}_{\eta}$ or $\mathcal{I}_{\psi}$ for the information of a single observation about the parameter named in the subscript.

The variance of the score is often awkward to compute directly, and the second expression of the following lemma is nearly always the quickest route to the information.

The second expression gives the name of the quantity: it is the expected curvature of the log-likelihood at the true parameter value. Large information means that the log-likelihood has a sharp maximum, well-defined by the data. Small information means the log-likelihood is flat. Note that the identity holds in expectation only: for a given sample the curvature $-\ell^{\prime\prime}(\theta)$ is random and in general differs from $\mathcal{I}_{n}(\theta)$. In lecture 17 this idea will be turned into a theorem: the approximate variance of the maximum likelihood estimator is $1/\mathcal{I}_{n}(\theta)$ for large sample size, and $\mathcal{I}_{n}(\theta)$ can be replaced there by the curvature $-\ell^{\prime\prime}(\hat{\theta})$ at the maximum likelihood estimate, which can be computed from the data, numerically if need be (lecture 18).

The proof also shows how the information of the sample relates to the information of one observation: each observation contributes the same amount, and $n$ observations carry $n$ times the information of one.

We now compute the information for four of the standard families of tables 1.1 and 1.2; the exercises cover the binomial, gamma and geometric distributions. By proposition 11.4 it suffices to work with a single observation $X=X_{1}$.

All four models from section 11.4 are exponential families. For such models, in the natural parametrisation, the information is the second derivative of the cumulant function $K$ from definition 7.9.

The assumptions on $\eta$ are those under which the conditions of section 11.1 hold, as discussed there, so that the information is defined.

The proposition turns the computation of the information into that of a variance, which for the standard families can be read off tables A.1 and A.2. As a check, the Bernoulli distribution has $T(x)=x$ and $\eta(\theta)=\log\bigl{(}\theta/(1-\theta)\bigr{)}$ by example 7.2, so that $\eta^{\prime}(\theta)^{2}\mathop{\mathrm{Var}}\nolimits_{\theta}(X)=\theta(1-\theta)/\bigl{(}\theta(1-\theta)\bigr{)}^{2}$ recovers example 11.5. Exercises 11.2 and 11.4 use the proposition in the same way.

The information depends on which parameter we use to describe a model, and the factor $\eta^{\prime}(\theta)^{2}$ in proposition 11.9 is an instance of the general rule.

For the exponential distribution, reparametrising from the rate $\theta$ to the mean $\mu=1/\theta$ changes the information $1/\theta^{2}$ from example 11.8 to $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$; see exercise 11.5. Thus, the information is not a property of the model in isolation, but depends on the parametrisation. In lecture 26 we will use the transformation rule of the proposition to construct a prior distribution, the Jeffreys prior, which does not depend on the parametrisation chosen.

The uniform distribution from example 1.2 shows what can go wrong if the conditions from section 11.1 are violated.

For this model the Fisher information is not defined, and the two results based on it do not apply. Their conclusions fail as well: by exercise 2.2 the mean squared error of the sample maximum, and with it the variance of the unbiased estimator $(n+1)\max_{i}X_{i}/n$, decreases like $1/n^{2}$, whereas the lower bound of lecture 13, where it applies, does not allow the variance of an unbiased estimator to decrease faster than $1/n$; and the maximum likelihood estimator $\max_{i}X_{i}$ is not approximately normal for large $n$, in contrast to lecture 17.