Lecture 11 Fisher Information

Since the data are random, the score function from lecture 4 is a random variable, and the Fisher information, given by the variance of the score, measures how much information the sample contains about the parameter. In lecture 13 we will use the Fisher information to obtain a lower bound for the variance of any unbiased estimator, and in lecture 17 we will see that the Fisher information equals the asymptotic variance of the maximum likelihood estimator. In this lecture we will define the Fisher information, prove two identities which allow us to compute the Fisher information, and we will compute the Fisher information for the standard families of models.

11.1 Setting and Assumptions

Throughout this lecture, $X_{1},\dots,X_{n}$ are i.i.d. with density or probability weights $f(x;\theta)$ , where the parameter space $\Theta\subseteq\mathbb{R}$ is an open interval. As in lecture 4, the log-likelihood is the sum $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$ of the terms $\ell_{i}(\theta)=\log f(X_{i};\theta)$ and the score of definition 4.4 is its derivative $\ell^{\prime}(\theta)=\sum_{i}\ell_{i}^{\prime}(\theta)$ . In this section we study the distribution of the score under $\mathbb{P}_{\theta}$ , for a fixed value of $\theta$ .

The results of this lecture are obtained by differentiating, with respect to $\theta$ , the identity $\int f(x;\theta)\,\mathrm{d}x=1$ , and we need this differentiation to be legitimate. We assume the following throughout. The support $\{x\mid f(x;\theta)>0\}$ of the distribution does not depend on $\theta$ . For every $x$ the function $\theta\mapsto f(x;\theta)$ is twice continuously differentiable on $\Theta$ . Finally, the integral of $f(x;\theta)$ over the support may be differentiated under the integral sign twice, so that

equation (11.1) (11.1)

\frac{\mathrm{d}^{k}}{\mathrm{d}\theta^{k}}\int f(x;\theta)\,\mathrm{d}x=\int% \frac{\partial^{k}}{\partial\theta^{k}}f(x;\theta)\,\mathrm{d}x\qquad\text{for% }k=1,2,

with sums in place of integrals for a discrete model. These three assumptions are referred to as the conditions of this section. Most standard families of models from tables 1.1 and 1.2 satisfy these conditions, except for the uniform distribution (in section 11.7). The exponential families in the sense of definition 7.1 with twice continuously differentiable natural parameter $\eta(\theta)$ also satisfy these conditions (since the exchange is the step in the proof of theorem 7.7). These conditions are included in the full list of regularity conditions in definition 16.1 of lecture 16.

11.2 The Score Has Mean Zero

The first identity is that, if $\theta$ is the correct parameter value, the data do not systematically deviate from this value: the score at the true parameter value is zero on average.

Lemma 11.1.

Let the situation be as described in section 11.1. Then

\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}=0\qquad\text{for % }i=1,\dots,n,\qquad\text{and thus}\qquad\mathbb{E}_{\theta}\bigl{(}\ell^{% \prime}(\theta)\bigr{)}=0

for all $\theta\in\Theta$ .

Proof.

Using the chain rule, we find the derivative of $\ell_{i}(\theta)=\log f(X_{i};\theta)$ as

equation (11.2) (11.2)

\ell_{i}^{\prime}(\theta)=\frac{\frac{\partial}{\partial\theta}f(X_{i};\theta)% }{f(X_{i};\theta)}.

Since the denominator is positive (because $X_{i}$ is in the support), we can cancel the density with the denominator to find

\displaystyle\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}

\displaystyle=\int\frac{\frac{\partial}{\partial\theta}f(x;\theta)}{f(x;\theta% )}\,f(x;\theta)\,\mathrm{d}x=\int\frac{\partial}{\partial\theta}f(x;\theta)\,% \mathrm{d}x

\displaystyle=\frac{\mathrm{d}}{\mathrm{d}\theta}\int f(x;\theta)\,\mathrm{d}x% =\frac{\mathrm{d}}{\mathrm{d}\theta}\,1=0,

where the integral is over the support of the distribution, which does not depend on $\theta$ , and the third equality comes from the case $k=1$ of (11.1). Summing over $i$ gives the statement for the whole sample. This completes the proof. ∎

If the parameter value is wrong, the score is not mean zero: for the exponential model from example 4.5, the score has expectation $n/\theta-n/\theta_{0}$ if the true parameter value is $\theta_{0}$ , positive for $\theta<\theta_{0}$ and negative for $\theta>\theta_{0}$ . The score on average points towards the correct parameter value from both sides. This is the mechanism behind the consistency proof of lecture 16.

11.3 Fisher Information

The next question we could ask is about the fluctuations of the score around zero. If the score is typically large in magnitude, this indicates that the data allow us to distinguish well between nearby parameter values. In contrast, if the score is typically close to zero, this indicates that the log-likelihood is nearly flat at the truth. The variance of the score is a measure for how informative the sample is.

Definition 11.2.

Under the conditions of section 11.1, the Fisher information of the sample $X_{1},\dots,X_{n}$ about $\theta$ is given by

\mathcal{I}_{n}(\theta)=\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell^{% \prime}(\theta)\bigr{)},

where the variance of the score is taken at $\theta$ . The Fisher information of a single observation is denoted by $\mathcal{I}(\theta)=\mathcal{I}_{1}(\theta)$ and is given by the variance of $\ell_{1}^{\prime}(\theta)$ .

The variance of the score is often awkward to compute directly, and the second expression of the following theorem is nearly always the quickest route to the information.

Theorem 11.3.

Suppose we are in the situation of section 11.1. Then

\mathcal{I}_{n}(\theta)=\mathbb{E}_{\theta}\bigl{(}\ell^{\prime}(\theta)^{2}% \bigr{)}=-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr{)}

for all $\theta\in\Theta$ .

Proof.

The first equality follows from the fact that $\ell^{\prime}(\theta)$ has mean zero by lemma 11.1, and thus the variance equals the second moment. The second equality is obtained by taking the derivative of the ratio (11.2) using the quotient rule:

\ell_{i}^{\prime\prime}(\theta)=\frac{\frac{\partial^{2}}{\partial\theta^{2}}f% (X_{i};\theta)}{f(X_{i};\theta)}-\Bigl{(}\frac{\frac{\partial}{\partial\theta}% f(X_{i};\theta)}{f(X_{i};\theta)}\Bigr{)}^{2}=\frac{\frac{\partial^{2}}{% \partial\theta^{2}}f(X_{i};\theta)}{f(X_{i};\theta)}-\ell_{i}^{\prime}(\theta)% ^{2}.

Similar to the proof of lemma 11.1, the expectation of the first term cancels out in this case, using the case $k=2$ of (11.1):

\mathbb{E}_{\theta}\Bigl{(}\frac{\frac{\partial^{2}}{\partial\theta^{2}}f(X_{i% };\theta)}{f(X_{i};\theta)}\Bigr{)}=\int\frac{\partial^{2}}{\partial\theta^{2}% }f(x;\theta)\,\mathrm{d}x=\frac{\mathrm{d}^{2}}{\mathrm{d}\theta^{2}}\int f(x;% \theta)\,\mathrm{d}x=0.

Thus we have $\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime\prime}(\theta)\bigr{)}=-\mathbb{E}% _{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)^{2}\bigr{)}$ for all $i$ . The terms $\ell_{1}^{\prime}(\theta),\dots,\ell_{n}^{\prime}(\theta)$ are independent of each other, since they are functions of different observations, and since they have mean zero, the variance of the sum is the sum of the variances:

\displaystyle\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell^{\prime}(% \theta)\bigr{)}

\displaystyle=\sum_{i=1}^{n}\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}% \ell_{i}^{\prime}(\theta)\bigr{)}=\sum_{i=1}^{n}\mathbb{E}_{\theta}\bigl{(}% \ell_{i}^{\prime}(\theta)^{2}\bigr{)}

\displaystyle=-\sum_{i=1}^{n}\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime\prime% }(\theta)\bigr{)}=-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr% {)},

where we used $\ell^{\prime\prime}(\theta)=\sum_{i}\ell_{i}^{\prime\prime}(\theta)$ in the last step. This completes the proof. ∎

The second expression gives the name of the quantity: it is the expected curvature of the log-likelihood at the true parameter value. Large information means that the log-likelihood has a sharp maximum, well-defined by the data. Small information means the log-likelihood is flat. In lecture 17 this idea will be turned into a theorem: the approximate variance of the maximum likelihood estimator is $1/\mathcal{I}_{n}(\theta)$ for large sample size, and the numerical value for the curvature $-\ell^{\prime\prime}(\hat{\theta})$ at the maximum likelihood estimate can be obtained by a numerical optimiser (lecture 18).

The proof also shows how the information of the sample relates to the information of one observation.

Proposition 11.4.

With the same assumptions as in section 11.1 we have

\mathcal{I}_{n}(\theta)=n\,\mathcal{I}(\theta)

for all $\theta\in\Theta$ .

Proof.

The score is the sum of independent terms and thus the variance can be found as the sum $\sum_{i}\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{i}^{\prime}(% \theta)\bigr{)}$ , as in the proof of theorem 11.3, and since the observations are identically distributed, each term in the sum equals

\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{% )}=\mathcal{I}(\theta).

This completes the proof. ∎

11.4 Examples

We now compute the information for the standard families of tables 1.1 and 1.2. By proposition 11.4 it suffices to work with a single observation $X=X_{1}$ .

Example 11.5.

For the Bernoulli distribution with success probability $\theta\in(0,1)$ we have $\ell_{1}(\theta)=X\log\theta+(1-X)\log(1-\theta)$ and thus

\ell_{1}^{\prime}(\theta)=\frac{X}{\theta}-\frac{1-X}{1-\theta},\qquad\ell_{1}% ^{\prime\prime}(\theta)=-\frac{X}{\theta^{2}}-\frac{1-X}{(1-\theta)^{2}}.

Using $\mathbb{E}_{\theta}(X)=\theta$ we find

\mathcal{I}(\theta)=-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta% )\bigr{)}=\frac{\theta}{\theta^{2}}+\frac{1-\theta}{(1-\theta)^{2}}=\frac{1}{% \theta(1-\theta)}.

Thus, the information of the sample is $\mathcal{I}_{n}(\theta)=n/\bigl{(}\theta(1-\theta)\bigr{)}$ . Exercise 11.1 checks these identities for this model directly.

Example 11.6.

For the Poisson distribution with mean $\theta>0$ we have $\ell_{1}(\theta)=X\log\theta-\theta-\log X!$ and thus

\ell_{1}^{\prime}(\theta)=\frac{X}{\theta}-1,\qquad\ell_{1}^{\prime\prime}(% \theta)=-\frac{X}{\theta^{2}},\qquad\mathcal{I}(\theta)=\frac{\mathbb{E}_{% \theta}(X)}{\theta^{2}}=\frac{1}{\theta}.

Example 11.7.

In the case of the normal distribution with unknown mean $\mu$ and known variance $\sigma^{2}$ , the parameter is $\mu$ and $\ell_{1}(\mu)=-\tfrac{1}{2}\log(2\pi\sigma^{2})-(X-\mu)^{2}/(2\sigma^{2})$ . Thus we have

\ell_{1}^{\prime}(\mu)=\frac{X-\mu}{\sigma^{2}},\qquad\ell_{1}^{\prime\prime}(% \mu)=-\frac{1}{\sigma^{2}},\qquad\mathcal{I}(\mu)=\frac{1}{\sigma^{2}}.

Since the second derivative does not depend on the data, we do not need to consider the expectation. Note that $\mathop{\mathrm{Var}}\nolimits_{\mu}(\bar{X})=\sigma^{2}/n=1/\mathcal{I}_{n}(\mu)$ by lemma 2.2 and from lecture 13 we know that no unbiased estimator can be better.

Example 11.8.

For the exponential distribution with rate $\theta>0$ , we have $\ell_{1}(\theta)=\log\theta-\theta X$ and thus

\ell_{1}^{\prime}(\theta)=\frac{1}{\theta}-X,\qquad\ell_{1}^{\prime\prime}(% \theta)=-\frac{1}{\theta^{2}},\qquad\mathcal{I}(\theta)=\frac{1}{\theta^{2}}.

Again, the second derivative is non-random. In fact, section 11.6 shows that the dependence of the information on the scale of the parameter is a general property.

11.5 Exponential Families

All four models from section 11.4 are exponential families. For such models, the information is the second derivative of the cumulant function $K$ , given by definition 7.6.

Proposition 11.9.

Let $f(x;\eta)=h(x)\exp\bigl{(}\eta T(x)-K(\eta)\bigr{)}$ be an exponential family in its natural parametrisation. Then the Fisher information of a single observation about $\eta$ is given by

\mathcal{I}(\eta)=K^{\prime\prime}(\eta)=\mathop{\mathrm{Var}}\nolimits_{\eta}% \bigl{(}T(X)\bigr{)}.

On the other hand, if the family is written as $f(x;\theta)=h(x)\exp\bigl{(}\eta(\theta)T(x)-B(\theta)\bigr{)}$ as in definition 7.1 and if $\eta$ is two times continuously differentiable, then the information about $\theta$ is given by

\mathcal{I}(\theta)=\eta^{\prime}(\theta)^{2}\,K^{\prime\prime}\bigl{(}\eta(% \theta)\bigr{)}=\eta^{\prime}(\theta)^{2}\,\mathop{\mathrm{Var}}\nolimits_{% \theta}\bigl{(}T(X)\bigr{)}.

Proof.

In the natural parametrisation we have $\ell_{1}(\eta)=\log h(X)+\eta T(X)-K(\eta)$ and thus $\ell_{1}^{\prime}(\eta)=T(X)-K^{\prime}(\eta)$ and $\ell_{1}^{\prime\prime}(\eta)=-K^{\prime\prime}(\eta)$ . By theorem 7.7 we have $\mathbb{E}_{\eta}\bigl{(}T(X)\bigr{)}=K^{\prime}(\eta)$ and $\mathop{\mathrm{Var}}\nolimits_{\eta}\bigl{(}T(X)\bigr{)}=K^{\prime\prime}(\eta)$ and thus, since the second derivative is non-random, we find $\mathcal{I}(\eta)=K^{\prime\prime}(\eta)=\mathop{\mathrm{Var}}\nolimits_{\eta}% \bigl{(}T(X)\bigr{)}$ . For the general parametrisation we have $B(\theta)=K\bigl{(}\eta(\theta)\bigr{)}$ , since both quantities are determined by the requirement that the density must integrate to one, and thus, using the chain rule, we find

\ell_{1}^{\prime}(\theta)=\eta^{\prime}(\theta)\,\Bigl{(}T(X)-K^{\prime}\bigl{% (}\eta(\theta)\bigr{)}\Bigr{)}.

The factor $\eta^{\prime}(\theta)$ is a constant and thus the variance of the score is

\mathcal{I}(\theta)=\eta^{\prime}(\theta)^{2}\,\mathop{\mathrm{Var}}\nolimits_% {\theta}\bigl{(}T(X)\bigr{)}=\eta^{\prime}(\theta)^{2}\,K^{\prime\prime}\bigl{% (}\eta(\theta)\bigr{)},

as claimed. ∎

The proposition turns the computation of the information into that of a variance, which for the standard families can be read off tables A.1 and A.2; exercises 11.2 and 11.4 use it in this way.

11.6 Reparametrisation

The information depends on which parameter we use to describe a model, and the factor $\eta^{\prime}(\theta)^{2}$ in proposition 11.9 is an instance of the general rule.

Proposition 11.10.

Let $\psi=g(\theta)$ where $g\colon\Theta\to g(\Theta)$ is a differentiable bijection with $g^{\prime}(\theta)\neq 0$ for all $\theta\in\Theta$ . Let $\mathcal{I}_{\theta}$ denote the Fisher information for a single observation when the model is parametrised by $\theta$ and let $\mathcal{I}_{\psi}$ be the information when the model is parametrised by $\psi$ . Then,

\mathcal{I}_{\psi}(\psi)=\frac{\mathcal{I}_{\theta}(\theta)}{g^{\prime}(\theta% )^{2}}\qquad\text{for }\psi=g(\theta).

Proof.

With the new parametrisation, the density is $\tilde{f}(x;\psi)=f\bigl{(}x;g^{-1}(\psi)\bigr{)}$ and the log-likelihood for a single observation is $\tilde{\ell}_{1}(\psi)=\ell_{1}\bigl{(}g^{-1}(\psi)\bigr{)}$ . Since $g^{\prime}$ never vanishes on the interval $\Theta$ , by Darboux’s theorem $g$ is strictly monotonic and $g^{-1}$ is differentiable. Using the chain rule and the derivative of the inverse function we get $(g^{-1})^{\prime}(\psi)=1/g^{\prime}(\theta)$ for $\theta=g^{-1}(\psi)$ . This gives

\tilde{\ell}_{1}^{\prime}(\psi)=\ell_{1}^{\prime}(\theta)\,(g^{-1})^{\prime}(% \psi)=\frac{\ell_{1}^{\prime}(\theta)}{g^{\prime}(\theta)}.

Since $\theta$ is fixed for fixed $\psi$ , $1/g^{\prime}(\theta)$ is a constant and taking variances under $\mathbb{P}_{\theta}$ gives $\mathcal{I}_{\psi}(\psi)=\mathcal{I}_{\theta}(\theta)/g^{\prime}(\theta)^{2}$ , as claimed. ∎

For the exponential distribution, reparametrising from the rate $\theta$ to the mean $\mu=1/\theta$ changes the information $1/\theta^{2}$ from example 11.8 to $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ ; see exercise 11.5. Thus, the information is not a property of the model in isolation, but depends on the parametrisation. What does not change under $\psi=g(\theta)$ is the combination $\mathcal{I}_{\theta}(\theta)\,(\mathrm{d}\theta)^{2}$ , since $\mathrm{d}\psi=g^{\prime}(\theta)\,\mathrm{d}\theta$ , and this invariance is the basis of the Jeffreys prior in lecture 26.

11.7 When the Identities Fail

The uniform distribution from example 1.2 shows what can go wrong if the conditions from section 11.1 are violated.

Example 11.11.

Let $X$ be uniformly distributed on $[0,\theta]$ , i.e. $f(x;\theta)=1/\theta$ for $0\leq x\leq\theta$ . The support of the distribution depends on $\theta$ , but we can still compute the derivatives on the support to see whether the identities hold. For $0<X<\theta$ we have $\ell_{1}(\theta)=-\log\theta$ and thus $\ell_{1}^{\prime}(\theta)=-1/\theta$ and $\ell_{1}^{\prime\prime}(\theta)=1/\theta^{2}$ . The score is a negative constant and thus we have $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}=-1/\theta\neq 0$ , in contradiction to lemma 11.1. The variance is zero, whereas $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)^{2}\bigr{)}=1/\theta^{2}$ and $-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta)\bigr{)}=-1/\theta^% {2}$ : the three expressions for theorem 11.3 take three different values, one of which is negative. The proof breaks down at the interchange between differentiation and integration: for this model, the two sides of (11.1) with $k=1$ differ by the term $f(\theta;\theta)=1/\theta$ from the moving upper limit, which is missed by the differentiation under the integral sign. Exercise 11.6 can be used to work out the two computations.

For this model, there is no Fisher information and the two results based on it are not applicable: the sample maximum does not satisfy the lower bound from lecture 13, the mean squared error decreases like $1/n^{2}$ per exercise 2.2 instead of like $1/n$ , and the asymptotic normality from lecture 17 also fails.

Summary.

•

Under the conditions of section 11.1 (a support free of $\theta$ , twice continuously differentiable density and differentiation under the integral sign), the score at the true parameter value has mean zero.
•

The Fisher information $\mathcal{I}_{n}(\theta)$ is the variance of the score, and it equals $\mathbb{E}_{\theta}\bigl{(}\ell^{\prime}(\theta)^{2}\bigr{)}$ and $-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr{)}$ , the expected curvature of the log-likelihood at the truth.
•

Information from independent observations can be combined, $\mathcal{I}_{n}(\theta)=n\mathcal{I}(\theta)$ ; for an exponential family we have $\eta^{\prime}(\theta)^{2}\mathop{\mathrm{Var}}\nolimits_{\theta}(T)$ ; under a reparametrisation $\psi=g(\theta)$ we get $\mathcal{I}_{\psi}(\psi)=\mathcal{I}_{\theta}(\theta)/g^{\prime}(\theta)^{2}$ .
•

We have learned the following standard values: $1/\bigl{(}\theta(1-\theta)\bigr{)}$ for the Bernoulli distribution, $1/\theta$ for the Poisson distribution, $1/\sigma^{2}$ for the mean of the normal distribution and $1/\theta^{2}$ for the rate parameter in the exponential distribution. For the uniform distribution these relations do not hold, since the support of the distribution moves with the parameter.
•

The information is the currency of the two results presented later: it gives a bound on the variance of unbiased estimators in lecture 13 and it determines the asymptotic variance of the MLE in lecture 17.

Exercise 11.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , and let $\ell_{1}^{\prime}(\theta)$ be the score of a single observation from example 11.5.

1.

Verify lemma 11.1 for this model by computing $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}$ directly from the two possible values of $X_{1}$ .
2.

Compute $\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}$ directly, and check that it agrees with the value of $\mathcal{I}(\theta)$ found in example 11.5.
3.

Compare $1/\mathcal{I}_{n}(\theta)$ with the variance of the estimator $\bar{X}$ for $\theta$ .

Exercise 11.2.

Let $X_{1},\dots,X_{n}$ be i.i.d. with density $f(x;\theta)=\theta^{-2}x\,e^{-x/\theta}$ for $x\geq 0$ , where $\theta>0$ , the gamma distribution with known shape $2$ and scale $\theta$ from exercise 5.2.

1.

Compute the score $\ell_{1}^{\prime}(\theta)$ of a single observation and verify that it has mean zero.
2.

Show that $\mathcal{I}(\theta)=2/\theta^{2}$ , once via $-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta)\bigr{)}$ and once via the variance of the score, using $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=2\theta^{2}$ .
3.

Write the density in the form of definition 7.1 with $T(x)=x$ , identify $\eta(\theta)$ , and recover $\mathcal{I}(\theta)$ from proposition 11.9.
4.

In exercise 5.2 we found that the MLE $\hat{\theta}=\bar{X}/2$ is unbiased with $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\theta^{2}/(2n)$ . Compare this with $1/\mathcal{I}_{n}(\theta)$ .

Exercise 11.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. from the geometric distribution with parameter $p\in(0,1)$ , i.e. $f(x;p)=(1-p)^{x-1}p$ for $x\in\{1,2,\dots\}$ , where $\mathbb{E}_{p}(X_{1})=1/p$ and $\mathop{\mathrm{Var}}\nolimits_{p}(X_{1})=(1-p)/p^{2}$ , as in exercises 4.2 and 5.1.

1.

Compute the score $\ell_{1}^{\prime}(p)$ of a single observation and verify that it has mean zero.
2.

Determine $\mathcal{I}(p)=1/\bigl{(}p^{2}(1-p)\bigr{)}$ and write down $\mathcal{I}_{n}(p)$ .
3.

Verify the value from the previous part by taking the variance of the score.

Exercise 11.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with mean $\theta>0$ .

1.

From lecture 7 we know that the natural parameter is $\eta=\log\theta$ and the cumulant function is $K(\eta)=e^{\eta}$ . Using proposition 11.9, find the information $\mathcal{I}_{\eta}(\eta)$ about $\eta$ in terms of $\theta$ .
2.

Recover $\mathcal{I}_{\theta}(\theta)=1/\theta$ from the previous part using proposition 11.10.
3.

In exercise 5.7 we have estimated the probability $p_{0}=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ . Determine the information $\mathcal{I}_{p_{0}}(p_{0})$ about $p_{0}$ and describe its behaviour as $\theta\to\infty$ . Give a one-sentence answer as to why this behaviour is reasonable.

Exercise 11.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ and let $\mu=1/\theta$ be the mean.

1.

Using proposition 11.10 and example 11.8, verify the value $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ from section 11.6.
2.

Show that the information of the sample about $\mu$ is $n/\mu^{2}$ and compare the reciprocal of this quantity to the variance of the estimator $\bar{X}$ for $\mu$ .

Exercise 11.6.

Let $X$ be uniformly distributed on the interval $[0,\theta]$ as described in example 11.11. Compute both sides of (11.1) for this model, using $k=1$ , and verify that the difference between the two quantities corresponds to the contribution $f(\theta;\theta)=1/\theta$ of the (moving) upper limit of the integral.