Lecture 11 Fisher Information

Since the data are random, the score function from lecture 4 is a random variable, and the Fisher information, given by the variance of the score, measures how much information the sample contains about the parameter. In lecture 13 we will use the Fisher information to obtain a lower bound for the variance of any unbiased estimator, and in lecture 17 we will see that the reciprocal of the Fisher information is the asymptotic variance of the maximum likelihood estimator. In this lecture we will define the Fisher information, prove two identities which allow us to compute the Fisher information, and we will compute the Fisher information for the standard families of models.

11.1 Setting and Assumptions

Throughout this lecture, $X_{1},\dots,X_{n}$ are i.i.d. with density or probability weights $f(x;\theta)$ , where the parameter space $\Theta\subseteq\mathbb{R}$ is an open interval. As in lecture 4, the log-likelihood is the sum $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$ of the terms $\ell_{i}(\theta)=\log f(X_{i};\theta)$ and the score of definition 4.4 is its derivative $\ell^{\prime}(\theta)=\sum_{i}\ell_{i}^{\prime}(\theta)$ . In this section we study the distribution of the score under $\mathbb{P}_{\theta}$ , for a fixed value of $\theta$ .

The results of this lecture are obtained by differentiating the identity

\int f(x;\theta)\,\mathrm{d}x=1

with respect to $\theta$ . Throughout the section we make the following assumptions:

1.

The support $\bigl{\{}x\bigm{|}f(x;\theta)>0\bigr{\}}$ of the distribution does not depend on $\theta$ .
2.

For every $x$ the function $\theta\mapsto f(x;\theta)$ is twice continuously differentiable on $\Theta$ .
3.

The integral of $f(x;\theta)$ over the support may be differentiated under the integral sign twice, so that

equation (11.1) (11.1)
$\frac{\mathrm{d}^{k}}{\mathrm{d}\theta^{k}}\int f(x;\theta)\,\mathrm{d}x=\int% \frac{\partial^{k}}{\partial\theta^{k}}f(x;\theta)\,\mathrm{d}x\qquad\text{for% }k=1,2,$

where the integrals on the right-hand side exist, with sums in place of integrals for a discrete model.
4.

The score of a single observation has finite variance, $\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{% )}<\infty$ for all $\theta\in\Theta$ , so that the quantities defined below are finite.

These four assumptions are referred to as the conditions of this section. The one-parameter models from tables 1.1 and 1.2, and the two-parameter models with one parameter held fixed (with $m$ known for the binomial distribution), satisfy these conditions, with the exception of the uniform distribution (see section 11.6). So do the exponential families of lecture 7, as explained after lemma 7.10 there.

11.2 The Score Has Mean Zero

The first identity is that, if $\theta$ is the correct parameter value, the data do not systematically deviate from this value: the score at the true parameter value is zero on average.

Lemma 11.1.

Let the situation be as described in section 11.1. Then

\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}=0

for $i=1,\dots,n$ , and thus

\mathbb{E}_{\theta}\bigl{(}\ell^{\prime}(\theta)\bigr{)}=0

for all $\theta\in\Theta$ .

Proof.

Using the chain rule, we find the derivative of $\ell_{i}(\theta)=\log f(X_{i};\theta)$ as

equation (11.2) (11.2)

\ell_{i}^{\prime}(\theta)=\frac{\frac{\partial}{\partial\theta}f(X_{i};\theta)% }{f(X_{i};\theta)}.

Since $X_{i}$ lies in the support with probability one, the denominator is positive and the ratio is well defined. Cancelling the density against the denominator we find

\displaystyle\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}

\displaystyle=\int\frac{\frac{\partial}{\partial\theta}f(x;\theta)}{f(x;\theta% )}\,f(x;\theta)\,\mathrm{d}x=\int\frac{\partial}{\partial\theta}f(x;\theta)\,% \mathrm{d}x

\displaystyle=\frac{\mathrm{d}}{\mathrm{d}\theta}\int f(x;\theta)\,\mathrm{d}x% =\frac{\mathrm{d}}{\mathrm{d}\theta}\,1=0,

where the integral is over the support of the distribution, which does not depend on $\theta$ (with a sum in place of the integral for a discrete model), and the third equality comes from the case $k=1$ of (11.1). Summing over $i$ gives the statement for the whole sample. This completes the proof. ∎

If the parameter value is wrong, the score is not mean zero: for the exponential model from example 4.5, the score $\ell^{\prime}(\theta)=n/\theta-\sum_{i}X_{i}$ has expectation $\mathbb{E}_{\theta_{0}}\bigl{(}\ell^{\prime}(\theta)\bigr{)}=n/\theta-n/\theta% _{0}$ if the true parameter value is $\theta_{0}$ , positive for $\theta<\theta_{0}$ and negative for $\theta>\theta_{0}$ . The score on average points towards the correct parameter value from both sides. In lecture 16 we will see the corresponding property of the log-likelihood, which on average is larger at the true parameter value than at any other value, and use it to prove that the maximum likelihood estimator is consistent.

11.3 Fisher Information

The next question we could ask is about the fluctuations of the score around zero. If the score is typically large in magnitude, this indicates that the data allow us to distinguish well between nearby parameter values. In contrast, if the score is typically close to zero, this indicates that the log-likelihood is nearly flat at the truth. The variance of the score is a measure for how informative the sample is.

Definition 11.2.

Under the conditions of section 11.1, the Fisher information of the sample $X_{1},\dots,X_{n}$ about $\theta$ is given by

\mathcal{I}_{n}(\theta)=\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell^{% \prime}(\theta)\bigr{)},

where the variance of the score is taken at $\theta$ . The Fisher information of a single observation is denoted by $\mathcal{I}(\theta)=\mathcal{I}_{1}(\theta)$ and is given by the variance of $\ell_{1}^{\prime}(\theta)$ .

The subscript in $\mathcal{I}_{n}$ is the sample size. Where we compare two different parametrisations of the same model, we write $\mathcal{I}_{\theta}$ , $\mathcal{I}_{\eta}$ or $\mathcal{I}_{\psi}$ for the information of a single observation about the parameter named in the subscript.

The variance of the score is often awkward to compute directly, and the second expression of the following lemma is nearly always the quickest route to the information.

Lemma 11.3.

Suppose we are in the situation of section 11.1. Then

\mathcal{I}_{n}(\theta)=\mathbb{E}_{\theta}\bigl{(}\ell^{\prime}(\theta)^{2}% \bigr{)}=-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr{)}

for all $\theta\in\Theta$ .

Proof.

The first equality follows from the fact that $\ell^{\prime}(\theta)$ has mean zero by lemma 11.1, and thus the variance equals the second moment. The second equality is obtained by taking the derivative of the ratio (11.2) using the quotient rule:

\ell_{i}^{\prime\prime}(\theta)=\frac{\frac{\partial^{2}}{\partial\theta^{2}}f% (X_{i};\theta)}{f(X_{i};\theta)}-\Bigl{(}\frac{\frac{\partial}{\partial\theta}% f(X_{i};\theta)}{f(X_{i};\theta)}\Bigr{)}^{2}=\frac{\frac{\partial^{2}}{% \partial\theta^{2}}f(X_{i};\theta)}{f(X_{i};\theta)}-\ell_{i}^{\prime}(\theta)% ^{2}.

Similar to the proof of lemma 11.1, the expectation of the first term vanishes, using the case $k=2$ of (11.1):

\mathbb{E}_{\theta}\Bigl{(}\frac{\frac{\partial^{2}}{\partial\theta^{2}}f(X_{i% };\theta)}{f(X_{i};\theta)}\Bigr{)}=\int\frac{\partial^{2}}{\partial\theta^{2}% }f(x;\theta)\,\mathrm{d}x=\frac{\mathrm{d}^{2}}{\mathrm{d}\theta^{2}}\int f(x;% \theta)\,\mathrm{d}x=0.

Since the second moment of $\ell_{i}^{\prime}(\theta)$ is finite by the conditions of section 11.1, we can take expectations term by term and get $\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime\prime}(\theta)\bigr{)}=-\mathbb{E}% _{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)^{2}\bigr{)}$ for all $i$ .

To go from $\ell_{i}$ to $\ell$ we also need to consider the cross terms: $\ell_{1}^{\prime}(\theta),\dots,\ell_{n}^{\prime}(\theta)$ are independent of each other, since the $X_{i}$ are independent. Thus the variance of the sum is the sum of the variances. Since each term also has mean zero we find

\displaystyle\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell^{\prime}(% \theta)\bigr{)}

\displaystyle=\sum_{i=1}^{n}\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}% \ell_{i}^{\prime}(\theta)\bigr{)}=\sum_{i=1}^{n}\mathbb{E}_{\theta}\bigl{(}% \ell_{i}^{\prime}(\theta)^{2}\bigr{)}

\displaystyle=-\sum_{i=1}^{n}\mathbb{E}_{\theta}\bigl{(}\ell_{i}^{\prime\prime% }(\theta)\bigr{)}=-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr% {)},

where we used $\ell^{\prime\prime}(\theta)=\sum_{i}\ell_{i}^{\prime\prime}(\theta)$ in the last step. This completes the proof. ∎

The second expression motivates the name Fisher Information: the quantity is the expected curvature of the log-likelihood at the true parameter value. Large information means that the log-likelihood has a sharp maximum, well-defined by the data. Small information means the log-likelihood is flat. Note that the identity holds in expectation only: for a given sample the curvature $-\ell^{\prime\prime}(\theta)$ is random and in general differs from $\mathcal{I}_{n}(\theta)$ .

The proof also shows how the information of the sample relates to the information of one observation: each observation contributes the same amount, and $n$ observations carry $n$ times the information of one.

Proposition 11.4.

With the same assumptions as in section 11.1 we have

\mathcal{I}_{n}(\theta)=n\,\mathcal{I}(\theta)

for all $\theta\in\Theta$ .

Proof.

The last display of the proof of lemma 11.3 writes $\mathcal{I}_{n}(\theta)$ as the sum of the $n$ variances $\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{i}^{\prime}(\theta)\bigr{)}$ , which are all equal to $\mathcal{I}(\theta)$ since the observations are identically distributed. This completes the proof. ∎

11.4 Examples

We now compute the information for four of the standard families of tables 1.1 and 1.2; the exercises cover the binomial, gamma and geometric distributions. By proposition 11.4 it suffices to work with a single observation $X=X_{1}$ .

Example 11.5.

For the Bernoulli distribution with success probability $\theta\in(0,1)$ we have $\ell_{1}(\theta)=X\log\theta+(1-X)\log(1-\theta)$ and thus

\ell_{1}^{\prime}(\theta)=\frac{X}{\theta}-\frac{1-X}{1-\theta},\qquad\ell_{1}% ^{\prime\prime}(\theta)=-\frac{X}{\theta^{2}}-\frac{1-X}{(1-\theta)^{2}}.

Using $\mathbb{E}_{\theta}(X)=\theta$ we find

\mathcal{I}(\theta)=-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta% )\bigr{)}=\frac{\theta}{\theta^{2}}+\frac{1-\theta}{(1-\theta)^{2}}=\frac{1}{% \theta(1-\theta)}.

Thus, the information of the sample is $\mathcal{I}_{n}(\theta)=n/\bigl{(}\theta(1-\theta)\bigr{)}$ . Exercise 11.1 checks these identities for this model directly.

Example 11.6.

For the Poisson distribution with mean $\theta>0$ we have $\ell_{1}(\theta)=X\log\theta-\theta-\log X!$ and thus

\ell_{1}^{\prime}(\theta)=\frac{X}{\theta}-1,\qquad\ell_{1}^{\prime\prime}(% \theta)=-\frac{X}{\theta^{2}},\qquad\mathcal{I}(\theta)=\frac{\mathbb{E}_{% \theta}(X)}{\theta^{2}}=\frac{1}{\theta}.

Example 11.7.

In the case of the normal distribution with unknown mean $\mu$ and known variance $\sigma^{2}$ , the parameter is $\mu$ and $\ell_{1}(\mu)=-\tfrac{1}{2}\log(2\pi\sigma^{2})-(X-\mu)^{2}/(2\sigma^{2})$ . Thus we have

\ell_{1}^{\prime}(\mu)=\frac{X-\mu}{\sigma^{2}},\qquad\ell_{1}^{\prime\prime}(% \mu)=-\frac{1}{\sigma^{2}},\qquad\mathcal{I}(\mu)=\frac{1}{\sigma^{2}}.

Since the second derivative does not depend on the data, we do not need to consider the expectation. Note that $\mathop{\mathrm{Var}}\nolimits_{\mu}(\bar{X})=\sigma^{2}/n=1/\mathcal{I}_{n}(\mu)$ by lemma 2.2, and in lecture 13 we will see that no unbiased estimator can have a smaller variance.

Example 11.8.

For the exponential distribution with rate $\theta>0$ , we have $\ell_{1}(\theta)=\log\theta-\theta X$ and thus

\ell_{1}^{\prime}(\theta)=\frac{1}{\theta}-X,\qquad\ell_{1}^{\prime\prime}(% \theta)=-\frac{1}{\theta^{2}},\qquad\mathcal{I}(\theta)=\frac{1}{\theta^{2}}.

Again, the second derivative is non-random. Section 11.5 shows how the information changes if we describe the model by the mean $1/\theta$ instead of the rate.

All four models from section 11.4 are exponential families, and exercise 11.2 shows that for such models the information is a variance which can be read off tables A.1 and A.2.

11.5 Reparametrisation

The information depends on which parameter we use to describe a model. The exponential distribution, for example, can be described by its rate or by its mean, and the following proposition gives the general rule for how the information changes.

Proposition 11.9.

Let $\psi=g(\theta)$ where $g\colon\Theta\to g(\Theta)$ is a twice continuously differentiable bijection with $g^{\prime}(\theta)\neq 0$ for all $\theta\in\Theta$ . Let $\mathcal{I}_{\theta}$ denote the Fisher information for a single observation when the model is parametrised by $\theta$ and let $\mathcal{I}_{\psi}$ be the information when the model is parametrised by $\psi$ . Then,

\mathcal{I}_{\psi}(\psi)=\frac{\mathcal{I}_{\theta}(\theta)}{g^{\prime}(\theta% )^{2}}\qquad\text{for }\psi=g(\theta).

Proof.

With the new parametrisation, the density is $\tilde{f}(x;\psi)=f\bigl{(}x;g^{-1}(\psi)\bigr{)}$ and the log-likelihood for a single observation is $\tilde{\ell}_{1}(\psi)=\ell_{1}\bigl{(}g^{-1}(\psi)\bigr{)}$ . Since $g^{\prime}$ is continuous and never vanishes on the interval $\Theta$ , it has the same sign throughout, so that $g$ is strictly monotonic and $g^{-1}$ is twice continuously differentiable. Using the chain rule, the conditions of section 11.1 for the parametrisation by $\theta$ carry over to the parametrisation by $\psi$ ; we omit the verification. The derivative of the inverse function is $(g^{-1})^{\prime}(\psi)=1/g^{\prime}(\theta)$ for $\theta=g^{-1}(\psi)$ . This gives

\tilde{\ell}_{1}^{\prime}(\psi)=\ell_{1}^{\prime}(\theta)\,(g^{-1})^{\prime}(% \psi)=\frac{\ell_{1}^{\prime}(\theta)}{g^{\prime}(\theta)}.

Since $\theta$ is fixed for fixed $\psi$ , $1/g^{\prime}(\theta)$ is a constant and taking variances under $\mathbb{P}_{\theta}$ gives $\mathcal{I}_{\psi}(\psi)=\mathcal{I}_{\theta}(\theta)/g^{\prime}(\theta)^{2}$ , as claimed. ∎

For the exponential distribution, reparametrising from the rate $\theta$ to the mean $\mu=1/\theta$ changes the information $1/\theta^{2}$ from example 11.8 to $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ ; see exercise 11.6. Thus, the information is not a property of the model in isolation, but depends on the parametrisation. In lecture 28 we will use the transformation rule of the proposition to construct a prior distribution, the Jeffreys prior, which does not depend on the parametrisation chosen.

11.6 When the Identities Fail

The uniform distribution from example 1.2 shows what can go wrong if the conditions from section 11.1 are violated.

Example 11.10.

Let $X$ be uniformly distributed on $[0,\theta]$ , i.e. $f(x;\theta)=1/\theta$ for $0\leq x\leq\theta$ . The support of the distribution depends on $\theta$ , but we can still compute the derivatives on the support to see whether the identities hold. For $0<X<\theta$ we have $\ell_{1}(\theta)=-\log\theta$ and thus $\ell_{1}^{\prime}(\theta)=-1/\theta$ and $\ell_{1}^{\prime\prime}(\theta)=1/\theta^{2}$ . The score is a negative constant and thus we have $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}=-1/\theta\neq 0$ , in contradiction to lemma 11.1. The variance is zero, whereas $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)^{2}\bigr{)}=1/\theta^{2}$ and $-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta)\bigr{)}=-1/\theta^% {2}$ : the three expressions for lemma 11.3 take three different values, one of which is negative. The proof breaks down at the interchange between differentiation and integration: for this model, the two sides of (11.1) with $k=1$ differ by the term $f(\theta;\theta)=1/\theta$ from the moving upper limit, which is missed by the differentiation under the integral sign. Exercise 11.7 can be used to work out the two computations.

For this model the Fisher information is not defined, and the two results based on it, in lectures 13 and 17, do not apply.

Summary.

•

Under the conditions of section 11.1 (a support free of $\theta$ , twice continuously differentiable density, differentiation under the integral sign and a score with finite variance), the score at the true parameter value has mean zero.
•

The Fisher information $\mathcal{I}_{n}(\theta)$ is the variance of the score, and it equals $\mathbb{E}_{\theta}\bigl{(}\ell^{\prime}(\theta)^{2}\bigr{)}$ and $-\mathbb{E}_{\theta}\bigl{(}\ell^{\prime\prime}(\theta)\bigr{)}$ , the expected curvature of the log-likelihood at the truth.
•

Information from independent observations can be combined, $\mathcal{I}_{n}(\theta)=n\mathcal{I}(\theta)$ . The information depends on the parametrisation, and under $\psi=g(\theta)$ we get $\mathcal{I}_{\psi}(\psi)=\mathcal{I}_{\theta}(\theta)/g^{\prime}(\theta)^{2}$ .
•

We have learned the following standard values: $1/\bigl{(}\theta(1-\theta)\bigr{)}$ for the Bernoulli distribution, $1/\theta$ for the Poisson distribution, $1/\sigma^{2}$ for the mean of the normal distribution and $1/\theta^{2}$ for the rate parameter in the exponential distribution. For the uniform distribution these relations do not hold, since the support of the distribution moves with the parameter.
•

The information is the currency of the two results presented later: it gives a bound on the variance of unbiased estimators in lecture 13 and it determines the asymptotic variance of the MLE in lecture 17.

Exercise 11.1.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ , and let $\ell_{1}^{\prime}(\theta)$ be the score of a single observation from example 11.5.

1.

Verify lemma 11.1 for this model by computing $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}$ directly from the two possible values of $X_{1}$ .
2.

Compute $\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}$ directly, and check that it agrees with the value of $\mathcal{I}(\theta)$ found in example 11.5.
3.

Compare $1/\mathcal{I}_{n}(\theta)$ with the variance of the estimator $\bar{X}$ for $\theta$ .
4.

Now let $Y$ be a single observation from the binomial distribution with known number of trials $m$ and success probability $\theta$ . Show that the information of $Y$ about $\theta$ is $m/\bigl{(}\theta(1-\theta)\bigr{)}$ , and explain the relation to the Bernoulli case using proposition 11.4.

Exercise 11.2.

Let $f(x;\eta)=h(x)\exp\bigl{(}\eta T(x)-K(\eta)\bigr{)}$ be an exponential family in its natural parametrisation from definition 7.9, where $\eta$ ranges over an open interval inside the natural parameter space $\mathcal{N}$ , and let $X$ be a single observation.

1.

Show that the Fisher information about $\eta$ is given by $\mathcal{I}_{\eta}(\eta)=K^{\prime\prime}(\eta)=\mathop{\mathrm{Var}}\nolimits% _{\eta}\bigl{(}T(X)\bigr{)}$ .
2.

Now let the family be written as $f(x;\theta)=h(x)\exp\bigl{(}\eta(\theta)T(x)-B(\theta)\bigr{)}$ as in definition 7.1, where $\eta$ is twice continuously differentiable and takes values in the interior of $\mathcal{N}$ . Show that $B(\theta)=K\bigl{(}\eta(\theta)\bigr{)}$ and that the information about $\theta$ is given by $\mathcal{I}_{\theta}(\theta)=\eta^{\prime}(\theta)^{2}\,\mathop{\mathrm{Var}}% \nolimits_{\theta}\bigl{(}T(X)\bigr{)}$ .
3.

Use the previous part to recover the information

$\mathcal{I}(\theta)=\frac{1}{\theta(1-\theta)}$

of the Bernoulli distribution from example 7.2.

Exercise 11.3.

Let $X_{1},\dots,X_{n}$ be i.i.d. with density $f(x;\theta)=\theta^{-2}x\,e^{-x/\theta}$ for $x\geq 0$ , where $\theta>0$ , the gamma distribution with known shape $2$ and scale $\theta$ from exercise 5.2, i.e. $\text{Gamma}(2,1/\theta)$ in the rate parametrisation from table A.2.

1.

Compute the score $\ell_{1}^{\prime}(\theta)$ of a single observation and verify that it has mean zero.
2.

Show that $\mathcal{I}(\theta)=2/\theta^{2}$ , once via $-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta)\bigr{)}$ and once via the variance of the score, using $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=2\theta^{2}$ .
3.

Write the density in the form of definition 7.1 with $T(x)=x$ , identify $\eta(\theta)$ , and recover $\mathcal{I}(\theta)$ from exercise 11.2.
4.

In exercise 5.2 we found that the MLE $\hat{\theta}=\bar{X}/2$ is unbiased with $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\theta^{2}/(2n)$ . Compare this with $1/\mathcal{I}_{n}(\theta)$ .

Exercise 11.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. from the geometric distribution with parameter $p\in(0,1)$ , i.e. $f(x;p)=(1-p)^{x-1}p$ for $x\in\mathbb{N}$ , where $\mathbb{E}_{p}(X_{1})=1/p$ and $\mathop{\mathrm{Var}}\nolimits_{p}(X_{1})=(1-p)/p^{2}$ , as in exercises 4.2 and 5.1.

1.

Compute the score $\ell_{1}^{\prime}(p)$ of a single observation and verify that it has mean zero.
2.

Show that $\mathcal{I}(p)=1/\bigl{(}p^{2}(1-p)\bigr{)}$ and write down $\mathcal{I}_{n}(p)$ .
3.

Verify the value from the previous part by taking the variance of the score.

Exercise 11.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. Poisson with mean $\theta>0$ .

1.

From lecture 7 we know that the natural parameter is $\eta=\log\theta$ and the cumulant function is $K(\eta)=e^{\eta}$ . Using exercise 11.2, find the information $\mathcal{I}_{\eta}(\eta)$ about $\eta$ in terms of $\theta$ .
2.

Recover $\mathcal{I}_{\theta}(\theta)=1/\theta$ from the previous part using proposition 11.9.
3.

In exercise 5.7 we have estimated the probability $p_{0}=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ . Determine the information $\mathcal{I}_{p_{0}}(p_{0})$ about $p_{0}$ and describe its behaviour as $\theta\to\infty$ . Give a one-sentence answer as to why this behaviour is reasonable.

Exercise 11.6.

Let $X_{1},\dots,X_{n}$ be i.i.d. exponential with rate $\theta>0$ and let $\mu=1/\theta$ be the mean.

1.

Using proposition 11.9 and example 11.8, verify the value $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ from section 11.5.
2.

Show that the information of the sample about $\mu$ is $n/\mu^{2}$ and compare the reciprocal of this quantity to the variance of the estimator $\bar{X}$ for $\mu$ .

Exercise 11.7.

Let $X$ be uniformly distributed on the interval $[0,\theta]$ as described in example 11.10. Compute both sides of (11.1) for this model, using $k=1$ , and verify that the difference between the two quantities corresponds to the contribution $f(\theta;\theta)=1/\theta$ of the (moving) upper limit of the integral.