Lecture 14 Multiparameter Fisher Information

Setting the stage as in lecture 11, we now assume that the parameter space $\Theta\subseteq\mathbb{R}^{k}$ is an open set and that the derivatives with respect to $\theta$ are partial derivatives, denoted by $\partial_{j}$ for the derivative with respect to $\theta_{j}$. The data $X_{1},\dots,X_{n}$ are now i.i.d. with density or probability weights $f(x;\theta)$ and the log-likelihood function is $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$, where $\ell_{i}(\theta)=\log f(X_{i};\theta)$, and we assume that the conditions of section 11.1 hold for each coordinate, allowing for differentiation under the integral sign for any pair of coordinates, and we assume that each coordinate of the score has finite variance under $\mathbb{P}_{\theta}$, so that the entries of the information matrix below are finite.

For $k=1$ the score vector is the score $\ell^{\prime}(\theta)$ from definition 4.4. Setting the score vector to zero leads to the system of likelihood equations for the normal distribution considered in example 5.4. As in lecture 11, we will consider the score vector to be a random vector under $\mathbb{P}_{\theta}$ and we will determine the covariance of this random vector. The covariance matrix $\mathop{\mathrm{Cov}}\nolimits(Y)$ of a random vector $Y=(Y_{1},\dots,Y_{k})$ has entries $\mathop{\mathrm{Cov}}\nolimits(Y_{j},Y_{l})$. The diagonal of this matrix contains the variances.

The diagonal element $\mathcal{I}_{n}(\theta)_{jj}$ is the Fisher information about $\theta_{j}$ in the sense of definition 11.2 with all other coordinates fixed to their true values; the off-diagonal elements are new and measure the strength of correlations between the components of the score. Both identities from lecture 11 carry over to this situation, separately for each coordinate.

The proof of this result is similar to the proof of lemma 11.1 and theorem 11.3, but with partial derivatives instead of $\mathrm{d}/\mathrm{d}\theta$. We omit the details of the proof here, but we can mention that the ratio $\partial_{j}f/f$ has mean zero since the density in the numerator and denominator cancels. Using the quotient rule we find $\partial_{j}\partial_{l}\ell_{i}=\partial_{j}\partial_{l}f(X_{i};\theta)/f(X_{i};\theta)-\partial_{j}\ell_{i}\,\partial_{l}\ell_{i}$ where the first term has mean zero. Finally, the independence of the observations allows us to carry over the resulting identity from one observation to the sample. As for the scalar case, the second expression is the more practical one, at least for $k=2$, because it requires only three second derivatives (one mixed derivative, occurring twice by symmetry).

The three properties of the scalar information which we used most often in lecture 11 have matrix counterparts. Recall that a symmetric matrix $A$ is positive semi-definite, if $a^{\top}Aa\geq 0$ for all $a\in\mathbb{R}^{k}$, and positive definite, if $a^{\top}Aa>0$ for all $a\neq 0$; a positive definite matrix has a positive definite inverse.

For $k=1$ the third statement is a consequence of proposition 11.10: If $\psi=g(\theta)$ we have $h=g^{-1}$ and $J=1/g^{\prime}(\theta)$. Positive definiteness is violated in cases where the log-likelihood does not respond to changes of $\theta$ in some direction, for example when the model is not identifiable in the sense of definition 4.10. From now on we assume that $\mathcal{I}_{n}(\theta)$ is positive definite so that the inverse exists.

Theorem 13.2 bounds the variance of an unbiased estimator for a scalar parameter from below. For a vector parameter the natural object to bound is the covariance matrix, and the natural reading of “at least” between symmetric matrices is that the difference is positive semi-definite.

The bound is almost always applied in its diagonal form: The variance of any unbiased estimator for $\theta_{j}$, when the other components are unknown, is at least the $j$th diagonal entry of the inverse information matrix. Note that this is the diagonal of the inverse, not the reciprocal of the diagonal; we will see the difference in section 14.5. An estimator which achieves the bound for $\theta_{j}$ is said to be efficient for $\theta_{j}$, as in lecture 13. The bound for a function $g(\theta)$ with gradient $\nabla g$ is $\nabla g(\theta)^{\top}\mathcal{I}_{n}(\theta)^{-1}\nabla g(\theta)$, which includes the statement of theorem 13.3 for $k=1$, as well as the linear case $g(\theta)=a^{\top}\theta$, which the theorem already covers. The result is also used in exercise 14.3.

The only example of a distribution which all students should be able to reproduce without notes is the normal distribution with unknown parameters. We will consider this example in detail.

The information matrix for the normal distribution is diagonal. The following section explains the significance of this fact and what happens when this is not the case.

Assume that we are interested in $\theta_{1}$ only, and that the remaining components $\theta_{2},\dots,\theta_{k}$ are nuisance parameters, i.e. unknown but not of interest. If the values of the nuisance parameters were known, the Cramer–Rao bound for $\theta_{1}$ would be $1/\mathcal{I}_{n}(\theta)_{11}$. Since the values are unknown, the bound is $\bigl{(}\mathcal{I}_{n}(\theta)^{-1}\bigr{)}_{11}$ by theorem 14.5. For $k=2$ we can compare the two bounds explicitly. For this we write

where $ac-b^{2}>0$ since the matrix is positive definite, and

is the correlation between the two components of the score vector. Since $0\leq\rho^{2}<1$, the bound where the nuisance parameter is unknown is $1/(1-\rho^{2})$ times larger than the bound where the nuisance parameter is known, and both bounds are equal if and only if $b=0$. Two parameters with $\mathcal{I}_{n}(\theta)_{12}=0$ are said to be orthogonal; in this case, not knowing $\theta_{2}$ does not incur any additional cost for the bound when estimating $\theta_{1}$. The same inequality $\bigl{(}\mathcal{I}_{n}(\theta)^{-1}\bigr{)}_{jj}\geq 1/\mathcal{I}_{n}(\theta)_{jj}$ holds for all $j$ and all $k$: not knowing a parameter can never be beneficial.

The normal distribution is the orthogonal case: in example 14.6 the mixed derivative has mean zero, and the bound for $\mu$ is $\sigma^{2}/n$ whether or not $\sigma^{2}$ is known. This is a special feature of the normal distribution, not the rule. For the gamma distribution with shape $\alpha>0$ and rate $\beta>0$ both unknown, as in exercise 7.3, all three second derivatives of the log-likelihood $\ell_{1}(\alpha,\beta)$ of one observation are non-random, and the information matrix of one observation is

where $\psi^{\prime}$, the second derivative of $\log\Gamma$, is the trigamma function; exercise 14.2 derives this matrix. The off-diagonal entry is non-zero and by  (14.2) we have $\rho^{2}=1/\bigl{(}\alpha\psi^{\prime}(\alpha)\bigr{)}$; for $\alpha=2$, with $\psi^{\prime}(2)=\pi^{2}/6-1\approx 0.645$, this gives $1/(1-\rho^{2})\approx 4.45$, so that the bound with the other parameter unknown is about four and a half times the bound with the other parameter known.

The inverse information matrix is needed when we estimate several parameters simultaneously. In lecture 17 we will see that, for large sample size, the maximum likelihood estimator is approximately normally distributed with covariance matrix $\mathcal{I}_{n}(\theta_{0})^{-1}$. In lecture 18 we will see how we can numerically find the standard errors of the estimator by inverting the observed Hessian matrix, and in lecture 23 we will see that the number of parameters $k$ can be seen as the number of degrees of freedom in this system.