Lecture 14 Multiparameter Fisher Information

Setting the stage as in lecture 11, we now assume that the parameter space is an open set $\Theta\subseteq\mathbb{R}^{k}$. The data $X_{1},\dots,X_{n}$ are again i.i.d. with density or probability weights $f(x;\theta)$ and the log-likelihood function is $\ell(\theta)=\sum_{i=1}^{n}\ell_{i}(\theta)$, where $\ell_{i}(\theta)=\log f(X_{i};\theta)$, but $\theta$ is now a vector. Derivatives with respect to $\theta$ are now partial derivatives, denoted by $\partial_{j}$ for the derivative with respect to $\theta_{j}$. We assume the conditions of section 11.1 with partial derivatives in place of $\mathrm{d}/\mathrm{d}\theta$, including the mixed derivatives $\partial_{j}\partial_{l}$, and we assume that each coordinate of the score has finite variance under $\mathbb{P}_{\theta}$.

For $k=1$ the score vector is the score $\ell^{\prime}(\theta)$ from definition 4.4. As in lecture 11, we consider the score vector as a random vector under $\mathbb{P}_{\theta}$. The covariance matrix $\mathop{\mathrm{Cov}}\nolimits(Y)$ of a random vector $Y=(Y_{1},\dots,Y_{k})$ is the $k\times k$ matrix with entries $\mathop{\mathrm{Cov}}\nolimits(Y_{j},Y_{l})$, and its diagonal contains the variances.

The diagonal element $\mathcal{I}_{n}(\theta)_{jj}$ is the Fisher information about $\theta_{j}$ in the sense of definition 11.2 with all other coordinates fixed to their true values; the off-diagonal elements are new and measure the strength of correlations between the components of the score. Both identities from lecture 11 carry over to this situation, separately for each coordinate.

The proof of this result is similar to the proof of lemma 11.1 and lemma 11.3, but with partial derivatives instead of $\mathrm{d}/\mathrm{d}\theta$. In outline, the ratio $\partial_{j}f/f$ has mean zero since the density in the numerator and denominator cancels. Using the quotient rule we find $\partial_{j}\partial_{l}\ell_{i}=\partial_{j}\partial_{l}f(X_{i};\theta)/f(X_{i};\theta)-\partial_{j}\ell_{i}\,\partial_{l}\ell_{i}$ where the first term has mean zero. Finally, the independence of the observations allows us to carry over the resulting identity from one observation to the sample. As for the scalar case, the second expression is often the more practical one.

Two properties of the scalar information from lecture 11 have matrix counterparts. Recall that a symmetric matrix $A$ is positive semi-definite, if $a^{\top}Aa\geq 0$ for all $a\in\mathbb{R}^{k}$, and positive definite, if $a^{\top}Aa>0$ for all $a\neq 0$; a positive definite matrix has a positive definite inverse.

Strict positive definiteness is violated in cases where the log-likelihood does not respond to changes of $\theta$ in some direction, for example for the model $N(\theta_{1}+\theta_{2},1)$. From now on we assume that $\mathcal{I}_{n}(\theta)$ is positive definite so that the inverse exists.

Theorem 13.2 bounds the variance of an unbiased estimator for a scalar parameter from below. For a vector parameter the natural object to bound is the covariance matrix, and the natural reading of “at least” between symmetric matrices is that the difference is positive semi-definite.

For real numbers, $a>b$ is the same as $a-b>0$. For symmetric matrices $A$ and $B$ we use the second form as the definition: we think of a matrix $A$ as larger than $B$, if $A-B$ is positive definite, and as at least as large as $B$, if $A-B$ is positive semi-definite. Thus $A$ is at least as large as $B$ exactly when $a^{\top}Aa\geq a^{\top}Ba$ for every $a\in\mathbb{R}^{k}$; unlike for numbers, two matrices may not be comparable at all.

The same argument gives a bound for a function $g(\theta)$ of the parameter. Let $W$ be an unbiased estimator for $g(\theta)$ with finite variance which satisfies (13.1) in every component of $\theta$. Then lemma 13.1 gives $\mathop{\mathrm{Cov}}\nolimits_{\theta}\bigl{(}W,b^{\top}\nabla\ell(\theta)\bigr{)}=b^{\top}\nabla g(\theta)$, and the proof above with $a^{\top}T$ replaced by $W$ and $a$ replaced by $\nabla g(\theta)$ shows

For $k=1$ this is theorem 13.2. The case $g(\theta)=a^{\top}\theta$ is used in exercise 14.3.

As an example, we apply the bound to the most important case, the normal distribution with both parameters unknown.

The information matrix for the normal distribution is diagonal. The following section explains the significance of this fact and what happens when this is not the case.

Assume that we are interested in $\theta_{1}$ only, and that the remaining components $\theta_{2},\dots,\theta_{k}$ are nuisance parameters, i.e. unknown but not of interest. If the values of the nuisance parameters were known, $\theta_{1}$ would be the only parameter and the Cramér–Rao bound for $\theta_{1}$ would be $1/\mathcal{I}_{n}(\theta)_{11}$. Since the values are unknown, the bound is $\bigl{(}\mathcal{I}_{n}(\theta)^{-1}\bigr{)}_{11}$ by proposition 14.5. The following lemma compares the two bounds.

Thus, not knowing the other parameters can never decrease the bound. For $k=2$ the bound for $\theta_{1}$ with $\theta_{2}$ unknown is $1/(1-\rho^{2})$ times the bound with $\theta_{2}$ known, and the two bounds agree if and only if $b=0$. This case has a name.

By the additivity of information, the condition does not depend on the sample size. For orthogonal parameters, not knowing $\theta_{2}$ does not increase the bound for $\theta_{1}$, and vice versa. We consider one example of each case.

Orthogonality is a special feature of the normal distribution, not the rule, as the following example shows.

The inverse information matrix will appear again in lecture 17, where we will see that for large samples the maximum likelihood estimator is approximately normal with covariance matrix $\mathcal{I}_{n}(\theta_{0})^{-1}$.