Lecture 7 Exponential Families

We start by considering a familiar distribution and then rewrite its probability weights so that the data and the parameter are as far apart as possible. For the Bernoulli distribution with $x\in\{0,1\}$ we can write

Here we have written the weights as the exponential of the product of one function of $\theta$ and one function of $x$, minus a function of only $\theta$. The Poisson weights and the exponential density can be written in a similar way: we have $\theta^{x}=e^{x\log\theta}$ and $\theta e^{-\theta x}=e^{-\theta x+\log\theta}$, up to a factor depending on $x$. We will give a name to this structure.

Since the exponential is always positive, $f(x;\theta)$ is positive if and only if $h(x)$ is. This proves the statement about the support. The representation is not unique, since we can replace $\eta$ by $c\eta$ and $T$ by $T/c$ for any constant $c\neq 0$. Finally, since the density must integrate (or sum) to one, the function $B$ is determined by the other three functions: we have

These rewritings give the following examples.

The normal distribution is also an exponential family, but since both parameters are unknown, we need to use the two-parameter version of the definition from section 7.3.

The shape survives the passage from one observation to a sample.

Read as a function of $\theta$, the joint density is the likelihood, and thus the log-likelihood of an exponential family sample is

The first term does not involve $\theta$ and thus, except for the term involving $n$, the likelihood depends on the data only via the single number $T_{n}=\sum_{i}T(x_{i})$. This is an extension of example 4.3 to the whole class; in lecture 8 we will see in which sense the value $T_{n}$ contains all the information about $\theta$.

Since the parameter enters the density only via $\eta(\theta)$, it is often convenient to use $\eta$ as the parameter. In this case, the normalising function (7.1) could be given a name, since its derivatives are the moments of the natural statistic.

Since $K$ makes $f(x;\eta)$ integrate to one, the formula defines a probability distribution for every $\eta\in\mathcal{N}$, whether or not $\eta$ is of the form $\eta(\theta)$; comparing with (7.1) shows that $\eta(\theta)\in\mathcal{N}$ and $B(\theta)=K\bigl{(}\eta(\theta)\bigr{)}$, so that the original family sits inside this larger one. For the three examples of section 7.1 we find $K(\eta)=\log(1+e^{\eta})$ with $\mathcal{N}=\mathbb{R}$ for the Bernoulli distribution, $K(\eta)=e^{\eta}$ with $\mathcal{N}=\mathbb{R}$ for the Poisson distribution and $K(\eta)=-\log(-\eta)$ with $\mathcal{N}=(-\infty,0)$ for the exponential distribution. In all three cases $\mathcal{N}$ is an open interval and $\theta\mapsto\eta(\theta)$ is a bijection onto $\mathcal{N}$, making the natural parametrisation a relabelling in the sense of theorem 5.6.

The first two derivatives of $K$ give the mean and the variance of $T(X)$. The name of $K$ is derived from the fact that $s\mapsto K(\eta+s)-K(\eta)$ is the cumulant generating function of $T(X)$, as given in exercise 7.6.

The theorem allows us to compute moments by differentiation: for the Poisson distribution $K(\eta)=e^{\eta}$ we find $K^{\prime}(\eta)=K^{\prime\prime}(\eta)=e^{\eta}=\theta$, the mean and variance from table A.1, and in exercise 7.4 we consider the exponential and normal distributions. For $T_{n}$, being a sum of $n$ i.i.d. copies of $T(X)$, we can use the theorem to find $\mathbb{E}_{\eta}(T_{n})=nK^{\prime}(\eta)$ and $\mathop{\mathrm{Var}}\nolimits_{\eta}(T_{n})=nK^{\prime\prime}(\eta)$.

The normal distribution where both $\mu$ and $\sigma^{2}$ are unknown does not fit definition 7.1, since there are two unknown quantities in the exponent. The extension here is to allow the sum of products in the exponent, one for each parameter.

The support of the distribution is again $\{x\mid h(x)>0\}$ and the proof of lemma 7.5 applies without change: an i.i.d. sample is again a $k$-parameter exponential family, where the natural statistic is the vector of the $k$ sums $\sum_{i}T_{j}(X_{i})$. The cumulant function can be extended to this situation, but we do not need this result here.

The gamma distribution with both parameters unknown is a second example of a two-parameter exponential family. If we write $x^{\alpha-1}=e^{(\alpha-1)\log x}$, then the natural statistic is $(\log x,\,x)$ and the details can be worked out using exercise 7.3. For a sample of size $n$, the natural statistic is the pair $\bigl{(}\sum_{i}\log x_{i},\sum_{i}x_{i}\bigr{)}$, which is different from the pair of moments used in example 4.9. This is the first sign that the moment estimator is missing some information. (We will see in lecture 8 that this is indeed the case.)

For the completeness statements in lecture 10 it is important that the natural parameter varies in as many directions as there are natural statistics. Mathematically, this can be expressed as the following condition.

In the one-parameter case, the condition is almost always satisfied, since the image of an interval $\Theta$ under a continuous, non-constant $\eta$ is an interval with more than one point; all the one-parameter examples above are of full rank. For several parameters, the condition can be more interesting: the natural parameter values of the normal distribution fill the open half-plane $\{(\eta_{1},\eta_{2})\mid\eta_{2}<0\}$ (example 7.9) and the natural parameter values of the gamma distribution fill an open quadrant (exercise 7.3), but the natural parameter values of $N(\theta,\theta^{2})$ with $\theta>0$, where the standard deviation equals the mean, form a curve in $\mathbb{R}^{2}$ which does not contain an open set and thus the two-parameter family in this case is not of full rank (exercise 7.7).

The definitions in this lecture have applications in many areas. In lecture 8 we see that the natural statistic $T_{n}$ of a sample is sufficient and lemma 7.5 is essentially the whole proof. In lecture 10 we see that full rank implies that $T_{n}$ is complete and we use the Lehmann–Scheffe theorem to find the best unbiased estimators. In lecture 11 we find that the Fisher information for the natural parameter equals $K^{\prime\prime}(\eta)$. In lecture 13 we see that the estimators which achieve the Cramer–Rao bound are, up to affine transformations, natural statistics. Finally, in lecture 26 we find that conjugate priors arise because the likelihood (7.2) depends on the data only via $T_{n}$ and $n$.

Exponential families also satisfy the regularity conditions from lectures 11, 16 and 17: the support of the distribution is fixed, the density is smooth in $\theta$ whenever $\eta$ and $B$ are, and we have differentiation under the integral sign as in the proof of theorem 7.7. The log-likelihood $\ell(\eta)=\sum_{i}\log h(x_{i})+\eta T_{n}-nK(\eta)$ has second derivative $-nK^{\prime\prime}(\eta)=-n\mathop{\mathrm{Var}}\nolimits_{\eta}\bigl{(}T(X)\bigr{)}<0$, since $T(X)$ is not constant, and thus is strictly concave in $\eta$. For this reason, the likelihood equation from exercise 7.5 has at most one root and this root is then the maximum likelihood estimator. This shows that the uniqueness hypothesis of the consistency theorem from lecture 16 holds for the whole class.