Lecture 7 Exponential Families

We start by considering a familiar distribution and then rewrite its probability weights so that the data and the parameter are as far apart as possible. For the Bernoulli distribution with $x\in\{0,1\}$ we can write

Here we have written the weights as the exponential of the product of one function of $\theta$ and one function of $x$, plus a function of only $\theta$. The Poisson weights and the exponential density can be written in a similar way: for $x\in\mathbb{N}_{0}$ and $x\geq 0$ respectively we have

For the Poisson weights an additional factor $1/x!$ appears, which depends on $x$ alone. We will give a name to this structure.

Since the exponential is always positive, $f(x;\theta)$ is positive if and only if $h(x)$ is. Thus the support $\{x\mid f(x;\theta)>0\}$ of the distribution is $\{x\mid h(x)>0\}$ and does not depend on $\theta$. The representation is not unique, since we can replace $\eta$ by $c\eta$ and $T$ by $T/c$ for any constant $c\neq 0$. Finally, since the density must integrate (or sum) to one, the function $B$ is determined by the other three functions: we have

These rewritings give the following examples.

The normal distribution is also an exponential family, but since both parameters are unknown, we need to use the two-parameter version of the definition from section 7.2.

The shape survives the passage from one observation to a sample.

Read as a function of $\theta$, the joint density is the likelihood, and thus the log-likelihood of an exponential family sample is

The first term does not involve $\theta$. Thus, up to a factor which does not depend on $\theta$, the likelihood depends on the data only through the single number $T_{n}=\sum_{i}T(x_{i})$. This is an extension of example 4.3 to the whole class; in lecture 8 we will see in which sense the value $T_{n}$ contains all the information about $\theta$.

The normal distribution where both $\mu$ and $\sigma^{2}$ are unknown does not fit definition 7.1, since the exponent contains the two products $x\mu/\sigma^{2}$ and $-x^{2}/(2\sigma^{2})$, which cannot be combined into a single product $\eta(\theta)\,T(x)$. The extension is to allow a sum of $k$ such products in the exponent.

The number $k$ counts the products in the exponent, not the components of $\theta$. In most of our examples the two agree, but they do not have to: the normal distribution $N(\theta,\theta^{2})$ considered below has the single parameter $\theta>0$, but its exponent needs two products. An exponential family in which $k$ is larger than the dimension of $\Theta$ is called a curved exponential family.

The support of the distribution is again $\{x\mid h(x)>0\}$ and the proof of lemma 7.5 applies without change: an i.i.d. sample is again a $k$-parameter exponential family, where the natural statistic is the vector of the $k$ sums $\sum_{i}T_{j}(X_{i})$.

The gamma distribution with both parameters unknown is a second example of a two-parameter exponential family. If we write $x^{\alpha-1}=e^{(\alpha-1)\log x}$, then the natural statistic is $(\log x,\,x)$ and the details are worked out in exercise 7.1. For a sample of size $n$, the natural statistic is the pair $\bigl{(}\sum_{i}\log x_{i},\sum_{i}x_{i}\bigr{)}$, which is different from the pair of moments used in example 4.9.

For the completeness statements in lecture 10 it is important that the natural parameter varies in as many directions as there are natural statistics. Mathematically, this can be expressed as the following condition.

In the one-parameter case, the condition is almost always satisfied, since the image of an interval $\Theta$ under a continuous, non-constant $\eta$ is an interval with more than one point; all the one-parameter examples above are of full rank. For several parameters, the condition can be more interesting: the natural parameter values of the normal distribution fill the open half-plane $\{(\eta_{1},\eta_{2})\mid\eta_{2}<0\}$ (example 7.7) and the natural parameter values of the gamma distribution fill an open quadrant (exercise 7.1), but the natural parameter values of $N(\theta,\theta^{2})$ with $\theta>0$, where the standard deviation equals the mean, form a curve in $\mathbb{R}^{2}$ which does not contain an open set, and thus this curved exponential family is not of full rank (exercise 7.2).

We return to one-parameter exponential families. Since the parameter enters the density only via $\eta(\theta)$, it is often convenient to use $\eta$ as the parameter. In this case, the normalising function (7.1) deserves a name, since its first two derivatives are the mean and the variance of the natural statistic.

Since $K$ makes $f(x;\eta)$ integrate to one, the formula defines a probability distribution for every $\eta\in\mathcal{N}$, whether or not $\eta$ is of the form $\eta(\theta)$; comparing with (7.1) shows that $\eta(\theta)\in\mathcal{N}$ and $B(\theta)=K\bigl{(}\eta(\theta)\bigr{)}$, so that the original family sits inside this larger one. For the three examples of section 7.1 we find $K(\eta)=\log(1+e^{\eta})$ with $\mathcal{N}=\mathbb{R}$ for the Bernoulli distribution, $K(\eta)=e^{\eta}$ with $\mathcal{N}=\mathbb{R}$ for the Poisson distribution and $K(\eta)=-\log(-\eta)$ with $\mathcal{N}=(-\infty,0)$ for the exponential distribution. In all three cases $\mathcal{N}$ is an open interval and $\theta\mapsto\eta(\theta)$ is a bijection onto $\mathcal{N}$, making the natural parametrisation a relabelling in the sense of proposition 5.6.

The first two derivatives of $K$ give the mean and the variance of $T(X)$, and the name of $K$ comes from a related object. The moment generating function of a random variable $Y$ is the function $s\mapsto\mathbb{E}(e^{sY})$, defined for all $s$ for which the expectation is finite. Its logarithm $s\mapsto\log\mathbb{E}(e^{sY})$ is called the cumulant generating function of $Y$, and the derivatives of this function at $s=0$ are called the cumulants of $Y$. The first two cumulants are the mean and the variance of $Y$. For a random variable $Y$ with values in $\mathbb{N}_{0}$ the closely related probability generating function is given by

defined for all $z\geq 0$ for which the sum is finite. Substituting $z=e^{s}$ shows that $s\mapsto G_{Y}(e^{s})$ is the moment generating function of $Y$. In exercise 7.6 we show that the function

is the cumulant generating function of $T(X)$.

The lemma allows us to compute moments by differentiation: for the Poisson distribution $K(\eta)=e^{\eta}$ we find $K^{\prime}(\eta)=K^{\prime\prime}(\eta)=e^{\eta}=\theta$, the mean and variance from table A.1, and in exercise 7.5 we consider the exponential and normal distributions. For $T_{n}$, being a sum of $n$ i.i.d. copies of $T(X)$, we can use the lemma to find $\mathbb{E}_{\eta}(T_{n})=nK^{\prime}(\eta)$ and $\mathop{\mathrm{Var}}\nolimits_{\eta}(T_{n})=nK^{\prime\prime}(\eta)$.

We conclude this lecture by collecting what the exponential family shape tells us about the questions of lectures 4 and 5, and by indicating where the shape will be used again.

The first consequence concerns the likelihood equation. By lemma 7.5, written in the natural parametrisation, the log-likelihood of a sample is

and thus we have $\ell^{\prime}(\eta)=T_{n}-nK^{\prime}(\eta)$ and, by lemma 7.10, $\ell^{\prime\prime}(\eta)=-nK^{\prime\prime}(\eta)=-n\mathop{\mathrm{Var}}\nolimits_{\eta}\bigl{(}T(X)\bigr{)}$. If $T(X)$ is not constant, this variance is positive and $\ell$ is strictly concave on the interior of $\mathcal{N}$. Since $\ell^{\prime}$ is then strictly decreasing, the likelihood equation $\ell^{\prime}(\eta)=0$ has at most one root, and if there is a root, it is the maximum likelihood estimator for $\eta$. When $\theta\mapsto\eta(\theta)$ is a bijection, proposition 5.6 transfers this estimator back to $\theta$. The argument shows uniqueness, but not existence: for a Bernoulli sample with no successes we have $T_{n}=0$, and the equation $K^{\prime}(\eta)=e^{\eta}/(1+e^{\eta})=0$ has no solution. This is the case $s=0$ from the remark after example 5.3.

The second consequence concerns smoothness. Many of the results later in the module are proved by differentiating with respect to the parameter, and they require the support of the distribution not to depend on $\theta$, the density to be smooth in $\theta$ and derivatives to be taken inside integrals. For an exponential family the support is fixed by definition, the density is smooth in $\theta$ whenever $\eta$ and $B$ are, and the argument in the proof of lemma 7.10 allows differentiation under the integral sign whenever $\eta(\theta)$ lies in the interior of $\mathcal{N}$. Similarly, if $\eta$ is one-to-one and $T(X)$ is not constant, then different values of $\theta$ give different distributions, and the parameter is identifiable in the sense of definition 4.10. Under these mild conditions on $\eta$, the exponential families are the standard examples for the theory in the rest of the module.

Finally, the observation that the likelihood depends on the data only through $T_{n}$ and $n$ suggests that, once $T_{n}$ is known, the individual observations carry no further information about $\theta$. In lecture 8 we make this idea precise, and lemma 7.5 will do most of the work there. From then on, the natural statistic $T_{n}$ will be the starting point whenever we look for good estimators, and again when we update our knowledge about $\theta$ using the Bayesian approach in the second half of the module.