Lecture 26 Bayesian Inference: Priors and Posteriors

So far the parameter $\theta$ has been a fixed but unknown number, and a sentence such as “ $\theta$ is probably close to $0.3$ ” had no meaning. In this lecture and in lecture 28 we will see that the parameter itself can be given a probability distribution: the prior, chosen before the data are seen, and the posterior, obtained after the data are seen. (We will see an update in pictures in lecture 27.)

26.1 Prior, Likelihood and Posterior

Since $\theta$ is now a random variable, the model density is a conditional density given the parameter, and in the Bayesian lectures we write it as $f(x\mskip 1.0mu|\mskip 1.0mu\theta)$ , where $x=(x_{1},\dots,x_{n})$ is the whole data set, so that $f(x\mskip 1.0mu|\mskip 1.0mu\theta)=\prod_{i}f(x_{i}\mskip 1.0mu|\mskip 1.0mu\theta)$ is the likelihood $L(\theta)$ of lecture 4.

Definition 26.1.

Let $\theta$ be a random variable with values in $\Theta$ and density $\pi(\theta)$ , and assume that given $\theta$ the data $X=(X_{1},\dots,X_{n})$ have joint density $f(x\mskip 1.0mu|\mskip 1.0mu\theta)$ . Then the density $\pi$ is called the prior density (or prior) of $\theta$ and the conditional density $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)$ of $\theta$ given $X=x$ is called the posterior density (or posterior). The parameters in the prior are called hyperparameters.

From now on, unless otherwise specified, $\pi$ will denote a prior or posterior density. In lecture 19, we used the same letter to denote the mixture weight for a two-component model.

The posterior density can be found using Bayes’ theorem as given in proposition A.20 in appendix A, where the parameter plays the role of $A$ and the data play the role of $B$ .

Theorem 26.2.

In the situation of definition 26.1, the posterior density is given by

equation (26.1) (26.1)

\pi(\theta\mskip 1.0mu|\mskip 1.0mux)=\frac{f(x\mskip 1.0mu|\mskip 1.0mu\theta% )\,\pi(\theta)}{m(x)},\qquad\text{where }m(x)=\int_{\Theta}f(x\mskip 1.0mu|% \mskip 1.0mu\vartheta)\,\pi(\vartheta)\,\mathrm{d}\vartheta

is the marginal density of the data. The formula holds for all $x$ with $m(x)>0$ .

Proof.

The joint density of $(\theta,X)$ is given by $f(x\mskip 1.0mu|\mskip 1.0mu\theta)\,\pi(\theta)$ . Integrating out $\theta$ gives the marginal density $m(x)$ of $X$ . Dividing the joint density by $m(x)$ gives the conditional density of $\theta$ given $X=x$ , as claimed. ∎

The denominator $m(x)$ makes the posterior integrate to one and does not involve $\theta$ ; in practice we never compute it directly, but write

equation (26.2) (26.2)

\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto f(x\mskip 1.0mu|\mskip 1.0mu% \theta)\,\pi(\theta),

where $\propto$ means “equal up to a factor free of $\theta$ ”, and we try to recognise the right-hand side as the kernel of a known density, whose normalising constant then comes from table A.2. In words, the posterior is proportional to the likelihood times the prior.

The posterior is the complete Bayesian answer: a statement like “ $\mathbb{P}(\theta>0.3\mskip 1.0mu|\mskip 1.0mux)=0.95$ ” now has a meaning. Point estimates are summaries of the posterior (section 26.3), and intervals and tests will be considered in lecture 28.

26.2 Conjugate Priors

The rule (26.2) works best if the posterior distribution belongs to the same family of distributions as the prior distribution, so that the update amounts to no more than a change of the hyperparameters.

Definition 26.3.

A family $\mathcal{P}$ of prior distributions is conjugate for a model $\bigl{(}f(x\mskip 1.0mu|\mskip 1.0mu\theta)\!\mathrel{\big{|}}\!\theta\in% \Theta\bigr{)}$ , if for every prior in $\mathcal{P}$ and every data set $x$ the posterior $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)$ also belongs to $\mathcal{P}$ .

For exponential families, conjugate priors can be written down by inspection: the likelihood depends on the natural parameter $\eta$ only through the exponent $\eta T_{n}-nK(\eta)$ (lemma 7.5), and a prior with an exponent of the same form reproduces itself.

Proposition 26.4.

Let $f(x\mskip 1.0mu|\mskip 1.0mu\eta)=h(x)\exp\bigl{(}\eta T(x)-K(\eta)\bigr{)}$ with $\eta\in\mathcal{N}$ be a one-parameter exponential family in its natural parametrisation, and for real numbers $a$ and $b$ let

\pi_{a,b}(\eta)\propto\exp\bigl{(}a\eta-bK(\eta)\bigr{)},\qquad\eta\in\mathcal% {N},

whenever the right-hand side has a finite integral over $\mathcal{N}$ . Then these priors are conjugate for the model: if the prior is $\pi_{a,b}$ , the data $x_{1},\dots,x_{n}$ have natural statistic $T_{n}=\sum_{i=1}^{n}T(x_{i})$ and the marginal density satisfies $0<m(x)<\infty$ , then the posterior is $\pi_{a+T_{n},\,b+n}$ .

Proof.

Using (26.2) and lemma 7.5 we find

\displaystyle\pi(\eta\mskip 1.0mu|\mskip 1.0mux)

\displaystyle\propto\prod_{i=1}^{n}h(x_{i})\,\exp\bigl{(}\eta T_{n}-nK(\eta)% \bigr{)}\,\exp\bigl{(}a\eta-bK(\eta)\bigr{)}

\displaystyle\propto\exp\bigl{(}(a+T_{n})\eta-(b+n)K(\eta)\bigr{)},

since $\prod_{i}h(x_{i})$ does not depend on $\eta$ . The right-hand side is the kernel of $\pi_{a+T_{n},b+n}$ and by assumption $m(x)<\infty$ this kernel has finite integral and thus is a density. This shows that the posterior is $\pi_{a+T_{n},b+n}$ . This completes the proof. ∎

The update adds the natural statistic to $a$ and the sample size to $b$ : the prior acts like $b$ earlier observations with statistic total $a$ . We now work the recipe for our simplest model in the original parameter.

Example 26.5.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta\in(0,1)$ and let the prior be the $\text{Beta}(a,b)$ distribution with $a,b>0$ . From table A.2 we know that this distribution has density

\pi(\theta)=\frac{\theta^{a-1}(1-\theta)^{b-1}}{B(a,b)},\qquad 0<\theta<1,

and the mean of this distribution is $a/(a+b)$ . If $T=\sum_{i}x_{i}$ is the number of successes in the sample, then the likelihood is $f(x\mskip 1.0mu|\mskip 1.0mu\theta)=\theta^{T}(1-\theta)^{n-T}$ , as in example 4.2, and thus we have

\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto\theta^{T}(1-\theta)^{n-T}\cdot% \theta^{a-1}(1-\theta)^{b-1}=\theta^{a+T-1}(1-\theta)^{b+n-T-1},

where we have omitted the constant $1/B(a,b)$ for simplicity. The right-hand side is the kernel of a $\text{Beta}(a+T,\,b+n-T)$ density and thus we have found the posterior distribution: the Beta distributions are conjugate for the Bernoulli model, and the update adds the successes to $a$ and the failures to $b$ . The posterior mean is given by

equation (26.3) (26.3)

\mathbb{E}(\theta\mskip 1.0mu|\mskip 1.0mux)=\frac{a+T}{a+b+n}=\frac{a+b}{a+b+% n}\cdot\frac{a}{a+b}+\frac{n}{a+b+n}\cdot\bar{x},

which is a weighted average of the prior mean $a/(a+b)$ and the sample proportion $\bar{x}=T/n$ , i.e. the maximum likelihood estimate from example 5.3. The prior can be interpreted as having observed $a+b$ additional samples, of which $a$ were successes. This equivalent sample size is a measure for how the hyperparameters should be chosen in practice (exercise 26.1). As $n\to\infty$ , the weight of the prior decreases and with enough data the prior will be completely forgotten. For $a=b=2$ , $n=10$ and $T=7$ , for example, the posterior distribution is $\text{Beta}(9,5)$ with mean $9/14\approx 0.643$ , between the prior mean $0.5$ and the sample proportion $0.7$ .

In terms of the log-odds $\eta$ , the example is an instance of proposition 26.4. Using the same arguments, we can show that normal priors are conjugate for the normal mean with known variance. The cumulant function in this case is a quadratic function of $\eta$ (exercise 7.4). The result is given here without proof, and the proof, based on completing the square, is given as exercise 26.3.

Proposition 26.6.

Let $X_{1},\dots,X_{n}$ be i.i.d. $N(\theta,\sigma^{2})$ with known variance $\sigma^{2}$ and let the prior be $\theta\sim N(m,\tau^{2})$ . Then the posterior distribution is $N(m_{n},\tau_{n}^{2})$ where

\frac{1}{\tau_{n}^{2}}=\frac{1}{\tau^{2}}+\frac{n}{\sigma^{2}}\qquad\text{and}% \qquad m_{n}=\frac{1/\tau^{2}}{1/\tau^{2}+n/\sigma^{2}}\,m+\frac{n/\sigma^{2}}% {1/\tau^{2}+n/\sigma^{2}}\,\bar{x}.

The reciprocal of a variance is called a precision, and in these terms the proposition is easy to remember: precisions add, and the posterior mean is the precision-weighted average of the prior mean and $\bar{x}$ . Again, the prior washes out as $n\to\infty$ .

26.3 Point Estimates from the Posterior

If a single number is required for $\theta$ , the posterior mean, median and mode can all be used. If the errors are measured in terms of squared error, as in lecture 2, the posterior mean is the optimal choice.

Theorem 26.7.

Assume that the posterior distribution has finite variance. Then the posterior mean $\hat{\theta}_{B}=\mathbb{E}(\theta\mskip 1.0mu|\mskip 1.0mux)$ minimises the posterior expected squared error, i.e. for every number $d$ we have

\mathbb{E}\bigl{(}(\theta-d)^{2}\!\mathrel{\big{|}}\!x\bigr{)}\geq\mathbb{E}% \bigl{(}(\theta-\hat{\theta}_{B})^{2}\!\mathrel{\big{|}}\!x\bigr{)}=\mathop{% \mathrm{Var}}\nolimits(\theta\mskip 1.0mu|\mskip 1.0mux),

and the equality holds only for $d=\hat{\theta}_{B}$ .

Proof.

Adding and subtracting $\hat{\theta}_{B}$ inside the square, as in the proof of theorem 2.7, we get

\mathbb{E}\bigl{(}(\theta-d)^{2}\!\mathrel{\big{|}}\!x\bigr{)}=\mathbb{E}\bigl% {(}(\theta-\hat{\theta}_{B})^{2}\!\mathrel{\big{|}}\!x\bigr{)}+2(\hat{\theta}_% {B}-d)\,\mathbb{E}(\theta-\hat{\theta}_{B}\mskip 1.0mu|\mskip 1.0mux)+(\hat{% \theta}_{B}-d)^{2},

where all expectations are taken with respect to the posterior. The middle term vanishes, because $\mathbb{E}(\theta\mskip 1.0mu|\mskip 1.0mux)=\hat{\theta}_{B}$ , and the last term is non-negative and zero only for $d=\hat{\theta}_{B}$ . This completes the proof. ∎

The estimator $\hat{\theta}_{B}$ is called the Bayes estimator (under squared error) for $\theta$ . The theorem is a statement about fixed data. For fixed $\theta$ , and varying data, exercise 26.4 shows that the Bernoulli $\hat{\theta}_{B}$ is biased but consistent, and that neither the Bernoulli estimator nor $\bar{X}$ has smaller mean squared error for all $\theta$ .

26.4 Flat, Improper and Jeffreys Priors

Without prior knowledge to encode, the obvious choice is the flat prior which is a constant density on $\Theta$ . This prior does not prefer any value of $\theta$ . For the Bernoulli model this prior is $\text{Beta}(1,1)$ and from example 26.5 we know that the posterior mean is $(T+1)/(n+2)$ , which is close to $\bar{x}$ but is shifted towards $1/2$ . Since a constant density on an unbounded parameter space has infinite integral, the flat prior is improper. Nevertheless, this prior is still in use today: if $f(x\mskip 1.0mu|\mskip 1.0mu\theta)\,\pi(\theta)$ has finite integral, it is normalised and used as the posterior. The finiteness of the integral must be checked in each case (exercise 26.2). For the normal mean, the flat prior on $\mathbb{R}$ is the limit $\tau\to\infty$ in proposition 26.6 and in this case leads to the proper posterior $N(\bar{x},\sigma^{2}/n)$ .

The flat prior has a more basic defect: flatness depends on the parametrisation. If $\theta$ is uniform on $(0,1)$ , then $\psi=\theta^{2}$ has density $1/(2\sqrt{\psi})$ by proposition A.19, and a prior which claims to know nothing about $\theta$ claims to know something about $\theta^{2}$ . The Fisher information provides a parametrisation-free rule, since proposition 11.10 shows how it transforms.

Definition 26.8.

In the situation of section 11.1, the Jeffreys prior for $\theta$ is given by

\pi_{J}(\theta)\propto\sqrt{\mathcal{I}(\theta)},\qquad\theta\in\Theta,

where $\mathcal{I}(\theta)$ is the Fisher information of an observation.

This prior is justified by the following invariance property, which the flat prior lacks.

Proposition 26.9.

Let $\psi=g(\theta)$ where $g\colon\Theta\to g(\Theta)$ is a continuously differentiable bijection with $g^{\prime}(\theta)\neq 0$ for all $\theta$ and let $h=g^{-1}$ .

1.

If $\theta$ has the flat prior on $\Theta$ , then $\psi$ has the density $|h^{\prime}(\psi)|$ on $g(\Theta)$ . The density is flat only if $g$ is affine.
2.

If $\theta$ has the Jeffreys prior $\sqrt{\mathcal{I}_{\theta}(\theta)}$ , then $\psi$ has the density $\sqrt{\mathcal{I}_{\psi}(\psi)}$ . This is the Jeffreys prior computed using the $\psi$ -parametrisation.

Proof.

Using the change-of-variable formula from proposition A.19, if a density $\pi_{\theta}$ for $\theta$ is given, then a density of $\pi_{\psi}(\psi)=\pi_{\theta}\bigl{(}h(\psi)\bigr{)}\,|h^{\prime}(\psi)|$ for $\psi$ can be derived. The same substitution can be applied to improper densities. In the case of the flat prior $\pi_{\theta}\equiv 1$ , we have $|h^{\prime}(\psi)|$ and the derivative of $h$ is constant only if $h$ , i.e. if $g$ , is affine. For the Jeffreys prior we have $\pi_{\theta}(\theta)=\sqrt{\mathcal{I}_{\theta}(\theta)}$ and using proposition 11.10 and $h^{\prime}(\psi)=1/g^{\prime}(\theta)$ for $\theta=h(\psi)$ we get

\pi_{\psi}(\psi)=\sqrt{\mathcal{I}_{\theta}(\theta)}\,\bigl{|}h^{\prime}(\psi)% \bigr{|}=\frac{\sqrt{\mathcal{I}_{\theta}(\theta)}}{|g^{\prime}(\theta)|}=% \sqrt{\frac{\mathcal{I}_{\theta}(\theta)}{g^{\prime}(\theta)^{2}}}=\sqrt{% \mathcal{I}_{\psi}(\psi)}.

This is the Jeffreys prior in the $\psi$ -parametrisation. This completes the proof. ∎

The Jeffreys prior is a property of the model, rather than of the way it is labelled, and as such it can be proper or improper.

Example 26.10.

For the Bernoulli model, from example 11.5 we know $\mathcal{I}(\theta)=1/\bigl{(}\theta(1-\theta)\bigr{)}$ and thus the Jeffreys prior is given by

\pi_{J}(\theta)\propto\theta^{-1/2}(1-\theta)^{-1/2},\qquad 0<\theta<1.

This is the density of the $\text{Beta}(1/2,1/2)$ distribution. The Jeffreys prior is proper and has more weight near the boundaries of the interval than the flat prior does. From example 26.5 we know that the posterior distribution is $\text{Beta}(T+1/2,\,n-T+1/2)$ and the mean of the posterior is $(T+1/2)/(n+1)$ . This value is between the maximum likelihood estimate $T/n$ and the estimate $(T+1)/(n+2)$ obtained using the flat prior. For large $n$ , all three values coincide.

For location and scale models the Jeffreys prior is the same for every base density $f_{0}$ .

Proposition 26.11.

Let $f_{0}$ be a fixed density such that the following models satisfy the conditions of section 11.1.

1.

Consider the location model $f(x\mskip 1.0mu|\mskip 1.0mu\theta)=f_{0}(x-\theta)$ with $\theta\in\mathbb{R}$ . The Fisher information in this model is constant as a function of $\theta$ and thus the Jeffreys prior is flat, $\pi_{J}(\theta)\propto 1$ on $\mathbb{R}$ .
2.

Consider the scale model $f(x\mskip 1.0mu|\mskip 1.0mu\sigma)=f_{0}(x/\sigma)/\sigma$ with $\sigma>0$ . The Fisher information in this model is $\mathcal{I}(\sigma)=c/\sigma^{2}$ for a constant $c>0$ . Thus, the Jeffreys prior is $\pi_{J}(\sigma)\propto 1/\sigma$ on $(0,\infty)$ .

Both priors are improper.

Exercise 26.5 verifies both statements by writing the score as a function of the standardised observation. The normal mean with known variance is a location model, so its Jeffreys prior is flat. Similarly, the normal standard deviation with known mean is a scale model with $\mathcal{I}(\sigma)=2/\sigma^{2}$ .

Summary.

•

The data can be used to update the prior density $\pi(\theta)$ to the posterior $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto f(x\mskip 1.0mu|\mskip 1.0mu% \theta)\,\pi(\theta)$ . The normalising constant can be found by recognising the kernel of a known density.
•

Priors are conjugate for a family if the posterior always belongs to the same family. For a one-parameter exponential family in its natural parametrisation, the priors $\exp(a\eta-bK(\eta))$ are conjugate. Beta priors are conjugate for the Bernoulli model, and normal priors are conjugate for the normal mean with known variance.
•

The posterior mean is a weighted average of the prior mean and the maximum likelihood estimate. As $n$ increases, the prior weight decreases. The posterior mean is the Bayes estimator for $\theta$ under squared error.
•

A flat prior is not invariant under reparametrisation, but the Jeffreys prior $\pi_{J}(\theta)\propto\sqrt{\mathcal{I}(\theta)}$ is. The Jeffreys prior is $\text{Beta}(1/2,1/2)$ for the Bernoulli model, is flat for a location parameter, and is proportional to $1/\sigma$ for a scale parameter. An improper prior can only be used if the posterior is proper.

Exercise 26.1.

A Bernoulli model is to be fitted with a $\text{Beta}(a,b)$ prior for the success probability $\theta$ .

1.

Find $a$ and $b$ such that the prior mean is $0.3$ and the equivalent sample size $a+b$ is $10$ . You can find the prior standard deviation in table A.2.
2.

Find $a$ and $b$ such that the prior has mean $0.3$ and standard deviation $0.1$ . What is the equivalent sample size for this prior?
3.

For the prior in the first part of the question, a sample of size $n=20$ has $T=9$ successes. Determine the posterior distribution and the posterior mean. Write the posterior mean as a weighted average of the prior mean and $\bar{x}$ .

Exercise 26.2.

The Haldane prior for the Bernoulli success probability is the limit $a,b\to 0$ of the $\text{Beta}(a,b)$ prior, i.e. the function $\pi(\theta)\propto\theta^{-1}(1-\theta)^{-1}$ for $0<\theta<1$ .

1.

Show that this prior is improper.
2.

Show that the formal posterior $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto\theta^{T-1}(1-\theta)^{n-T-1}$ is a proper distribution if and only if $0<T<n$ . Identify the resulting distribution.
3.

For $0<T<n$ , determine the posterior mean and compare it to the maximum likelihood estimate.
4.

Describe what happens for $T=0$ .

Exercise 26.3.

Using the following steps, prove proposition 26.6.

1.

Show that the likelihood, as a function of $\theta$ , satisfies $f(x\mskip 1.0mu|\mskip 1.0mu\theta)\propto\exp\bigl{(}-n(\theta-\bar{x})^{2}/(% 2\sigma^{2})\bigr{)}$ .
2.

Multiply the prior density with this function and show, by completing the square in $\theta$ , that

$\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto\exp\Bigl{(}-\frac{(\theta-m_{n})^% {2}}{2\tau_{n}^{2}}\Bigr{)}$

where $m_{n}$ and $\tau_{n}^{2}$ are as in the proposition.
3.

What happens to $m_{n}$ and $\tau_{n}^{2}$ when $\tau\to\infty$ with $n$ fixed? And when $n\to\infty$ with $\tau$ fixed?

Exercise 26.4.

Let $X_{1},\dots,X_{n}$ be i.i.d. Bernoulli with success probability $\theta$ and let $\hat{\theta}_{B}=(a+T)/(a+b+n)$ with $T=\sum_{i}X_{i}$ be the Bayes estimator from example 26.5. Now fix $\theta$ and let $\hat{\theta}_{B}$ be an estimator as in lecture 2.

1.

What are the bias and variance of $\hat{\theta}_{B}$ ? Show that the estimator is biased for every $\theta\neq a/(a+b)$ , but consistent.
2.

For $a=b=1$ and $n=10$ , compare the estimators $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{B})$ and $\mathop{\mathrm{MSE}}\nolimits(\bar{X})$ for $\theta=1/2$ and $\theta=0.1$ .

Exercise 26.5.

Let $f_{0}$ be a fixed density as in proposition 26.11 and let $\ell_{0}(z)=\log f_{0}(z)$ .

1.

For the location model $f(x\mskip 1.0mu|\mskip 1.0mu\theta)=f_{0}(x-\theta)$ , show that the score is $-\ell_{0}^{\prime}(x-\theta)$ and thus that the Fisher information does not depend on $\theta$ .
2.

For the scale model $f(x\mskip 1.0mu|\mskip 1.0mu\sigma)=f_{0}(x/\sigma)/\sigma$ , show that the score is $-\bigl{(}1+z\ell_{0}^{\prime}(z)\bigr{)}/\sigma$ where $z=x/\sigma$ and that $\mathcal{I}(\sigma)=c/\sigma^{2}$ where $c$ does not depend on $\sigma$ .
3.

For the $N(0,\sigma^{2})$ model with parameter $\sigma$ , show that $\mathcal{I}(\sigma)=2/\sigma^{2}$ .