C.1 Statistical Models and the Estimation Problem

Here we solve the exercises at the end of lecture 1.

Solution to exercise 1.1.

1.

Since the tosses are independent and since each toss has probability $\theta$ of being heads, we can describe $X_{1},\dots,X_{n}$ as a sequence of independent and identically distributed Bernoulli trials with probability weights $f(x;\theta)=\theta^{x}(1-\theta)^{1-x}$ for $x\in\{0,1\}$ and parameter space $\Theta=(0,1)$ .
2.

We can use the proportion of heads in the sample, $\hat{\theta}=\bar{X}=(1/n)\sum_{i=1}^{n}X_{i}$ , as an estimator.
3.

The value $\hat{\theta}(X_{1},\dots,X_{n})$ is a function of the random variables $X_{i}$ before the coin tosses occur, so the value is still indeterminate. After the tosses have been made and the numbers $x_{1},\dots,x_{n}$ are known, $\hat{\theta}(x_{1},\dots,x_{n})$ is a function of numbers, i.e. a single number.

∎

Solution to exercise 1.2.

A statistic is a function of the data which does not use the unknown parameter $\theta=(\mu,\sigma^{2})$ .

1.

The range $\max_{i}X_{i}-\min_{i}X_{i}$ is computed from the data alone, so it is a statistic.
2.

The value $(1/n)\sum_{i}(X_{i}-\mu)^{2}$ contains the unknown mean $\mu$ and thus cannot be computed from the data, so it is not a statistic. If $\mu$ was known, then it would be a statistic.
3.

The sample variance $\bigl{(}1/(n-1)\bigr{)}\sum_{i}(X_{i}-\bar{X})^{2}$ uses the sample mean $\bar{X}$ , which is computed from the data, in place of $\mu$ . Thus, it is a statistic.

∎

Solution to exercise 1.3.

1.

Since the observations are independent, the joint density is the product of the marginal densities:

$f(x_{1},\dots,x_{n};\theta)=\prod_{i=1}^{n}\frac{1}{\theta}=\frac{1}{\theta^{% \,n}},$

whenever all observations are in the support, i.e. when $0\leq x_{i}\leq\theta$ for all $i$ . Otherwise, the density is zero. This can also be written as the density being non-zero only when $\min_{i}x_{i}\geq 0$ and $\max_{i}x_{i}\leq\theta$ .
2.

Each observation satisfies $X_{i}\leq\theta$ with probability one, since the support is $[0,\theta]$ . The maximum of numbers, none of which are larger than $\theta$ , cannot exceed $\theta$ . Thus, $\hat{\theta}=\max_{i}X_{i}\leq\theta$ .
3.

Since the distribution of each $X_{i}$ is continuous, we have $\mathbb{P}_{\theta}(X_{i}=\theta)=0$ for all $i$ . The maximum equals $\theta$ only if at least one observation equals this value, and thus we get

$\mathbb{P}_{\theta}(\hat{\theta}=\theta)\leq\sum_{i=1}^{n}\mathbb{P}_{\theta}(% X_{i}=\theta)=0.$

Together with the first part of the solution this shows that $\hat{\theta}<\theta$ with probability one. Thus, the random variable $\theta-\hat{\theta}$ is non-negative and is zero with probability zero. A non-negative random variable with expectation zero equals zero with probability one, and thus we find $\mathbb{E}_{\theta}(\theta-\hat{\theta})>0$ , which is the same as $\mathbb{E}_{\theta}(\hat{\theta})<\theta$ . Thus, the estimator never overestimates $\theta$ and underestimates $\theta$ on average.

∎

C.2 Bias, Mean Squared Error and Consistency

Here we solve the exercises at the end of lecture 2.

Solution to exercise 2.1.

1.

Since $\mathbb{E}_{\theta}(X_{i})=\theta$ , we can use lemma 2.2 to conclude $\mathbb{E}_{\theta}(\bar{X})=\theta$ . Thus we have $\mathop{\mathrm{bias}}\nolimits(\hat{\theta})=\mathbb{E}_{\theta}(\bar{X})-% \theta=0$ for all $\theta$ and the estimator $\hat{\theta}$ is unbiased.
2.

Since the estimator is unbiased, the mean squared error equals the variance. Using lemma 2.2 and the fact that $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{i})=\theta$ for the Poisson distribution, we find

$\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\mathop{\mathrm{Var}}\nolimits_{% \theta}(\bar{X})=\frac{\theta}{n}.$

Since this converges to zero as $n\to\infty$ , we can use theorem 2.10 to conclude that the estimator is consistent.

∎

Solution to exercise 2.2.

1.

Using the given density

$\mathbb{E}_{\theta}(\hat{\theta})=\int_{0}^{\theta}t\cdot\frac{n\,t^{n-1}}{% \theta^{n}}\,\mathrm{d}t=\frac{n}{\theta^{n}}\cdot\frac{\theta^{n+1}}{n+1}=% \frac{n\theta}{n+1},$

we find the bias to be $\mathop{\mathrm{bias}}\nolimits(\hat{\theta})=n\theta/(n+1)-\theta=-\theta/(n+1)$ . Since this converges to zero as $n\to\infty$ , the estimator is asymptotically unbiased. Since the bias is negative, the maximum systematically underestimates $\theta$ , as expected from exercise 1.3.
2.

We have

$\mathbb{E}_{\theta}(\hat{\theta}^{2})=\int_{0}^{\theta}t^{2}\cdot\frac{n\,t^{n% -1}}{\theta^{n}}\,\mathrm{d}t=\frac{n\theta^{2}}{n+2},$

and thus

$\mathop{\mathrm{Var}}\nolimits(\hat{\theta})=\frac{n\theta^{2}}{n+2}-\Bigl{(}% \frac{n\theta}{n+1}\Bigr{)}^{2}=\frac{n\theta^{2}}{(n+2)(n+1)^{2}},$

where we used $(n+1)^{2}-n(n+2)=1$ in the last step. By theorem 2.7 we find

$\displaystyle\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})$

$\displaystyle=\mathop{\mathrm{Var}}\nolimits(\hat{\theta})+\mathop{\mathrm{% bias}}\nolimits(\hat{\theta})^{2}$

$\displaystyle=\frac{n\theta^{2}}{(n+2)(n+1)^{2}}+\frac{\theta^{2}}{(n+1)^{2}}$

$\displaystyle=\frac{2\theta^{2}}{(n+1)(n+2)}.$

Since this converges to zero as $n\to\infty$ , we can conclude that $\hat{\theta}$ is consistent by theorem 2.10.
3.

The mean squared error is of order $1/n^{2}$ , while the mean squared error of the sample mean is of order $1/n$ . Thus, the maximum converges to the target faster than the average does. The very fast convergence is caused by the support of the uniform model changing as a function of the parameter. This effect, mentioned in example 1.2, will be seen again in lecture 13.

∎

Solution to exercise 2.3.

Since $\hat{\theta}=n/T$ , we have $\mathbb{E}_{\theta}(\hat{\theta})=n\,\mathbb{E}_{\theta}(1/T)$ and using the gamma density of $T$ we get

\mathbb{E}_{\theta}(1/T)=\int_{0}^{\infty}\frac{1}{x}\cdot\frac{\theta^{n}x^{n% -1}e^{-\theta x}}{(n-1)!}\,\mathrm{d}x=\frac{\theta^{n}}{(n-1)!}\int_{0}^{% \infty}x^{n-2}e^{-\theta x}\,\mathrm{d}x.

For $n\geq 2$ , the remaining integral can be evaluated as the gamma integral $\int_{0}^{\infty}x^{n-2}e^{-\theta x}\,\mathrm{d}x=(n-2)!/\theta^{n-1}$ . Thus we get

\mathbb{E}_{\theta}(\hat{\theta})=\frac{n\,\theta^{n}}{(n-1)!}\cdot\frac{(n-2)% !}{\theta^{n-1}}=\frac{n\theta}{n-1}.

The bias is given by

\mathop{\mathrm{bias}}\nolimits(\hat{\theta})=\frac{n\theta}{n-1}-\theta=\frac% {\theta}{n-1}.

Since this is strictly positive, we see that $\hat{\theta}$ systematically overestimates $\theta$ on average and thus is biased. As $n\to\infty$ , the bias converges to zero and thus $\hat{\theta}$ is asymptotically unbiased. This completes the solution. ∎

C.3 Likelihood and the Method of Moments

Here we solve the exercises at the end of lecture 4.

Solution to exercise 4.1.

1.

Using the Poisson weights from table 1.1, we find the likelihood to be

$L(\theta)=\prod_{i=1}^{n}\frac{\theta^{x_{i}}}{x_{i}!}\,e^{-\theta}=\frac{% \theta^{\sum_{i}x_{i}}}{\prod_{i}x_{i}!}\,e^{-n\theta}.$

Taking logarithms we get

$\ell(\theta)=\log\theta\sum_{i=1}^{n}x_{i}-n\theta-\sum_{i=1}^{n}\log(x_{i}!),$

as claimed.
2.

Taking derivatives with respect to $\theta$ , and noting that the last term does not depend on $\theta$ , we find the score function to be

$\ell^{\prime}(\theta)=\frac{1}{\theta}\sum_{i=1}^{n}x_{i}-n.$

Setting this equal to zero and solving gives $\theta=(1/n)\sum_{i}x_{i}=\bar{x}$ . Thus, the score function is zero at the sample mean.
3.

For the Poisson distribution we have $\mu_{1}(\theta)=\mathbb{E}_{\theta}(X_{1})=\theta$ . Thus, the only moment equation we have is $\hat{\theta}=\bar{X}$ : the method of moments estimator is the sample mean. This is the value at which the score function is zero, i.e. for the Poisson distribution the method of moments coincides with the maximum likelihood estimator from lecture 5.

∎

Solution to exercise 4.2.

1.

Since we have $\mu_{1}(p)=\mathbb{E}_{p}(X_{1})=1/p$ and the parameter is scalar, we get the moment equation $1/\hat{p}=\bar{X}$ . Solving this equation we find

$\hat{p}=\frac{1}{\bar{X}}.$
2.

Since each observation satisfies $X_{i}\geq 1$ , we have $\bar{X}\geq 1$ , with equality if and only if all observations equal one. Thus, the interval of $\hat{p}=1/\bar{X}$ values where the estimate is possible is $(0,1]$ . Since for every $p\in(0,1)$ the dataset with values in $\{1,2,\dots\}$ has positive probability, no estimate in this range is incompatible with the data (this is in contrast to the example of the uniform distribution in the text).

∎

Solution to exercise 4.3.

1.

From table A.2 we know that $\mathbb{E}_{\theta}(X_{1})=\mu$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=\sigma^{2}$ . Thus, the first population moment is $\mu_{1}(\theta)=\mu$ . Rewriting the relation $\mathop{\mathrm{Var}}\nolimits(X_{1})=\mathbb{E}(X_{1}^{2})-\mathbb{E}(X_{1})^% {2}$ we find

$\mu_{2}(\theta)=\mathbb{E}_{\theta}\bigl{(}X_{1}^{2}\bigr{)}=\mathop{\mathrm{% Var}}\nolimits_{\theta}(X_{1})+\mathbb{E}_{\theta}(X_{1})^{2}=\sigma^{2}+\mu^{% 2},$

as claimed.
2.

Using the moments from the first part, the moment equations from definition 4.6 are

$\hat{\mu}=m_{1}=\bar{X}\qquad\text{and}\qquad\hat{\sigma}^{2}+\hat{\mu}^{2}=m_% {2}=\frac{1}{n}\sum_{i=1}^{n}X_{i}^{2}.$

From the first equation we find $\hat{\mu}=\bar{X}$ directly. Substituting this value into the second equation we find

$\hat{\sigma}^{2}=\frac{1}{n}\sum_{i=1}^{n}X_{i}^{2}-\bar{X}^{2}=\frac{1}{n}% \sum_{i=1}^{n}(X_{i}-\bar{X})^{2},$

where the last equality sign comes from expanding the square $(X_{i}-\bar{X})^{2}=X_{i}^{2}-2X_{i}\bar{X}+\bar{X}^{2}$ and using the relation $\sum_{i}X_{i}=n\bar{X}$ . These are the estimators given in example 4.8 and thus the solution is complete.

∎

Solution to exercise 4.4.

1.

From table A.2 we know that the uniform model has $\mathbb{E}_{\theta}(X_{1})=\theta/2$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=\theta^{2}/12$ . Using lemma 2.2 we find $\mathbb{E}_{\theta}(\hat{\theta})=2\,\mathbb{E}_{\theta}(\bar{X})=2\cdot\theta% /2=\theta$ and thus the estimator $\hat{\theta}$ is unbiased. Since the estimator is unbiased, the mean squared error equals the variance by theorem 2.7, and using lemma 2.2 again we get

$\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\mathop{\mathrm{Var}}\nolimits(2% \bar{X})=4\cdot\frac{\theta^{2}/12}{n}=\frac{\theta^{2}}{3n}.$
2.

The maximum has smaller mean squared error, if and only if

$\frac{2\theta^{2}}{(n+1)(n+2)}<\frac{\theta^{2}}{3n}.$

This inequality is equivalent to $6n<(n+1)(n+2)$ , i.e. to $0<n^{2}-3n+2=(n-1)(n-2)$ for all $n\geq 3$ . (The equality signs hold for $n=1$ and $n=2$ .) Thus, for three or more observations the biased maximum is better than the unbiased method of moments estimator, and the $1/n^{2}$ rate of the maximum makes the difference increase quickly with $n$ . This is another example of the lesson from lecture 2 that unbiasedness is not sufficient: the moment estimator squashes the data into the sample mean, whereas the maximum uses the fact that the support of the distribution moves with the parameter.

∎

Solution to exercise 4.5.

1.

The distribution $N(0,\theta^{2})$ depends on $\theta$ only via $\theta^{2}$ . Thus, for different parameter values $\theta$ and $-\theta$ , the distribution is the same, e.g. $\theta_{1}=1$ and $\theta_{2}=-1$ both lead to $N(0,1)$ . Thus, the model is not identifiable.
2.

To make the model identifiable, we can restrict ourselves to $\Theta^{\prime}=(0,\infty)$ : for $0<\theta_{1}<\theta_{2}$ the distributions $N(0,\theta_{1}^{2})$ and $N(0,\theta_{2}^{2})$ have different variances and thus are different. (The other half line $(-\infty,0)$ would work equally well.)
3.

Since the first moment $\mathbb{E}_{\theta}(X_{1})=0$ for all $\theta$ , we cannot use this moment to derive any information about the parameter. We can use the second moment instead. Since $\mu_{2}(\theta)=\mathbb{E}_{\theta}(X_{1}^{2})=\theta^{2}$ , the moment equation is $\hat{\theta}^{2}=(1/n)\sum_{i}X_{i}^{2}$ and taking the positive root, as required for $\Theta^{\prime}=(0,\infty)$ , we get

$\hat{\theta}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}X_{i}^{2}}\,.$

This completes the solution.

∎

C.4 Maximum Likelihood Estimation

Here we solve the exercises at the end of lecture 5.

Solution to exercise 5.1.

1.

Using the geometric weights, the likelihood function is

$L(p)=\prod_{i=1}^{n}(1-p)^{x_{i}-1}p=p^{n}\,(1-p)^{\sum_{i}x_{i}-n}.$

Taking logarithms we get $\ell(p)=n\log p+\bigl{(}\sum_{i}x_{i}-n\bigr{)}\log(1-p)$ , as claimed.
2.

Taking derivatives we find the score as

$\ell^{\prime}(p)=\frac{n}{p}-\frac{\sum_{i}x_{i}-n}{1-p}.$

Setting this equal to zero and clearing the denominators we find $n(1-p)=p\bigl{(}\sum_{i}x_{i}-n\bigr{)}$ , i.e. $n=p\sum_{i}x_{i}$ . The only solution to this equation is $p=n/\sum_{i}x_{i}=1/\bar{x}$ . Since all observations satisfy $x_{i}\geq 1$ , we have $\sum_{i}x_{i}\geq n$ and thus

$\ell^{\prime\prime}(p)=-\frac{n}{p^{2}}-\frac{\sum_{i}x_{i}-n}{(1-p)^{2}}<0$

for all $p\in(0,1)$ . Thus, $\ell$ is strictly concave and the root is the global maximum. Thus, the MLE is $\hat{p}=1/\bar{X}$ . (If all observations equal one, i.e. if $\sum_{i}x_{i}=n$ , then $\ell(p)=n\log p$ is strictly increasing and no maximiser exists in the open interval $(0,1)$ . The same boundary phenomenon as for the Bernoulli distribution in the text occurs. In this case, the formula $\hat{p}=1/\bar{X}$ gives the boundary value $1$ .)
3.

In exercise 4.2 we have found the method of moments estimator to be $\hat{p}=1/\bar{X}$ . Thus, for the geometric distribution, as for the Bernoulli, Poisson and exponential distributions, the two different construction methods coincide.

∎

Solution to exercise 5.2.

1.

The log-likelihood function is

$\ell(\theta)=-2n\log\theta+\sum_{i=1}^{n}\log x_{i}-\frac{1}{\theta}\sum_{i=1}% ^{n}x_{i},$

and the score function is

$\ell^{\prime}(\theta)=-\frac{2n}{\theta}+\frac{1}{\theta^{2}}\sum_{i=1}^{n}x_{% i}.$

Setting the score equal to zero, we find $\sum_{i}x_{i}=2n\theta$ and thus the only root is $\theta=\bar{x}/2$ . Since the score is positive for $\theta<\bar{x}/2$ and negative for $\theta>\bar{x}/2$ , this root is the global maximum and the MLE is $\hat{\theta}=\bar{X}/2$ .
2.

Since the model is described by $\text{Gamma}(2,1/\theta)$ in the rate parametrisation, we can use table A.2 to find $\mathbb{E}_{\theta}(X_{1})=2\theta$ . Using lemma 2.2 we find $\mathbb{E}_{\theta}(\bar{X})=2\theta$ and thus $\mathbb{E}_{\theta}(\hat{\theta})=(1/2)\cdot 2\theta=\theta$ for all $\theta$ . Thus, $\hat{\theta}$ is unbiased.
3.

From table A.2 we find the variance to be $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=2\theta^{2}$ . Since the estimator is unbiased, the mean squared error equals the variance and using lemma 2.2 again, we find

$\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\mathop{\mathrm{Var}}\nolimits% \Bigl{(}\frac{\bar{X}}{2}\Bigr{)}=\frac{1}{4}\cdot\frac{2\theta^{2}}{n}=\frac{% \theta^{2}}{2n}.$

Since this converges to zero as $n\to\infty$ , we can conclude that the estimator is consistent by theorem 2.10.

∎

Solution to exercise 5.3.

1.

Since $\hat{\theta}=n/T$ and since $T$ has gamma density $\theta^{n}x^{n-1}e^{-\theta x}/(n-1)!$ on $(0,\infty)$ , we get the same integral as in exercise 2.3, but with one fewer power of $x$ , i.e. for $n\geq 3$ we have

$\displaystyle\mathbb{E}_{\theta}(\hat{\theta}^{2})$

$\displaystyle=n^{2}\int_{0}^{\infty}\frac{1}{x^{2}}\cdot\frac{\theta^{n}x^{n-1% }e^{-\theta x}}{(n-1)!}\,\mathrm{d}x$

$\displaystyle=\frac{n^{2}\theta^{n}}{(n-1)!}\cdot\frac{(n-3)!}{\theta^{n-2}}=% \frac{n^{2}}{(n-1)(n-2)}\,\theta^{2},$

as claimed.
2.

Expanding the square and using the linearity of the expectation, we find

$\mathop{\mathrm{MSE}}\nolimits(c\hat{\theta})=\mathbb{E}_{\theta}\bigl{(}(c% \hat{\theta}-\theta)^{2}\bigr{)}=c^{2}\,\mathbb{E}_{\theta}(\hat{\theta}^{2})-% 2c\theta\,\mathbb{E}_{\theta}(\hat{\theta})+\theta^{2}.$

In the text we have seen that $\mathbb{E}_{\theta}(\hat{\theta})=n\theta/(n-1)$ , and using this value and the second moment from the previous part of the question, we get

$\mathop{\mathrm{MSE}}\nolimits(c\hat{\theta})=\Bigl{(}\frac{n^{2}}{(n-1)(n-2)}% \,c^{2}-\frac{2n}{n-1}\,c+1\Bigr{)}\theta^{2}.$

For the MLE we set $c=1$ and thus we get

$\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\Bigl{(}\frac{n^{2}}{(n-1)(n-2)}-% \frac{2n}{n-1}+1\Bigr{)}\theta^{2}=\frac{n+2}{(n-1)(n-2)}\,\theta^{2}.$

For the unbiased estimator we set $c=(n-1)/n$ and thus we get the mean squared error as the variance

$\mathop{\mathrm{MSE}}\nolimits(\tilde{\theta})=\Bigl{(}\frac{n-1}{n-2}-2+1% \Bigr{)}\theta^{2}=\frac{\theta^{2}}{n-2}.$

Dividing the two expressions we get $\mathop{\mathrm{MSE}}\nolimits(\tilde{\theta})/\mathop{\mathrm{MSE}}\nolimits(% \hat{\theta})=(n-1)/(n+2)$ , which is always smaller than $1$ for $n\geq 3$ .
3.

The expression inside the brackets is a quadratic function of $c$ with positive leading coefficient, so the minimum is at the point where the derivative is zero. This point is

$c=\frac{n}{n-1}\cdot\frac{(n-1)(n-2)}{n^{2}}=\frac{n-2}{n}.$

Substituting this value back into the expression inside the brackets we find

$\mathop{\mathrm{MSE}}\nolimits\Bigl{(}\frac{n-2}{n}\,\hat{\theta}\Bigr{)}=% \Bigl{(}1-\frac{n-2}{n-1}\Bigr{)}\theta^{2}=\frac{\theta^{2}}{n-1}.$

Since $\theta^{2}/(n-1)<\theta^{2}/(n-2)$ , this value is better than the unbiased estimator ( $c=(n-1)/n$ ) and the MLE ( $c=1$ ), as shown in the previous part of the question. Since the winning estimator is smaller than the MLE, we know that the estimator is biased downwards. This is the same effect as we have seen for the normal variance in section 2.3, where the optimal divisor was $n+1$ instead of the unbiased value $n-1$ or the value found by maximum likelihood, i.e. $n$ : the mean squared error can be minimised by a deliberately biased estimator, and the amount of bias depends on the family of distributions under consideration. There is no one-size-fits-all solution for this kind of problem.

∎

Solution to exercise 5.4.

1.

Up to the factor $\nu+1$ , which we can cancel, the score is $a/(\nu+a^{2})+b/(\nu+b^{2})$ . Multiplying through by the positive quantity $(\nu+a^{2})(\nu+b^{2})$ , the likelihood equation becomes

$a(\nu+b^{2})+b(\nu+a^{2})=\nu(a+b)+ab(a+b)=(a+b)(\nu+ab)=0,$

as claimed.
2.

For $\theta=m+t$ we have $a=d-t$ and $b=-d-t$ , and up to an additive constant the log-likelihood is $-\frac{\nu+1}{2}\bigl{(}\log(\nu+a^{2})+\log(\nu+b^{2})\bigr{)}$ . The product inside the logarithms is

$\displaystyle(\nu+a^{2})(\nu+b^{2})$

$\displaystyle=\bigl{(}\nu+d^{2}+t^{2}-2dt\bigr{)}\bigl{(}\nu+d^{2}+t^{2}+2dt% \bigr{)}$

$\displaystyle=(\nu+d^{2}+t^{2})^{2}-4d^{2}t^{2}=(\nu+d^{2})^{2}+2(\nu-d^{2})\,% t^{2}+t^{4},$

which is the polynomial $P(t)$ and thus $\ell(m+t)=C-\frac{\nu+1}{2}\log P(t)$ .
3.

Since the logarithm is increasing, $\ell(m+t)$ takes its largest value when $P(t)$ is smallest. Let $u=t^{2}\geq 0$ . Then we have $P=(\nu+d^{2})^{2}+2(\nu-d^{2})u+u^{2}$ , i.e. we get a quadratic function of $u$ with minimum at $u=d^{2}-\nu$ . If $d^{2}<\nu$ , this minimum is at a negative value of $u$ and thus $P$ is increasing on $u\geq 0$ and takes its minimum at $t=0$ . The midpoint $m$ is then the only maximiser of $\ell$ and, since the factor $\nu+ab=\nu-d^{2}+t^{2}$ from the first part of the solution has no real zero, $m$ is the only root of the likelihood equation. If $d^{2}>\nu$ , then $P$ takes its minimum at $u=d^{2}-\nu$ , i.e. at $t=\pm\sqrt{d^{2}-\nu}$ which are the zeros of $\nu+ab$ . These two values are the maximum likelihood estimates, and since $P$ is even in $t$ , they have equal likelihood. Between these two values, at $t=0$ , the polynomial $P$ has a local maximum, because the coefficient $2(\nu-d^{2})$ of $t^{2}$ is negative. Thus, $\ell$ has a local minimum at the midpoint. This completes the proof.

∎

Solution to exercise 5.5.

Since $g$ is a bijection, there exists an inverse function $g^{-1}\colon\Psi\to\Theta$ and we can re-index the model, using $\psi$ instead of $\theta$ , i.e. if we let $\tilde{f}(x;\psi)=f\bigl{(}x;g^{-1}(\psi)\bigr{)}$ the family of distributions is unchanged, but the likelihood function is $\tilde{L}(\psi)=L\bigl{(}g^{-1}(\psi)\bigr{)}$ . For any $\psi\in\Psi$ , the value $g^{-1}(\psi)$ falls into the set $\Theta$ and by the definition of $\hat{\theta}$ we have

\tilde{L}(\psi)=L\bigl{(}g^{-1}(\psi)\bigr{)}\leq L(\hat{\theta})=\tilde{L}% \bigl{(}g(\hat{\theta})\bigr{)}.

Thus, $g(\hat{\theta})$ is a maximum of the function $\tilde{L}$ on the set $\Psi$ and consequently is a maximum likelihood estimator for $\psi$ . This completes the proof. ∎

Solution to exercise 5.6.

1.

We have $s+t\geq s$ and thus the event $\{X>s+t\}$ is contained in the event $\{X>s\}$ . We get

$\mathbb{P}_{\theta}(X>s+t\mskip 1.0mu|\mskip 1.0muX>s)=\frac{\mathbb{P}_{% \theta}(X>s+t)}{\mathbb{P}_{\theta}(X>s)}=\frac{e^{-\theta(s+t)}}{e^{-\theta s% }}=e^{-\theta t}=\mathbb{P}_{\theta}(X>t),$

i.e. the unconditional probability to survive beyond time $t$ . We can conclude from this that, if a lifetime has already passed time $s$ , the remaining lifetime has the same distribution as a fresh lifetime does. In other words, the time already elapsed does not change the distribution of the remaining lifetime.
2.

The conditional density of $X$ given the event $\{X>c\}$ is the density restricted to the set $(c,\infty)$ , normalised to be a density again. We get $f(x;\theta)/\mathbb{P}_{\theta}(X>c)$ for $x>c$ . Thus we have

$\mathbb{E}_{\theta}(X\mskip 1.0mu|\mskip 1.0muX>c)=\frac{1}{e^{-\theta c}}\int% _{c}^{\infty}x\,\theta e^{-\theta x}\,\mathrm{d}x.$

Using integration by parts, with $u=x$ and $v^{\prime}=\theta e^{-\theta x}$ , we find $v=-e^{-\theta x}$ and thus

$\int_{c}^{\infty}x\,\theta e^{-\theta x}\,\mathrm{d}x=\Bigl{[}-x\,e^{-\theta x% }\Bigr{]}_{c}^{\infty}+\int_{c}^{\infty}e^{-\theta x}\,\mathrm{d}x=c\,e^{-% \theta c}+\frac{e^{-\theta c}}{\theta}.$

Dividing by $e^{-\theta c}$ gives $\mathbb{E}_{\theta}(X\mskip 1.0mu|\mskip 1.0muX>c)=c+1/\theta$ as required. This result is consistent with the first part of the solution: given survival until time $c$ , the remaining lifetime $X-c$ is distributed like a fresh exponential lifetime. Thus, the conditional expectation is the unconditional mean $1/\theta$ , plus the time $c$ already elapsed, giving $c+1/\theta$ . This completes the solution.

∎

Solution to exercise 5.7.

1.

From the solution to exercise 4.1 we know that the score is given by $\ell^{\prime}(\theta)=(1/\theta)\sum_{i}x_{i}-n$ and this score equals zero at $\theta=\bar{x}$ . Let $S=\sum_{i}x_{i}\geq 1$ . Then the second derivative is $\ell^{\prime\prime}(\theta)=-S/\theta^{2}<0$ for all $\theta>0$ and thus $\ell$ is strictly concave. This shows that the root is the unique global maximum and thus the MLE is given by $\hat{\theta}=\bar{X}$ .
2.

The map $g(\theta)=e^{-\theta}$ is continuous and strictly decreasing and therefore maps $(0,\infty)$ onto $(0,1)$ bijectively. Thus, by theorem 5.6, the MLE for $p_{0}=g(\theta)$ is given by

$\hat{p}_{0}=e^{-\hat{\theta}}=e^{-\bar{X}}.$
3.

Using the independence of the observations and the given identity with $z=e^{-1/n}$ we get

$\mathbb{E}_{\theta}\bigl{(}e^{-\bar{X}}\bigr{)}=\mathbb{E}_{\theta}\Bigl{(}% \prod_{i=1}^{n}e^{-X_{i}/n}\Bigr{)}=\prod_{i=1}^{n}\mathbb{E}_{\theta}\Bigl{(}% \bigl{(}e^{-1/n}\bigr{)}^{X_{i}}\Bigr{)}=\exp\bigl{(}n\theta(e^{-1/n}-1)\bigr{% )}.$

Since $e^{u}>1+u$ for all $u\neq 0$ , we can choose $u=-1/n$ to get $e^{-1/n}-1>-1/n$ and thus $n(e^{-1/n}-1)>-1$ . This gives

$\mathbb{E}_{\theta}(\hat{p}_{0})=\exp\bigl{(}n\theta(e^{-1/n}-1)\bigr{)}>e^{-% \theta}=p_{0},$

and thus the estimator systematically overestimates $p_{0}$ on average and is biased for all $n$ . On the other hand, expanding the exponential we find $n(e^{-1/n}-1)=-1+1/(2n)-\dotsb\to-1$ as $n\to\infty$ and thus $\mathbb{E}_{\theta}(\hat{p}_{0})\to e^{-\theta}$ and the estimator is asymptotically unbiased. There is no contradiction to the unbiasedness of $\bar{X}$ : unbiasedness is not preserved under the nonlinear map $g$ , as discussed after theorem 5.6, whereas the maximum likelihood property is.

∎

Solution to exercise 5.8.

1.

Solving the equation $\mathbb{P}_{\theta}(X_{1}\leq m)=1-e^{-\theta m}=1/2$ for $m$ we find $e^{-\theta m}=1/2$ and taking logarithms we find $m=(\log 2)/\theta$ . The map $g(\theta)=(\log 2)/\theta$ is continuous and strictly decreasing on $(0,\infty)$ and since $g(\theta)\to\infty$ as $\theta\to 0$ and $g(\theta)\to 0$ as $\theta\to\infty$ , it maps $(0,\infty)$ onto itself in a bijective way.
2.

From example 5.2 we know that the MLE for the rate is given by $\hat{\theta}=1/\bar{X}$ . Since $g$ is a bijection, theorem 5.6 shows that the MLE for the median $m=g(\theta)$ is given by

$\hat{m}=g(\hat{\theta})=\frac{\log 2}{\hat{\theta}}=(\log 2)\,\bar{X}.$
3.

From lemma 2.2 we know $\mathbb{E}_{\theta}(\bar{X})=\mathbb{E}_{\theta}(X_{1})=1/\theta$ and thus $\mathbb{E}_{\theta}(\hat{m})=(\log 2)/\theta=m$ for all $\theta$ and thus $\hat{m}$ is unbiased. There is no contradiction: unbiasedness is a property of a parametrisation and is in general lost under nonlinear transformations of the parameter. Thus, the bias of the MLE $\hat{\theta}=1/\bar{X}$ for the rate is not relevant for the bias of the estimator $\hat{m}=(\log 2)\,\bar{X}$ for the median, which is a linear function of $\bar{X}$ .

∎

C.5 Exponential Families

Solution to exercise 7.1.

1.

For $x\in\{1,2,\dots\}$ we can write

$\displaystyle f(x;p)$

$\displaystyle=(1-p)^{x-1}p=\exp\bigl{(}(x-1)\log(1-p)+\log p\bigr{)}$

$\displaystyle=\exp\Bigl{(}x\log(1-p)-\log\frac{1-p}{p}\Bigr{)}.$

This is the form of definition 7.1 with $h(x)=1$ for $x\in\{1,2,\dots\}$ and $h(x)=0$ otherwise, natural statistic $T(x)=x$ , natural parameter $\eta(p)=\log(1-p)$ and $B(p)=\log\bigl{(}(1-p)/p\bigr{)}$ . By lemma 7.5 the natural statistic of a sample is $T_{n}=\sum_{i=1}^{n}X_{i}$ .
2.

For $\eta<0$ the geometric series gives

$\sum_{x=1}^{\infty}e^{\eta x}=\frac{e^{\eta}}{1-e^{\eta}},$

but for $\eta\geq 0$ the series diverges. Thus we have $\mathcal{N}=(-\infty,0)$ and

$K(\eta)=\eta-\log(1-e^{\eta}),\qquad\eta<0.$

Substituting $\eta(p)=\log(1-p)$ , so that $e^{\eta}=1-p$ , we get $K\bigl{(}\eta(p)\bigr{)}=\log(1-p)-\log p=B(p)$ as required. As $p$ ranges over $(0,1)$ , the natural parameter $\log(1-p)$ ranges over the entire open interval $(-\infty,0)$ , and thus the family has full rank.
3.

We have

$K^{\prime}(\eta)=1+\frac{e^{\eta}}{1-e^{\eta}}=\frac{1}{1-e^{\eta}}\qquad\text% {and}\qquad K^{\prime\prime}(\eta)=\frac{e^{\eta}}{(1-e^{\eta})^{2}}.$

With $e^{\eta}=1-p$ , by theorem 7.7, we find $\mathbb{E}_{p}(X)=1/p$ and $\mathop{\mathrm{Var}}\nolimits_{p}(X)=(1-p)/p^{2}$ , the values given in table A.1. The mean $1/p$ is the moment considered in exercise 4.2.

∎

Solution to exercise 7.2.

1.

For $x\in\{0,1,\dots,m\}$ we can write

$f(x;\theta)=\binom{m}{x}\theta^{x}(1-\theta)^{m-x}=\binom{m}{x}\exp\Bigl{(}x% \log\frac{\theta}{1-\theta}+m\log(1-\theta)\Bigr{)}.$

This is the form of definition 7.1 with $h(x)=\binom{m}{x}$ for $x\in\{0,1,\dots,m\}$ and $h(x)=0$ otherwise, natural statistic $T(x)=x$ , natural parameter $\eta(\theta)=\log\bigl{(}\theta/(1-\theta)\bigr{)}$ , the same log-odds as in example 7.2, and $B(\theta)=-m\log(1-\theta)$ .
2.

Using the binomial theorem, we find

$\sum_{x=0}^{m}\binom{m}{x}e^{\eta x}=(1+e^{\eta})^{m}$

for all $\eta\in\mathbb{R}$ . Thus, $\mathcal{N}=\mathbb{R}$ and $K(\eta)=m\log(1+e^{\eta})$ . This is $m$ times the cumulant function of the Bernoulli distribution.
3.

Taking derivatives, we get

$K^{\prime}(\eta)=\frac{me^{\eta}}{1+e^{\eta}}\qquad\text{and}\qquad K^{\prime% \prime}(\eta)=\frac{me^{\eta}}{(1+e^{\eta})^{2}}.$

Since $e^{\eta}/(1+e^{\eta})=\theta$ and $1/(1+e^{\eta})=1-\theta$ , theorem 7.7 gives $\mathbb{E}_{\theta}(X)=m\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(X)=m\theta(1-\theta)$ , in agreement with table A.1. For a sample of size $n$ the natural statistic is $T_{n}=\sum_{i}X_{i}$ and, by the remark after theorem 7.7, we have $\mathbb{E}_{\theta}(T_{n})=nm\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(T_{n})=nm\theta(1-\theta)$ .

∎

Solution to exercise 7.3.

1.

We can write $x^{\alpha-1}=e^{(\alpha-1)\log x}$ and, collecting all factors into one exponential, for $x>0$ we find

$\displaystyle f(x;\alpha,\beta)$

$\displaystyle=\frac{\beta^{\alpha}}{\Gamma(\alpha)}\,e^{(\alpha-1)\log x}\,e^{% -\beta x}$

$\displaystyle=\exp\Bigl{(}(\alpha-1)\log x+(-\beta)\,x-\bigl{(}\log\Gamma(% \alpha)-\alpha\log\beta\bigr{)}\Bigr{)}.$

This is the form of definition 7.8 with $k=2$ , with $h(x)=1$ for $x>0$ and $h(x)=0$ otherwise, natural statistic $\bigl{(}T_{1}(x),T_{2}(x)\bigr{)}=(\log x,\,x)$ , natural parameter $\bigl{(}\eta_{1}(\alpha,\beta),\eta_{2}(\alpha,\beta)\bigr{)}=(\alpha-1,\,-\beta)$ and $B(\alpha,\beta)=\log\Gamma(\alpha)-\alpha\log\beta$ .
2.

As $\alpha$ varies over $(0,\infty)$ , the first coordinate $\eta_{1}=\alpha-1$ varies over $(-1,\infty)$ and as $\beta$ varies over $(0,\infty)$ , the second coordinate $\eta_{2}=-\beta$ varies over $(-\infty,0)$ , independently of each other. Thus, the set of natural parameter values is the open quadrant $\{(\eta_{1},\eta_{2})\mid\eta_{1}>-1,\ \eta_{2}<0\}$ . This is a non-empty open subset of $\mathbb{R}^{2}$ and the family has full rank as in definition 7.10.
3.

Knowing $\alpha$ , we can include the factor $x^{\alpha-1}$ into the function $h$ : for $x>0$ we have

$f(x;\beta)=x^{\alpha-1}\exp\Bigl{(}(-\beta)\,x-\bigl{(}\log\Gamma(\alpha)-% \alpha\log\beta\bigr{)}\Bigr{)},$

This is a one-parameter exponential family in the sense of definition 7.1, with $h(x)=x^{\alpha-1}$ for $x>0$ , natural statistic $T(x)=x$ and natural parameter $-\beta$ .

∎

Solution to exercise 7.4.

1.

Using the formula $K(\eta)=-\log(-\eta)$ on $\mathcal{N}=(-\infty,0)$ we find $K^{\prime}(\eta)=-1/\eta$ and $K^{\prime\prime}(\eta)=1/\eta^{2}$ . Substituting the natural parameter $\eta=-\theta$ from example 7.4, we can use theorem 7.7 to find $\mathbb{E}_{\theta}(X)=1/\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(X)=1/\theta^{2}$ . This is the mean and variance of the exponential distribution with rate $\theta$ .
2.

For $h(x)=(2\pi\sigma^{2})^{-1/2}e^{-x^{2}/(2\sigma^{2})}$ and $T(x)=x$ we have to evaluate

$Z(\eta)=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\Bigl{(}-% \frac{x^{2}-2\sigma^{2}\eta x}{2\sigma^{2}}\Bigr{)}\,\mathrm{d}x.$

We can complete the square to find $x^{2}-2\sigma^{2}\eta x=(x-\sigma^{2}\eta)^{2}-\sigma^{4}\eta^{2}$ and thus

$Z(\eta)=\exp\Bigl{(}\frac{\sigma^{2}\eta^{2}}{2}\Bigr{)}\int_{-\infty}^{\infty% }\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\Bigl{(}-\frac{(x-\sigma^{2}\eta)^{2}}{2% \sigma^{2}}\Bigr{)}\,\mathrm{d}x=\exp\Bigl{(}\frac{\sigma^{2}\eta^{2}}{2}\Bigr% {)},$

since the remaining integrand is the density of $N(\sigma^{2}\eta,\sigma^{2})$ and thus integrates to one. The integral is finite for all $\eta$ and thus $\mathcal{N}=\mathbb{R}$ and $K(\eta)=\log Z(\eta)=\sigma^{2}\eta^{2}/2$ . Taking derivatives we find $K^{\prime}(\eta)=\sigma^{2}\eta$ and $K^{\prime\prime}(\eta)=\sigma^{2}$ and substituting $\eta=\mu/\sigma^{2}$ in theorem 7.7 we get $\mathbb{E}_{\mu}(X)=\mu$ and $\mathop{\mathrm{Var}}\nolimits_{\mu}(X)=\sigma^{2}$ , as expected. Finally, $K\bigl{(}\eta(\mu)\bigr{)}=\sigma^{2}(\mu/\sigma^{2})^{2}/2=\mu^{2}/(2\sigma^{% 2})$ , which is the function $B(\mu)$ from example 7.9.

∎

Solution to exercise 7.5.

1.

By lemma 7.5, applied in the natural parametrisation where $B=K$ , the log-likelihood is

$\ell(\eta)=\sum_{i=1}^{n}\log h(x_{i})+\eta\,T_{n}-nK(\eta),\qquad T_{n}=\sum_% {i=1}^{n}T(x_{i}),$

and thus the score is $\ell^{\prime}(\eta)=T_{n}-nK^{\prime}(\eta)$ . The likelihood equation $\ell^{\prime}(\eta)=0$ is therefore $K^{\prime}(\eta)=T_{n}/n$ , and by theorem 7.7 the left-hand side equals $\mathbb{E}_{\eta}\bigl{(}T(X_{1})\bigr{)}$ . This is the stated equation.
2.

Differentiating once more and using theorem 7.7 again, we find

$\ell^{\prime\prime}(\eta)=-nK^{\prime\prime}(\eta)=-n\mathop{\mathrm{Var}}% \nolimits_{\eta}\bigl{(}T(X_{1})\bigr{)}<0$

for all $\eta\in\mathcal{N}$ . Thus $\ell$ is strictly concave on the interval $\mathcal{N}$ , and a strictly concave function has at most one critical point, which is then its global maximum. Any root of the likelihood equation is therefore the unique maximiser of $\ell$ , i.e. the unique MLE for $\eta$ .
3.

For the Poisson distribution we have $T(x)=x$ and $K^{\prime}(\eta)=e^{\eta}$ , and thus the likelihood equation reads $e^{\eta}=\bar{x}$ , with root $\hat{\eta}=\log\bar{x}$ whenever $\bar{x}>0$ . Since $\theta=e^{\eta}$ is a bijection between $\mathcal{N}=\mathbb{R}$ and $\Theta=(0,\infty)$ , theorem 5.6 gives $\hat{\theta}=e^{\hat{\eta}}=\bar{X}$ , the MLE of exercise 5.7. For the Bernoulli distribution we have $T(x)=x$ and $K^{\prime}(\eta)=e^{\eta}/(1+e^{\eta})$ , and thus the likelihood equation reads $e^{\eta}/(1+e^{\eta})=\bar{x}$ , with root $\hat{\eta}=\log\bigl{(}\bar{x}/(1-\bar{x})\bigr{)}$ whenever $0<\bar{x}<1$ . Since $\theta=e^{\eta}/(1+e^{\eta})$ is a bijection between $\mathbb{R}$ and $(0,1)$ , theorem 5.6 gives $\hat{\theta}=\bar{X}$ , the MLE of example 5.3. The restriction $0<\bar{x}<1$ is the condition $0<s<n$ discussed after that example: when it fails the likelihood equation has no root and the MLE does not exist in the open parameter space.
4.

The equation of the first part says that the MLE is the parameter value under which the population mean of the natural statistic $T$ agrees with its sample mean, i.e. the MLE is the method of moments estimator based on the moment $\mathbb{E}_{\theta}\bigl{(}T(X_{1})\bigr{)}$ rather than on $\mathbb{E}_{\theta}(X_{1})$ . Whenever $T(x)=x$ , as for the Bernoulli, Poisson and exponential distributions and for the normal mean with known variance, the two moments are the same, and the MLE coincides with the method of moments estimator from definition 4.6; this is the coincidence observed in the remark at the end of the method of moments section of lecture 4. For the gamma distribution of exercise 7.3 the natural statistic is $(\log x,\,x)$ , and thus maximum likelihood matches the means of $\log X_{1}$ and $X_{1}$ , whereas the method of moments in example 4.9 matched the means of $X_{1}$ and $X_{1}^{2}$ . Different moments are matched, and thus the two estimators differ.

∎

Solution to exercise 7.6.

1.

Writing the expectation as an integral (or a sum, for a discrete model), we find

$\displaystyle\mathbb{E}_{\eta}\bigl{(}e^{sT(X)}\bigr{)}$

$\displaystyle=\int e^{sT(x)}\,h(x)\,e^{\eta T(x)-K(\eta)}\,\mathrm{d}x$

$\displaystyle=e^{-K(\eta)}\int h(x)\,e^{(\eta+s)T(x)}\,\mathrm{d}x=e^{K(\eta+s% )-K(\eta)},$

where the last step uses the definition of $K$ at the point $\eta+s\in\mathcal{N}$ . Taking logarithms gives $\log\mathbb{E}_{\eta}(e^{sT(X)})=K(\eta+s)-K(\eta)=M(s)$ , as claimed.
2.

By the first part we have $e^{M(s)}=\mathbb{E}_{\eta}(e^{sT(X)})$ . Differentiating both sides with respect to $s$ , and taking the derivative inside the expectation on the right, we get

$\displaystyle M^{\prime}(s)\,e^{M(s)}$

$\displaystyle=\mathbb{E}_{\eta}\bigl{(}T(X)\,e^{sT(X)}\bigr{)}\qquad\text{and}$

$\displaystyle\bigl{(}M^{\prime\prime}(s)+M^{\prime}(s)^{2}\bigr{)}e^{M(s)}$

$\displaystyle=\mathbb{E}_{\eta}\bigl{(}T(X)^{2}\,e^{sT(X)}\bigr{)}.$

At $s=0$ we have $M(0)=0$ , and thus $M^{\prime}(0)=\mathbb{E}_{\eta}(T(X))$ and $M^{\prime\prime}(0)=\mathbb{E}_{\eta}(T(X)^{2})-\mathbb{E}_{\eta}(T(X))^{2}=% \mathop{\mathrm{Var}}\nolimits_{\eta}(T(X))$ . On the other hand, from $M(s)=K(\eta+s)-K(\eta)$ we get directly $M^{\prime}(0)=K^{\prime}(\eta)$ and $M^{\prime\prime}(0)=K^{\prime\prime}(\eta)$ . Comparing the two computations gives $\mathbb{E}_{\eta}(T(X))=K^{\prime}(\eta)$ and $\mathop{\mathrm{Var}}\nolimits_{\eta}(T(X))=K^{\prime\prime}(\eta)$ , which is theorem 7.7.
3.

For the Poisson distribution we have $T(x)=x$ , $\eta=\log\theta$ and $K(\eta)=e^{\eta}$ by example 7.3 and the computation following definition 7.6, and $\mathcal{N}=\mathbb{R}$ , so that $\eta+s\in\mathcal{N}$ for every $s$ . Writing $z=e^{s}$ , the first part gives

$\mathbb{E}_{\theta}\bigl{(}z^{X}\bigr{)}=\mathbb{E}_{\eta}\bigl{(}e^{sX}\bigr{% )}=\exp\bigl{(}e^{\eta+s}-e^{\eta}\bigr{)}=\exp\bigl{(}\theta z-\theta\bigr{)}% =e^{\theta(z-1)},$

which is the identity used in exercise 5.7.

∎

Solution to exercise 7.7.

1.

Using the notation $\mu=\theta$ and $\sigma^{2}=\theta^{2}$ from example 7.9, we see that the density of $N(\theta,\theta^{2})$ is of the form given in definition 7.8 with $k=2$ , where $h(x)=1$ , the natural statistic is $(x,x^{2})$ and the natural parameter is

$\Bigl{(}\frac{\mu}{\sigma^{2}},\,-\frac{1}{2\sigma^{2}}\Bigr{)}=\Bigl{(}\frac{% 1}{\theta},\,-\frac{1}{2\theta^{2}}\Bigr{)}.$
2.

Let $\eta_{1}=1/\theta$ and $\eta_{2}=-1/(2\theta^{2})$ . As $\theta$ varies over the interval $(0,\infty)$ , the first coordinate $\eta_{1}$ also varies over the interval $(0,\infty)$ . Substituting $\theta=1/\eta_{1}$ for the second coordinate, we find

$\eta_{2}=-\frac{1}{2\theta^{2}}=-\frac{\eta_{1}^{2}}{2}.$

Thus, the set of natural parameter values is the curve $\bigl{\{}(\eta_{1},-\eta_{1}^{2}/2)\mathrel{\big{|}}\eta_{1}>0\bigr{\}}$ . Any open subset of $\mathbb{R}^{2}$ which contains a point on the curve also contains a small disc around this point, and thus points off the curve, obtained by slightly changing $\eta_{2}$ while keeping $\eta_{1}$ fixed. Thus the curve does not contain any non-empty open subsets of $\mathbb{R}^{2}$ . By definition 7.10, the family is not of full rank, as claimed.

∎

C.6 Sufficiency

Here we solve the exercises at the end of lecture 8.

Solution to exercise 8.1.

1.

If $\sum_{i}x_{i}=t$ , the event $\{X=x\}$ is contained in the event $\{T=t\}$ and we have

$\mathbb{P}_{\theta}(X=x\mskip 1.0mu|\mskip 1.0muT=t)=\frac{\mathbb{P}_{\theta}% (X=x)}{\mathbb{P}_{\theta}(T=t)}=\frac{\theta^{t}(1-\theta)^{n-t}}{\binom{n}{t% }\theta^{t}(1-\theta)^{n-t}}=\frac{1}{\binom{n}{t}}.$
2.

If $\sum_{i}x_{i}\neq t$ , the events $\{X=x\}$ and $\{T=t\}$ are disjoint and the conditional probability is zero. Thus the result does not depend on $\theta$ and $T$ is sufficient for $\theta$ by definition 8.1.
3.

The joint probability weights are given by

$f(x;\theta)=\prod_{i=1}^{n}\theta^{x_{i}}(1-\theta)^{1-x_{i}}=\theta^{\sum_{i}% x_{i}}(1-\theta)^{n-\sum_{i}x_{i}}$

for $x\in\{0,1\}^{n}$ . This function is of the form (8.1) with $g(t;\theta)=\theta^{t}(1-\theta)^{n-t}$ and $h(x)=1$ . By theorem 8.2 the statistic $T$ is sufficient, which confirms our direct computation.

∎

Solution to exercise 8.2.

1.

The density of a single observation is $f(x_{i};\theta)=\theta^{-1}\mathbf{1}_{\{0\leq x_{i}\leq\theta\}}$ . Since the joint density is the product of these densities, and since a product of indicators can be written as the indicator of the intersection of the corresponding events, we find that all $n$ conditions $0\leq x_{i}\leq\theta$ are satisfied if and only if the smallest observation is greater than or equal to $0$ and the largest observation is less than or equal to $\theta$ . Thus we have

$f(x;\theta)=\theta^{-n}\prod_{i=1}^{n}\mathbf{1}_{\{0\leq x_{i}\leq\theta\}}=% \theta^{-n}\,\mathbf{1}_{\{\max_{i}x_{i}\leq\theta\}}\,\mathbf{1}_{\{\min_{i}x% _{i}\geq 0\}}.$
2.

Let $g(t;\theta)=\theta^{-n}\mathbf{1}_{\{t\leq\theta\}}$ and $h(x)=\mathbf{1}_{\{\min_{i}x_{i}\geq 0\}}$ . Then $g$ depends on the data only through $t=\max_{i}x_{i}$ , the function $h$ does not depend on $\theta$ and from the first part of the solution we know that $f(x;\theta)=g(\max_{i}x_{i};\theta)\,h(x)$ for all $x$ and $\theta$ . By theorem 8.2 the statistic $X_{(n)}=\max_{i}X_{i}$ is sufficient for $\theta$ .
3.

The factor $\mathbf{1}_{\{\min_{i}x_{i}\geq 0\}}$ depends on the data but does not depend on $\theta$ and thus can be included in $h$ . The factor $\mathbf{1}_{\{\max_{i}x_{i}\leq\theta\}}$ depends on $\theta$ and thus a function involving this factor cannot be in $h$ . This factor must be included in $g$ and, since it only depends on the data through $\max_{i}x_{i}$ , the maximum is the sufficient statistic resulting from this factorisation. It is a common mistake to write the joint density as $\theta^{-n}$ and to conclude that no additional statistic is needed, since the indicator is where the data and the parameter interact.

∎

Solution to exercise 8.3.

1.

For $x_{1},\dots,x_{n}\geq 0$ the joint density is

$f(x;\theta)=\prod_{i=1}^{n}\theta e^{-\theta x_{i}}=\theta^{n}\exp\Bigl{(}-% \theta\sum_{i=1}^{n}x_{i}\Bigr{)},$

and it is zero if any $x_{i}$ is negative. Thus the factorisation (8.1) holds with $g(t;\theta)=\theta^{n}e^{-\theta t}$ and $h(x)=\mathbf{1}_{\{\min_{i}x_{i}\geq 0\}}$ , and $T=\sum_{i}X_{i}$ is sufficient by theorem 8.2.
2.

For two data vectors $x$ and $y$ with non-negative entries, the first part gives

$\frac{f(x;\theta)}{f(y;\theta)}=\exp\Bigl{(}-\theta\Bigl{(}\sum_{i}x_{i}-\sum_% {i}y_{i}\Bigr{)}\Bigr{)}.$

If $\sum_{i}x_{i}=\sum_{i}y_{i}$ , the ratio equals $1$ for every $\theta$ . If the sums differ by $d\neq 0$ , the ratio is $e^{-\theta d}$ , which is a strictly monotone function of $\theta$ and thus not constant on $(0,\infty)$ . The ratio is therefore free of $\theta$ if and only if $T(x)=T(y)$ and by theorem 8.6 we find that $T$ is minimal sufficient.
3.

Since $\bar{X}=T/n$ , the MLE is given by $\hat{\theta}=1/\bar{X}=n/T$ , a function of $T$ . For fixed data, the likelihood function is $L(\theta)=g(T;\theta)\,h(x)=\theta^{n}e^{-\theta T}\,h(x)$ , where $h(x)$ is a positive constant not depending on $\theta$ . Thus, the maximiser of $L(\theta)$ over $\theta$ is the same as the maximiser of $\theta^{n}e^{-\theta T}$ , which depends on the data only through $T$ . Whatever the MLE is, it can be written as a function of the sufficient statistic by the factorisation in theorem 8.2.

∎

Solution to exercise 8.4.

1.

With $\mu$ known, the joint density

$f(x;\sigma^{2})=(2\pi\sigma^{2})^{-n/2}\exp\Bigl{(}-\frac{1}{2\sigma^{2}}\sum_% {i=1}^{n}(x_{i}-\mu)^{2}\Bigr{)}$

depends on the data only through $t=\sum_{i}(x_{i}-\mu)^{2}$ , so that (8.1) holds with $g(t;\sigma^{2})=(2\pi\sigma^{2})^{-n/2}\exp\bigl{(}-t/(2\sigma^{2})\bigr{)}$ and $h(x)=1$ . By theorem 8.2 the statistic $\sum_{i}(X_{i}-\mu)^{2}$ is sufficient for $\sigma^{2}$ .
2.

Expanding the squares in the exponent, the joint density is

$\displaystyle f(x;\mu,\sigma^{2})$

$\displaystyle=(2\pi\sigma^{2})^{-n/2}\exp\Bigl{(}-\frac{1}{2\sigma^{2}}\sum_{i% =1}^{n}(x_{i}-\mu)^{2}\Bigr{)}$

$\displaystyle=(2\pi\sigma^{2})^{-n/2}\exp\Bigl{(}\frac{\mu}{\sigma^{2}}\sum_{i% }x_{i}-\frac{1}{2\sigma^{2}}\sum_{i}x_{i}^{2}-\frac{n\mu^{2}}{2\sigma^{2}}% \Bigr{)},$

which depends on the data only through the pair $T=\bigl{(}\sum_{i}x_{i},\sum_{i}x_{i}^{2}\bigr{)}$ , and thus the factorisation (8.1) holds with $h(x)=1$ and with $g$ equal to the whole expression. By theorem 8.2 the pair $T$ is sufficient for $(\mu,\sigma^{2})$ .
3.

Using the expanded form of the joint density from the previous part, and writing $a=\sum_{i}x_{i}-\sum_{i}y_{i}$ and $b=\sum_{i}x_{i}^{2}-\sum_{i}y_{i}^{2}$ , the ratio of the densities of two data vectors is

$\frac{f(x;\mu,\sigma^{2})}{f(y;\mu,\sigma^{2})}=\exp\Bigl{(}\frac{a\mu-b/2}{% \sigma^{2}}\Bigr{)},$

since the factors $(2\pi\sigma^{2})^{-n/2}$ and $\exp\bigl{(}-n\mu^{2}/(2\sigma^{2})\bigr{)}$ cancel. If $a=b=0$ , which is the statement $T(x)=T(y)$ , the ratio equals $1$ for all parameter values. Conversely, suppose the ratio does not depend on $(\mu,\sigma^{2})$ , so that the exponent $(a\mu-b/2)/\sigma^{2}$ equals a constant $c$ for all $\mu\in\mathbb{R}$ and all $\sigma^{2}>0$ . Taking $\sigma^{2}=1$ , the expression $a\mu-b/2$ is constant in $\mu$ , which forces $a=0$ and then $b=-2c$ . With $a=0$ the exponent is $-b/(2\sigma^{2})$ , and this is constant in $\sigma^{2}$ only if $b=0$ . Thus the ratio is free of the parameters if and only if $T(x)=T(y)$ , and $T$ is minimal sufficient by theorem 8.6.
4.

No. By the previous part $T$ is minimal sufficient, and thus, by definition 8.5, $T$ would have to be a function of $\bar{X}$ if $\bar{X}$ were sufficient. It is not: for $n=2$ the data sets $x=(1,-1)$ and $y=(2,-2)$ have the same sample mean $0$ , but $\sum_{i}x_{i}^{2}=2$ and $\sum_{i}y_{i}^{2}=8$ , so that $T(x)\neq T(y)$ . Two data sets with the same value of $\bar{X}$ can therefore have non-proportional likelihood functions, and $\bar{X}$ alone loses information about $\sigma^{2}$ .

∎

Solution to exercise 8.5.

1.

The exponential distribution with rate $\theta$ has distribution function $F(y)=1-e^{-\theta y}$ for $y\geq 0$ , and thus $1-F(y)=e^{-\theta y}$ . Proposition 8.9 gives

$f_{(1)}(y)=n\,\bigl{(}e^{-\theta y}\bigr{)}^{n-1}\,\theta e^{-\theta y}=n% \theta\,e^{-n\theta y},\qquad y\geq 0,$

which is the exponential density with rate $n\theta$ . Directly, the minimum exceeds $y$ if and only if every observation does, and by independence

$\mathbb{P}_{\theta}(X_{(1)}>y)=\prod_{i=1}^{n}\mathbb{P}_{\theta}(X_{i}>y)=% \bigl{(}e^{-\theta y}\bigr{)}^{n}=e^{-n\theta y},$

which is the probability that an exponential random variable with rate $n\theta$ exceeds $y$ . The two computations agree.
2.

From table A.2, an exponential random variable with rate $n\theta$ has mean $1/(n\theta)$ and variance $1/(n\theta)^{2}$ . Thus $\mathbb{E}_{\theta}(nX_{(1)})=n/(n\theta)=1/\theta$ , so $nX_{(1)}$ is unbiased for $1/\theta$ , and $\mathop{\mathrm{Var}}\nolimits_{\theta}(nX_{(1)})=n^{2}/(n\theta)^{2}=1/\theta% ^{2}$ .
3.

By lemma 2.2 the sample mean has $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})=1/(n\theta^{2})$ , and thus we find that $\mathop{\mathrm{Var}}\nolimits_{\theta}(nX_{(1)})$ equals $n\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})$ , so that the estimator based on the minimum has $n$ times the variance of the sample mean. Worse, its variance does not decrease at all as $n$ grows. Indeed, by the first part $nX_{(1)}$ has the exponential distribution with rate $\theta$ for every $n$ , and thus its sampling distribution does not change with the sample size, and $nX_{(1)}$ is not a consistent estimator for $1/\theta$ . It is not a function of $\sum_{i}X_{i}$ either: for $n=2$ the data sets $(1,3)$ and $(2,2)$ have the same sum but minima $1$ and $2$ . The estimator discards the information which the sufficient statistic retains, and lecture 10 shows how conditioning on a sufficient statistic repairs such an estimator.

∎

Solution to exercise 8.6.

1.

Since $X_{1}$ and $X_{2}$ are both Bernoulli with success probability $\theta$ , we have $\mathbb{E}_{\theta}(X_{1}-X_{2})=\theta-\theta=0$ for all $\theta\in(0,1)$ and thus $g(T)=X_{1}-X_{2}$ is an unbiased estimator for zero. The variable $g(T)$ equals $1$ on the event $\{X_{1}=1,X_{2}=0\}$ and, since the two events are independent, this happens with probability $\theta(1-\theta)>0$ . Thus we have $\mathbb{P}_{\theta}\bigl{(}g(T)=0\bigr{)}\leq 1-\theta(1-\theta)<1$ .
2.

The function $g$ satisfies $\mathbb{E}_{\theta}\bigl{(}g(T)\bigr{)}=0$ for all $\theta$ , but $g(T)$ does not equal zero with probability one, and thus the condition from definition 8.8 is violated: the whole sample is not complete. Nevertheless, the statistic $T$ is sufficient, because once the whole sample is given, there is no variability left. This completes the proof.

∎

C.7 Rao–Blackwell and Lehmann–Scheffe

Solution to exercise 10.1.

1.

Since $U$ takes only the values $0$ and $1$ , we have $\mathbb{E}_{\theta}(U)=\mathbb{P}_{\theta}(X_{1}=0)=e^{-\theta}$ for all $\theta$ and thus $U$ is an unbiased estimator for $e^{-\theta}$ . The indicator function is a Bernoulli random variable with success probability $e^{-\theta}$ , and thus we have $\mathop{\mathrm{Var}}\nolimits_{\theta}(U)=e^{-\theta}(1-e^{-\theta})$ .
2.

Let $T=X_{1}+R$ where $R=\sum_{i=2}^{n}X_{i}$ . By the given fact, $R$ is Poisson with parameter $(n-1)\theta$ and $T$ is Poisson with parameter $n\theta$ , and $R$ is independent of $X_{1}$ . For $0\leq x\leq t$ , the event $\{X_{1}=x,\ T=t\}$ is the event $\{X_{1}=x,\ R=t-x\}$ and thus we get

$\displaystyle\mathbb{P}_{\theta}(X_{1}=x\mskip 1.0mu|\mskip 1.0muT=t)$

$\displaystyle=\frac{\mathbb{P}_{\theta}(X_{1}=x)\,\mathbb{P}_{\theta}(R=t-x)}{% \mathbb{P}_{\theta}(T=t)}$

$\displaystyle=\frac{e^{-\theta}\theta^{x}}{x!}\cdot\frac{e^{-(n-1)\theta}\bigl% {(}(n-1)\theta\bigr{)}^{t-x}}{(t-x)!}\cdot\frac{t!}{e^{-n\theta}(n\theta)^{t}}$

$\displaystyle=\frac{t!}{x!\,(t-x)!}\cdot\frac{(n-1)^{t-x}}{n^{t}}=\binom{t}{x}% \Bigl{(}\frac{1}{n}\Bigr{)}^{x}\Bigl{(}1-\frac{1}{n}\Bigr{)}^{t-x}.$

As expected, the parameter $\theta$ cancels since $T$ is sufficient. Given $T=t$ , the first observation can be seen as the number of successes in $t$ independent trials with success probability $1/n$ , and in the model each of the $t$ counted events is equally likely to have been any of the $n$ observations.
3.

The conditional expectation of an indicator function is a conditional probability. Thus, using the second part of the question, we find

$V=\mathbb{E}_{\theta}(U\mskip 1.0mu|\mskip 1.0muT)=\mathbb{P}_{\theta}(X_{1}=0% \mskip 1.0mu|\mskip 1.0muT)=\binom{T}{0}\Bigl{(}1-\frac{1}{n}\Bigr{)}^{T}=% \Bigl{(}1-\frac{1}{n}\Bigr{)}^{T}.$
4.

Using the probability generating function with $s=1-1/n$ , we find

$\mathbb{E}_{\theta}(V)=\mathbb{E}_{\theta}\Bigl{(}\Bigl{(}1-\frac{1}{n}\Bigr{)% }^{T}\Bigr{)}=\exp\Bigl{(}n\theta\Bigl{(}1-\frac{1}{n}-1\Bigr{)}\Bigr{)}=e^{-% \theta}.$

This agrees with the second statement of the Rao–Blackwell theorem, theorem 10.1.
5.

From example 10.6 we know that the sum $T$ is a complete and sufficient statistic, and $V$ is a function of $T$ which is unbiased for $e^{-\theta}$ . Thus, by the Lehmann–Scheffe theorem, theorem 10.4, $V$ is the UMVUE for $e^{-\theta}$ . The MLE $e^{-\bar{X}}=e^{-T/n}$ is also a function of $T$ , but in exercise 5.7 we have seen that $\mathbb{E}_{\theta}(e^{-\bar{X}})=\exp\bigl{(}n\theta(e^{-1/n}-1)\bigr{)}>e^{-\theta}$ , and thus it is biased and cannot be the UMVUE. Nevertheless, for large $n$ , the two estimators are close since $(1-1/n)^{T}=\exp\bigl{(}T\log(1-1/n)\bigr{)}$ and $\log(1-1/n)=-1/n-1/(2n^{2})-\dotsb$ and thus $V\approx e^{-\bar{X}}e^{-\bar{X}/(2n)}$ : the UMVUE is obtained by applying a small downward correction to the MLE in order to compensate for the upward bias.

∎

Solution to exercise 10.2.

1.

The density $\theta e^{-\theta x}$ is that of the exponential distribution with rate $\theta$ , whose mean is $1/\theta$ by table A.2, and thus $X_{1}$ is unbiased for $1/\theta$ . The joint density of the sample is

$f(x_{1},\dots,x_{n};\theta)=\prod_{i=1}^{n}\theta e^{-\theta x_{i}}=\theta^{n}% \exp\Bigl{(}-\theta\sum_{i=1}^{n}x_{i}\Bigr{)}$

for all $x_{i}\geq 0$ , which depends on the data only through $\sum_{i}x_{i}$ , and thus $T$ is sufficient by the factorisation theorem, theorem 8.2, with $h\equiv 1$ .
2.

The random variables $X_{1}$ and $R$ are independent, $X_{1}$ has density $f(x;\theta)=\theta e^{-\theta x}$ and $R$ has the quoted gamma density with $k=n-1$ , and thus the pair $(X_{1},R)$ has joint density $f(x;\theta)f_{n-1}(r)$ for $x,r\geq 0$ . The map $(x,r)\mapsto(x,s)=(x,x+r)$ is linear with determinant one, and thus by the change of variable formula, proposition A.19, the pair $(X_{1},T)$ has joint density

$\displaystyle f_{X_{1},T}(x,s)$

$\displaystyle=f(x;\theta)\,f_{n-1}(s-x)=\theta e^{-\theta x}\cdot\frac{\theta^% {n-1}(s-x)^{n-2}e^{-\theta(s-x)}}{(n-2)!}$

$\displaystyle=\frac{\theta^{n}(s-x)^{n-2}e^{-\theta s}}{(n-2)!}$

for $0<x<s$ , and zero otherwise. Dividing by the density $f_{n}(s)=\theta^{n}s^{n-1}e^{-\theta s}/(n-1)!$ of $T$ , we obtain the conditional density

$f(x\mskip 1.0mu|\mskip 1.0mus)=\frac{f_{X_{1},T}(x,s)}{f_{n}(s)}=\frac{(n-1)!}% {(n-2)!}\cdot\frac{(s-x)^{n-2}}{s^{n-1}}=\frac{(n-1)(s-x)^{n-2}}{s^{n-1}}$

for $0<x<s$ . Both $\theta^{n}$ and $e^{-\theta s}$ have cancelled, and thus the conditional density does not depend on $\theta$ , which is the sufficiency of $T$ seen from definition 8.1. For $n=2$ the density is $1/s$ on $(0,s)$ : given the total of two observations, the first is uniformly distributed over the possible values.
3.

Substituting $x=su$ , we find

$\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT=s)=\int_{0}^{s}x\,\frac{(n% -1)(s-x)^{n-2}}{s^{n-1}}\,\mathrm{d}x=(n-1)\,s\int_{0}^{1}u(1-u)^{n-2}\,% \mathrm{d}u.$

Integrating by parts, with $u$ differentiated and $(1-u)^{n-2}$ integrated, gives

$\int_{0}^{1}u(1-u)^{n-2}\,\mathrm{d}u=\Bigl{[}-\frac{u(1-u)^{n-1}}{n-1}\Bigr{]% }_{0}^{1}+\frac{1}{n-1}\int_{0}^{1}(1-u)^{n-1}\,\mathrm{d}u=\frac{1}{(n-1)n},$

and thus $\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT=s)=s/n$ . Consequently $\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT)=T/n=\bar{X}$ : Rao–Blackwellising the single observation $X_{1}$ , which has variance $1/\theta^{2}$ , gives the sample mean, which has variance $1/(n\theta^{2})$ , and the reduction promised by theorem 10.1 is by a factor of $n$ .
4.

Since the $X_{i}$ are i.i.d. and $T$ is a symmetric function of them, the pair $(X_{i},T)$ has the same joint distribution for every $i$ , and thus the conditional expectations $\mathbb{E}_{\theta}(X_{i}\mskip 1.0mu|\mskip 1.0muT)$ are the same random variable for all $i$ . Summing over $i$ and using the linearity of conditional expectation, we find

$n\,\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT)=\sum_{i=1}^{n}\mathbb{% E}_{\theta}(X_{i}\mskip 1.0mu|\mskip 1.0muT)=\mathbb{E}_{\theta}(T\mskip 1.0mu% |\mskip 1.0muT)=T,$

and thus $\mathbb{E}_{\theta}(X_{1}\mskip 1.0mu|\mskip 1.0muT)=T/n$ . This argument uses only the symmetry of an i.i.d. sample and none of the exponential form, and so it applies to every model in which $\sum_{i}X_{i}$ is sufficient.

∎

Solution to exercise 10.3.

1.

In exercise 2.2 we have found $\mathbb{E}_{\theta}(M)=n\theta/(n+1)$ and thus

$\mathbb{E}_{\theta}(\hat{\theta})=\frac{n+1}{n}\cdot\frac{n\theta}{n+1}=\theta$

for all $\theta$ . Thus, $\hat{\theta}$ is unbiased.
2.

From exercise 8.2 we know that the maximum $M$ is sufficient for $\theta$ and by the second statement of proposition 10.5 it is complete. Since $\hat{\theta}$ is a function of $M$ and is unbiased for $\theta$ , we can use the Lehmann–Scheffe theorem, theorem 10.4, to conclude that this is the UMVUE for $\theta$ .
3.

From the solution to exercise 2.2 we know $\mathop{\mathrm{Var}}\nolimits_{\theta}(M)=n\theta^{2}/\bigl{(}(n+2)(n+1)^{2}% \bigr{)}$ and thus

$\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta})=\Bigl{(}\frac{n+1}{n}% \Bigr{)}^{2}\cdot\frac{n\theta^{2}}{(n+2)(n+1)^{2}}=\frac{\theta^{2}}{n(n+2)}.$

The variance of a single observation is $\theta^{2}/12$ , as shown in table A.2, and thus, using lemma 2.2, we find

$\mathop{\mathrm{Var}}\nolimits_{\theta}(\tilde{\theta})=4\,\mathop{\mathrm{Var% }}\nolimits_{\theta}(\bar{X})=\frac{4\theta^{2}}{12n}=\frac{\theta^{2}}{3n}.$

The ratio $\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta})/\mathop{\mathrm{Var}}% \nolimits_{\theta}(\tilde{\theta})=3/(n+2)$ , which is always less than or equal to one for all $n\geq 1$ , equals one only for $n=1$ , where both estimators equal $2X_{1}$ . For large $n$ , the variance of the UMVUE is of order $1/n^{2}$ , compared to the order $1/n$ for $2\bar{X}$ , similar to the result in exercise 2.2.
4.

From theorem 10.1, the Rao–Blackwell theorem, we know that the conditional expectation $\mathbb{E}_{\theta}(\tilde{\theta}\mskip 1.0mu|\mskip 1.0muM)$ is a random variable, which is a function of $M$ and is unbiased for $\theta$ . Since $M$ is complete, the first step of the proof of theorem 10.4 shows that there can only be one such function and thus we can conclude $\mathbb{E}_{\theta}(\tilde{\theta}\mskip 1.0mu|\mskip 1.0muM)=(n+1)M/n$ with probability one. For $n=1$ we have $M=X_{1}$ and $\tilde{\theta}=2X_{1}=2M$ , which coincides with the formula $(n+1)M/n=2M$ .

∎

Solution to exercise 10.4.

1.

Using the definition of the exponential family, we can write the density as

$f(x;\mu,\sigma^{2})=\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\Bigl{(}-\frac{(x-\mu)^% {2}}{2\sigma^{2}}\Bigr{)}=\frac{e^{-\mu^{2}/(2\sigma^{2})}}{\sqrt{2\pi\sigma^{% 2}}}\exp\Bigl{(}\frac{\mu}{\sigma^{2}}\,x-\frac{1}{2\sigma^{2}}\,x^{2}\Bigr{)},$

which is of the form given in definition 7.8 with $h(x)=1$ , natural statistic $T(x)=(x,x^{2})$ and natural parameter

$\eta(\mu,\sigma^{2})=\Bigl{(}\frac{\mu}{\sigma^{2}},\ -\frac{1}{2\sigma^{2}}% \Bigr{)}.$
2.

As $(\mu,\sigma^{2})$ varies over $\mathbb{R}\times(0,\infty)$ , the second component $\eta_{2}=-1/(2\sigma^{2})$ can take any value in $(-\infty,0)$ . For fixed $\sigma^{2}$ , the first component $\eta_{1}=\mu/\sigma^{2}$ can take any value in $\mathbb{R}$ as $\mu$ varies. Thus, the range of $\eta$ is the open half-plane $\mathbb{R}\times(-\infty,0)$ , which is an open, non-empty subset of $\mathbb{R}^{2}$ and thus the family has full rank in the sense of definition 7.10.
3.

From corollary 8.3 and the first statement of proposition 10.5 we know that the natural statistic $\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$ of the sample is complete and sufficient. From the pair $\bigl{(}\sum_{i}X_{i},\sum_{i}X_{i}^{2}\bigr{)}$ we can compute $\bar{X}=\sum_{i}X_{i}/n$ and $S^{2}=\bigl{(}\sum_{i}X_{i}^{2}-n\bar{X}^{2}\bigr{)}/(n-1)$ , and conversely $\sum_{i}X_{i}=n\bar{X}$ and $\sum_{i}X_{i}^{2}=(n-1)S^{2}+n\bar{X}^{2}$ , and thus the map between the two pairs is one-to-one. Since a one-to-one function of a complete sufficient statistic is itself complete and sufficient, the pair $(\bar{X},S^{2})$ is complete and sufficient.

∎

Solution to exercise 10.5.

1.

Let $Z_{i}=X_{i}-\mu$ , so that $Z_{1},\dots,Z_{n}$ are i.i.d. $N(0,\sigma^{2})$ whatever the value of $\mu$ . Since subtracting the same constant from every observation shifts the maximum and the minimum by that constant, we have

$R=\max_{i}X_{i}-\min_{i}X_{i}=\max_{i}Z_{i}-\min_{i}Z_{i},$

and the right-hand side is a function of random variables whose joint distribution does not involve $\mu$ . Thus the distribution of $R$ is the same under every $\mathbb{P}_{\mu}$ , and $R$ is ancillary for $\mu$ in the sense of definition 10.8.
2.

In the proof of corollary 10.10 we have seen that, for known $\sigma^{2}$ , the normal distribution is a full-rank exponential family with natural statistic $T(x)=x$ , and thus $\bar{X}$ is complete and sufficient for $\mu$ by corollary 8.3 and proposition 10.5. Since $R$ is ancillary by the first part, Basu’s theorem, theorem 10.9, shows that $\bar{X}$ and $R$ are independent.
3.

Now let $\mu$ be known and $\sigma^{2}$ unknown, and write $X_{i}=\mu+\sigma W_{i}$ with $W_{1},\dots,W_{n}$ i.i.d. standard normal. Then $R=\sigma\,(\max_{i}W_{i}-\min_{i}W_{i})$ , and thus for example $\mathbb{E}_{\sigma^{2}}(R)=\sigma\,\mathbb{E}(\max_{i}W_{i}-\min_{i}W_{i})$ is proportional to $\sigma$ , and this expectation is strictly positive rather than zero, since $W_{1}$ and $W_{2}$ are independent and continuous and thus differ with probability one. The distribution of $R$ changes with the parameter, and $R$ is not ancillary for $\sigma^{2}$ . The scaled range $R/\sigma$ does have a distribution free of $\sigma^{2}$ , but it is not a statistic in the sense of definition 1.3, because computing it requires the unknown $\sigma$ . Quantities of this kind, whose distribution is known although they involve the parameter, return in lecture 25, where they are used to build confidence intervals.

∎

C.8 Fisher Information

Solution to exercise 11.1.

1.

The score for the observation is given by $\ell_{1}^{\prime}(\theta)=X_{1}/\theta-(1-X_{1})/(1-\theta)$ . This takes values $1/\theta$ if $X_{1}=1$ (with probability $\theta$ ) and $-1/(1-\theta)$ if $X_{1}=0$ (with probability $1-\theta$ ). Thus we have

$\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}=\theta\cdot\frac{% 1}{\theta}+(1-\theta)\cdot\Bigl{(}-\frac{1}{1-\theta}\Bigr{)}=1-1=0,$

as given in lemma 11.1.
2.

Combining the two fractions we get

$\ell_{1}^{\prime}(\theta)=\frac{X_{1}(1-\theta)-(1-X_{1})\theta}{\theta(1-% \theta)}=\frac{X_{1}-\theta}{\theta(1-\theta)},$

which is a constant multiple of $X_{1}-\theta$ . Since $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})=\theta(1-\theta)$ we find

$\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{% )}=\frac{\theta(1-\theta)}{\theta^{2}(1-\theta)^{2}}=\frac{1}{\theta(1-\theta)}.$

This is the value of $\mathcal{I}(\theta)$ in example 11.5 and thus, by theorem 11.3, the correct value.
3.

From proposition 11.4 we know $\mathcal{I}_{n}(\theta)=n/\bigl{(}\theta(1-\theta)\bigr{)}$ and thus $1/\mathcal{I}_{n}(\theta)=\theta(1-\theta)/n$ . By lemma 2.2 the sample mean has $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})=\mathop{\mathrm{Var}}% \nolimits_{\theta}(X_{1})/n=\theta(1-\theta)/n$ and thus the variance of $\bar{X}$ equals the reciprocal of the information, as it did for the normal mean in example 11.7.

∎

Solution to exercise 11.2.

1.

The log-likelihood of a single observation is $\ell_{1}(\theta)=-2\log\theta+\log X_{1}-X_{1}/\theta$ and thus the score is given by

$\ell_{1}^{\prime}(\theta)=-\frac{2}{\theta}+\frac{X_{1}}{\theta^{2}}.$

From table A.2 or from the solution to exercise 5.2 we know $\mathbb{E}_{\theta}(X_{1})=2\theta$ and thus $\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{)}=-2/\theta+2\theta% /\theta^{2}=0$ .
2.

Taking derivatives again, we find $\ell_{1}^{\prime\prime}(\theta)=2/\theta^{2}-2X_{1}/\theta^{3}$ and thus

$\mathcal{I}(\theta)=-\mathbb{E}_{\theta}\bigl{(}\ell_{1}^{\prime\prime}(\theta% )\bigr{)}=-\frac{2}{\theta^{2}}+\frac{2\cdot 2\theta}{\theta^{3}}=\frac{2}{% \theta^{2}}.$

On the other hand, since the score is $X_{1}/\theta^{2}$ plus a constant, we can find the variance as

$\mathop{\mathrm{Var}}\nolimits_{\theta}\bigl{(}\ell_{1}^{\prime}(\theta)\bigr{% )}=\frac{\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})}{\theta^{4}}=\frac{2% \theta^{2}}{\theta^{4}}=\frac{2}{\theta^{2}}.$

Both approaches give the same result.
3.

Since the density can be written as

$f(x;\theta)=x\,\exp\Bigl{(}-\frac{x}{\theta}-2\log\theta\Bigr{)},$

we have $h(x)=x$ , $T(x)=x$ , $\eta(\theta)=-1/\theta$ and $B(\theta)=2\log\theta$ . Thus, the model is an exponential family as in definition 7.1. The natural parameter takes values $\eta<0$ and the cumulant function is

$K(\eta)=\log\int_{0}^{\infty}x\,e^{\eta x}\,\mathrm{d}x=\log\frac{1}{\eta^{2}}% =-2\log(-\eta),$

where $K^{\prime}(\eta)=-2/\eta=2\theta$ is the mean and $K^{\prime\prime}(\eta)=2/\eta^{2}=2\theta^{2}$ is the variance, as predicted by theorem 7.7. Since $\eta^{\prime}(\theta)=1/\theta^{2}$ , we find by proposition 11.9 $\mathcal{I}(\theta)=\eta^{\prime}(\theta)^{2}K^{\prime\prime}\bigl{(}\eta(% \theta)\bigr{)}=2\theta^{2}/\theta^{4}=2/\theta^{2}$ . This is the same value as we obtained above.
4.

From proposition 11.4 we know that the information of the sample is $\mathcal{I}_{n}(\theta)=2n/\theta^{2}$ and thus $1/\mathcal{I}_{n}(\theta)=\theta^{2}/(2n)=\mathop{\mathrm{MSE}}\nolimits(\hat{% \theta})$ . Since $\hat{\theta}$ is unbiased, its mean squared error equals its variance and thus the variance of the MLE is the reciprocal of the Fisher information for all sample sizes, as it was for the sample mean in exercise 11.1 and example 11.7. From lecture 13 we know that this is the smallest variance an unbiased estimator can have.

∎

Solution to exercise 11.3.

1.

The log-likelihood of a single observation is $\ell_{1}(p)=(X_{1}-1)\log(1-p)+\log p$ and thus the score is given by

$\ell_{1}^{\prime}(p)=\frac{1}{p}-\frac{X_{1}-1}{1-p}.$

Using $\mathbb{E}_{p}(X_{1}-1)=1/p-1=(1-p)/p$ we find

$\mathbb{E}_{p}\bigl{(}\ell_{1}^{\prime}(p)\bigr{)}=\frac{1}{p}-\frac{(1-p)/p}{% 1-p}=\frac{1}{p}-\frac{1}{p}=0.$
2.

Taking derivatives again, we find $\ell_{1}^{\prime\prime}(p)=-1/p^{2}-(X_{1}-1)/(1-p)^{2}$ and taking expectations we get

$\displaystyle\mathcal{I}(p)$

$\displaystyle=-\mathbb{E}_{p}\bigl{(}\ell_{1}^{\prime\prime}(p)\bigr{)}=\frac{% 1}{p^{2}}+\frac{(1-p)/p}{(1-p)^{2}}$

$\displaystyle=\frac{1}{p^{2}}+\frac{1}{p(1-p)}=\frac{(1-p)+p}{p^{2}(1-p)}=% \frac{1}{p^{2}(1-p)}.$

From proposition 11.4 we know that the sample has information $\mathcal{I}_{n}(p)=n/\bigl{(}p^{2}(1-p)\bigr{)}$ .
3.

Since the score is $-X_{1}/(1-p)$ plus a constant, we find

$\mathop{\mathrm{Var}}\nolimits_{p}\bigl{(}\ell_{1}^{\prime}(p)\bigr{)}=\frac{% \mathop{\mathrm{Var}}\nolimits_{p}(X_{1})}{(1-p)^{2}}=\frac{(1-p)/p^{2}}{(1-p)% ^{2}}=\frac{1}{p^{2}(1-p)}.$

This agrees with the result from the previous part.

∎

Solution to exercise 11.4.

1.

Since $K(\eta)=e^{\eta}$ we have $K^{\prime\prime}(\eta)=e^{\eta}$ and using proposition 11.9 we find $\mathcal{I}_{\eta}(\eta)=e^{\eta}=\theta$ . This is the variance of the natural statistic $T(X)=X$ , as expected.
2.

Using proposition 11.10 with $\eta$ (the original parameter) and $\theta=g(\eta)=e^{\eta}$ (the new parameter), we find $g^{\prime}(\eta)=e^{\eta}=\theta$ . Thus we have

$\mathcal{I}_{\theta}(\theta)=\frac{\mathcal{I}_{\eta}(\eta)}{g^{\prime}(\eta)^% {2}}=\frac{\theta}{\theta^{2}}=\frac{1}{\theta},$

the value from example 11.6.
3.

We have $p_{0}=g(\theta)=e^{-\theta}$ with $g^{\prime}(\theta)=-e^{-\theta}$ and using proposition 11.10 we find

$\mathcal{I}_{p_{0}}(p_{0})=\frac{1/\theta}{e^{-2\theta}}=\frac{e^{2\theta}}{% \theta},$

or using the notation $p_{0}$ we find $\mathcal{I}_{p_{0}}(p_{0})=1/\bigl{(}p_{0}^{2}\log(1/p_{0})\bigr{)}$ . As $\theta\to\infty$ , the information goes to infinity. This makes sense, since $p_{0}=e^{-\theta}$ converges to zero, and the information measures accuracy on the absolute scale of the parameter: a probability which is close to zero is estimated with small absolute error by any reasonable estimator, and the approximate variance $1/\mathcal{I}_{n}(p_{0})=\theta e^{-2\theta}/n$ from lecture 17 is tiny, even if the error relative to the size of $p_{0}$ is not small.

∎

Solution to exercise 11.5.

1.

The map $g(\theta)=1/\theta$ maps $(0,\infty)$ to $(0,\infty)$ and $g^{\prime}(\theta)=-1/\theta^{2}\neq 0$ . From example 11.8 we know that $\mathcal{I}_{\theta}(\theta)=1/\theta^{2}$ . Thus, by proposition 11.10, we have

$\mathcal{I}_{\mu}(\mu)=\frac{\mathcal{I}_{\theta}(\theta)}{g^{\prime}(\theta)^% {2}}=\frac{1/\theta^{2}}{1/\theta^{4}}=\theta^{2}=\frac{1}{\mu^{2}}.$

This is the value given in section 11.6.
2.

Using proposition 11.4, in the $\mu$ -parametrisation, the information of the sample about $\mu$ is given by $n\mathcal{I}_{\mu}(\mu)=n/\mu^{2}$ . From table A.2 we know that $\mathop{\mathrm{Var}}\nolimits_{\mu}(X_{1})=1/\theta^{2}=\mu^{2}$ and thus we have $\mathop{\mathrm{Var}}\nolimits_{\mu}(\bar{X})=\mu^{2}/n$ , by lemma 2.2. The variance of the unbiased estimator $\bar{X}$ for $\mu$ equals the reciprocal $\mu^{2}/n$ of the information of the sample, and from lecture 13 we know that no unbiased estimator can be better.

∎

Solution to exercise 11.6.

The right-hand side of (11.1) with $k=1$ is an integral over the whole real line, but $f(x;\theta)=0$ and thus $\partial f/\partial\theta=0$ for $x<0$ and for $x>\theta$ ; since $x=\theta$ is a single point, it does not contribute to the integral; only $0\leq x\leq\theta$ , where $f(x;\theta)=1/\theta$ , is left. We have

\int_{0}^{\theta}\frac{\partial}{\partial\theta}f(x;\theta)\,\mathrm{d}x=\int_% {0}^{\theta}\Bigl{(}-\frac{1}{\theta^{2}}\Bigr{)}\mathrm{d}x=-\frac{1}{\theta},

while the left-hand side can be found as

\frac{\mathrm{d}}{\mathrm{d}\theta}\int_{0}^{\theta}f(x;\theta)\,\mathrm{d}x=% \frac{\mathrm{d}}{\mathrm{d}\theta}\,1=0.

The difference between the two sides is $0-(-1/\theta)=1/\theta$ , which is the value $f(\theta;\theta)=1/\theta$ of the density at the upper limit of the integral. The derivative of the integral picks up this boundary term (from the moving upper limit) in addition to the derivative of the integrand, but the differentiation under the integral sign ignores this boundary term. ∎

C.9 The Cramer–Rao Inequality

Solution to exercise 13.1.

1.

For $y\geq 0$ we have

$\mathbb{P}_{\theta}(Y_{1}\leq y)=\mathbb{P}_{\theta}\Bigl{(}X_{1}\leq\frac{y}{% 2\theta}\Bigr{)}=1-e^{-y/2},$

and taking derivatives we find the density $\tfrac{1}{2}e^{-y/2}$ for $y\geq 0$ . This is the exponential density with rate $1/2$ , i.e. the $\text{Gamma}(1,1/2)$ density from table A.2 and $\text{Gamma}(1,1/2)$ is $\chi^{2}_{2}$ by the statement quoted in the question. Thus we have $Y_{1}\sim\chi^{2}_{2}$ .
2.

From the first part and definition A.2 we know that each $Y_{i}=2\theta X_{i}$ has the distribution of $Z_{2i-1}^{2}+Z_{2i}^{2}$ for independent, standard normally distributed $Z_{2i-1}$ and $Z_{2i}$ . Since the $X_{i}$ are independent, we can assume that all $2n$ are independent normal variables. Thus, $2\theta T=\sum_{i=1}^{n}Y_{i}$ has the distribution of $Z_{1}^{2}+\dots+Z_{2n}^{2}$ , i.e. $\chi^{2}_{2n}$ by definition A.2.
3.

The density of $T\sim\text{Gamma}(n,\theta)$ is given by $f_{T}(t)=\theta^{n}t^{n-1}e^{-\theta t}/\Gamma(n)$ for $t\geq 0$ . For $V=2\theta T$ , using proposition A.19 with inverse map $t=v/(2\theta)$ and derivative $1/(2\theta)$ , we find

$f_{V}(v)=f_{T}\Bigl{(}\frac{v}{2\theta}\Bigr{)}\,\frac{1}{2\theta}=\frac{% \theta^{n}\bigl{(}v/(2\theta)\bigr{)}^{n-1}e^{-v/2}}{\Gamma(n)}\cdot\frac{1}{2% \theta}=\frac{(1/2)^{n}\,v^{n-1}e^{-v/2}}{\Gamma(n)}$

for $v\geq 0$ . This is the $\text{Gamma}(n,1/2)$ density again, by table A.2. Since $n=(2n)/2$ , this is the $\chi^{2}_{2n}$ density and it agrees with the result from the previous part.
4.

From definition A.2 we know $\mathbb{E}_{\theta}(2\theta T)=2n$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(2\theta T)=4n$ , and thus $\mathbb{E}_{\theta}(T)=n/\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(T)=4n/(4\theta^{2})=n/\theta^{2}$ . For a single observation, the mean is $1/\theta$ and the variance is $1/\theta^{2}$ , as found in table A.2, and thus the sum of $n$ independent copies has mean $n/\theta$ and variance $n/\theta^{2}$ , as shown above.

∎

Solution to exercise 13.2.

1.

From example 11.5 we know that $\mathcal{I}_{n}(\theta)=n/\bigl{(}\theta(1-\theta)\bigr{)}$ . Thus, the bound from theorem 13.2 is $1/\mathcal{I}_{n}(\theta)=\theta(1-\theta)/n$ .
2.

The sample mean is an unbiased estimator for $\theta$ and using lemma 2.2 we find $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})=\theta(1-\theta)/n$ , since the variance of a Bernoulli distribution is $\theta(1-\theta)$ . This bound coincides with the bound from the definition, and thus $\bar{X}$ is efficient by definition 13.4, since the exchange identity (13.1) holds because the Bernoulli distribution is a regular exponential family and $\bar{X}$ has finite variance, as noted after lemma 13.1.
3.

The score of the sample in example 11.5 is given by

$\ell^{\prime}(\theta)=\frac{T_{n}}{\theta}-\frac{n-T_{n}}{1-\theta}=\frac{T_{n% }-n\theta}{\theta(1-\theta)}=\frac{n}{\theta(1-\theta)}\,(\bar{X}-\theta),$

and thus we have $\bar{X}-\theta=c(\theta)\,\ell^{\prime}(\theta)$ with $c(\theta)=\theta(1-\theta)/n$ . Since $g(\theta)=\theta$ has $g^{\prime}(\theta)=1$ , this is $g^{\prime}(\theta)/\mathcal{I}_{n}(\theta)$ as predicted by proposition 13.5.
4.

For $g(\theta)=\theta/(1-\theta)$ we have $g^{\prime}(\theta)=1/(1-\theta)^{2}$ and thus using theorem 13.3 we get the bound

$\frac{g^{\prime}(\theta)^{2}}{\mathcal{I}_{n}(\theta)}=\frac{1}{(1-\theta)^{4}% }\cdot\frac{\theta(1-\theta)}{n}=\frac{\theta}{n(1-\theta)^{3}}.$

The Bernoulli distribution is an exponential family with natural statistic $T(x)=x$ and $\mu(\theta)=\theta$ . From the discussion after proposition 13.5 we know that only affine functions of the form $a+b\theta$ can be used to construct an efficient estimator. Since the odds are not an affine function of $\theta$ , there can be no efficient estimator for the odds.
5.

Any estimator $W=W(X_{1},\dots,X_{n})$ can take at most $2^{n}$ different values, one for each possible value of the sample $x\in\{0,1\}^{n}$ , and the expectation of the estimator is

$\mathbb{E}_{\theta}(W)=\sum_{x\in\{0,1\}^{n}}W(x)\,\theta^{\sum_{i}x_{i}}(1-% \theta)^{n-\sum_{i}x_{i}},$

which is a polynomial in $\theta$ of degree at most $n$ . A polynomial function is bounded on $(0,1)$ , whereas $\theta/(1-\theta)\to\infty$ as $\theta\to 1$ and thus no choice of $W$ makes the estimator $\mathbb{E}_{\theta}(W)$ equal to the odds for all $\theta$ . Thus there is no unbiased estimator for the odds, efficient or otherwise.

∎

Solution to exercise 13.3.

1.

From example 11.6 we know that $\mathcal{I}_{n}(\theta)=n/\theta$ . Thus, the Cramer–Rao bound for unbiased estimators for $\theta$ is $\theta/n$ . Since the Poisson distribution has mean and variance $\theta$ , lemma 2.2 shows that $\mathbb{E}_{\theta}(\bar{X})=\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(\bar{X})=\theta/n$ . Thus, $\bar{X}$ is an unbiased estimator and satisfies the bound: it is an efficient estimator for $\theta$ , since the exchange identity (13.1) holds because the Poisson distribution is a regular exponential family and $\bar{X}$ has finite variance, as noted after lemma 13.1.
2.

The log-likelihood function is $\ell(\theta)=T_{n}\log\theta-n\theta-\sum_{i}\log(X_{i}!)$ and thus the score function is

$\ell^{\prime}(\theta)=\frac{T_{n}}{\theta}-n=\frac{n}{\theta}\,(\bar{X}-\theta).$

Re-arranging this we find $\bar{X}-\theta=(\theta/n)\,\ell^{\prime}(\theta)$ , which is condition (13.2) with $W=\bar{X}$ , $g(\theta)=\theta$ and $c(\theta)=\theta/n$ ; indeed $g^{\prime}(\theta)/\mathcal{I}_{n}(\theta)=1/(n/\theta)=\theta/n$ .
3.

From the discussion after proposition 13.5 we know that in a one-parameter exponential family, only affine functions of the mean $\mu(\theta)$ of the natural statistic allow for efficient estimators. In this case we have $\mu(\theta)=\theta$ and $g(\theta)=e^{-\theta}$ is not an affine function of $\theta$ . Thus, no efficient estimator for $e^{-\theta}$ exists. There is no contradiction in this result: example 10.6 shows that $(1-1/n)^{T_{n}}$ has the smallest variance amongst all unbiased estimators for $e^{-\theta}$ , but this smallest variance is strictly larger than the Cramer–Rao bound. Efficiency is a stronger property than being UMVUE, and in this case the bound cannot be achieved.

∎

Solution to exercise 13.4.

1.

Example 11.8 gives $\mathcal{I}(\theta)=1/\theta^{2}$ , and thus $\mathcal{I}_{n}(\theta)=n/\theta^{2}$ and the bound is $1/\mathcal{I}_{n}(\theta)=\theta^{2}/n$ .
2.

Since $\tilde{\theta}$ is unbiased, its mean squared error is its variance, $\mathop{\mathrm{Var}}\nolimits_{\theta}(\tilde{\theta})=\theta^{2}/(n-2)$ . The efficiency is

$\operatorname{eff}_{\theta}(\tilde{\theta})=\frac{\theta^{2}/n}{\theta^{2}/(n-% 2)}=\frac{n-2}{n},$

which is less than one for every $n\geq 3$ and tends to one as $n\to\infty$ : the estimator falls short of the bound for every finite sample size, but only by a factor which disappears in the limit.
3.

The exponential distribution with rate $\theta$ is an exponential family with $h(x)=1$ , $\eta(\theta)=-\theta$ , $T(x)=x$ and $B(\theta)=-\log\theta$ , as in example 7.4, and thus $\mu(\theta)=\mathbb{E}_{\theta}(X_{1})=1/\theta$ . By the discussion after proposition 13.5, an efficient estimator exists only for affine functions $a+b/\theta$ of $\mu(\theta)$ . The rate $g(\theta)=\theta$ is not of this form, and thus no unbiased estimator for $\theta$ is efficient. The same conclusion can be reached directly: condition (13.2) with $\ell^{\prime}(\theta)=n/\theta-T$ would read $W=\theta+c(\theta)n/\theta-c(\theta)T$ , and since $W$ does not depend on $\theta$ , the coefficients $-c(\theta)$ and $\theta+c(\theta)n/\theta$ would have to be constants $b$ and $a$ , giving $\mathbb{E}_{\theta}(W)=a+bn/\theta$ , which cannot equal $\theta$ for all $\theta$ .
4.

The natural parameter $\eta=-\theta$ ranges over the open interval $(-\infty,0)$ , and thus the family is of full rank and $T$ is complete and sufficient by corollary 8.3 and proposition 10.5. Since $\tilde{\theta}$ is a function of $T$ and unbiased for $\theta$ , the Lehmann–Scheffe theorem, theorem 10.4, shows that it is the UMVUE for $\theta$ . Together with the previous part this is an example of a UMVUE which is not efficient: the bound $\theta^{2}/n$ is not attained by any unbiased estimator, and the best unbiased estimator has variance $\theta^{2}/(n-2)$ .
5.

For $g(\theta)=1/\theta$ we have $g^{\prime}(\theta)=-1/\theta^{2}$ , and theorem 13.3 gives the bound

$\frac{g^{\prime}(\theta)^{2}}{\mathcal{I}_{n}(\theta)}=\frac{1/\theta^{4}}{n/% \theta^{2}}=\frac{1}{n\theta^{2}}=\frac{\mu^{2}}{n}.$

The sample mean is unbiased for $\mu$ and has variance $\mathop{\mathrm{Var}}\nolimits_{\theta}(X_{1})/n=\mu^{2}/n$ by lemma 2.2, and thus $\bar{X}$ is efficient for $\mu$ , since the exchange identity (13.1) holds because the exponential distribution is a regular exponential family and $\bar{X}$ has finite variance, as noted after lemma 13.1. This is consistent with the previous parts: $\mu(\theta)=1/\theta$ is the mean of the natural statistic, for which proposition 13.5 guarantees an efficient estimator, whereas the rate is a non-affine function of it. Whether an efficient estimator exists depends on which function of the parameter we set out to estimate.

∎

Solution to exercise 13.5.

1.

Since $\mu$ is given and $\tau=\sigma^{2}$ is the parameter, the log-likelihood function is $\ell(\tau)=-\tfrac{n}{2}\log(2\pi\tau)-\frac{1}{2\tau}\sum_{i}(X_{i}-\mu)^{2}$ and thus

$\ell^{\prime}(\tau)=-\frac{n}{2\tau}+\frac{1}{2\tau^{2}}\sum_{i=1}^{n}(X_{i}-% \mu)^{2},\qquad\ell^{\prime\prime}(\tau)=\frac{n}{2\tau^{2}}-\frac{1}{\tau^{3}% }\sum_{i=1}^{n}(X_{i}-\mu)^{2}.$

Since $\mathbb{E}\bigl{(}(X_{i}-\mu)^{2}\bigr{)}=\tau$ , we can use theorem 11.3 to find

$\mathcal{I}_{n}(\tau)=-\mathbb{E}\bigl{(}\ell^{\prime\prime}(\tau)\bigr{)}=-% \frac{n}{2\tau^{2}}+\frac{n\tau}{\tau^{3}}=\frac{n}{2\tau^{2}}=\frac{n}{2% \sigma^{4}}.$

Thus, the Cramer–Rao bound is $2\sigma^{4}/n$ .
2.

The $(X_{i}-\mu)/\sigma$ are i.i.d. standard normally distributed, i.e. $\sum_{i}(X_{i}-\mu)^{2}/\sigma^{2}\sim\chi^{2}_{n}$ by definition A.2, with mean $n$ and variance $2n$ . Thus, $\hat{\sigma}^{2}=\frac{\sigma^{2}}{n}\sum_{i}(X_{i}-\mu)^{2}/\sigma^{2}$ has mean $\sigma^{2}$ and variance $\sigma^{4}\cdot 2n/n^{2}=2\sigma^{4}/n$ . This is the bound and thus $\hat{\sigma}^{2}$ is efficient, since the exchange identity (13.1) holds because the normal distribution with known mean is a regular exponential family and $\hat{\sigma}^{2}$ has finite variance, as noted after lemma 13.1. Alternatively, the score for the first part can be written as $\ell^{\prime}(\sigma^{2})=\frac{n}{2\sigma^{4}}(\hat{\sigma}^{2}-\sigma^{2})$ , which satisfies condition (13.2) with $c(\sigma^{2})=2\sigma^{4}/n$ .
3.

From proposition A.3 we know that $(n-1)S^{2}/\sigma^{2}\sim\chi^{2}_{n-1}$ , with variance $2(n-1)$ . Thus we have

$\mathop{\mathrm{Var}}\nolimits(S^{2})=\frac{\sigma^{4}}{(n-1)^{2}}\cdot 2(n-1)% =\frac{2\sigma^{4}}{n-1}>\frac{2\sigma^{4}}{n}.$

Thus, the sample variance does not achieve the bound, and the efficiency of the sample variance is $(n-1)/n$ . Since the bound $2\sigma^{4}/n$ still holds for unknown $\mu$ , and since $S^{2}$ is the UMVUE for $\sigma^{2}$ by example 10.7, this is a second example of a best unbiased estimator which is not efficient: no unbiased estimator for $\sigma^{2}$ can achieve the bound if $\mu$ also needs to be estimated, and the loss of one degree of freedom is the price to be paid for not knowing $\mu$ .

∎

Solution to exercise 13.6.

1.

By proposition 11.4 the information of the sample is $n$ times the information of a single observation, in either parametrisation. Proposition 11.10 gives the relation for a single observation, and multiplying both sides by $n$ we find

$\mathcal{I}_{\psi}(\psi)=\frac{\mathcal{I}_{\theta}(\theta)}{g^{\prime}(\theta% )^{2}}\qquad\text{for }\psi=g(\theta).$
2.

Since $\psi=g(\theta)$ , the estimator $W$ is an unbiased estimator for the parameter $\psi$ , and theorem 13.2, applied in the $\psi$ -parametrisation, bounds its variance by $1/\mathcal{I}_{\psi}(\psi)$ . Using the first part we get

$\frac{1}{\mathcal{I}_{\psi}(\psi)}=\frac{g^{\prime}(\theta)^{2}}{\mathcal{I}_{% \theta}(\theta)},$

which is the bound of theorem 13.3. The two theorems thus give the same bound, and the bound does not depend on how the model is parametrised. This completes the proof.

∎

C.10 Multiparameter Fisher Information

Solution to exercise 14.1.

1.

The Jacobian matrix of $(\mu,\sigma)\mapsto(\mu,\sigma^{2})$ is given by

$J=\begin{pmatrix}\dfrac{\partial\mu}{\partial\mu}&\dfrac{\partial\mu}{\partial% \sigma}\\[9.42914pt] \dfrac{\partial\sigma^{2}}{\partial\mu}&\dfrac{\partial\sigma^{2}}{\partial% \sigma}\end{pmatrix}=\begin{pmatrix}1&0\\ 0&2\sigma\end{pmatrix}.$

This matrix is invertible for $\sigma>0$ . Since all three matrices are diagonal, the product $J^{\top}\mathcal{I}_{n}(\mu,\sigma^{2})J$ is also diagonal and consists of the entries $1\cdot(n/\sigma^{2})\cdot 1=n/\sigma^{2}$ and $2\sigma\cdot n/(2\sigma^{4})\cdot 2\sigma=2n/\sigma^{2}$ . Thus, by proposition 14.4, we have $\mathcal{I}^{\varphi}_{n}(\mu,\sigma)=\mathrm{diag}(n/\sigma^{2},\,2n/\sigma^{% 2})$ .
2.

The log-likelihood function in terms of $(\mu,\sigma)$ is

$\ell(\mu,\sigma)=-\frac{n}{2}\log(2\pi)-n\log\sigma-\frac{1}{2\sigma^{2}}\sum_% {i=1}^{n}(X_{i}-\mu)^{2}.$

Taking derivatives we get $\partial\ell/\partial\mu=\sum_{i}(X_{i}-\mu)/\sigma^{2}$ and $\partial\ell/\partial\sigma=-n/\sigma+\sum_{i}(X_{i}-\mu)^{2}/\sigma^{3}$ . Taking derivatives again, we find

$\displaystyle\frac{\partial^{2}\ell}{\partial\mu^{2}}$

$\displaystyle=-\frac{n}{\sigma^{2}},$

$\displaystyle\frac{\partial^{2}\ell}{\partial\mu\,\partial\sigma}$

$\displaystyle=-\frac{2}{\sigma^{3}}\sum_{i=1}^{n}(X_{i}-\mu),$

$\displaystyle\frac{\partial^{2}\ell}{\partial\sigma^{2}}$

$\displaystyle=\frac{n}{\sigma^{2}}-\frac{3}{\sigma^{4}}\sum_{i=1}^{n}(X_{i}-% \mu)^{2}.$

Using $\mathbb{E}(X_{i}-\mu)=0$ and $\mathbb{E}\bigl{(}(X_{i}-\mu)^{2}\bigr{)}=\sigma^{2}$ we find that the mixed derivative has expectation zero, and the last term has expectation $n/\sigma^{2}-3n/\sigma^{2}=-2n/\sigma^{2}$ . Thus, by proposition 14.3, the information matrix is given by $\mathrm{diag}(n/\sigma^{2},\,2n/\sigma^{2})$ . This agrees with the result from part (a).
3.

The inverse of the matrix from part (a) is given by $\mathrm{diag}\bigl{(}\sigma^{2}/n,\,\sigma^{2}/(2n)\bigr{)}$ . Thus, by theorem 14.5, any unbiased estimator for $\sigma$ has variance of at least $\sigma^{2}/(2n)$ . The sample variance $S^{2}$ is an unbiased estimator for $\sigma^{2}$ by lemma 2.3, but the square root of the sample variance is not an unbiased estimator for $\sigma$ : The function $g(x)=-\sqrt{x}$ is strictly convex on $(0,\infty)$ and by Jensen’s inequality, lemma A.7, we have $-\sqrt{\mathbb{E}(S^{2})}\leq\mathbb{E}(-\sqrt{S^{2}})$ , i.e. $\mathbb{E}(S)\leq\sqrt{\mathbb{E}(S^{2})}=\sigma$ . The inequality is strict, since $S^{2}$ is not constant with probability one (by proposition A.3 it is a multiple of a chi-squared random variable). Thus, $\mathbb{E}(S)<\sigma$ , the estimator $S$ is biased for $\sigma$ and the Cramer–Rao bound, which only concerns unbiased estimators, cannot be compared to $\mathop{\mathrm{Var}}\nolimits(S)$ directly.

∎

Solution to exercise 14.2.

1.

We have $\ell_{1}(\alpha,\beta)=\alpha\log\beta-\log\Gamma(\alpha)+(\alpha-1)\log X-\beta X$ and

$\frac{\partial\ell_{1}}{\partial\alpha}=\log\beta-\psi(\alpha)+\log X\qquad% \text{and}\qquad\frac{\partial\ell_{1}}{\partial\beta}=\frac{\alpha}{\beta}-X,$

where $\psi=(\log\Gamma)^{\prime}$ is the derivative of $\log\Gamma$ , which is smooth and thus differentiable on $(0,\infty)$ . Taking derivatives again, we get $\partial^{2}\ell_{1}/\partial\alpha^{2}=-\psi^{\prime}(\alpha)$ , $\partial^{2}\ell_{1}/\partial\alpha\,\partial\beta=1/\beta$ and $\partial^{2}\ell_{1}/\partial\beta^{2}=-\alpha/\beta^{2}$ , as given in section 14.5. To verify the first identity of proposition 14.3, we note that the second component of the score has mean $\alpha/\beta-\mathbb{E}(X)=0$ , since $\mathbb{E}(X)=\alpha/\beta$ by table A.2.
2.

If $\alpha=2$ is known, the model consists of one parameter $\beta$ and by theorem 11.3 the Fisher information is $\mathcal{I}(\beta)=-\mathbb{E}\bigl{(}\partial^{2}\ell_{1}/\partial\beta^{2}% \bigr{)}=\alpha/\beta^{2}=2/\beta^{2}$ . The lower right corner of $\mathcal{I}(\alpha,\beta)$ is the diagonal element $\mathcal{I}(\alpha,\beta)_{22}$ , which by definition is the variance of $\partial\ell_{1}/\partial\beta$ with $\alpha$ fixed to the true value, i.e. for known shape. The diagonal element of the inverse answers a different question, asking about the estimation of $\beta$ when $\alpha$ is unknown. Part (d) shows that this value is larger.
3.

The scale is $\theta=g(\beta)=1/\beta$ where $g^{\prime}(\beta)=-1/\beta^{2}$ and by proposition 11.10 we have

$\mathcal{I}_{\theta}(\theta)=\frac{\mathcal{I}_{\beta}(\beta)}{g^{\prime}(% \beta)^{2}}=\frac{2/\beta^{2}}{1/\beta^{4}}=2\beta^{2}=\frac{2}{\theta^{2}}.$

This is the same value as in exercise 11.2.
4.

We have $a=\psi^{\prime}(\alpha)$ , $b=-1/\beta$ and $c=\alpha/\beta^{2}$ and thus the determinant of $\mathcal{I}(\alpha,\beta)$ is $ac-b^{2}=\bigl{(}\alpha\psi^{\prime}(\alpha)-1\bigr{)}/\beta^{2}$ . The lower right element of the inverse is given by

$\bigl{(}\mathcal{I}(\alpha,\beta)^{-1}\bigr{)}_{22}=\frac{a}{ac-b^{2}}=\frac{% \psi^{\prime}(\alpha)\,\beta^{2}}{\alpha\psi^{\prime}(\alpha)-1}.$

For a sample of size $n$ , the information matrix is $n\mathcal{I}(\alpha,\beta)$ , the inverse is $\mathcal{I}(\alpha,\beta)^{-1}/n$ and by theorem 14.5 we find the bound $\beta^{2}\psi^{\prime}(\alpha)/\bigl{(}n(\alpha\psi^{\prime}(\alpha)-1)\bigr{)}$ for an unbiased estimator for $\beta$ as claimed. The bound for known shape is $1/\mathcal{I}_{n}(\beta)=\beta^{2}/(n\alpha)$ and the ratio of the two is

$\frac{\beta^{2}\psi^{\prime}(\alpha)/\bigl{(}n(\alpha\psi^{\prime}(\alpha)-1)% \bigr{)}}{\beta^{2}/(n\alpha)}=\frac{\alpha\psi^{\prime}(\alpha)}{\alpha\psi^{% \prime}(\alpha)-1}=\frac{1}{1-\rho^{2}},$

where $\rho^{2}=1/\bigl{(}\alpha\psi^{\prime}(\alpha)\bigr{)}$ is as in (14.2). For $\alpha=2$ we have $\psi^{\prime}(2)=\pi^{2}/6-1\approx 0.6449$ and thus $\alpha\psi^{\prime}(\alpha)\approx 1.2899$ and the ratio is approximately $1.2899/0.2899\approx 4.45$ : not knowing the shape increases the bound for the rate by a factor of approximately $4.45$ , as in section 14.5.

∎

Solution to exercise 14.3.

1.

Since the two samples are independent, the log-likelihood is the sum of the two sample log-likelihoods:

$\ell(\mu_{1},\mu_{2})=-\frac{m+n}{2}\log(2\pi\sigma^{2})-\frac{1}{2\sigma^{2}}% \sum_{i=1}^{m}(X_{i}-\mu_{1})^{2}-\frac{1}{2\sigma^{2}}\sum_{j=1}^{n}(Y_{j}-% \mu_{2})^{2},$

with partial derivatives $\partial\ell/\partial\mu_{1}=\sum_{i}(X_{i}-\mu_{1})/\sigma^{2}$ and $\partial\ell/\partial\mu_{2}=\sum_{j}(Y_{j}-\mu_{2})/\sigma^{2}$ . The second partial derivatives are $\partial^{2}\ell/\partial\mu_{1}^{2}=-m/\sigma^{2}$ , $\partial^{2}\ell/\partial\mu_{2}^{2}=-n/\sigma^{2}$ and $\partial^{2}\ell/\partial\mu_{1}\,\partial\mu_{2}=0$ , since each first partial derivative involves only one of the two parameters. None of these second derivatives are random, and thus proposition 14.3 gives $\mathcal{I}(\mu_{1},\mu_{2})=\mathrm{diag}(m/\sigma^{2},\,n/\sigma^{2})$ , without any expectations.
2.

The difference is $\mu_{1}-\mu_{2}=a^{\top}\theta$ with $a=(1,-1)$ and the inverse information matrix is $\mathrm{diag}(\sigma^{2}/m,\,\sigma^{2}/n)$ . By theorem 14.5 any unbiased estimator for $\mu_{1}-\mu_{2}$ has variance at least

$a^{\top}\mathcal{I}(\mu_{1},\mu_{2})^{-1}a=\frac{\sigma^{2}}{m}+\frac{\sigma^{% 2}}{n}.$

The estimator $\bar{X}-\bar{Y}$ is unbiased for $\mu_{1}-\mu_{2}$ by lemma 2.2 and, since the two samples are independent, the variance of this estimator is $\mathop{\mathrm{Var}}\nolimits(\bar{X})+\mathop{\mathrm{Var}}\nolimits(\bar{Y}% )=\sigma^{2}/m+\sigma^{2}/n$ . This is the bound. Thus, $\bar{X}-\bar{Y}$ is an efficient estimator for the difference of the means.
3.

If we include $\sigma^{2}$ as an additional parameter, the first derivative is

$\frac{\partial\ell}{\partial\sigma^{2}}=-\frac{m+n}{2\sigma^{2}}+\frac{1}{2% \sigma^{4}}\Bigl{(}\sum_{i=1}^{m}(X_{i}-\mu_{1})^{2}+\sum_{j=1}^{n}(Y_{j}-\mu_% {2})^{2}\Bigr{)}.$

The mixed derivatives with respect to $\mu_{1}$ and $\sigma^{2}$ and with respect to $\mu_{2}$ and $\sigma^{2}$ are $-\sum_{i}(X_{i}-\mu_{1})/\sigma^{4}$ and $-\sum_{j}(Y_{j}-\mu_{2})/\sigma^{4}$ , both with expectation zero, and the second derivative with respect to $\sigma^{2}$ has expectation $(m+n)/(2\sigma^{4})-(m+n)\sigma^{2}/\sigma^{6}=-(m+n)/(2\sigma^{4})$ , as shown in example 14.6. Thus, the information matrix is

$\mathcal{I}(\mu_{1},\mu_{2},\sigma^{2})=\mathrm{diag}\Bigl{(}\frac{m}{\sigma^{% 2}},\,\frac{n}{\sigma^{2}},\,\frac{m+n}{2\sigma^{4}}\Bigr{)},$

still diagonal, and thus $\sigma^{2}$ is orthogonal to both means. The upper left $2\times 2$ block of the inverse is the inverse information matrix from part (b), and the Cramer–Rao bound for $\mu_{1}-\mu_{2}$ is still $\sigma^{2}/m+\sigma^{2}/n$ , and this bound is achieved by $\bar{X}-\bar{Y}$ , whose definition does not depend on $\sigma^{2}$ . For point estimation of the difference, not knowing the variance does not change anything. However, once we want to determine the accuracy of an estimate, the unknown variance starts to matter, since the bound depends on $\sigma^{2}$ and thus needs to be estimated. This is the reason why we go from the normal distribution to the $t$ distribution in lectures 25 and 29.

∎

C.11 Consistency of the MLE

Solution to exercise 16.1.

1.

For (R1) we have two different rates and thus two different distributions: for example, the means $1/\theta$ are different. For (R2) the support is $[0,\infty)$ for all $\theta$ . For (R3) we have $\log f(x;\theta)=\log\theta-\theta x$ and the derivatives w.r.t. $\theta$ are $1/\theta-x$ , $-1/\theta^{2}$ and $2/\theta^{3}$ , all of which are continuous at $\theta$ . The derivatives of $f$ are of the form (polynomial in $x$ and $\theta$ ) times $e^{-\theta x}$ and thus are integrable over $[0,\infty)$ uniformly for $\theta$ in any compact subinterval of $(0,\infty)$ . We can therefore interchange the order of differentiation and integration, and $\mathcal{I}(\theta)=1/\theta^{2}$ is continuous and lies strictly between zero and infinity, so that (R3) holds. Finally, for (R4), the third derivative $2/\theta^{3}$ does not depend on $x$ and for $|\theta-\theta_{0}|<\delta$ with $\delta=\theta_{0}/2$ we have $\theta>\theta_{0}/2$ and thus $2/\theta^{3}\leq 16/\theta_{0}^{3}$ . The constant $M(x)=16/\theta_{0}^{3}$ bounds the third derivative and has finite expectation.
2.

The likelihood equation $n/\theta-\sum_{i}x_{i}=0$ has only one solution $\theta=1/\bar{x}$ , since $\sum_{i}x_{i}>0$ with probability one. Since $\ell^{\prime\prime}(\theta)=-n/\theta^{2}<0$ , the log-likelihood function is strictly concave and thus the root is the global maximum of the likelihood function, i.e. the MLE as in example 5.2. Since all conditions of theorem 16.3 are satisfied, we find that $1/\bar{X}$ is consistent for $\theta$ .
3.

By the law of large numbers, theorem A.14, we have $\bar{X}\xrightarrow{\ \mathrm{p}\ }\mathbb{E}_{\theta}(X_{1})=1/\theta$ . Since the function $g(y)=1/y$ is continuous at $y=1/\theta>0$ , the continuous mapping theorem, theorem A.17, applies and we find $1/\bar{X}\xrightarrow{\ \mathrm{p}\ }g(1/\theta)=\theta$ . This proves the consistency of the MLE.

∎

Solution to exercise 16.2.

1.

Since $f(x;\theta)=\theta^{x}(1-\theta)^{1-x}$ for $x\in\{0,1\}$ , we have

$\log\frac{f(X_{1};\theta)}{f(X_{1};\theta_{0})}=X_{1}\log\frac{\theta}{\theta_% {0}}+(1-X_{1})\log\frac{1-\theta}{1-\theta_{0}},$

and taking expectations with $\mathbb{E}_{\theta_{0}}(X_{1})=\theta_{0}$ gives

$D(\theta)=\theta_{0}\log\frac{\theta}{\theta_{0}}+(1-\theta_{0})\log\frac{1-% \theta}{1-\theta_{0}}.$
2.

Using the rule $\log y\leq y-1$ for logarithms, we find

$D(\theta)\leq\theta_{0}\Bigl{(}\frac{\theta}{\theta_{0}}-1\Bigr{)}+(1-\theta_{% 0})\Bigl{(}\frac{1-\theta}{1-\theta_{0}}-1\Bigr{)}=(\theta-\theta_{0})+(\theta% _{0}-\theta)=0.$

For $D(\theta)=0$ to hold, we need to have equality in both applications of the inequality, i.e. we need $\theta/\theta_{0}=1$ and $(1-\theta)/(1-\theta_{0})=1$ . Both of these conditions state $\theta=\theta_{0}$ . Thus we have $D(\theta)<0$ for all $\theta\neq\theta_{0}$ , as stated in theorem 16.2.

∎

Solution to exercise 16.3.

1.

Since all observations are at most $\theta$ , we have $\hat{\theta}_{n}\leq\theta$ and thus $|\hat{\theta}_{n}-\theta|>\varepsilon$ is the same as $\hat{\theta}_{n}<\theta-\varepsilon$ . This is the case if and only if all $n$ observations are less than $\theta-\varepsilon$ . Using independence we find

$\mathbb{P}_{\theta}\bigl{(}|\hat{\theta}_{n}-\theta|>\varepsilon\bigr{)}=\Bigl% {(}\frac{\theta-\varepsilon}{\theta}\Bigr{)}^{n}=\Bigl{(}1-\frac{\varepsilon}{% \theta}\Bigr{)}^{n}\longrightarrow 0\qquad(n\to\infty),$

since $0<1-\varepsilon/\theta<1$ . For $\varepsilon\geq\theta$ , the probability of this event is zero. Thus $\hat{\theta}_{n}\xrightarrow{\ \mathrm{p}\ }\theta$ and the MLE is consistent.
2.

Condition (R2) is violated, since the support $(0,\theta)$ depends on $\theta$ and (R3) is also violated: for fixed $x$ , the function $\theta\mapsto f(x;\theta)$ has a jump from $0$ to $1/\theta$ at $\theta=x$ and thus is not differentiable there. The uniqueness of the MLE fails, since the likelihood equation has no roots: on the region $\theta\geq\max_{i}x_{i}$ where $L$ is positive, the derivative $-n\theta^{-n-1}$ never equals zero. Since the proof of theorem 16.3 requires a differentiable likelihood and a root of the derivative to apply Rolle’s theorem, neither of these conditions are satisfied. Nevertheless, the statement is true, as can be seen by the direct computation in the first part of the question, or using the mean squared error from exercise 2.2.

∎

C.12 Asymptotic Normality of the MLE

Solution to exercise 17.1.

1.

Since we have $\mathcal{I}(\theta)=1/\bigl{(}\theta(1-\theta)\bigr{)}$ , theorem 17.1 gives that $\sqrt{n}\,(\bar{X}-\theta)\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,\theta(1-% \theta)\bigr{)}$ . Since the mean and variance of the Bernoulli distribution equal $\theta$ and $\theta(1-\theta)$ , respectively, this is the central limit theorem, theorem A.15, for $\bar{X}$ and thus the two results coincide.
2.

From theorem 5.6 we know that the MLE for $\psi$ is given by $\hat{\psi}_{n}=\log\bigl{(}\bar{X}/(1-\bar{X})\bigr{)}$ . Taking derivatives we find the derivative of $g(\theta)=\log\theta-\log(1-\theta)$ to be $g^{\prime}(\theta)=1/\theta+1/(1-\theta)=1/\bigl{(}\theta(1-\theta)\bigr{)}$ and using the delta method, theorem A.18, we find

$\sqrt{n}\,(\hat{\psi}_{n}-\psi)\xrightarrow{\ \mathrm{d}\ }N\Bigl{(}0,\ \frac{% \theta(1-\theta)}{\theta^{2}(1-\theta)^{2}}\Bigr{)}=N\Bigl{(}0,\frac{1}{\theta% (1-\theta)}\Bigr{)}.$
3.

Substituting $\theta$ by $\hat{\theta}_{n}$ in the asymptotic variance, we find the standard error $\mathop{\mathrm{se}}\nolimits(\hat{\psi}_{n})=1/\sqrt{n\hat{\theta}_{n}(1-\hat% {\theta}_{n})}$ and thus the approximate $95\%$ interval $\hat{\psi}_{n}\pm 1.96/\sqrt{n\hat{\theta}_{n}(1-\hat{\theta}_{n})}$ . Both of these quantities are undefined if all observations are equal, i.e. at the boundary of the domain discussed in section 17.5.

∎

Solution to exercise 17.2.

1.

Using the result $\mathcal{I}(\theta)=1/\theta$ from the theorem, we find $\sqrt{n}\,(\bar{X}-\theta)\xrightarrow{\ \mathrm{d}\ }N(0,\theta)$ . Since the Poisson distribution has mean and variance equal to $\theta$ , this is the central limit theorem for $\bar{X}$ .
2.

For $g(\theta)=e^{-\theta}$ we have $g^{\prime}(\theta)=-e^{-\theta}$ and using the delta method we find

$\sqrt{n}\,(\hat{p}_{0}-p_{0})\xrightarrow{\ \mathrm{d}\ }N\bigl{(}0,\ e^{-2% \theta}\theta\bigr{)}.$

In exercise 5.7 we found $\mathbb{E}_{\theta}(\hat{p}_{0})=\exp\bigl{(}n\theta(e^{-1/n}-1)\bigr{)}$ and expanding $e^{-1/n}=1-1/n+1/(2n^{2})-\dotsb$ we find that the bias is approximately $\theta e^{-\theta}/(2n)$ , which is of order $1/n$ . Since it is multiplied by $\sqrt{n}$ , the bias goes to zero and thus is of a smaller order than the spread $1/\sqrt{n}$ and does not contribute in the limit: the estimator is biased for finite $n$ , but is asymptotically normal with mean $p_{0}$ .

∎

Solution to exercise 17.3.

1.

For $x\geq 0$ and $n>x/\theta$ , the event $n(\theta-\hat{\theta}_{n})>x$ is the event $\hat{\theta}_{n}<\theta-x/n$ , i.e. that all observations are less than $\theta-x/n$ . Thus we have

$\mathbb{P}_{\theta}\bigl{(}n(\theta-\hat{\theta}_{n})>x\bigr{)}=\Bigl{(}1-% \frac{x}{n\theta}\Bigr{)}^{n}\longrightarrow e^{-x/\theta}\qquad(n\to\infty),$

where we used the standard limit $(1-a/n)^{n}\to e^{-a}$ . The right-hand side is the probability that an exponential random variable with mean $\theta$ is greater than $x$ . Thus, $n(\theta-\hat{\theta}_{n})$ converges in distribution to the corresponding exponential distribution.
2.

Since $\sqrt{n}\,(\hat{\theta}_{n}-\theta)=-n^{-1/2}\cdot n(\theta-\hat{\theta}_{n})$ , the second factor converges in distribution to the result from the first part, and the first factor is a constant which converges to zero. Thus, by Slutsky’s theorem (theorem A.16), the product converges in distribution to $0\cdot Y=0$ . This is equivalent to convergence in probability to zero. Thus we cannot have a limit $N(0,\tau^{2})$ where $\tau^{2}>0$ , since the error of the MLE is of order $1/n$ and not of order $1/\sqrt{n}$ . This is a manifestation of the super-efficiency from lecture 13, seen through the lens of the limiting distribution: the variance of the bias-corrected maximum is of order $1/n^{2}$ , smaller than allowed by the Cramer–Rao bound, and thus neither the bound nor theorem 17.1 apply, since the model violates (R2).

∎

C.13 The EM Algorithm

Solution to exercise 19.1.

1.

Using the formula for the sum of a geometric series, we find

$\displaystyle\mathbb{P}_{p}(X\geq m)$

$\displaystyle=\sum_{x=m}^{\infty}(1-p)^{x-1}p=p\,(1-p)^{m-1}\sum_{j=0}^{\infty% }(1-p)^{j}$

$\displaystyle=p\,(1-p)^{m-1}\cdot\frac{1}{p}=(1-p)^{m-1}.$

Since the event $\{X\geq m+k\}$ is contained in the event $\{X\geq m\}$ , we can use the definition of conditional probability to get

$\mathbb{P}_{p}(X\geq m+k\mskip 1.0mu|\mskip 1.0muX\geq m)=\frac{(1-p)^{m+k-1}}% {(1-p)^{m-1}}=(1-p)^{k}=\mathbb{P}_{p}(X\geq k+1).$

This shows that, given that the first $m-1$ trials were failures, the number $X-(m-1)$ of additional trials required until the first success is still geometrically distributed. The information about the number of failures so far does not change the distribution of the number of additional failures required. This is the discrete analogue of the statement $\mathbb{P}_{\theta}(X>s+t\mskip 1.0mu|\mskip 1.0muX>s)=\mathbb{P}_{\theta}(X>t)$ for the exponential distribution in exercise 5.6. The shift by one is due to the fact that the support of the geometric distribution starts at $1$ instead of at $0$ .
2.

Given $X\geq m$ , the conditional probability weights of $X$ are

$\frac{f(x;p)}{\mathbb{P}_{p}(X\geq m)}=(1-p)^{x-m}p$

for $x\geq m$ . Substituting $x=m-1+j$ with $j\in\{1,2,\dots\}$ we get

$\displaystyle\mathbb{E}_{p}(X\mskip 1.0mu|\mskip 1.0muX\geq m)$

$\displaystyle=\sum_{j=1}^{\infty}(m-1+j)(1-p)^{j-1}p$

$\displaystyle=(m-1)\sum_{j=1}^{\infty}(1-p)^{j-1}p+\sum_{j=1}^{\infty}j(1-p)^{% j-1}p$

$\displaystyle=(m-1)+\mathbb{E}_{p}(X),$

where the first sum is the total probability $1$ and the second sum is the mean of the geometric distribution. Using $\mathbb{E}_{p}(X)=1/p$ we get the claim. Alternatively we can get the same result from the first part of the solution: given $X\geq m$ , the value $X-(m-1)$ is geometrically distributed with parameter $p$ and thus has mean $1/p$ .

∎

Solution to exercise 19.2.

1.

The log-likelihood function for the given data is

$\ell(\pi)=\sum_{i=1}^{n}\log\bigl{(}\pi f_{1}(x_{i})+(1-\pi)f_{2}(x_{i})\bigr{% )}.$

Taking derivatives we get the likelihood equation

$\ell^{\prime}(\pi)=\sum_{i=1}^{n}\frac{f_{1}(x_{i})-f_{2}(x_{i})}{\pi f_{1}(x_% {i})+(1-\pi)f_{2}(x_{i})}=0.$

This equation cannot be solved analytically for general densities.
2.

The density of the complete data is $f_{c}(x,1;\pi)=\pi f_{1}(x)$ and $f_{c}(x,2;\pi)=(1-\pi)f_{2}(x)$ . Thus we have

$\ell_{c}(\pi)=\sum_{i=1}^{n}\Bigl{[}\mathbf{1}_{\{Z_{i}=1\}}\bigl{(}\log\pi+% \log f_{1}(x_{i})\bigr{)}+\mathbf{1}_{\{Z_{i}=2\}}\bigl{(}\log(1-\pi)+\log f_{% 2}(x_{i})\bigr{)}\Bigr{]}.$

Using Bayes’ rule we find the conditional probability of $Z_{i}=1$ given $X_{i}=x_{i}$ under $\pi_{k}$ as $\gamma_{i}(\pi_{k})=\pi_{k}f_{1}(x_{i})/\bigl{(}\pi_{k}f_{1}(x_{i})+(1-\pi_{k}% )f_{2}(x_{i})\bigr{)}$ . Substituting the indicators using these responsibilities we get

$Q(\pi\mskip 1.0mu|\mskip 1.0mu\pi_{k})=\log\pi\sum_{i=1}^{n}\gamma_{i}(\pi_{k}% )+\log(1-\pi)\sum_{i=1}^{n}\bigl{(}1-\gamma_{i}(\pi_{k})\bigr{)}+\text{const},$

where the constant term combines the terms $\log f_{1}(x_{i})$ and $\log f_{2}(x_{i})$ , which do not depend on $\pi$ . Setting the derivative $\sum_{i}\gamma_{i}/\pi-\sum_{i}(1-\gamma_{i})/(1-\pi)$ equal to zero we find $\pi_{k+1}=\frac{1}{n}\sum_{i}\gamma_{i}(\pi_{k})$ and the second derivative $-\sum_{i}\gamma_{i}/\pi^{2}-\sum_{i}(1-\gamma_{i})/(1-\pi)^{2}$ is negative for every $\pi\in(0,1)$ , so $Q(\,\mathord{\cdot}\,\mskip 1.0mu|\mskip 1.0mu\pi_{k})$ is concave on $(0,1)$ and the stationary point is the global maximum.
3.

Using the notation $f_{i}=f(x_{i};\pi)=\pi f_{1}(x_{i})+(1-\pi)f_{2}(x_{i})$ for the denominator, we have

$\displaystyle\gamma_{i}(\pi)-\pi$

$\displaystyle=\frac{\pi f_{1}(x_{i})-\pi\bigl{(}\pi f_{1}(x_{i})+(1-\pi)f_{2}(% x_{i})\bigr{)}}{f_{i}}$

$\displaystyle=\frac{\pi(1-\pi)\bigl{(}f_{1}(x_{i})-f_{2}(x_{i})\bigr{)}}{f_{i}}.$

Summing over $i$ and comparing to the first part of the solution we find $\frac{1}{n}\sum_{i}\gamma_{i}(\pi)-\pi=\pi(1-\pi)\,\ell^{\prime}(\pi)/n$ . A value $\pi^{*}\in(0,1)$ is a fixed point of the iteration if and only if the left-hand side is zero at $\pi^{*}$ and since $\pi^{*}(1-\pi^{*})>0$ this is the case if and only if $\ell^{\prime}(\pi^{*})=0$ . Thus the fixed points of EM are the roots of the likelihood equation as given in proposition 19.3.

∎

Solution to exercise 19.3.

At $\theta_{0}=(1/2,2,4)$ , the mixture density of observation $i$ is $\frac{1}{2}\varphi_{1}(x_{i}-2)+\frac{1}{2}\varphi_{1}(x_{i}-4)$ . Using the standard normal density values $\varphi_{1}(1)=0.2420$ , $\varphi_{1}(2)=0.05399$ , $\varphi_{1}(3)=0.004432$ , $\varphi_{1}(4)=0.0001338$ and $\varphi_{1}(5)=0.0000015$ , we find the five densities to be $0.027062$ , $0.12320$ , $0.12320$ , $0.027062$ and $0.0022167$ . The corresponding logarithms are

-3.6096,\quad-2.0939,\quad-2.0939,\quad-3.6096,\quad-6.1118

and thus $\ell(\theta_{0})=-17.5188$ . At $\theta_{1}=(0.4001,0.5445,5.9709)$ the densities are $0.13763$ , $0.14389$ , $0.14939$ , $0.23922$ and $0.14094$ . The corresponding logarithms are

-1.9832,\quad-1.9387,\quad-1.9012,\quad-1.4304,\quad-1.9595

and thus $\ell(\theta_{1})=-9.2129$ . The log-likelihood has increased by $8.31$ as required by theorem 19.2. The largest contribution, $4.15$ , comes from observation $x_{5}=7$ : under $\theta_{0}$ this observation is three standard deviations away from the nearer mean $4$ and has tiny density, whereas under $\theta_{1}$ it is about one standard deviation away from the mean $5.97$ . The observations $x_{1}=0$ and $x_{4}=6$ contribute $1.63$ and $2.18$ , respectively, for the same reason. The observations $x_{2}=1$ and $x_{3}=5$ , which were always within one standard deviation of a mean, contribute little. ∎

Solution to exercise 19.4.

1.

Since we know $\pi$ , the complete-data log-likelihood is the same as in example 19.4 and the terms $\log\pi$ and $\log(1-\pi)$ are now constants. The E step determines the responsibilities $\gamma_{i}=\pi\varphi_{\sigma}(x_{i}-\mu_{1,k})/\bigl{(}\pi\varphi_{\sigma}(x_% {i}-\mu_{1,k})+(1-\pi)\varphi_{\sigma}(x_{i}-\mu_{2,k})\bigr{)}$ . Note that $Q(\theta\mskip 1.0mu|\mskip 1.0mu\theta_{k})$ depends on $\theta=(\mu_{1},\mu_{2})$ only via the terms $-\frac{1}{2\sigma^{2}}\sum_{i}\bigl{[}\gamma_{i}(x_{i}-\mu_{1})^{2}+(1-\gamma_% {i})(x_{i}-\mu_{2})^{2}\bigr{]}$ . The partial derivatives are as shown in the example, and setting them equal to zero gives the two mean updates of (19.2). The update for the weight is no longer needed.
2.

Since $\varphi_{\sigma}^{\prime}(u)=-(u/\sigma^{2})\varphi_{\sigma}(u)$ , we have $\frac{\partial}{\partial\mu_{1}}\varphi_{\sigma}(x-\mu_{1})=\frac{x-\mu_{1}}{% \sigma^{2}}\varphi_{\sigma}(x-\mu_{1})$ and thus we can take derivatives of $\ell(\theta)=\sum_{i}\log\bigl{(}\pi\varphi_{\sigma}(x_{i}-\mu_{1})+(1-\pi)% \varphi_{\sigma}(x_{i}-\mu_{2})\bigr{)}$ to get

$\displaystyle\frac{\partial\ell}{\partial\mu_{1}}(\theta)$

$\displaystyle=\sum_{i=1}^{n}\frac{\pi\varphi_{\sigma}(x_{i}-\mu_{1})}{\pi% \varphi_{\sigma}(x_{i}-\mu_{1})+(1-\pi)\varphi_{\sigma}(x_{i}-\mu_{2})}\cdot% \frac{x_{i}-\mu_{1}}{\sigma^{2}}$

$\displaystyle=\frac{1}{\sigma^{2}}\sum_{i=1}^{n}\gamma_{i}(\theta)\,(x_{i}-\mu% _{1}),$

and similarly we find $\frac{\partial\ell}{\partial\mu_{2}}(\theta)=\frac{1}{\sigma^{2}}\sum_{i}\bigl% {(}1-\gamma_{i}(\theta)\bigr{)}(x_{i}-\mu_{2})$ . Setting both of these equal to zero gives $\mu_{1}=\sum_{i}\gamma_{i}(\theta)x_{i}/\sum_{i}\gamma_{i}(\theta)$ and the corresponding equation for $\mu_{2}$ , stating that $\theta$ is mapped to itself by the updates of the first part. Thus, the likelihood equations and the fixed-point equations of the EM algorithm coincide as predicted by proposition 19.3, and we also find the identity of this proposition directly: the derivative of $\ell$ equals the derivative of $\theta^{\prime}\mapsto Q(\theta^{\prime}\mskip 1.0mu|\mskip 1.0mu\theta)$ at $\theta^{\prime}=\theta$ .
3.

As $\pi\to 1$ , the responsibilities converge to $\gamma_{i}=1$ for all observations since $\varphi_{\sigma}(x_{i}-\mu_{1,k})>0$ . Thus, the update for $\mu_{1}$ is now $\mu_{1,k+1}=\bar{x}$ independent of the initial value, and this is the MLE for the mean of a single normal sample in example 5.4, reached in one step. The update for $\mu_{2}$ is now the undefined ratio $0/0$ : a component with no weight has no observations to use to estimate its mean, and the mean of this component is not identifiable as discussed in section 19.5.

∎

C.14 Hypothesis Testing

Solution to exercise 20.1.

1.

By equation (20.2) the power at $\theta_{1}$ is $\beta(\theta_{1})=\Phi(d-z_{1-\alpha})$ , where $d=\sqrt{n}\,(\theta_{1}-\theta_{0})/\sigma$ . Since $\Phi$ is strictly increasing with $\Phi(z_{1-\gamma})=1-\gamma$ , we have $\beta(\theta_{1})\geq 1-\gamma$ if and only if $d-z_{1-\alpha}\geq z_{1-\gamma}$ , i.e. if and only if

$\sqrt{n}\geq\frac{(z_{1-\alpha}+z_{1-\gamma})\,\sigma}{\theta_{1}-\theta_{0}}.$

Since $1-\gamma>\alpha$ we have $z_{1-\gamma}>z_{\alpha}=-z_{1-\alpha}$ , and thus the right-hand side is positive; squaring both sides gives the stated condition.
2.

With $z_{0.95}=1.645$ and $z_{0.9}=1.282$ the bound is

$n\geq\Bigl{(}\frac{(1.645+1.282)\cdot 2}{0.5}\Bigr{)}^{2}=11.708^{2}=137.1,$

and thus $n=138$ observations are needed.
3.

The bound is proportional to $\sigma^{2}/(\theta_{1}-\theta_{0})^{2}$ . Halving the difference $\theta_{1}-\theta_{0}$ multiplies the required sample size by four, to $n=549$ in the numerical example (the bound becomes $548.3$ ), and doubling $\sigma$ has the same effect. Detecting a difference half as large costs four times as many observations.

∎

Solution to exercise 20.2.

1.

Under $H_{0}$ the total $T$ is binomially distributed with parameters $10$ and $1/2$ . Thus, the test size is

$\beta(1/2)=\mathbb{P}_{1/2}(T\geq c)=\frac{1}{2^{10}}\sum_{k=c}^{10}\binom{10}% {k}.$
2.

We can compute the binomial coefficients $\binom{10}{10}=1$ , $\binom{10}{9}=10$ and $\binom{10}{8}=45$ to find the sizes $1/1024=0.0010$ for $c=10$ , $11/1024=0.0107$ for $c=9$ and $56/1024=0.0547$ for $c=8$ . The smallest value $c$ with size less than or equal to $0.05$ is $c=9$ with size $0.0107$ . Since $T$ can only take the values $0,\dots,10$ , the size can only take the eleven values computed in the first part of the question, and $0.05$ is not one of these values (the size jumps from $0.0107$ to $0.0547$ between $c=9$ and $c=8$ ).
3.

The power at $p$ is $\mathbb{P}_{p}(T\geq 9)=10p^{9}(1-p)+p^{10}=p^{9}(10-9p)$ . This gives the values $0.149$ for $p=0.7$ , $0.376$ for $p=0.8$ and $0.736$ for $p=0.9$ . A coin which shows heads four times in five experiments is detected in fewer than $40\%$ of the experiments; for ten experiments the test can only detect large deviations from fairness with reasonable probability, and a failure to reject the hypothesis is not very meaningful.
4.

The $p$ -value is the probability of the event that the total under $H_{0}$ is larger than or equal to the observed value, i.e. the probability of $\mathbb{P}_{1/2}(T\geq 9)=11/1024=0.0107$ . Since this probability is less than or equal to $0.05$ , we can reject $H_{0}$ at a significance level of $0.05$ . This agrees with the critical region $T\geq 9$ from the second part of the question.

∎

Solution to exercise 20.3.

1.

The maximum of the distribution is at most $m$ , if and only if all observations are at most $m$ . Since the observations are independent, for $0\leq m\leq\theta$ we have

$\mathbb{P}_{\theta}(M\leq m)=\prod_{i=1}^{n}\mathbb{P}_{\theta}(X_{i}\leq m)=% \Bigl{(}\frac{m}{\theta}\Bigr{)}^{n},$

This is also the integral of the density from example 8.10.
2.

For $c\leq\theta_{0}$ the size is $\mathbb{P}_{\theta_{0}}(M>c)=1-(c/\theta_{0})^{n}$ , as found in the first part. Setting this equal to $\alpha$ gives $c=\theta_{0}(1-\alpha)^{1/n}$ .
3.

For $\theta>\theta_{0}\geq c$ the power function is

$\beta(\theta)=\mathbb{P}_{\theta}(M>c)=1-\Bigl{(}\frac{c}{\theta}\Bigr{)}^{n}=% 1-(1-\alpha)\Bigl{(}\frac{\theta_{0}}{\theta}\Bigr{)}^{n},$

This function increases from $\alpha$ for $\theta=\theta_{0}$ to $1$ as $\theta\to\infty$ . For $n=5$ , $\theta_{0}=1$ and $\alpha=0.05$ we have $c=0.95^{1/5}=0.990$ , and for $\theta=1.2$ the power is $1-0.95/1.2^{5}=1-0.95\cdot 0.402=0.618$ .
4.

Under $H_{0}$ all observations are in the interval $[0,\theta_{0}]$ and thus we have $\mathbb{P}_{\theta_{0}}(M>\theta_{0})=0$ : The second test has size $0$ and is a level $\alpha$ test for every $\alpha$ . The power function of this test is $\mathbb{P}_{\theta}(M>\theta_{0})=1-(\theta_{0}/\theta)^{n}$ for $\theta>\theta_{0}$ . This function equals $0.598$ for $\theta=1.2$ in the numerical example. The difference between the two power functions is $\alpha(\theta_{0}/\theta)^{n}$ , and this is positive for every alternative, i.e. the first test is more powerful everywhere on $H_{1}$ , but the first test has type I error probability $\alpha$ which the second test avoids. In the context of section 20.3, the first test is to be preferred: we have agreed on a level $\alpha$ , and a test which does not use the level would be a waste of power. The power gain is at most $\alpha$ , and decreases quickly as $\theta$ increases, whereas for the second test a rejection is certain if the observation is above $\theta_{0}$ , which is impossible under $H_{0}$ , and so we can reasonably assume that this is worth the small loss of power. The example shows that the choice of level is a matter for the statistician, and is not a mathematical necessity.

∎

Solution to exercise 20.4.

1.

The observed value of the standardised statistic is $z=\sqrt{16}\,(52.3-50)/4=2.3$ and the one-sided $p$ -value is $p=1-\Phi(2.3)=0.0107$ . Since $0.0107\leq 0.05$ we can reject $H_{0}$ at level $0.05$ , but since $0.0107>0.01$ we cannot reject at level $0.01$ .
2.

The two-sided $p$ -value is $2\bigl{(}1-\Phi(2.3)\bigr{)}=0.0214$ . Again we reject at level $0.05$ and we cannot reject at level $0.01$ .
3.

The mean $\theta$ is a fixed, unknown number and not a random variable, so we cannot determine a probability for the statement $\theta=50$ . The $p$ -value is a probability about the data, assuming that $\theta=50$ . A correct interpretation is: if the true mean were $50$ , then a sample mean of at least $52.3$ would occur with probability $0.011$ , and thus the data are significant at level $0.05$ but not at level $0.01$ .
4.

Under $H_{0}$ the statistic $Z$ is standard normally distributed. Since $\Phi$ is continuous and strictly increasing, for $u\in(0,1)$ we have

$\displaystyle\mathbb{P}_{\theta_{0}}\bigl{(}1-\Phi(Z)\leq u\bigr{)}$

$\displaystyle=\mathbb{P}_{\theta_{0}}\bigl{(}\Phi(Z)\geq 1-u\bigr{)}$

$\displaystyle=\mathbb{P}_{\theta_{0}}(Z\geq z_{1-u})=1-\Phi(z_{1-u})=u,$

and thus the $p$ -value is uniformly distributed on $(0,1)$ under $H_{0}$ . In particular we have $\mathbb{P}_{\theta_{0}}\bigl{(}p(X)\leq\alpha\bigr{)}=\alpha$ , which is the statement that the test has size $\alpha$ .

∎

C.15 The Neyman–Pearson Lemma

Solution to exercise 22.1.

1.

The joint probability weights are $f(x;\theta)=\theta^{s}e^{-n\theta}/\prod_{i}x_{i}!$ with $s=\sum_{i}x_{i}$ . Thus the likelihood ratio is

$R(x)=\Bigl{(}\frac{\theta_{1}}{\theta_{0}}\Bigr{)}^{s}e^{-n(\theta_{1}-\theta_% {0})}.$

Since $\theta_{1}/\theta_{0}>1$ , this function is strictly increasing as a function of $s$ and the inequality $R(x)>k$ is equivalent to $s>c$ with $c=\bigl{(}\log k+n(\theta_{1}-\theta_{0})\bigr{)}/\log(\theta_{1}/\theta_{0})$ . By theorem 22.2 the most powerful test at this level is the one which rejects when $S>c$ . The size of this test, $\mathbb{P}_{\theta_{0}}(S>c)$ , can be found using the Poisson distribution with mean $n\theta_{0}$ , which contains the parameters $n$ and $\theta_{0}$ only. For a given size the threshold $c$ is the same for all $\theta_{1}>\theta_{0}$ . By proposition 22.4 the test is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ .
2.

Under $H_{0}$ , the sum $S$ is Poisson-distributed with mean $5$ and thus we have $\mathbb{P}_{\theta_{0}}(S>8)=1-0.9319=0.0681>0.05$ , whereas $\mathbb{P}_{\theta_{0}}(S>9)=1-0.9682=0.0318\leq 0.05$ . The smallest acceptable threshold for the test is $c=9$ : we can reject $S\geq 10$ and the size of this test is $0.0318$ .
3.

Under $\theta_{1}=2$ , the sum is Poisson-distributed with mean $10$ and thus the power of the test is $\mathbb{P}_{\theta_{1}}(S>9)=1-0.4579=0.5421$ .
4.

The size of the test $\{S>c\}$ is $1-\mathbb{P}_{\theta_{0}}(S\leq c)$ . This value can only take the values listed for $c=0,1,2,\dots$ and the jumps between these values are from $c=8$ to $c=9$ , i.e. $0.0681$ to $0.0318$ . The value $0.05$ is not one of the possible values. The test from part (b) has size $0.0318$ and the lemma shows that this is the most powerful test at level $0.0318$ . The test is also a test of level $0.05$ , but the lemma does not show that this is the most powerful test among all tests with level $0.05$ . As the remark in section 22.2 points out, a randomised test could have size exactly $0.05$ and higher power.

∎

Solution to exercise 22.2.

1.

The joint probability weights are $f(x;p)=p^{s}(1-p)^{n-s}$ with $s=\sum_{i}x_{i}$ . Under $H_{0}$ , these weights equal $2^{-n}$ for all $x$ . Thus we have

$R(x)=\frac{p_{1}^{s}(1-p_{1})^{n-s}}{2^{-n}}=\Bigl{(}\frac{p_{1}}{1-p_{1}}% \Bigr{)}^{s}\bigl{(}2(1-p_{1})\bigr{)}^{n},$

as claimed. For $p_{1}>1/2$ , we have $p_{1}/(1-p_{1})>1$ and thus $R(x)$ is strictly increasing in $s$ . The inequality $R(x)>k$ is equivalent to $s>c$ for some $c$ . By theorem 22.2, the most powerful test is to reject when $S>c$ , where $c$ is determined by the distribution of $S$ under $H_{0}$ , i.e. the binomial distribution with parameters $n$ and $1/2$ . Since this distribution does not depend on $p_{1}$ , the same region of $p_{1}>1/2$ is optimal for all alternatives and by proposition 22.4 the test from exercise 20.2 is uniformly most powerful against $H_{1}\colon p>1/2$ .
2.

Since $\mathbb{P}_{1/2}(S\geq 14)=0.0577>0.05$ and $\mathbb{P}_{1/2}(S\geq 15)=0.0207\leq 0.05$ , the largest critical region which does not exceed size $0.05$ is $\{S\geq 15\}$ . This region has size $0.0207$ . The power of this test at $p_{1}=0.7$ is $\mathbb{P}_{0.7}(S\geq 15)=1-0.5836=0.4164$ . For $n=10$ , the corresponding region $\{S\geq 9\}$ from exercise 20.2 has smaller size $11/1024=0.0107$ but the power at $p_{1}=0.7$ is $10\cdot 0.7^{9}\cdot 0.3+0.7^{10}=0.1493$ . By doubling the sample size, we can use a test with size closer to the nominal value $0.05$ and nearly triple the power at this alternative.
3.

For $p_{1}<1/2$ , we have $p_{1}/(1-p_{1})<1$ and thus $R(x)$ is strictly decreasing in $s$ . The inequality $R(x)>k$ is now equivalent to $s<c$ and the most powerful test will be a test which rejects for small values of $S$ . For $n=20$ , with the same size $0.0207$ , the test will reject for $S\leq 5$ , by the symmetry of the binomial distribution with parameter $1/2$ . Thus, the two one-sided problems will result in tests which reject on the opposite tails of the distribution of $S$ and there can be no single critical region which satisfies both tests.

∎

Solution to exercise 22.3.

1.

For $x_{1},\dots,x_{n}\geq 0$ we have

$f(x;\theta_{0})=\theta_{0}^{-n}\mathbf{1}_{\{m\leq\theta_{0}\}}\qquad\text{and% }\qquad f(x;\theta_{1})=\theta_{1}^{-n}\mathbf{1}_{\{m\leq\theta_{1}\}},$

and since $\theta_{0}<\theta_{1}$ there are three cases. If $m\leq\theta_{0}$ , both densities are positive and

$f(x;\theta_{1})=\theta_{1}^{-n}=(\theta_{0}/\theta_{1})^{n}\theta_{0}^{-n}=k\,% f(x;\theta_{0}).$

If $\theta_{0}<m\leq\theta_{1}$ , we have $f(x;\theta_{0})=0<f(x;\theta_{1})$ , and thus $f(x;\theta_{1})>k\,f(x;\theta_{0})$ . If $m>\theta_{1}$ , both densities vanish and equality holds. Consequently

$\displaystyle\{x\mid f(x;\theta_{1})>k\,f(x;\theta_{0})\}$

$\displaystyle=\{\theta_{0}<m\leq\theta_{1}\},$

$\displaystyle\{x\mid f(x;\theta_{1})=k\,f(x;\theta_{0})\}$

$\displaystyle=\{m\leq\theta_{0}\}\cup\{m>\theta_{1}\}.$
2.

Since $0<c<\theta_{0}$ , the size of $C$ is $\mathbb{P}_{\theta_{0}}(M>c)=1-(c/\theta_{0})^{n}=1-(1-\alpha)=\alpha$ . The region $C=\{m>c\}$ contains the set $\{\theta_{0}<m\leq\theta_{1}\}$ of part (a), because $c<\theta_{0}$ , and it is contained in the union of the two sets of part (a), which is the whole sample space. Thus $C$ satisfies the condition of theorem 22.2 with $k=(\theta_{0}/\theta_{1})^{n}$ and by the first statement of the theorem it is most powerful at level $\alpha$ . The power of this test is

$\mathbb{P}_{\theta_{1}}(M>c)=1-\Bigl{(}\frac{c}{\theta_{1}}\Bigr{)}^{n}=1-(1-% \alpha)\Bigl{(}\frac{\theta_{0}}{\theta_{1}}\Bigr{)}^{n}.$

The threshold $c=\theta_{0}(1-\alpha)^{1/n}$ does not depend on $\theta_{1}$ and the argument from above applies to all $\theta_{1}>\theta_{0}$ with the corresponding $k$ and thus, by proposition 22.4, the test is uniformly most powerful against $H_{1}\colon\theta>\theta_{0}$ .
3.

The two parts of $C^{\prime\prime}$ are disjoint and thus the size of this region is $\mathbb{P}_{\theta_{0}}(M>\theta_{0})+\mathbb{P}_{\theta_{0}}(M\leq\theta_{0}% \alpha^{1/n})=0+\alpha=\alpha$ . The power of this test is

$\mathbb{P}_{\theta_{1}}(M>\theta_{0})+\mathbb{P}_{\theta_{1}}(M\leq\theta_{0}% \alpha^{1/n})=1-\Bigl{(}\frac{\theta_{0}}{\theta_{1}}\Bigr{)}^{n}+\alpha\Bigl{% (}\frac{\theta_{0}}{\theta_{1}}\Bigr{)}^{n}=1-(1-\alpha)\Bigl{(}\frac{\theta_{% 0}}{\theta_{1}}\Bigr{)}^{n},$

This is the same power as for $C$ . Thus both regions are most powerful at level $\alpha$ and, by the second statement of theorem 22.2, they can only differ on the boundary set. And indeed they do: the two regions differ on the sets $C\setminus C^{\prime\prime}=\{c<m\leq\theta_{0}\}$ and $C^{\prime\prime}\setminus C=\{m\leq\theta_{0}\alpha^{1/n}\}$ , both of which are contained in $\{m\leq\theta_{0}\}$ where $f(x;\theta_{1})=k\,f(x;\theta_{0})$ . The argument in the uniqueness comment in section 22.2 used the fact that the likelihood ratio has a continuous distribution, but in this case this is not true: on the set $\{M\leq\theta_{0}\}$ , where an event with probability $1-\alpha$ occurs under $H_{0}$ , the ratio $R(X)$ is constant and thus the boundary set has positive probability, despite the model having densities.

∎

Solution to exercise 22.4.

Since $T$ is sufficient, we can use the factorisation theorem 8.2 to find functions $g$ and $h$ such that

f(x;\theta)=g\bigl{(}T(x);\theta\bigr{)}\,h(x)

for all $x$ and all parameter values $\theta$ . Now let us choose an $x$ with $f(x;\theta_{0})>0$ . Then neither of the two factors in the expression for $f(x;\theta_{0})=g\bigl{(}T(x);\theta_{0}\bigr{)}h(x)$ can be zero and we get

R(x)=\frac{f(x;\theta_{1})}{f(x;\theta_{0})}=\frac{g\bigl{(}T(x);\theta_{1}% \bigr{)}\,h(x)}{g\bigl{(}T(x);\theta_{0}\bigr{)}\,h(x)}=\frac{g\bigl{(}T(x);% \theta_{1}\bigr{)}}{g\bigl{(}T(x);\theta_{0}\bigr{)}}.

Since we can cancel the factor $h(x)$ on both sides, the right-hand side only depends on $x$ via $T(x)$ . This completes the proof. ∎

C.16 Likelihood-Ratio Tests

Here we solve the exercises at the end of lecture 23.

Solution to exercise 23.1.

1.

By example 5.4 the log-likelihood

$\ell(\mu,\sigma^{2})=-\frac{n}{2}\log(2\pi\sigma^{2})-\frac{1}{2\sigma^{2}}% \sum_{i=1}^{n}(X_{i}-\mu)^{2}$

is maximised over $\Theta$ at $\hat{\mu}=\bar{X}$ and $\hat{\sigma}^{2}=Q/n$ . Substituting these values, the sum of squares becomes $Q$ and the second term becomes $Q/(2Q/n)=n/2$ , and thus the maximised log-likelihood is

$\ell(\hat{\mu},\hat{\sigma}^{2})=-\frac{n}{2}\log\Bigl{(}\frac{2\pi Q}{n}\Bigr% {)}-\frac{n}{2}.$
2.

Under $H_{0}$ the mean is fixed at $\mu_{0}$ and only $v=\sigma^{2}$ is free. Writing $Q_{0}=\sum_{i}(X_{i}-\mu_{0})^{2}$ , the function to maximise is $-\frac{n}{2}\log(2\pi v)-Q_{0}/(2v)$ , which is the function of example 5.4 with $Q_{0}$ in place of $Q$ , and thus by the same derivative computation it is maximised at $\hat{\sigma}_{0}^{2}=Q_{0}/n$ , with maximum $-\frac{n}{2}\log(2\pi Q_{0}/n)-\frac{n}{2}$ . The identity (2.1) with $\mu_{0}$ in place of $\mu$ gives $Q_{0}=Q+n(\bar{X}-\mu_{0})^{2}$ , and thus

$\hat{\sigma}_{0}^{2}=\frac{Q+n(\bar{X}-\mu_{0})^{2}}{n}.$
3.

By (23.1) and the two previous parts, the terms $-n/2$ cancel and we get

$W=2\Bigl{(}-\frac{n}{2}\log\frac{2\pi Q}{n}+\frac{n}{2}\log\frac{2\pi Q_{0}}{n% }\Bigr{)}=n\log\frac{Q_{0}}{Q}=n\log\Bigl{(}1+\frac{n(\bar{X}-\mu_{0})^{2}}{Q}% \Bigr{)}.$

Since $Q=(n-1)S^{2}$ , we have $n(\bar{X}-\mu_{0})^{2}/Q=n(\bar{X}-\mu_{0})^{2}/\bigl{(}(n-1)S^{2}\bigr{)}=t^{% 2}/(n-1)$ , which gives the stated formula. The function $t^{2}\mapsto n\log\bigl{(}1+t^{2}/(n-1)\bigr{)}$ is strictly increasing, and thus the event $W\geq c$ is the event $|t|\geq c^{\prime}$ for a suitable $c^{\prime}$ : the generalised likelihood-ratio test rejects for large values of $|t|$ .
4.

The full parameter $\theta=(\mu,\sigma^{2})$ has $k=2$ coordinates, and $H_{0}$ fixes one of them, $\mu=\mu_{0}$ , leaving $\sigma^{2}$ free, so that $r=1$ . The normal model satisfies the regularity conditions, and thus theorem 23.3 gives $W\xrightarrow{\ \mathrm{d}\ }\chi^{2}_{1}$ under $H_{0}$ . The approximate test rejects when $W\geq 3.841$ , i.e. when $t^{2}\geq(n-1)\bigl{(}e^{3.841/n}-1\bigr{)}$ . For $n=10$ this is $t^{2}\geq 9\,(e^{0.3841}-1)=9\cdot 0.4683=4.215$ , or $|t|\geq 2.053$ , whereas the exact test rejects when $|t|\geq 2.262$ . The approximate test rejects more often than the exact one, and its true size is $\mathbb{P}(|t_{9}|\geq 2.053)=0.070$ instead of the nominal $0.05$ : the chi-squared approximation is optimistic for small $n$ , because the $t_{9}$ distribution has heavier tails than the normal limit which the approximation assumes. For large $n$ the two tests merge, since $n\log\bigl{(}1+t^{2}/(n-1)\bigr{)}\to t^{2}$ by $\log(1+x)\approx x$ and the $t_{n-1}$ distribution tends to the standard normal one; for $n=100$ the two thresholds are $1.969$ and $1.984$ . Whenever the exact distribution is available, as here, it is the one to use, and lecture 29 does so.

∎

Solution to exercise 23.2.

1.

Taking logarithms of the probability weights gives

$\ell(p)=\log\frac{n!}{n_{1}!\cdots n_{m}!}+\sum_{j=1}^{m}n_{j}\log p_{j},$

where the first term does not depend on $p$ . To maximise $\sum_{j}n_{j}\log p_{j}$ subject to $\sum_{j}p_{j}=1$ we consider the Lagrangian $\sum_{j}n_{j}\log p_{j}-\lambda\bigl{(}\sum_{j}p_{j}-1\bigr{)}$ . Setting its partial derivative with respect to $p_{j}$ to zero gives $n_{j}/p_{j}=\lambda$ , i.e. $p_{j}=n_{j}/\lambda$ for every $j$ , and summing over $j$ and using the constraint we find $1=n/\lambda$ , so that $\lambda=n$ and $\hat{p}_{j}=n_{j}/n$ . This stationary point is the maximum, because $\sum_{j}n_{j}\log p_{j}$ is a concave function of $p$ and the constraint set is convex. If some $n_{j}$ is zero, the corresponding $\hat{p}_{j}=0$ lies on the boundary of the parameter space; the formula still gives the maximum, since any probability assigned to an unobserved outcome only lowers the likelihood.
2.

Since $H_{0}$ is simple, the restricted maximiser is $p^{0}$ , and by (23.1) we get

$W=2\bigl{(}\ell(\hat{p})-\ell(p^{0})\bigr{)}=2\Bigl{(}\sum_{j=1}^{m}n_{j}\log% \frac{n_{j}}{n}-\sum_{j=1}^{m}n_{j}\log p_{j}^{0}\Bigr{)}=2\sum_{j=1}^{m}n_{j}% \log\frac{n_{j}}{n\,p_{j}^{0}},$

where the combinatorial constant cancels. A term with $n_{j}=0$ contributes nothing to either sum, which is the convention $0\log 0=0$ .
3.

The parameter space $\Theta=\{p\mid p_{j}>0,\ \sum_{j}p_{j}=1\}$ is described by the $m-1$ free coordinates $p_{1},\dots,p_{m-1}$ , with $p_{m}$ determined by the constraint, and thus has dimension $k=m-1$ . The null hypothesis fixes all of them, so that $r=k=m-1$ . The counts are the sums of $n$ i.i.d. indicator vectors, one per repetition, whose distribution is a full-rank exponential family with support not depending on $p$ , so that the regularity conditions hold at every interior point $p^{0}$ . Theorem 23.3 therefore gives $W\xrightarrow{\ \mathrm{d}\ }\chi^{2}_{m-1}$ under $H_{0}$ .
4.

Under $H_{0}\colon p_{j}^{0}=1/6$ the expected counts are $np_{j}^{0}=10$ for every face. The terms $n_{j}\log(n_{j}/10)$ are

$\displaystyle 13\log 1.3=3.411,\quad 8\log 0.8=-1.785,\quad 11\log 1.1=1.048,$

$\displaystyle 6\log 0.6=-3.065,\quad 12\log 1.2=2.188,\quad 10\log 1=0,$

with sum $1.797$ , and thus $w=2\cdot 1.797=3.59$ . Since $3.59<11.07=\chi^{2}_{5,0.95}$ , we do not reject $H_{0}$ at level $0.05$ ; the approximate $p$ -value is $\mathbb{P}(\chi^{2}_{5}\geq 3.59)=0.61$ . The observed counts are entirely consistent with a fair die.
5.

Write $x_{j}=d_{j}/(np_{j}^{0})$ , so that $n_{j}=np_{j}^{0}(1+x_{j})$ and $n_{j}/(np_{j}^{0})=1+x_{j}$ . Using $\log(1+x)=x-x^{2}/2+O(x^{3})$ we get

$\displaystyle n_{j}\log\frac{n_{j}}{np_{j}^{0}}$

$\displaystyle=np_{j}^{0}\,(1+x_{j})\Bigl{(}x_{j}-\frac{x_{j}^{2}}{2}+O(x_{j}^{% 3})\Bigr{)}$

$\displaystyle=np_{j}^{0}\Bigl{(}x_{j}+\frac{x_{j}^{2}}{2}+O(x_{j}^{3})\Bigr{)}% =d_{j}+\frac{d_{j}^{2}}{2np_{j}^{0}}+O(x_{j}^{3}),$

since $np_{j}^{0}x_{j}=d_{j}$ and $np_{j}^{0}x_{j}^{2}=d_{j}^{2}/(np_{j}^{0})$ . Summing over $j$ , the linear terms add up to $\sum_{j}d_{j}=0$ , and doubling the result gives

$W\approx\sum_{j=1}^{m}\frac{d_{j}^{2}}{np_{j}^{0}}=\sum_{j=1}^{m}\frac{(n_{j}-% np_{j}^{0})^{2}}{np_{j}^{0}},$

up to terms of third order in the relative deviations $x_{j}$ . For the die the deviations from $10$ are $3,-2,1,-4,2,0$ , and Pearson’s statistic is $(9+4+1+16+4+0)/10=3.4$ , close to the value $3.59$ of $W$ ; the two statistics agree to the extent that the relative deviations are small, which for these data they are.

∎

Solution to exercise 23.3.

1.

By example 4.2 the likelihood is $L(p)=p^{S}(1-p)^{n-S}$ , so that $\ell(p)=S\log p+(n-S)\log(1-p)$ , and by example 5.3 it is maximised over $(0,1)$ at $\hat{p}=S/n$ when $0<S<n$ . Since $H_{0}$ is simple, the restricted maximiser is $p_{0}$ , and (23.1) gives

$W=2\bigl{(}\ell(\hat{p})-\ell(p_{0})\bigr{)}=2\Bigl{(}S\log\frac{\hat{p}}{p_{0% }}+(n-S)\log\frac{1-\hat{p}}{1-p_{0}}\Bigr{)}.$

For $S=0$ or $S=n$ the supremum of $L$ over $(0,1)$ is approached at the boundary, where one of the two terms is absent, and the convention $0\log 0=0$ gives the correct value.
2.

The parameter is one-dimensional and $H_{0}$ fixes it, so that $k=r=1$ ; the Bernoulli model satisfies the regularity conditions for $p_{0}\in(0,1)$ , as noted in lecture 16. Theorem 23.3 gives $W\xrightarrow{\ \mathrm{d}\ }\chi^{2}_{1}$ under $H_{0}$ , and the approximate level $\alpha$ test rejects when $W\geq\chi^{2}_{1,1-\alpha}$ , which for $\alpha=0.05$ means $W\geq 3.841$ .
3.

We have $\hat{p}=32/50=0.64$ and $p_{0}=0.5$ , and thus

$w=2\bigl{(}32\log 1.28+18\log 0.72\bigr{)}=2\,(7.900-5.913)=3.97.$

Since $3.97\geq 3.841$ , we reject $H_{0}\colon p=1/2$ at level $0.05$ . The approximate $p$ -value is $\mathbb{P}(\chi^{2}_{1}\geq 3.97)=2\bigl{(}1-\Phi(\sqrt{3.97})\bigr{)}=2\bigl{% (}1-\Phi(1.99)\bigr{)}=0.047$ , just below $0.05$ , so that the evidence against a fair coin is real but not strong.
4.

The map $g(p)=\log\bigl{(}p/(1-p)\bigr{)}$ is a strictly increasing bijection from $(0,1)$ onto $\mathbb{R}$ with inverse $p=e^{\psi}/(1+e^{\psi})$ , and the likelihood in the new parametrisation is $L_{\psi}(\psi)=L\bigl{(}g^{-1}(\psi)\bigr{)}$ . The null hypothesis $\psi=\psi_{0}$ is the same set of distributions as $p=p_{0}$ with $p_{0}=g^{-1}(\psi_{0})$ , and thus the numerator of the ratio is $L_{\psi}(\psi_{0})=L(p_{0})$ in both cases. Since the supremum $\sup_{p\in(0,1)}L(p)$ is attained at the MLE $\hat{p}$ by the first part, and $g$ is a bijection from $(0,1)$ onto $\mathbb{R}$ , the two suprema $\sup_{\psi\in\mathbb{R}}L_{\psi}(\psi)$ and $\sup_{p\in(0,1)}L(p)$ run over the same set of values, and thus the denominator is $\sup_{\psi\in\mathbb{R}}L_{\psi}(\psi)=\sup_{p\in(0,1)}L(p)=L(\hat{p})$ . Both ratios are therefore equal, and so are the two statistics $W$ . This is the invariance noted in section 23.1; by contrast the standard errors of exercise 17.1 do depend on the parametrisation.

∎

C.17 Confidence Intervals

Solution to exercise 25.1.

1.

The function $Q=2\theta T$ is a function of the data and the parameter, and from exercise 13.1 we know that under $\mathbb{P}_{\theta}$ the distribution of $Q$ is $\chi^{2}_{2n}$ for all $\theta$ , independent of $\theta$ . Thus, $Q$ is a pivot. Using $a=\chi^{2}_{2n,\alpha/2}$ and $b=\chi^{2}_{2n,1-\alpha/2}$ we find $\mathbb{P}_{\theta}(a\leq 2\theta T\leq b)=1-\alpha$ and since the function $Q$ is increasing in $\theta$ , the two inequalities can be solved for $\theta$ directly:

$a\leq 2\theta T\leq b\quad\Longleftrightarrow\quad\frac{\chi^{2}_{2n,\alpha/2}% }{2T}\leq\theta\leq\frac{\chi^{2}_{2n,1-\alpha/2}}{2T}.$

Thus, the interval

$\bigl{[}\chi^{2}_{2n,\alpha/2}/(2T),\ \chi^{2}_{2n,1-\alpha/2}/(2T)\bigr{]}$

is a confidence interval for $\theta$ with level $1-\alpha$ and the probability of this interval covering the correct value is exactly $1-\alpha$ .
2.

We have $2T=50$ and $2n=20$ and thus the $95\%$ interval is

$[9.591/50,\ 34.170/50]=[0.192,\ 0.683].$
3.

The MLE for the rate is $\hat{\theta}=1/\bar{x}=10/25=0.4$ and the standard error of the MLE is $0.4/\sqrt{10}=0.126$ . Thus the Wald interval is $0.4\pm 1.96\cdot 0.126$ , which is the interval $[0.152,0.648]$ . The exact interval is asymmetric about the estimate, extending further to the right than to the left. This is caused by the right skew of the distribution of $\hat{\theta}$ for small exponential samples; the Wald interval, being symmetric, extends to lower values than the exact interval does. At $n=10$ the normal approximation used in the Wald interval is rough, as noted in section 17.5, and the exact interval should be used instead.

∎

Solution to exercise 25.2.

1.

From example 8.10 we know that the maximum has density $f_{M}(t)=nt^{n-1}/\theta^{n}$ for $0\leq t\leq\theta$ . Thus, for $0\leq u\leq 1$ we have

$\mathbb{P}_{\theta}\Bigl{(}\frac{M}{\theta}\leq u\Bigr{)}=\mathbb{P}_{\theta}(% M\leq u\theta)=\int_{0}^{u\theta}\frac{nt^{n-1}}{\theta^{n}}\,\mathrm{d}t=% \frac{(u\theta)^{n}}{\theta^{n}}=u^{n}.$

The distribution function of $M/\theta$ does not depend on $\theta$ and thus $M/\theta$ is a pivot.
2.

Since $M\leq\theta$ always holds, we have

$\mathbb{P}_{\theta}\Bigl{(}M\leq\theta\leq\frac{M}{\alpha^{1/n}}\Bigr{)}=% \mathbb{P}_{\theta}\Bigl{(}\alpha^{1/n}\leq\frac{M}{\theta}\leq 1\Bigr{)}=1-% \bigl{(}\alpha^{1/n}\bigr{)}^{n}=1-\alpha,$

where we used the first part of the solution. Thus, $[M,M/\alpha^{1/n}]$ is a confidence interval with level $1-\alpha$ . For $n=5$ and $\alpha=0.05$ we have $\alpha^{1/n}=0.05^{0.2}=0.549$ and thus the interval is $[3.2,\ 3.2/0.549]=[3.2,5.83]$ .
3.

The condition for coverage is $b^{n}-a^{n}=1-\alpha$ . This condition can be used to find $a=(b^{n}-1+\alpha)^{1/n}$ as a function of $b$ and thus, by choosing the interval $[M/b,M/a]$ , we can find the length $M(1/a-1/b)$ of the interval. Taking derivatives we find $b^{n-1}=a^{n-1}\,\mathrm{d}a/\mathrm{d}b$ and thus

$\frac{\mathrm{d}}{\mathrm{d}b}\Bigl{(}\frac{1}{a}-\frac{1}{b}\Bigr{)}=-\frac{1% }{a^{2}}\,\frac{\mathrm{d}a}{\mathrm{d}b}+\frac{1}{b^{2}}=-\frac{b^{n-1}}{a^{n% +1}}+\frac{1}{b^{2}}=\frac{1}{b^{2}}\Bigl{(}1-\Bigl{(}\frac{b}{a}\Bigr{)}^{n+1% }\Bigr{)}<0,$

since $b>a$ . The length of the interval decreases as $b$ increases and thus is smallest for the largest possible value $b=1$ , where $a=\alpha^{1/n}$ . This is the interval from the previous part of the solution. The reason for this result is that the density $nu^{n-1}$ of the pivot is increasing on $[0,1]$ and thus the probability $1-\alpha$ is concentrated on the shortest possible interval which is placed at the upper end of the density.

∎

Solution to exercise 25.3.

1.

The standard error is $\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})=1/\sqrt{n\mathcal{I}(\bar{X})}% =\sqrt{\bar{X}/n}$ , and the Wald interval is $\bar{X}\pm z_{1-\alpha/2}\sqrt{\bar{X}/n}$ . For the data we have $\bar{x}=1.5$ and $\sqrt{1.5/20}=0.274$ , and the $95\%$ interval is $1.5\pm 1.96\cdot 0.274$ , which is $[0.96,2.04]$ .
2.

In exercise 17.2 we found $\sqrt{n}\,(\hat{p}_{0}-p_{0})\xrightarrow{\ \mathrm{d}\ }N(0,e^{-2\theta}\theta)$ , and replacing $\theta$ by $\bar{X}$ in the variance gives the standard error $\mathop{\mathrm{se}}\nolimits(\hat{p}_{0})=e^{-\bar{X}}\sqrt{\bar{X}/n}$ , in agreement with the delta-method formula $|g^{\prime}(\hat{\theta}_{n})|\,\mathop{\mathrm{se}}\nolimits(\hat{\theta}_{n})$ for $g(\theta)=e^{-\theta}$ . The approximate interval is $e^{-\bar{X}}\pm z_{1-\alpha/2}\,e^{-\bar{X}}\sqrt{\bar{X}/n}$ . For the data we have $\hat{p}_{0}=e^{-1.5}=0.223$ and $\mathop{\mathrm{se}}\nolimits(\hat{p}_{0})=0.223\cdot 0.274=0.061$ , and the $95\%$ interval is $0.223\pm 1.96\cdot 0.061$ , which is $[0.103,0.343]$ .
3.

Write $[L,U]$ for the interval of the first part. Since $g$ is decreasing, the event $L\leq\theta\leq U$ is the same as the event $g(U)\leq g(\theta)\leq g(L)$ , and thus the interval $[e^{-U},e^{-L}]$ covers $p_{0}$ exactly when $[L,U]$ covers $\theta$ . Its coverage probability is that of the Wald interval, which converges to $1-\alpha$ by proposition 25.7. For the data we get $[e^{-2.04},e^{-0.96}]=[0.130,0.383]$ . The two intervals differ because $g$ is not linear: the delta-method interval is symmetric about $\hat{p}_{0}$ , while the transformed interval is shifted towards larger values and is asymmetric about $\hat{p}_{0}=0.223$ . Both have asymptotic coverage $1-\alpha$ , and for large $n$ the two nearly coincide, since $g$ is close to linear over a short interval. A practical advantage of the transformed interval is that it always lies inside $(0,1)$ , whereas the delta-method interval can extend below zero when $\hat{p}_{0}$ is small.

∎

Solution to exercise 25.4.

1.

Under $H_{0}$ , the test statistic $\sqrt{n}\,(\bar{X}-\mu_{0})/S$ is the pivot $T$ from example 25.4 and has $t_{n-1}$ distribution, independent of the value of $\sigma^{2}$ . Since the $t$ distribution is symmetric, we have

$\displaystyle\mathbb{P}_{\mu_{0},\sigma^{2}}\bigl{(}|T|>t_{n-1,1-\alpha/2}% \bigr{)}$

$\displaystyle=\mathbb{P}(T>t_{n-1,1-\alpha/2})+\mathbb{P}(T<-t_{n-1,1-\alpha/2})$

$\displaystyle=\frac{\alpha}{2}+\frac{\alpha}{2}=\alpha,$

and the test has size $\alpha$ .
2.

The acceptance region of the test for $\mu_{0}$ is $A(\mu_{0})=\{x\mid|\bar{x}-\mu_{0}|\leq t_{n-1,1-\alpha/2}\,s/\sqrt{n}\}$ . By theorem 25.6, the set of values not rejected,

$\displaystyle C(X)$

$\displaystyle=\Bigl{\{}\mu_{0}\mathrel{\big{|}}|\bar{X}-\mu_{0}|\leq t_{n-1,1-% \alpha/2}\,\frac{S}{\sqrt{n}}\Bigr{\}}$

$\displaystyle=\Bigl{[}\,\bar{X}-t_{n-1,1-\alpha/2}\,\frac{S}{\sqrt{n}},\ \bar{% X}+t_{n-1,1-\alpha/2}\,\frac{S}{\sqrt{n}}\,\Bigr{]},$

is a confidence set with level $1-\alpha$ and it is the $t$ -interval from example 25.4.
3.

The $95\%$ $t$ -interval for the given data is $[9.18,11.42]$ . By duality, $H_{0}\colon\mu=\mu_{0}$ is rejected at level $5\%$ if and only if $\mu_{0}$ is outside this interval. Thus, $H_{0}\colon\mu=9$ is rejected since $9<9.18$ , and $H_{0}\colon\mu=10$ is not rejected since $10$ is inside the interval.
4.

The one-sided test does not reject $\mu_{0}$ if and only if $\sqrt{n}\,(\bar{X}-\mu_{0})/S\leq t_{n-1,1-\alpha}$ , i.e. if $\mu_{0}\geq\bar{X}-t_{n-1,1-\alpha}\,S/\sqrt{n}$ . Thus, the set of values not rejected is the one-sided interval $\bigl{[}\bar{X}-t_{n-1,1-\alpha}\,S/\sqrt{n},\ \infty\bigr{)}$ , whose left-hand boundary is a lower confidence bound for $\mu$ with level $1-\alpha$ . Under $H_{0}$ , the test statistic is $t_{n-1}$ distributed, and thus the test has size $\alpha$ and the bound has coverage probability exactly $1-\alpha$ , by theorem 25.6.

∎

C.18 Bayesian Inference: Priors and Posteriors

Solution to exercise 26.1.

1.

The prior mean is $a/(a+b)=0.3$ and since $a+b=10$ we have $a=3$ and $b=7$ . From table A.2 we find that the variance of the $\text{Beta}(3,7)$ distribution is

$\frac{ab}{(a+b)^{2}(a+b+1)}=\frac{21}{100\cdot 11}\approx 0.0191,$

and thus the prior standard deviation is approximately $0.138$ .
2.

Let $\mu=a/(a+b)$ denote the prior mean and $s=a+b$ the equivalent sample size. Then we have $a=\mu s$ and $b=(1-\mu)s$ . Using the formula for the variance from the table, we get

$\frac{ab}{(a+b)^{2}(a+b+1)}=\frac{\mu(1-\mu)}{s+1}.$

Given $\mu=0.3$ and a variance of $0.1^{2}=0.01$ we find that we need $s+1=0.21/0.01=21$ , i.e. $s=20$ and thus we have $a=6$ and $b=14$ . The equivalent sample size is $20$ : the smaller spread corresponds to a prior which gives twice the number of observations.
3.

From example 26.5 we know that the posterior distribution is $\text{Beta}(3+9,\,7+11)=\text{Beta}(12,18)$ . The mean of this distribution is $12/30=0.4$ . In the form (26.3) we can write this as

$\frac{10}{30}\cdot 0.3+\frac{20}{30}\cdot 0.45=0.1+0.3=0.4,$

where $\bar{x}=9/20=0.45$ : The data have twice the weight as the prior, and the posterior mean is two thirds of the way between the prior mean and the sample proportion.

∎

Solution to exercise 26.2.

1.

On the interval $(0,1/2]$ we have $1-\theta\geq 1/2$ and thus $\theta^{-1}(1-\theta)^{-1}\geq\theta^{-1}$ there. Since $\int_{0}^{1/2}\theta^{-1}\,\mathrm{d}\theta=\infty$ , the integral of the prior over $(0,1)$ is infinite and the prior is improper.
2.

Consider the kernel $\theta^{\alpha-1}(1-\theta)^{\beta-1}$ for real $\alpha$ and $\beta$ . Near $\theta=0$ the factor $(1-\theta)^{\beta-1}$ is bounded between two positive constants and thus the kernel is integrable near $0$ if and only if $\theta^{\alpha-1}$ is, which is equivalent to $\alpha>0$ . By symmetry the kernel is integrable near $\theta=1$ if and only if $\beta>0$ and thus, since it is bounded away from the endpoints, the integral over $(0,1)$ is finite if and only if $\alpha>0$ and $\beta>0$ , in which case it equals $B(\alpha,\beta)$ . For the formal posterior we have $\alpha=T$ and $\beta=n-T$ and thus the posterior is proper if and only if $T>0$ and $n-T>0$ , i.e. if and only if $0<T<n$ . In this case it is the $\text{Beta}(T,n-T)$ distribution.
3.

From table A.2 we find that the posterior mean is $T/(T+n-T)=T/n=\bar{x}$ . This is the maximum likelihood estimate from example 5.3. The Haldane prior is the choice of prior where the Bayes estimator and the maximum likelihood estimator coincide. This happens for an equivalent sample size of zero in (26.3).
4.

For $T=0$ the likelihood $(1-\theta)^{n}$ has its maximum at $\theta=0$ and does not vanish there, while the prior has a non-integrable singularity at $\theta=0$ . Since the data does not contain any successes, the infinite mass of the prior near zero cannot be compensated and thus the integral of $\theta^{-1}(1-\theta)^{n-1}$ is infinite: there is no posterior distribution in this case. In informal language, the improper prior suggests that $\theta$ is very close to either $0$ or $1$ and a data set with no successes cannot exclude the first of these two possibilities.

∎

Solution to exercise 26.3.

1.

We have

$\displaystyle f(x\mskip 1.0mu|\mskip 1.0mu\theta)$

$\displaystyle=\prod_{i=1}^{n}\frac{1}{\sqrt{2\pi\sigma^{2}}}\exp\Bigl{(}-\frac% {(x_{i}-\theta)^{2}}{2\sigma^{2}}\Bigr{)}$

$\displaystyle=(2\pi\sigma^{2})^{-n/2}\exp\Bigl{(}-\frac{1}{2\sigma^{2}}\sum_{i% =1}^{n}(x_{i}-\theta)^{2}\Bigr{)}.$

Using the identity (2.1) from lecture 2, which holds for all numbers in place of the mean, and using $\theta$ , we find $\sum_{i}(x_{i}-\theta)^{2}=\sum_{i}(x_{i}-\bar{x})^{2}+n(\bar{x}-\theta)^{2}$ . Since the first term does not depend on $\theta$ and since the factor $(2\pi\sigma^{2})^{-n/2}$ does not depend on $\theta$ , we have

$f(x\mskip 1.0mu|\mskip 1.0mu\theta)\propto\exp\bigl{(}-n(\theta-\bar{x})^{2}/(% 2\sigma^{2})\bigr{)}.$
2.

Since the prior density, as a function of $\theta$ , is proportional to $\exp\bigl{(}-(\theta-m)^{2}/(2\tau^{2})\bigr{)}$ and since we can group the terms in the exponent of the posterior density as powers of $\theta$ , we get

$\displaystyle\pi(\theta\mskip 1.0mu|\mskip 1.0mux)$

$\displaystyle\propto\exp\Bigl{(}-\frac{n(\theta-\bar{x})^{2}}{2\sigma^{2}}-% \frac{(\theta-m)^{2}}{2\tau^{2}}\Bigr{)}$

$\displaystyle\propto\exp\Bigl{(}-\frac{1}{2}\Bigl{(}\frac{n}{\sigma^{2}}+\frac% {1}{\tau^{2}}\Bigr{)}\theta^{2}+\Bigl{(}\frac{n\bar{x}}{\sigma^{2}}+\frac{m}{% \tau^{2}}\Bigr{)}\theta\Bigr{)},$

where we have omitted the terms which do not contain $\theta$ . Using the abbreviations $1/\tau_{n}^{2}=1/\tau^{2}+n/\sigma^{2}$ and $m_{n}=\tau_{n}^{2}\bigl{(}m/\tau^{2}+n\bar{x}/\sigma^{2}\bigr{)}$ we can write the exponent as $-\theta^{2}/(2\tau_{n}^{2})+\theta m_{n}/\tau_{n}^{2}$ and completing the square we find

$-\frac{\theta^{2}}{2\tau_{n}^{2}}+\frac{\theta m_{n}}{\tau_{n}^{2}}=-\frac{(% \theta-m_{n})^{2}}{2\tau_{n}^{2}}+\frac{m_{n}^{2}}{2\tau_{n}^{2}},$

where we have omitted the last term which does not contain $\theta$ . Thus we have $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\propto\exp\bigl{(}-(\theta-m_{n})^{2}/(2% \tau_{n}^{2})\bigr{)}$ and this is the kernel of the $N(m_{n},\tau_{n}^{2})$ density and, since the posterior is a density, this is the corresponding normal distribution. If we multiply out $m_{n}$ we get the weighted form of the posterior given in proposition 26.6.
3.

As $\tau\to\infty$ with $n$ fixed, the prior precision $1/\tau^{2}$ goes to zero and thus $\tau_{n}^{2}\to\sigma^{2}/n$ and $m_{n}\to\bar{x}$ : the posterior converges to $N(\bar{x},\sigma^{2}/n)$ , i.e. to the posterior under the flat prior discussed in section 26.4. As $n\to\infty$ with $\tau$ fixed, the weight $(1/\tau^{2})/(1/\tau^{2}+n/\sigma^{2})$ of the prior mean goes to zero and thus $m_{n}-\bar{x}\to 0$ and $\tau_{n}^{2}\leq\sigma^{2}/n$ goes to zero: the posterior is now concentrated around $\bar{x}$ and the prior is washed out.

∎

Solution to exercise 26.4.

1.

Under $\mathbb{P}_{\theta}$ , the total number $T$ of successes is binomially distributed with $\mathbb{E}_{\theta}(T)=n\theta$ and $\mathop{\mathrm{Var}}\nolimits_{\theta}(T)=n\theta(1-\theta)$ . Since $\hat{\theta}_{B}$ is an affine function of $T$ , we have

$\displaystyle\mathbb{E}_{\theta}(\hat{\theta}_{B})$

$\displaystyle=\frac{a+n\theta}{a+b+n}\qquad\text{and}$

$\displaystyle\mathop{\mathrm{bias}}\nolimits(\hat{\theta}_{B})$

$\displaystyle=\frac{a+n\theta-(a+b+n)\theta}{a+b+n}=\frac{a-(a+b)\theta}{a+b+n},$

which equals zero if and only if $\theta=a/(a+b)$ , the prior mean, and

$\mathop{\mathrm{Var}}\nolimits_{\theta}(\hat{\theta}_{B})=\frac{n\theta(1-% \theta)}{(a+b+n)^{2}}.$

The bias is of order $1/n$ and the variance is of order $1/n$ , both tending to zero as $n\to\infty$ . Thus, by corollary 2.11, the estimator is consistent, although for fixed $n$ it is biased for all $\theta\neq a/(a+b)$ .
2.

For $a=b=1$ and $n=10$ we have $\hat{\theta}_{B}=(T+1)/12$ and thus, by theorem 2.7,

$\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{B})=\frac{10\theta(1-\theta)+(1-2% \theta)^{2}}{144},\qquad\mathop{\mathrm{MSE}}\nolimits(\bar{X})=\frac{\theta(1% -\theta)}{10}.$

At $\theta=1/2$ the bias is zero and we have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{B})=2.5/144\approx 0.0174$ compared to $\mathop{\mathrm{MSE}}\nolimits(\bar{X})=0.025$ . Thus, the Bayes estimator is better. At $\theta=0.1$ we have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{B})=(0.9+0.64)/144\approx 0.0107$ compared to $\mathop{\mathrm{MSE}}\nolimits(\bar{X})=0.009$ and thus $\bar{X}$ is better. Neither estimator has smaller mean squared error for all $\theta$ : the Bayes estimator is better in the middle of the parameter range, where the prior mean is close to the truth, and the reduced variance does not compensate for the bias introduced by the shrinkage towards $1/2$ at the ends of the range.

∎

Solution to exercise 26.5.

For the first part of the question we can take derivatives. The log-likelihood function for a single observation is $\ell_{0}(x-\theta)$ and thus the score function is

\frac{\partial}{\partial\theta}\,\log f(x\mskip 1.0mu|\mskip 1.0mu\theta)=% \frac{\partial}{\partial\theta}\,\ell_{0}(x-\theta)=-\ell_{0}^{\prime}(x-% \theta).

Since $Z=X-\theta$ , with density $f_{0}$ , for all values of $\theta$ , the Fisher information is

\mathcal{I}(\theta)=\mathbb{E}\bigl{(}\ell_{0}^{\prime}(X-\theta)^{2}\bigr{)}=% \int\ell_{0}^{\prime}(z)^{2}\,f_{0}(z)\,\mathrm{d}z.

This does not depend on $\theta$ .

For the second part of the question we have $\log f(x\mskip 1.0mu|\mskip 1.0mu\sigma)=\ell_{0}(x/\sigma)-\log\sigma$ and thus the score function is

\frac{\partial}{\partial\sigma}\bigl{(}\ell_{0}(x/\sigma)-\log\sigma\bigr{)}=-% \ell_{0}^{\prime}(x/\sigma)\,\frac{x}{\sigma^{2}}-\frac{1}{\sigma}=-\frac{1+z% \ell_{0}^{\prime}(z)}{\sigma}

where $z=x/\sigma$ . Since $Z=X/\sigma$ has density $f_{0}$ for all $\sigma$ , we find

\mathcal{I}(\sigma)=\frac{1}{\sigma^{2}}\,\mathbb{E}\Bigl{(}\bigl{(}1+Z\ell_{0% }^{\prime}(Z)\bigr{)}^{2}\Bigr{)}=\frac{c}{\sigma^{2}},\qquad\text{where }c=% \int\bigl{(}1+z\ell_{0}^{\prime}(z)\bigr{)}^{2}f_{0}(z)\,\mathrm{d}z

does not depend on $\sigma$ .

Finally, the $N(0,\sigma^{2})$ model is the scale model with base density the standard normal density and with $\ell_{0}(z)=-z^{2}/2-\log\sqrt{2\pi}$ and thus $\ell_{0}^{\prime}(z)=-z$ . In this case we have $1+z\ell_{0}^{\prime}(z)=1-z^{2}$ and using the substitutions $\mathbb{E}(Z^{2})=1$ and $\mathbb{E}(Z^{4})=3$ for the standard normal density $Z$ we find

c=\mathbb{E}\bigl{(}(1-Z^{2})^{2}\bigr{)}=1-2\mathbb{E}(Z^{2})+\mathbb{E}(Z^{4% })=2.

Thus $\mathcal{I}(\sigma)=2/\sigma^{2}$ , as claimed. ∎

C.19 Credible Intervals and Bayesian Testing

Solution to exercise 28.1.

1.

Since the posterior density is monotone on each half-line and integrates to one, it converges to zero as $\theta\to\pm\infty$ . The condition $\mathbb{P}(\theta\in H_{c}\mskip 1.0mu|\mskip 1.0mux)=1-\alpha\in(0,1)$ excludes $c\geq\pi(\mu\mskip 1.0mu|\mskip 1.0mux)$ , where $H_{c}$ contains at most the point $\mu$ and has probability zero, and thus $0<c<\pi(\mu\mskip 1.0mu|\mskip 1.0mux)$ . On $(-\infty,\mu]$ the density is continuous and strictly increasing from $0$ to $\pi(\mu\mskip 1.0mu|\mskip 1.0mux)$ and thus there is exactly one $l<\mu$ with $\pi(l\mskip 1.0mu|\mskip 1.0mux)=c$ and $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\geq c$ for $\theta\leq\mu$ if and only if $\theta\geq l$ . Similarly, there is exactly one $u>\mu$ with $\pi(u\mskip 1.0mu|\mskip 1.0mux)=c$ and the density is at least $c$ on $[\mu,\infty)$ exactly on $[\mu,u]$ . Thus we have $H_{c}=[l,u]$ with $\pi(l\mskip 1.0mu|\mskip 1.0mux)=c=\pi(u\mskip 1.0mu|\mskip 1.0mux)$ .
2.

Let $l=\mu-s$ and $u=\mu+t$ with $s,t>0$ . By symmetry we have $\pi(\mu+s\mskip 1.0mu|\mskip 1.0mux)=\pi(\mu-s\mskip 1.0mu|\mskip 1.0mux)=c=% \pi(\mu+t\mskip 1.0mu|\mskip 1.0mux)$ and since the density is strictly decreasing on $[\mu,\infty)$ we find $s=t$ . The two tails $(-\infty,\mu-t)$ and $(\mu+t,\infty)$ have equal posterior probability by symmetry and together they have probability $\alpha$ , i.e. each tail has probability $\alpha/2$ . Thus we have $l=q_{\alpha/2}$ and $u=q_{1-\alpha/2}$ , i.e. we have found the equal-tailed interval.
3.

The set $\{\theta\mid\pi(\theta\mskip 1.0mu|\mskip 1.0mux)\geq c\}$ is for $0<c<\pi(0\mskip 1.0mu|\mskip 1.0mux)$ the interval $[0,u]$ with $\pi(u\mskip 1.0mu|\mskip 1.0mux)=c$ , obtained by using the same argument as in the first part, but only for the decreasing branch of the density, and using the condition $\mathbb{P}(\theta\leq u\mskip 1.0mu|\mskip 1.0mux)=1-\alpha$ to get $u=q_{1-\alpha}$ . The equal-tailed interval $[q_{\alpha/2},q_{1-\alpha/2}]$ is then obtained from $[0,q_{1-\alpha}]$ by removing the piece $[0,q_{\alpha/2})$ and adding the piece $(q_{1-\alpha},q_{1-\alpha/2}]$ , both of which have posterior probability $\alpha/2$ . The density is at least $d_{1}=\pi(q_{\alpha/2}\mskip 1.0mu|\mskip 1.0mux)$ on the first piece and at most $d_{2}=\pi(q_{1-\alpha}\mskip 1.0mu|\mskip 1.0mux)$ on the second piece, where $d_{1}>d_{2}$ since the density is strictly decreasing. Writing $\lambda_{1}$ and $\lambda_{2}$ for the lengths of the two pieces we find $\alpha/2\geq d_{1}\lambda_{1}$ and $\alpha/2\leq d_{2}\lambda_{2}$ and thus $\lambda_{1}\leq\alpha/(2d_{1})<\alpha/(2d_{2})\leq\lambda_{2}$ : the added piece is longer than the removed piece and the equal-tailed interval is longer than the HPD interval.

∎

Solution to exercise 28.2.

1.

The equal-tailed $95\%$ credible interval is $[0.386,0.861]$ , with posterior probability $2.5\%$ cut off in each tail. The midpoint of this interval is $(0.386+0.861)/2=0.624$ , which is slightly below the posterior mean $9/14=0.643$ .
2.

The posterior density is proportional to $\theta^{8}(1-\theta)^{4}$ on $(0,1)$ . Setting the derivative of the logarithm of the posterior density, i.e. $8\log\theta+4\log(1-\theta)$ , equal to zero, gives the maximum of the density at $8/\theta=4/(1-\theta)$ and thus the mode is at $\theta=8/12=2/3\approx 0.667$ . Since the mode is above the mean $0.643$ , the posterior distribution is skewed.
3.

By definition 28.2 of the HPD interval, the HPD interval contains the values with the highest posterior density and thus is centred around the mode, so that it lies to the right of the equal-tailed interval. The HPD interval is the shortest interval which has posterior probability $0.95$ , in this case of length $0.473$ against $0.475$ . As $n$ increases, the posterior distribution approaches a symmetric, normal distribution and the mode and mean move together. In the limit, for a symmetric, unimodal posterior, both intervals coincide; this is a result from exercise 28.1.

∎

Solution to exercise 28.3.

1.

Since the prior is flat, we have $\text{Beta}(1,1)$ and using $n=3$ and $T=2$ from example 26.5 we find the posterior $\text{Beta}(3,2)$ . Since $B(3,2)=\Gamma(3)\Gamma(2)/\Gamma(5)=2/24=1/12$ , the density of this posterior is $\pi(\theta\mskip 1.0mu|\mskip 1.0mux)=12\theta^{2}(1-\theta)$ for $0<\theta<1$ .
2.

We can find the posterior probability of $H_{0}$ by integrating the density:

$\mathbb{P}(\theta\leq 1/2\mskip 1.0mu|\mskip 1.0mux)=12\int_{0}^{1/2}(\theta^{% 2}-\theta^{3})\,\mathrm{d}\theta=12\Bigl{(}\frac{1}{24}-\frac{1}{64}\Bigr{)}=% \frac{5}{16}=0.3125.$

Thus we have $\mathbb{P}(H_{1}\mskip 1.0mu|\mskip 1.0mux)=11/16$ and the posterior odds for $H_{0}$ against $H_{1}$ are $5/11\approx 0.45$ .
3.

The data have multiplied the odds by $5/11$ . Using Bayes’ theorem for the events $H_{0}$ and $H_{1}$ , we find that the posterior probability of $H_{j}$ is $\mathbb{P}(H_{j})$ times the average of the likelihoods over $H_{j}$ , with respect to the prior conditioned on $H_{j}$ , divided by the marginal density of the data. The factor by which the odds change is the ratio of these two averaged likelihoods. In this case, the likelihood is $\theta^{2}(1-\theta)$ and the conditional priors are uniform on $(0,1/2)$ and on $(1/2,1)$ , and thus the factor is

$\frac{2\int_{0}^{1/2}\theta^{2}(1-\theta)\,\mathrm{d}\theta}{2\int_{1/2}^{1}% \theta^{2}(1-\theta)\,\mathrm{d}\theta}=\frac{5/192}{1/12-5/192}=\frac{5}{11}.$

This agrees with the result from the previous part. Note that this is not the ratio of likelihoods at two values, since each hypothesis includes a range of parameter values, and these need to be weighted by the prior before the two hypotheses can be compared. Only for two simple hypotheses do the averages coincide with single likelihood values, as in case (28.3).
4.

The bounds $l$ and $u$ satisfy $12\int_{0}^{l}(\theta^{2}-\theta^{3})\,\mathrm{d}\theta=0.025$ and $12\int_{0}^{u}(\theta^{2}-\theta^{3})\,\mathrm{d}\theta=0.975$ , i.e. we have

$4l^{3}-3l^{4}=0.025\qquad\text{and}\qquad 4u^{3}-3u^{4}=0.975.$

These are polynomial equations of degree four and while they can be solved explicitly, the solutions are cumbersome and not very instructive; for a general Beta posterior, the bounds will not be algebraic. The quantiles are best found numerically, for example using the R function qbeta to get $l=0.194$ and $u=0.932$ and thus the $95\%$ equal-tailed credible interval is $[0.194,0.932]$ .

∎

Solution to exercise 28.4.

1.

Using the formula from example 28.7, with $n=25$ , $\sigma^{2}=4$ , $\theta_{0}=10$ , $\theta_{1}=11$ and $\bar{x}=10.9$ , we find that the Bayes factor in favour of $H_{0}$ is

$\frac{f(x\mskip 1.0mu|\mskip 1.0mu\theta_{0})}{f(x\mskip 1.0mu|\mskip 1.0mu% \theta_{1})}=\exp\Bigl{(}\frac{25\cdot 1\cdot(21-21.8)}{8}\Bigr{)}=e^{-2.5}=0.% 082.$

The data give evidence in favour of $H_{1}$ over $H_{0}$ by a factor of $e^{2.5}\approx 12.2$ .
2.

The prior odds are $3$ and thus the posterior odds are $3\cdot 0.082=0.246$ and $\mathbb{P}(H_{0}\mskip 1.0mu|\mskip 1.0mux)=0.246/1.246=0.198$ . The data have changed the prior opinion, which was three to one in favour of $H_{0}$ , to a posterior opinion of approximately four to one against it.
3.

The $p$ -value $0.0122$ is the probability, under $\theta=10$ , of obtaining a sample mean of at least $10.9$ . This value refers to the data, includes outcomes more extreme than the one observed and makes no reference to any alternative. In contrast, the posterior probability $0.198$ is a probability for the hypothesis; it uses only the observed data, compares the likelihoods at exactly $10$ and $11$ and takes into account the prior odds, which were chosen in favour of $H_{0}$ . Even if the prior odds had been $1$ , the posterior probability would have been $0.082/1.082=0.076$ , or six times the $p$ -value. Both numbers show that the data are evidence against $H_{0}$ , but in different ways. Neither number is a rewording of the other.
4.

The posterior odds are the prior odds times $e^{-2.5}$ , and the posterior odds are greater than $1$ if and only if the prior odds are greater than $e^{2.5}\approx 12.2$ . Only a prior which assumed $H_{0}$ to be more likely than $H_{1}$ , by a factor of more than twelve, would result in $H_{0}$ being the more likely hypothesis after these data are observed.

∎

Solution to exercise 28.5.

1.

By (28.1) we have $1-w=(n/\sigma^{2})/(1/\tau^{2}+n/\sigma^{2})=(n/\sigma^{2})\,\tau_{n}^{2}$ , and thus $\tau_{n}^{2}=(1-w)\sigma^{2}/n$ . The posterior mean is $m_{n}=wm+(1-w)\bar{X}$ , and thus $m_{n}-\theta=w(m-\theta)+(1-w)(\bar{X}-\theta)$ . Under $\mathbb{P}_{\theta}$ we have $\bar{X}-\theta\sim N(0,\sigma^{2}/n)$ by proposition A.3, and an affine function of a normal variable is normal, so that $m_{n}-\theta\sim N\bigl{(}w(m-\theta),(1-w)^{2}\sigma^{2}/n\bigr{)}$ .
2.

Write $z=z_{1-\alpha/2}$ and $\delta=\sqrt{n}\,(m-\theta)/\sigma$ for the standardised distance between the prior mean and the true value. The interval covers $\theta$ if and only if $|m_{n}-\theta|\leq z\tau_{n}=z\sqrt{1-w}\,\sigma/\sqrt{n}$ . Subtracting the mean $w(m-\theta)$ and dividing by the standard deviation $(1-w)\sigma/\sqrt{n}$ turns $m_{n}-\theta$ into a standard normal variable, and the bounds into $(\pm z\sqrt{1-w}-w\delta)/(1-w)$ . Thus the coverage probability is

$\mathbb{P}_{\theta}\bigl{(}\theta\in\text{interval}\bigr{)}=\Phi\Bigl{(}\frac{% z\sqrt{1-w}-w\delta}{1-w}\Bigr{)}-\Phi\Bigl{(}\frac{-z\sqrt{1-w}-w\delta}{1-w}% \Bigr{)}.$

It depends on $\theta$ only through $\delta$ , and it is symmetric in $\delta$ .
3.

Here $1/\tau^{2}=1$ and $n/\sigma^{2}=4$ , and thus $w=0.2$ , $1-w=0.8$ , $\sqrt{1-w}=0.894$ and $z\sqrt{1-w}=1.753$ . At $\theta=m=12$ we have $\delta=0$ and the coverage probability is $2\Phi(1.753/0.8)-1=2\Phi(2.19)-1=0.971$ , above the nominal $0.95$ . At $\theta=10$ we have $\delta=4\cdot 2/2=4$ and $w\delta=0.8$ , and the coverage probability is $\Phi(0.953/0.8)-\Phi(-2.553/0.8)=\Phi(1.19)-\Phi(-3.19)=0.883-0.001=0.882$ , below the nominal value. As $|\theta-m|\to\infty$ we have $|\delta|\to\infty$ , both arguments of $\Phi$ tend to the same infinity and the coverage probability tends to zero: the interval is pulled towards the prior mean by a fixed fraction $w$ of the distance, and once that pull exceeds the half-width the interval almost never contains the truth.
4.

It is not a confidence interval with level $0.95$ , because definition 25.1 requires coverage probability at least $0.95$ for every $\theta$ , and the previous part exhibits values of $\theta$ where it is smaller. The credible interval trades the uniform guarantee for better behaviour near the prior mean, which is where a well-chosen prior expects the truth to be. As $n\to\infty$ with $\theta$ fixed we have $w\approx\sigma^{2}/(n\tau^{2})\to 0$ , and thus $w\delta\approx\sigma(m-\theta)/(\tau^{2}\sqrt{n})\to 0$ and $\sqrt{1-w}\to 1$ , so that the coverage probability tends to $2\Phi(z)-1=1-\alpha$ for every $\theta$ . This is the large-sample agreement of section 28.3 seen from the frequentist side.

∎

C.20 Standard Tests Revisited

Solution to exercise 29.1.

1.

We have the sum of observations $154$ and thus $\bar{x}=154/8=19.25$ . The sum of squared deviations from $\bar{x}$ equals $3.30$ and thus we have $s^{2}=3.30/7=0.471$ and $s=0.687$ . The observed value of the test statistic is

$t=\frac{\sqrt{8}\,(19.25-20)}{0.687}=\frac{2.828\cdot(-0.75)}{0.687}=-3.09.$
2.

From proposition 29.1 we know that the test against $\mu<20$ rejects if $t<-t_{7,1-\alpha}$ . For level $0.05$ the critical value is $-t_{7,0.95}=-1.895$ and for level $0.01$ the critical value is $-t_{7,0.99}=-2.998$ ; since $-3.09$ lies below both of these critical values, the hypothesis $H_{0}$ is rejected at both levels. The $p$ -value is given by $\mathbb{P}(T\leq-3.09)=\mathbb{P}(T\geq 3.09)$ for $T\sim t_{7}$ , by symmetry, and since $3.09$ is between $t_{7,0.99}=2.998$ and $t_{7,0.995}=3.499$ , the value is between $0.005$ and $0.01$ .
3.

The two-sided test rejects if $|t|>t_{7,1-\alpha/2}$ . For level $0.05$ we have $|t|=3.09>t_{7,0.975}=2.365$ and thus $H_{0}$ is rejected. For level $0.01$ we have $3.09<t_{7,0.995}=3.499$ and thus $H_{0}$ is not rejected. The two-sided $p$ -value is $2\,\mathbb{P}(T\geq 3.09)$ , which is twice the value of the one-sided test, so it is between $0.01$ and $0.02$ .
4.

The $99\%$ interval is $\bar{x}\pm t_{7,0.995}\,s/\sqrt{8}=19.25\pm 3.499\cdot 0.687/2.828=19.25\pm 0.% 849$ . This interval is $[18.40,20.10]$ . The interval contains the value $20$ and by theorem 25.6 this is equivalent to the non-rejection of $H_{0}\colon\mu=20$ by the two-sided test with level $0.01$ in the previous part. The $95\%$ interval $19.25\pm 2.365\cdot 0.243=[18.68,19.82]$ does not contain $20$ and thus the rejection at level $0.05$ is consistent with the results of the other parts.

∎

Solution to exercise 29.2.

1.

The value of the test statistic (29.2) is $v=14\cdot 2.9/1.5=27.07$ . We can compare this value to quantiles of the $\chi^{2}_{14}$ distribution: at level $0.05$ the critical value is $\chi^{2}_{14,0.95}=23.685$ and since $27.07>23.685$ we can reject $H_{0}$ , whereas at level $0.01$ the critical value is $\chi^{2}_{14,0.99}=29.141$ and since $27.07<29.141$ we cannot reject $H_{0}$ . The $p$ -value, $\mathbb{P}(W\geq 27.07)$ for $W\sim\chi^{2}_{14}$ , is between $0.01$ and $0.025$ , since $v$ is between $\chi^{2}_{14,0.975}=26.119$ and $\chi^{2}_{14,0.99}=29.141$ .
2.

The two-sided test with level $0.05$ does not reject, if

$v\in[\chi^{2}_{14,0.025},\chi^{2}_{14,0.975}]=[5.629,26.119].$

Since $27.07>26.119$ , we reject $H_{0}$ .
3.

Let $c=\chi^{2}_{n-1,1-\alpha}$ . Assuming the true variance is $\sigma^{2}$ , the value $W=(n-1)S^{2}/\sigma^{2}$ is $\chi^{2}_{n-1}$ distributed by proposition A.3 and thus $V=(n-1)S^{2}/\sigma_{0}^{2}=W\sigma^{2}/\sigma_{0}^{2}$ . We have

$\beta(\sigma^{2})=\mathbb{P}_{\sigma^{2}}(V>c)=\mathbb{P}\Bigl{(}W>\frac{c\,% \sigma_{0}^{2}}{\sigma^{2}}\Bigr{)}.$

As $\sigma^{2}$ increases, the threshold $c\sigma_{0}^{2}/\sigma^{2}$ decreases and the probability of $W$ exceeding a smaller number increases, i.e. $\beta$ increases as a function of $\sigma^{2}$ . By definition 20.5, the size of the test for the composite null hypothesis $\sigma^{2}\leq\sigma_{0}^{2}$ is $\sup_{\sigma^{2}\leq\sigma_{0}^{2}}\beta(\sigma^{2})=\beta(\sigma_{0}^{2})=% \mathbb{P}(W>c)=\alpha$ , where the bound is achieved at the boundary.
4.

The two-sided test with level $\alpha$ rejects large values of $V$ only if $V>\chi^{2}_{n-1,1-\alpha/2}$ . Since this is a larger threshold than the $\chi^{2}_{n-1,1-\alpha}$ of the one-sided test, the power of the two-sided test against the alternative $\sigma^{2}>\sigma_{0}^{2}$ is smaller than the power of the one-sided test. An observed value between the two thresholds is rejected by the one-sided test but not by the two-sided test. The two-sided test has power above $\alpha$ against the alternative $\sigma^{2}<\sigma_{0}^{2}$ , where the one-sided test has probability of rejection below $\alpha$ . For the given data, both tests reject the hypothesis at level $0.05$ , but at level $0.025$ the one-sided test, with threshold $\chi^{2}_{14,0.975}=26.119$ , still rejects, whereas the two-sided test compares $27.07$ to $\chi^{2}_{14,0.9875}=28.42$ and does not reject.

∎

Solution to exercise 29.3.

1.

We can describe the differences $D_{i}=X_{i}-Y_{i}$ using i.i.d. $N(\mu_{D},\sigma_{D}^{2})$ , where both parameters are unknown. We test $H_{0}\colon\mu_{D}=0$ against $H_{1}\colon\mu_{D}>0$ , since we expect the reaction time to be smaller after training. The differences are $4,-1,6,3,6,5$ and the sum of these differences is $23$ , i.e. we have $\bar{d}=3.833$ . The sum of squared deviations from $\bar{d}$ is $34.83$ , and thus we have $s_{D}^{2}=34.83/5=6.967$ and $s_{D}=2.639$ . The test statistic from proposition 29.1, for $\mu_{0}=0$ , is

$t=\frac{\sqrt{6}\cdot 3.833}{2.639}=3.56,$

where $5$ is the number of degrees of freedom. Since $3.56>t_{5,0.95}=2.015$ we can reject the hypothesis at level $0.05$ and since $3.56>t_{5,0.99}=3.365$ we can also reject the hypothesis at level $0.01$ : the data show a reduction in reaction time.
2.

The $95\%$ $t$ -interval from example 25.4 for $\mu_{D}$ is $\bar{d}\pm t_{5,0.975}\,s_{D}/\sqrt{6}=3.833\pm 2.571\cdot 2.639/2.449=3.833% \pm 2.770$ , which is $[1.06,6.60]$ .
3.

The row means are $\bar{x}=419/6=69.83$ and $\bar{y}=396/6=66.00$ , the pooled variance is $s_{p}^{2}=(5\cdot 59.77+5\cdot 42.80)/10=51.28$ and thus we have $s_{p}=7.161$ . The test statistic (29.3) is

$t=\frac{69.83-66.00}{7.161\sqrt{1/6+1/6}}=\frac{3.833}{4.135}=0.93,$

where $10$ is the number of degrees of freedom. Since $0.93<t_{10,0.95}=1.812$ the colleague does not reject the hypothesis at level $0.05$ . The two-sample test is wrong, because the two measurements for one subject are not independent (a slow person is likely to be slow after training, too), whereas the two-sample test assumes that the twelve observations are independent. The difference in opinion is caused by the variability of the data. Between subjects the standard deviation of the reaction times is about $7$ . The two-sample test compares the mean difference $3.83$ to this spread. Inside subjects the differences have standard deviation only $2.64$ . The paired test compares $3.83$ to this smaller spread. By taking differences we remove the effect of the subjects from the comparison, and this is the idea of the paired test.

∎

Solution to exercise 29.4.

1.

For $\mu_{0}=9$ the value of the test statistic (29.1) is $t=4\,(10.3-9)/2.1=2.48$ and, since $|t|>t_{15,0.975}=2.131$ , the two-sided test rejects at level $0.05$ , as found using duality in exercise 25.4. For $\mu_{0}=11.5$ we have $t=4\,(10.3-11.5)/2.1=-2.29$ and $|t|=2.29>2.131$ , so $H_{0}$ is rejected again. This agrees with the interval, since $11.5$ is outside the interval $[9.18,11.42]$ .
2.

From theorem 25.6 we know that the two-sided test from proposition 29.3 rejects $\sigma_{0}^{2}$ at level $0.05$ if and only if $\sigma_{0}^{2}$ is outside the $95\%$ confidence interval $[2.41,10.56]$ from example 25.5. This is the inverse of the test, so $\sigma_{0}^{2}=2$ is rejected and $\sigma_{0}^{2}=9$ is not. Looking at the values directly, for $(n-1)s^{2}=15\cdot 4.41=66.15$ the test statistic corresponding to $\sigma_{0}^{2}=2$ is $v=66.15/2=33.08$ , which is greater than $\chi^{2}_{15,0.975}=27.488$ , so $H_{0}$ is rejected. For $\sigma_{0}^{2}=9$ the test statistic $v=66.15/9=7.35$ falls into the interval $[6.262,27.488]$ and thus $H_{0}$ is not rejected.
3.

For $\mu_{0}=9.5$ the test statistic is $t=4\,(10.3-9.5)/2.1=1.52$ and, since $1.52\leq t_{15,0.95}=1.753$ , the one-sided test does not reject at level $0.05$ . The two-sided interval is the inverse of the two-sided test with level $\alpha$ , i.e. the upper tail of the two-sided test has probability $\alpha/2$ . If we decided a one-sided test by checking whether $\mu_{0}$ is less than the lower bound of the interval, we would obtain the one-sided test at level $\alpha/2=0.025$ , not at level $0.05$ . Thus, the one-sided test with $0.05$ is not the same as the two-sided test. The confidence set corresponding to the one-sided test with $0.05$ is, as in exercise 25.4, the one-sided interval $\bigl{[}\bar{x}-t_{15,0.95}\,s/\sqrt{n},\ \infty\bigr{)}=[10.3-1.753\cdot 0.52% 5,\ \infty)=[9.38,\infty)$ , and $9.5$ is contained in this interval, as expected.

∎

C.21 The Exponential Model: From Start to Finish

Here we solve the exercises at the end of lecture 31.

Solution to exercise 31.1.

1.

The log-likelihood function is $\ell(\theta)=n\log\theta-\theta T$ , with $T=\sum_{i}x_{i}=n\bar{x}$ , and the MLE is $\hat{\theta}=n/T$ . Using $\hat{\theta}T=n$ and $\theta_{0}T=n\theta_{0}/\hat{\theta}$ we find

$W=2\bigl{(}\ell(\hat{\theta})-\ell(\theta_{0})\bigr{)}=2\bigl{(}n\log\hat{% \theta}-n-n\log\theta_{0}+n\theta_{0}/\hat{\theta}\bigr{)}=2n\Bigl{(}\log\frac% {\hat{\theta}}{\theta_{0}}-1+\frac{\theta_{0}}{\hat{\theta}}\Bigr{)}.$

Using $u=\theta_{0}\bar{x}=\theta_{0}/\hat{\theta}$ we have $\log(\hat{\theta}/\theta_{0})=-\log u$ and $\theta_{0}/\hat{\theta}=u$ , and thus $W=2n(u-1-\log u)$ .
2.

Let $g(u)=u-1-\log u$ for $u>0$ . Then $g(1)=0$ and $g^{\prime}(u)=1-1/u$ . Since this value is negative for $u<1$ and positive for $u>1$ , $u=1$ must be the unique minimum of $g$ and $g(u)>0$ for $u\neq 1$ . Since $W=2ng(u)$ , we have $W\geq 0$ , with equality if and only if $u=1$ , i.e. if and only if $\hat{\theta}=\theta_{0}$ .
3.

For $\theta_{0}=0.5$ we have $u=1.25$ and $W=20\,(1.25-1-\log 1.25)=20\cdot 0.0269=0.54$ . This value is less than $\chi^{2}_{1,0.95}=3.84$ , so $H_{0}\colon\theta=0.5$ is not rejected at level $0.05$ . For $\theta_{0}=1$ we have $u=2.5$ and $W=20\,(2.5-1-\log 2.5)=20\cdot 0.584=11.67$ . This value is greater than $3.84$ , so $H_{0}\colon\theta=1$ is rejected: the data are not compatible with a mean lifetime of one year.

This completes the solution. ∎

Solution to exercise 31.2.

1.

From example 11.8 we know that $\mathcal{I}(\theta)=1/\theta^{2}$ . Thus, the Jeffreys prior is $\pi_{J}(\theta)\propto\sqrt{\mathcal{I}(\theta)}=1/\theta$ . Since $\int_{0}^{1}\theta^{-1}\,\mathrm{d}\theta=\infty$ , the prior has infinite integral and thus is improper.
2.

The posterior density is proportional to $\theta^{-1}\cdot\theta^{n}e^{-\theta T}=\theta^{n-1}e^{-\theta T}$ . This is the Gamma $(n,T)$ density up to the constant $T^{n}/\Gamma(n)$ . Since $T>0$ , this constant is finite, so the posterior is proper and corresponds to the Gamma $(n,T)$ distribution.
3.

From exercise 11.5 we know that the information about the mean is $\mathcal{I}_{\mu}(\mu)=1/\mu^{2}$ . Thus, the Jeffreys prior for $\mu$ is $\pi_{J}(\mu)\propto 1/\mu$ . Using the transformation $\pi_{J}(\theta)$ with $\theta=1/\mu$ and $|\mathrm{d}\theta/\mathrm{d}\mu|=1/\mu^{2}$ we find the transformed density $\pi_{J}(1/\mu)/\mu^{2}\propto\mu/\mu^{2}=1/\mu$ . This is the same prior as $\pi_{J}(\mu)$ above, as expected from proposition 26.9.
4.

For $\theta\sim\text{Gamma}(n,T)$ we have $2T\theta\sim\text{Gamma}(n,1/2)=\chi^{2}_{2n}$ and thus the equal-tailed $95\%$ credible interval is $[\chi^{2}_{2n,0.025}/(2T),\ \chi^{2}_{2n,0.975}/(2T)]$ . These are the same numbers as for the exact confidence interval. The confidence interval covers the fixed value $\theta$ in $95\%$ of repeated samples. The credible interval, on the other hand, covers the random variable $\theta$ with posterior probability $0.95$ given the observed data.