We write $\mathbb{E}(X)$ for the expectation of a random variable $X$ and $\mathop{\mathrm{Var}}\nolimits(X)$ for its variance. We assume that the basic rules of algebra for expectations and variances are known. For reference, we state the following rules for expectations and variances.

More generally, for independent $X_{1},\dots,X_{n}$ the variance of the sum is the sum of the individual variances. We use this fact, for example, when we average a random sample. Since these identities are standard, we do not give proofs of these results here.

We will consider a small number of distributions in the module. Tables A.1 and A.2 summarise the probability weights or density, support, mean and variance of these distributions, for the discrete and for the continuous distributions respectively. The families of distributions considered in tables 1.1 and 1.2 are all included, together with their moments. The negative binomial, Pareto and Rayleigh distributions do not appear in tables 1.1 and 1.2; these distributions are used in the R sessions as a source of simulated data. The tables use the usual parameter names, but in the main text these families of distributions will be considered as parametric models where the unknown parameter is denoted by $\theta$. For example, the rate parameter $\lambda$ of the exponential distribution below is the $\theta$ of table 1.2. Throughout the module the exponential and gamma distributions are parametrised by their rate, as in the table and as in R’s dexp and dgamma; some textbooks use the mean $1/\lambda$, or equivalently the scale, so a formula from elsewhere may need to be translated.

In the module we often need the distribution of a sum of independent random variables, for example when we compute the distribution of a total or of a sample mean. For several of the standard distributions the sum of independent copies again has one of the standard distributions, with only the parameters changed.

Another family, introduced in exercise 23.2 of lecture 23, is the multinomial distribution. If each of $n$ independent trials has one of $k$ possible categories, with probabilities $p_{1},\dots,p_{k}$, where the probabilities must sum to one, then the vector of counts $(N_{1},\dots,N_{k})$ is multinomial distributed with $\mathbb{E}(N_{j})=np_{j}$ and $\mathop{\mathrm{Var}}\nolimits(N_{j})=np_{j}(1-p_{j})$. The individual counts are Binomial$(n,p_{j})$ and the multinomial distribution is a generalisation of the binomial distribution to the case where there are more than two categories.

The following distributions are needed to describe the sampling distributions of statistics computed from a normal sample. They form the basis of the confidence intervals in lecture 25 and of the standard tests in lecture 29. The first of these distributions also arose in the comparison of estimators for the variance in lecture 2. We start our discussion by considering the chi-squared distribution.

The chi-squared distribution is a special case of the gamma distribution: the $\chi^{2}_{k}$ distribution corresponds to the $\text{Gamma}(k/2,1/2)$ distribution from table A.2, i.e. it has density $x^{k/2-1}e^{-x/2}/\bigl{(}2^{k/2}\Gamma(k/2)\bigr{)}$ for $x>0$. In statistics, the parametrisation in terms of degrees of freedom is more commonly used. The most important property of the chi-squared distribution for applications in statistics is that the sample mean and sample variance, being the two most common statistics for normally distributed samples, have known and independent distributions.

The independence of $\bar{X}$ and $S^{2}$ is the least obvious of these statements; we will prove this statement in lecture 10, as an application of Basu’s theorem.

We now construct two more distributions, using the normal distribution and the chi-squared distribution as building blocks. These distributions describe what happens when trying to estimate an unknown variance.

Using definition A.4 and proposition A.3 one can understand the origin of the $t$ statistic for a normal sample: by dividing the standardised mean by the square root of the scaled sample variance, the unknown $\sigma$ is replaced with its estimate and a $t_{n-1}$ variable is obtained. We will see this argument in lecture 25.

A small number of basic inequalities are used throughout the module, mostly to estimate the probability of an estimator being away from its target. The most basic of these inequalities is an upper tail bound for non-negative variables, in terms of the mean.

The following application of Markov’s inequality, showing that a bound on the mean can be converted into a bound on the spread, completes this section.

The following inequality, which establishes a relation between the expectation of a convex function and the function of the expectation, is the basis of the information inequality for maximum likelihood and of the ascent property in the EM algorithm.

Finally, the Cauchy–Schwarz inequality can be used to estimate a covariance in terms of variances. This result is, for example, used in the derivation of the Cramer–Rao lower bound in lecture 13.

When one random variable carries information about another, we summarise it through conditional expectation. Given random variables $X$ and $Y$, the conditional expectation $\mathbb{E}(X\mskip 1.0mu|\mskip 1.0muY)$ is the expectation of $X$ computed as if the value of $Y$ were known; because that value is itself random, $\mathbb{E}(X\mskip 1.0mu|\mskip 1.0muY)$ is a function of $Y$, and hence a random variable in its own right. Its own average recovers the ordinary expectation of $X$.

Since the value of $\mathbb{E}(X\mskip 1.0mu|\mskip 1.0muY)$ is computed from the value of $Y$, we can pull out all factors which are functions of $Y$ from the conditional expectation.

We already know the tower property and this rule. The only effect of conditioning can be to reduce the variance on average. The exact argument is given by the following decomposition, which we will use in lecture 10 when we improve an estimator by conditioning.

The large-sample theory of the module is based on two different concepts of a sequence of random variables converging to a limit. The weaker of the two concepts requires that the random variables are unlikely to differ from the limit by more than a fixed amount.

The weaker (but still useful) convergence of distribution functions is used to approximate a whole distribution:

The two great theorems of large-sample statistics describe the behaviour of a sample mean of independent, identically distributed observations. The first theorem gives the mean of the sample, the second describes the size and shape of the fluctuations around the mean.

The assumption of finite mean is important. The Cauchy distribution from table A.2 has no finite mean and the distribution of the sample mean is the same as the distribution of a single observation for all $n$. Thus, in this case nothing converges and this example is used in lecture 2 to argue that the sample mean is not consistent.

In order to combine these limits we will need rules which allow us to manipulate limits. Slutsky’s theorem allows us to treat a term which converges to a constant as the constant itself, and the continuous mapping theorem allows us to apply a continuous function to a limit.

An important application of this theorem is to apply the central limit theorem after applying a smooth transformation to the data, in order to determine the limiting distribution of a function of an asymptotically normal statistic. This method is known as the delta method. We will use the delta method in lecture 17 to compute standard errors for functions of the maximum likelihood estimator.

These five theorems are given without proof, and are from the material covered in a first course in probability. We use the theorems as a ready-made toolkit.

We sometimes know the distribution of a random variable and want to find the distribution of a smooth, one-to-one transformation of this random variable. The change-of-variable formula is a mathematical tool which allows us to find the density of the transformed random variable. This formula is, for example, used when we reparametrise a model, as discussed in the context of the Jeffreys prior in lecture 26.

The same argument applies to a vector of random variables: if $X$ has a joint density $f_{X}$ on $\mathbb{R}^{k}$ and $Y=g(X)$ for a differentiable, one-to-one map $g$ with inverse $h$, then $Y$ has joint density $f_{Y}(y)=f_{X}\bigl{(}h(y)\bigr{)}\,|\det h^{\prime}(y)|$, where $h^{\prime}(y)$ is the Jacobian matrix of $h$. For a linear map with determinant one, as in the solution to exercise 10.2 in appendix C, the factor is $1$.

The final tool reverses the direction of conditioning, expressing the conditional density of one variable given another in terms of the reverse conditional. It is the mechanism behind every posterior distribution in the Bayesian lectures.

If one of the two variables is discrete, the same formula holds with its probability weights in place of the density, and the integral in $f(b)$ becomes a sum over the values of $A$ when $A$ is discrete.

In the Bayesian lectures of the module, the unknown parameter plays the role of $A$ with prior density $f(a)$ and the data are given by $B$. In this situation, proposition A.20 states that the posterior density is proportional to the product of the likelihood and the prior density. The denominator $f(B)$ does not depend on the parameter. This viewpoint is further developed in lectures 26 and 28.