Lecture 1 Statistical Models and the Estimation Problem

The data are given as $x_{1},\dots,x_{n}$, a list of $n$ numbers. Once we have computed an estimate from these numbers, we want to know how far it is from the truth, and this only makes sense if there is a true value to compare the estimate to. The data themselves do not contain a ‘true value’. Instead, this ground truth is provided by a model, a mathematical description of how the data were generated.

Our description of how the data were generated is called a model. We use a model to describe the data as the outcome of a random experiment. The data we have observed, $x_{1},\dots,x_{n}$, are seen as the observed values of random variables $X_{1},\dots,X_{n}$, where the distribution of the random variables depends on the unknown true value. Here we use the convention that upper case letters like $X_{i}$ stand for random variables, and that lower case letters like $x_{i}$ stand for the observed values for our dataset.

In the simplest and most common situation the observations are a random sample: the $X_{i}$ are independent and identically distributed (i.i.d.), but we do not know the distribution. Instead, we assume that the distribution belongs to a known family of distributions, labelled by a parameter.

Our assumption here is that the actually observed data, $x_{1},\dots,x_{n}$, are a sample from $\mathbb{P}_{\theta}$ for one (unknown to us) $\theta$. This value is called the true parameter value and is denoted by $\theta_{0}$. This is the value which our estimates are compared to: an estimate is good, if it is close to $\theta_{0}$, and bad if it is far away from $\theta_{0}$.

Since the distribution is determined by the parameter, learning about the distribution also means learning about the parameter. Thus, the diagram from the beginning of this lecture can be updated to

The true parameter value can never be observed directly, and all information about the true parameter value comes from $X_{1},\dots,X_{n}$.

The parameter $\theta$ does not necessarily have to be a single number. For example, if we assume that the data are normally distributed with unknown mean $\mu$ and unknown variance $\sigma^{2}$, then $\mathbb{P}_{\theta}=N(\mu,\sigma^{2})$ and the parameter is the pair $\theta=(\mu,\sigma^{2})$. In this case, $\Theta=\mathbb{R}\times(0,\infty)$ is a subset of the plane. While we allow for more general parameters from the start, most of the examples in this text will involve a scalar parameter $\theta$.

The distribution $\mathbb{P}_{\theta}$ describes, for every set $A$ of possible values, the probability $\mathbb{P}_{\theta}(A)$ that an observation falls into $A$. In practice, a distribution is almost always given by a function $f(x;\theta)$ of the possible values $x$, and this can be done in two different ways. For discrete distributions, where the observations can take finitely or countably many values, e.g. for $\{0,1\}$ or $\{0,1,2,\dots\}$, we have $f(x;\theta)=\mathbb{P}_{\theta}(\{x\})$, giving the probability of observing the value $x$. These numbers are called the probability weights of the distribution (sometimes also called the probability mass function). These weights are numbers in the interval $[0,1]$, summing to one, and satisfying

For continuous distributions, $f(x;\theta)$ is a density: a non-negative function, which is integrable and has integral one, such that

For continuous distributions, each value $x$ has probability $\mathbb{P}_{\theta}\bigl{(}\{x\}\bigr{)}=0$, and a density can take values larger than one. The function values $f(x;\theta)$ of a density are not probabilities, but integrals of the density over sets give probabilities.

In both cases, the family of functions $\bigl{(}f(x;\theta)\!\mathrel{\big{|}}\!\theta\in\Theta\bigr{)}$ determines the model, and we will often write the model in this form. Much of the theory in this module is common to both cases. Where a statement is true for both cases, we sometimes only state it for densities, and then the discrete case can be deduced by interpreting $f(x;\theta)$ as the probability weights and all integrals over $x$ as sums.

We use the notation $\mathbb{P}_{\theta}$ also for the probabilities of events involving the data, and we write $\mathbb{E}_{\theta}$ for expectations, to indicate that the result depends on the value of the parameter. For example, $\mathbb{P}_{\theta}(X_{1}\leq t)=\mathbb{P}_{\theta}\bigl{(}(-\infty,t]\bigr{)}$ is the probability that a single observation is at most $t$, when the data have distribution $\mathbb{P}_{\theta}$.

We will consider a small number of families of models throughout the module. The discrete families are listed in table 1.1 and the continuous ones in table 1.2. The corresponding means and variances, which we will need frequently in calculations, are shown in tables A.1 and A.2 in appendix A.

For families where $\theta$ is a vector, usually one component is assumed known and fixed, and only the remaining component is estimated. For example, for the binomial distribution the number $m$ of trials is often assumed to be known, and for the normal distribution the variance $\sigma^{2}$ is often assumed to be known.

Once a model is fixed, we want to extract information about the unknown $\theta$ from the data. Since we do not know the value of $\theta$, we need to consider quantities which can be computed from the data.

Under this restriction the sample mean $\bar{X}=(X_{1}+\dots+X_{n})/n$ is a statistic, but $X_{1}-\theta$ is not, since we need to know the parameter value to compute it.

Throughout the notes we will use hats to denote estimators: $\hat{\theta}$ is an estimator for $\theta$, $\hat{\sigma}^{2}$ is an estimator for $\sigma^{2}$, and so on. If two estimators for the same quantity are compared, the second one is denoted by a tilde: $\tilde{\theta}$ is a competitor of $\hat{\theta}$.

Most estimators are defined by a formula which works for all sample sizes, e.g. $\bar{X}=(X_{1}+\dots+X_{n})/n$. Technically, this formula defines a family of estimators $(\hat{\theta}_{n})_{n\in\mathbb{N}}$, one for each sample size $n$. When we need to consider the dependence on $n$, and in particular when we study the behaviour of an estimator as the amount of data increases, we write $\hat{\theta}_{n}$ instead of $\hat{\theta}$.

An important note is that the definition does not imply anything about quality. Any statistic is an estimator, for example the constant function $\hat{\theta}=0$ is a (bad) estimator for a Bernoulli success probability. We will discuss how to find and recognise good estimators in the following lectures.

The difference between an estimator and an estimate is a subtle but important one. An estimate is a single number, computed from the data, for example $\hat{\theta}=0.7$. An estimator is a rule for computing such a number, typically a function of the random variables $X_{1},\dots,X_{n}$, such that different outcomes of the experiment give rise to different values of the estimate.

We can only make statements about how good an estimator is, because $\hat{\theta}(X_{1},\dots,X_{n})$ is a random variable. The sampling distribution of an estimator is the distribution of the resulting estimates. We can ask questions like the following: If the data are generated by the true value $\theta_{0}$, how are the estimates distributed around $\theta_{0}$? Is the distribution centred around the truth? If so, how large is the spread? A good estimator will have a sampling distribution which is concentrated around the true parameter value, for all possible values of the true parameter. In the next lecture we will learn how to quantify these ideas using the concepts of bias and mean squared error.

Now that we have established the model and the estimator, we can start to build the module. The topics we will cover include point estimation (both likelihood and method of moments), measurement and comparison of estimators, the best possible estimators (sufficiency, information in a sample), and the limits of estimation. We will then consider statistical tests and the related question of compatibility of data with a given hypothesis about $\theta$. Finally, we will consider both estimation and testing from a Bayesian perspective, where the parameter is assumed to be random.