Lecture 2 Bias, Mean Squared Error and Consistency

The most basic question we can ask about an estimator is whether it gives the correct answer on average.

The bias in general depends on the true value $\theta$, as shown by the expectation with $\theta$ as a subscript.

The workhorse example is the sample mean. Since we will use the first two moments frequently, we state them here for reference.

Whenever the parameter of interest is the mean of the observations, as for the Bernoulli and Poisson models from table 1.1, lemma 2.2 shows that $\mathbb{E}_{\theta}(\bar{X})=\theta$ and thus the sample mean is unbiased.

A more delicate example, which we will consider again in section 2.3, is the problem of estimating a variance. If a sample is observed with unknown mean, then one can estimate the variance $\sigma^{2}$ using the sample variance

The unexpected divisor $n-1$ instead of the seemingly more obvious $n$ is required to make this estimator unbiased.

The calculation also shows what is wrong with the divisor $n$: the estimator $(1/n)\sum_{i}(X_{i}-\bar{X})^{2}$ has expectation $(n-1)\sigma^{2}/n$ and thus systematically underestimates the variance, since the observations are on average slightly closer to the sample mean $\bar{X}$ than the true mean $\mu$ would be. We will return to this comparison, and to the surprising fact that the biased estimator is in some sense better, in section 2.3.

The bias of this estimator is small if the sample is large. A property which is weaker than unbiasedness, but which holds for large sample sizes, is that the bias goes to zero in the limit.

If an estimator is unbiased, this does not mean that it is accurate. The mean squared error is a numerical measure of the typical size of the error $\hat{\theta}-\theta$.

We want the mean squared error to be small. It combines two different kinds of defects, a systematic offset and random scatter, and the following decomposition separates the two.

The decomposition allows us to understand the rôle of unbiasedness: for an unbiased estimator, the bias term disappears and we have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\mathop{\mathrm{Var}}\nolimits(\hat{\theta})$, and thus, when comparing two unbiased estimators, we can only consider the variances. This is the basis of all optimality theory: among the unbiased estimators, the best one is the one with the smallest variance. In lecture 13 we will see that there is a lower bound for the variance of an unbiased estimator.

Using theorem 2.7 and lemma 2.2, we find that the mean squared error of the sample mean is given by $\mathop{\mathrm{MSE}}\nolimits(\bar{X})=\sigma^{2}/n$. For Bernoulli samples this gives $\mathop{\mathrm{MSE}}\nolimits(\bar{X})=\theta(1-\theta)/n$ and for Poisson samples $\mathop{\mathrm{MSE}}\nolimits(\bar{X})=\theta/n$. In both cases the mean squared error decreases as $1/n$: more data give a proportionally smaller error.

Unbiasedness is an appealing property, but it is a constraint, not a goal, and insisting on it can cost us accuracy. We will show that for the variance of a normal sample the unbiased sample variance is beaten, in terms of mean squared error, by a biased competitor.

Let $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ be i.i.d. and consider the family of estimators

for every constant $c>0$. For $c=1/(n-1)$ this is the unbiased sample variance $S^{2}$ and for $c=1/n$ it is the underestimating version from the previous section. To compare these estimators, we need to know the distribution of the sum of squares. From proposition A.3 in appendix A we know that the scaled sum of squares satisfies

If we write $W=\sum_{i}(X_{i}-\bar{X})^{2}$ and recall from definition A.2 that $\chi^{2}_{n-1}$ has mean $n-1$ and variance $2(n-1)$, we get

Thus, the estimator $\hat{\sigma}^{2}_{c}=cW$ has bias $\mathop{\mathrm{bias}}\nolimits(\hat{\sigma}^{2}_{c})=\bigl{(}c(n-1)-1\bigr{)}\sigma^{2}$ and, using theorem 2.7, we get

If we choose $c=1/(n-1)$, the bias term disappears and we get

If we choose $c=1/n$, then simplification of (2.2) gives

The biased estimator wins, since $(2n-1)/n^{2}<2/(n-1)$. A little algebra shows that this inequality is equivalent to $(2n-1)(n-1)<2n^{2}$, i.e. to $2n^{2}-3n+1<2n^{2}$, which holds for all $n\geq 2$.

By allowing a small, controlled bias, we have been able to reduce the variance and thus have strictly improved the mean squared error. The lesson to be learned here is not that bias is good, but that an estimator should be evaluated using its mean squared error instead of just the bias.

The mean squared errors we have computed all shrink as the sample grows. Consistency is the property which describes this limiting behaviour: as the number of observations increases, a consistent estimator converges to the true value.

Here we write $\xrightarrow{\ \mathrm{p}\ }$ for convergence in probability, as recalled in definition A.12 in appendix A: the probability of the estimate being off the truth by any fixed amount eventually becomes negligible. The most basic example of a consistent estimator is the sample mean, where consistency is a manifestation of the law of large numbers.

The argument used almost nothing about the sample mean except that its mean squared error converges to zero, and thus the same argument applies to any estimator with this property.

Combining this criterion with the bias-variance decomposition gives a sufficient condition in terms of the two quantities we know how to compute.

All estimators in this lecture where we have computed a mean squared error have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{n})$ of order $1/n$ and thus are consistent. Consistency is a coarse property: two consistent estimators can have very different rates of convergence and the mean squared error is a more detailed measure. We will see numerical evidence of consistency in the R session of lecture 9, and in lecture 16 we will see a proof that the maximum likelihood estimator is consistent for a wide class of cases.