Lecture 2 Bias, Mean Squared Error and Consistency

The most basic question we can ask about an estimator is whether it gives the correct answer on average.

The bias in general depends on the true value $\theta$, as shown by the expectation with $\theta$ as a subscript.

The following lemma shows that the sample mean is an unbiased estimator for the mean. For future reference, we also compute the variance of the sample mean here.

Whenever the parameter of interest is the mean of the distribution $P_{\theta}$, i.e. $\theta=\mu$, then lemma 2.2 shows that $\mathbb{E}_{\theta}(\bar{X})=\theta$ and thus $\mathop{\mathrm{bias}}\nolimits(\bar{X})=\mu-\mu=0$.

A more delicate example, which we will consider again in section 2.3, is the problem of estimating a variance. If a sample is observed with unknown mean, then one can estimate the variance $\sigma^{2}$ using the sample variance

The unexpected divisor $n-1$ instead of the seemingly more obvious $n$ is required to make this estimator unbiased.

The calculation also shows what is wrong with the divisor $n$: the estimator $(1/n)\sum_{i}(X_{i}-\bar{X})^{2}$ has expectation $(n-1)\sigma^{2}/n$ and thus systematically underestimates the variance, since the observations are on average slightly closer to the sample mean $\bar{X}$ than the true mean $\mu$ would be. We will return to this comparison, and to the surprising fact that the biased estimator is in some sense better, in section 2.3.

The bias of this estimator is small if the sample is large. A property which is weaker than unbiasedness, but which holds for large sample sizes, is that the bias goes to zero in the limit.

If an estimator is unbiased, this does not mean that it is accurate. The mean squared error is a numerical measure of the typical size of the error $\hat{\theta}-\theta$.

We want the mean squared error to be small. It combines two different kinds of defects, a systematic offset and random scatter, and the following decomposition separates the two.

The decomposition allows us to understand the rôle of unbiasedness: for an unbiased estimator, the bias term disappears and we have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta})=\mathop{\mathrm{Var}}\nolimits(\hat{\theta})$, and thus, when comparing two unbiased estimators, we can only consider the variances. This is the basis of all optimality theory: among the unbiased estimators, the best one is the one with the smallest variance. In lecture 13 we will see that there is a lower bound for the variance of an unbiased estimator.

Unbiasedness is an appealing property, but it is a constraint, not a goal, and insisting on it can cost us accuracy. We will show that for the variance of a normal sample the unbiased sample variance is beaten, in terms of mean squared error, by a biased competitor.

Let $X_{1},\dots,X_{n}\sim N(\mu,\sigma^{2})$ be i.i.d. and consider the family of estimators

for every constant $c>0$. For $c=1/(n-1)$ this is the unbiased sample variance $S^{2}$ and for $c=1/n$ it is the underestimating version from the previous section. To compare these estimators, we need to know the distribution of the sum of squares. From proposition A.4 in appendix A we know that the scaled sum of squares satisfies

We first use this result to find the mean squared error of $\hat{\sigma}^{2}_{c}$ for a general constant $c$. We write $W=\sum_{i=1}^{n}(X_{i}-\bar{X})^{2}$, so that $\hat{\sigma}^{2}_{c}=cW$. Since the $\chi^{2}_{n-1}$ distribution has mean $n-1$ and variance $2(n-1)$ by lemma A.3, we have

Multiplying by $c$, we find that the estimator has expectation $\mathbb{E}(\hat{\sigma}^{2}_{c})=c(n-1)\sigma^{2}$ and variance $\mathop{\mathrm{Var}}\nolimits(\hat{\sigma}^{2}_{c})=c^{2}\mathop{\mathrm{Var}}\nolimits(W)=2(n-1)c^{2}\sigma^{4}$. Thus its bias is given by

and lemma 2.7 gives

This formula holds for every $c>0$, and we can now use it to compare the two estimators.

If we choose $c=1/(n-1)$, the bias term disappears and we get

If we choose $c=1/n$, then $c(n-1)-1=-1/n$ and equation (2.2) gives

An easy calculation shows that $(2n-1)/n^{2}<2/(n-1)$ for all $n$. Thus, for every sample size and every value of $\sigma^{2}$, the biased estimator $\hat{\sigma}^{2}_{1/n}$ has smaller MSE than the unbiased estimator $S^{2}$.

By allowing a small, controlled bias, we have been able to reduce the variance and thus have strictly improved the mean squared error. The lesson to be learned here is not that bias is good, but that an estimator should be evaluated using its mean squared error instead of just the bias.

The mean squared errors we have computed all shrink as the sample grows. Consistency is the property which describes this limiting behaviour: as the number of observations increases, a consistent estimator converges to the true value.

Here we write $\xrightarrow{\ \mathrm{p}\ }$ for convergence in probability, as recalled in definition A.12 in appendix A: the probability of the estimate being off the truth by any fixed amount eventually becomes negligible.

To show that an estimator is consistent, we need an upper bound for the probability in definition 2.9. The basic tool for this is Markov’s inequality, which bounds the probability of a non-negative random variable being large in terms of its mean.

Applying Markov’s inequality to the squared distance between the estimator and the parameter gives a criterion for consistency in terms of the mean squared error.

The most basic example of a consistent estimator is the sample mean, where consistency is a manifestation of the law of large numbers.

Combining this criterion with the bias-variance decomposition gives a sufficient condition in terms of the two quantities we know how to compute.

All estimators in this lecture where we have computed a mean squared error have $\mathop{\mathrm{MSE}}\nolimits(\hat{\theta}_{n})$ of order $1/n$ and thus are consistent. Consistency is a coarse property: two consistent estimators can have very different rates of convergence and the mean squared error is a more detailed measure.