Skip to content.
MATH5703M Statistical Theory
Introduction
1
Statistical Models and Estimation
2
Bias, Mean Squared Error and Consistency
3
R Basics and a First Simulation
4
Likelihood and the Method of Moments
5
Maximum Likelihood Estimation
6
Estimators by Simulation
7
Exponential Families
8
Sufficiency
9
Consistency by Simulation
10
Improving Estimators: Rao–Blackwell and Lehmann–Scheffé
11
Fisher Information
12
Sampling Distributions and Standard Errors
13
The Cramér–Rao Inequality
14
Multiparameter Fisher Information
15
Cramér–Rao and Efficiency in R
16
Consistency of the MLE
17
Asymptotic Normality and Efficiency of the MLE
18
Maximum Likelihood in R
19
The EM Algorithm
20
Properties of the EM Algorithm
21
EM in R
22
Hypothesis Testing
23
The Neyman–Pearson Lemma
24
Power and Size by Simulation
25
Likelihood-Ratio Tests
26
Confidence Intervals
27
Coverage and Level by Simulation
28
Bayesian Inference: Priors and Posteriors
29
Credible Intervals and Bayesian Testing
30
Confidence Versus Credibility
31
Standard Tests Revisited
32
The Exponential Model: Start to Finish
Preparing for the Exam
A
Background from Probability
B
Introduction to R
C
Solutions to the Exercises
Bibliography
Index
MATH5703M Statistical Theory
Jochen Voss
Contents
Introduction
1
Statistical Models and Estimation
1.1
Statistical Models
1.2
Statistics and Estimators
2
Bias, Mean Squared Error and Consistency
2.1
Bias and Unbiasedness
2.2
Mean Squared Error
2.3
When Unbiasedness Is Not Enough
2.4
Consistency
3
R Basics and a First Simulation
3.1
First Steps in R
3.2
Simulating a Normal Sample
3.3
The Sample Mean as a Random Quantity
4
Likelihood and the Method of Moments
4.1
The Likelihood Function
4.2
The Score Function
4.3
The Method of Moments
4.4
Identifiability
5
Maximum Likelihood Estimation
5.1
The Maximum Likelihood Estimator
5.2
Worked Examples
5.3
When the Recipe Fails
5.4
Invariance Under Reparametrisation
6
Estimators by Simulation
6.1
Functions and Loops
6.2
Empirical Bias and Mean Squared Error
6.3
Comparing Two Estimators
7
Exponential Families
7.1
Recognising the Shape
7.2
Several Parameters
7.3
The Natural Parametrisation and the Cumulant Function
7.4
What the Structure Buys
8
Sufficiency
8.1
Sufficient Statistics
8.2
The Factorisation Theorem
8.3
Sufficiency in Exponential Families
8.4
The Sample Maximum and Minimum
8.5
Minimal Sufficiency and Completeness
9
Consistency by Simulation
9.1
A Running Estimate
9.2
The Sampling Distribution as
n
n
Grows
9.3
The Mean Squared Error as a Function of the Sample Size
9.4
A Sufficient Statistic in R
9.5
A Task to Take Home
10
Improving Estimators: Rao–Blackwell and Lehmann–Scheffé
10.1
The Rao–Blackwell Theorem
10.2
Completeness and the Lehmann–Scheffé Theorem
11
Fisher Information
11.1
Setting and Assumptions
11.2
The Score Has Mean Zero
11.3
Fisher Information
11.4
Examples
11.5
Reparametrisation
11.6
When the Identities Fail
12
Sampling Distributions and Standard Errors
12.1
The Sampling Distribution of an Estimator
12.2
Standard Errors and a Wald Interval
12.3
Estimating a Probability by Simulation
12.4
A Task to Take Home
13
The Cramér–Rao Inequality
13.1
The Inequality
13.2
Efficiency and Attainment
13.3
When the Conditions Fail
14
Multiparameter Fisher Information
14.1
The Score Vector and the Information Matrix
14.2
Properties of the Information Matrix
14.3
The Cramér–Rao Bound for Vector Parameters
14.4
Nuisance Parameters
15
Cramér–Rao and Efficiency in R
15.1
An Efficient Estimator Against the Bound
15.2
An Estimator Not Attaining the Bound
15.3
A Task to Take Home
16
Consistency of the MLE
16.1
The Regularity Conditions
16.2
Comparing the True Value with a Wrong Value
16.3
Consistency
16.4
When the Assumptions Fail
17
Asymptotic Normality and Efficiency of the MLE
17.1
The Theorem
17.2
Proof of the Theorem
17.3
Standard Errors
17.4
Examples
17.5
Quality of the Approximation
17.6
Vector Parameters
18
Maximum Likelihood in R
18.1
The Log-Likelihood as an R Function
18.2
Standard Errors from the Observed Information
18.3
Starting Values
18.4
Coverage of the Wald Interval
18.5
A Task to Take Home
19
The EM Algorithm
19.1
Incomplete Data
19.2
The Algorithm
20
Properties of the EM Algorithm
20.1
The Ascent Property
20.2
Practical Matters
21
EM in R
21.1
The E and M Steps as R Functions
21.2
A Larger Sample
21.3
Label Switching and a Poor Start
21.4
A Task to Take Home
22
Hypothesis Testing
22.1
Hypotheses
22.2
Tests, Critical Regions and Errors
22.3
The Power Function, Size and Level
22.4
Testing the Mean of a Normal Sample
22.5
The
p
p
-Value
23
The Neyman–Pearson Lemma
23.1
Simple Hypotheses and Most Powerful Tests
23.2
The Lemma
23.3
The Normal Mean
23.4
One-Sided Alternatives
23.5
Two-Sided Alternatives
24
Power and Size by Simulation
24.1
The Size of a Test by Simulation
24.2
The Power Curve
24.3
Looking at the Data More Than Once
24.4
A Task to Take Home
25
Likelihood-Ratio Tests
25.1
The Generalised Likelihood Ratio
25.2
The Two-Sided Test for the Mean of a Normal Distribution
25.3
Wilks’ Theorem
25.4
Using the Theorem
26
Confidence Intervals
26.1
Definition and Interpretation
26.2
Pivotal Quantities
26.3
Inverting a Test
26.4
Approximate Intervals from the MLE
26.5
One-Sided Intervals and the Choice of Interval
27
Coverage and Level by Simulation
27.1
Coverage by Simulation
27.2
Wilks’ Theorem in Action
27.3
A Task to Take Home
28
Bayesian Inference: Priors and Posteriors
28.1
Prior, Likelihood and Posterior
28.2
Conjugate Priors
28.3
Point Estimates from the Posterior
28.4
Flat, Improper and Jeffreys Priors
29
Credible Intervals and Bayesian Testing
29.1
Credible Intervals
29.2
Bayesian Testing
29.3
Comparing Frequentist and Bayesian Answers
30
Confidence Versus Credibility
30.1
The Same Data, Two Intervals
30.2
From Prior to Posterior
30.3
A Task to Take Home
31
Standard Tests Revisited
31.1
Tests from Pivots
31.2
The One-Sample
t
t
-Test
31.3
The Chi-Squared Test for a Variance
31.4
Paired and Two-Sample Problems
31.5
Which Test?
32
The Exponential Model: Start to Finish
32.1
The Model and the Data
32.2
Point Estimation
32.3
Interval Estimation
32.4
Testing
32.5
The Bayesian Answer
32.6
Looking Back and Looking Forward
Preparing for the Exam
A
Background from Probability
A.1
Expectation, Variance and Independence
A.2
Standard Distributions
A.3
Sampling from a Normal Population
A.4
Some Inequalities
A.5
Conditional Expectation
A.6
Convergence and Limit Theorems
A.7
Transformation of a Random Variable
A.8
Bayes’ Theorem
B
Introduction to R
B.1
Getting Started
B.2
Vectors and Arithmetic
B.3
Lists
B.4
Random Numbers and Distributions
B.5
Writing Your Own Code
B.6
Estimation by Simulation
B.7
Plotting, and Getting Results into a Report
B.8
Numerical Maximum Likelihood
C
Solutions to the Exercises
C.1
Statistical Models and the Estimation Problem
C.2
Bias, Mean Squared Error and Consistency
C.3
Likelihood and the Method of Moments
C.4
Maximum Likelihood Estimation
C.5
Exponential Families
C.6
Sufficiency
C.7
Rao–Blackwell and Lehmann–Scheffé
C.8
Fisher Information
C.9
The Cramér–Rao Inequality
C.10
Multiparameter Fisher Information
C.11
Consistency of the MLE
C.12
Asymptotic Normality of the MLE
C.13
The EM Algorithm
C.14
Properties of the EM Algorithm
C.15
Hypothesis Testing
C.16
The Neyman–Pearson Lemma
C.17
Likelihood-Ratio Tests
C.18
Confidence Intervals
C.19
Bayesian Inference: Priors and Posteriors
C.20
Credible Intervals and Bayesian Testing
C.21
Standard Tests Revisited
C.22
The Exponential Model: From Start to Finish
Bibliography
Index