Skip to content.
MATH5703M Statistical Theory
Introduction
1
Statistical Models and Estimation
2
Bias, Mean Squared Error and Consistency
3
R Basics and a First Simulation
4
Likelihood and the Method of Moments
5
Maximum Likelihood Estimation
6
Estimators by Simulation
7
Exponential Families
8
Sufficiency
9
Consistency by Simulation
10
Improving Estimators: Rao–Blackwell and Lehmann–Scheffe
11
Fisher Information
12
Sampling Distributions and Standard Errors
13
The Cramer–Rao Inequality
14
Multiparameter Fisher Information
15
Cramer–Rao and Efficiency in R
16
Consistency of the MLE
17
Asymptotic Normality and Efficiency of the MLE
18
Maximum Likelihood in R
19
The EM Algorithm
20
Hypothesis Testing
21
EM in R
22
The Neyman–Pearson Lemma
23
Likelihood-Ratio Tests
24
Power and Size by Simulation
25
Confidence Intervals
26
Bayesian Inference: Priors and Posteriors
27
Confidence Versus Credibility
28
Credible Intervals and Bayesian Testing
29
Standard Tests Revisited
30
Do the Standard Tests Work?
31
The Exponential Model: Start to Finish
Preparing for the Exam
A
Background from Probability
B
Introduction to R
C
Solutions to the Exercises
Bibliography
Index
MATH5703M Statistical Theory
Jochen Voss
Contents
Introduction
1
Statistical Models and Estimation
1.1
Statistical Models
1.2
Statistics and Estimators
1.3
The Road Ahead
2
Bias, Mean Squared Error and Consistency
2.1
Bias and Unbiasedness
2.2
Mean Squared Error
2.3
When Unbiasedness Is Not Enough
2.4
Consistency
3
R Basics and a First Simulation
3.1
First Steps in R
3.2
Simulating a Normal Sample
3.3
The Sample Mean as a Random Quantity
4
Likelihood and the Method of Moments
4.1
The Likelihood Function
4.2
The Score Function
4.3
The Method of Moments
4.4
Identifiability
5
Maximum Likelihood Estimation
5.1
The Maximum Likelihood Estimator
5.2
Worked Examples
5.3
When the Recipe Fails
5.4
Bias of the Maximum Likelihood Estimator
5.5
Invariance Under Reparametrisation
6
Estimators by Simulation
6.1
Functions and Loops
6.2
Empirical Bias and Mean Squared Error
6.3
Comparing Two Estimators
7
Exponential Families
7.1
Recognising the Shape
7.2
The Natural Parametrisation and the Cumulant Function
7.3
Several Parameters
7.4
What the Structure Buys
8
Sufficiency
8.1
Sufficient Statistics
8.2
The Factorisation Theorem
8.3
Sufficiency in Exponential Families
8.4
Minimal Sufficiency and Completeness
8.5
The Sample Maximum and Minimum
9
Consistency by Simulation
9.1
A Running Estimate
9.2
The Mean Squared Error as a Function of the Sample Size
9.3
A Sufficient Statistic in R
9.4
A Task to Take Home
10
Improving Estimators: Rao–Blackwell and Lehmann–Scheffe
10.1
The Rao–Blackwell Theorem
10.2
Completeness and the Lehmann–Scheffe Theorem
10.3
Ancillary Statistics and Basu’s Theorem
11
Fisher Information
11.1
Setting and Assumptions
11.2
The Score Has Mean Zero
11.3
Fisher Information
11.4
Examples
11.5
Exponential Families
11.6
Reparametrisation
11.7
When the Identities Fail
12
Sampling Distributions and Standard Errors
12.1
The Sampling Distribution of an Estimator
12.2
Standard Errors and a Wald Interval
12.3
Estimating a Probability by Simulation
12.4
Skewed Data
13
The Cramer–Rao Inequality
13.1
The Inequality
13.2
Estimating a Function of the Parameter
13.3
Efficiency and Attainment
13.4
When the Conditions Fail
14
Multiparameter Fisher Information
14.1
The Score Vector and the Information Matrix
14.2
Properties of the Information Matrix
14.3
The Cramer–Rao Bound for Vector Parameters
14.4
The Normal Distribution
14.5
Nuisance Parameters
15
Cramer–Rao and Efficiency in R
15.1
An Efficient Estimator Against the Bound
15.2
Attaining and Missing the Bound
15.3
Beating the Bound Without Breaking It
15.4
A Task to Take Home
16
Consistency of the MLE
16.1
The Regularity Conditions
16.2
The Likelihood Ratio Detects the Truth
16.3
Consistency
16.4
When the Assumptions Fail
17
Asymptotic Normality and Efficiency of the MLE
17.1
The Theorem
17.2
Proof of the Theorem
17.3
Standard Errors
17.4
Examples
17.5
Quality of the Approximation
18
Maximum Likelihood in R
18.1
The Log-Likelihood as an R Function
18.2
Standard Errors from the Observed Information
18.3
Starting Values
18.4
Coverage of the Wald Interval
18.5
A Task to Take Home
19
The EM Algorithm
19.1
Incomplete Data
19.2
The Algorithm
19.3
The Ascent Property
19.4
A Two-Component Normal Mixture
19.5
Practical Matters
20
Hypothesis Testing
20.1
Hypotheses
20.2
Tests, Critical Regions and Errors
20.3
The Power Function, Size and Level
20.4
Testing the Mean of a Normal Sample
20.5
The
p
p
-Value
21
EM in R
21.1
The E and M Steps as R Functions
21.2
A Larger Sample
21.3
Label Switching and a Bad Start
21.4
A Task to Take Home
22
The Neyman–Pearson Lemma
22.1
Simple Hypotheses and Most Powerful Tests
22.2
The Lemma
22.3
The Normal Mean
22.4
One-Sided Alternatives
22.5
Two-Sided Alternatives
23
Likelihood-Ratio Tests
23.1
The Generalised Likelihood Ratio
23.2
The Two-Sided Test for the Mean of a Normal Distribution
23.3
Wilks’ Theorem
23.4
Using the Theorem
24
Power and Size by Simulation
24.1
The Size of a Test by Simulation
24.2
The Power Curve
24.3
Wilks’ Theorem in Action
24.4
A Task to Take Home
25
Confidence Intervals
25.1
Definition and Interpretation
25.2
Pivotal Quantities
25.3
Inverting a Test
25.4
Approximate Intervals from the MLE
25.5
One-Sided Intervals and the Choice of Interval
26
Bayesian Inference: Priors and Posteriors
26.1
Prior, Likelihood and Posterior
26.2
Conjugate Priors
26.3
Point Estimates from the Posterior
26.4
Flat, Improper and Jeffreys Priors
27
Confidence Versus Credibility
27.1
Coverage by Simulation
27.2
The Same Data, Two Intervals
27.3
From Prior to Posterior
28
Credible Intervals and Bayesian Testing
28.1
Credible Intervals
28.2
Bayesian Testing
28.3
Comparing Frequentist and Bayesian Answers
29
Standard Tests Revisited
29.1
Tests from Pivots
29.2
The One-Sample
t
t
-Test
29.3
The Chi-Squared Test for a Variance
29.4
Paired and Two-Sample Problems
29.5
Which Test?
30
Do the Standard Tests Work?
30.1
Real Data, Two Answers
30.2
The Replication Crisis in Miniature
30.3
A Stress Test
30.4
A Task to Take Home
31
The Exponential Model: Start to Finish
31.1
The Model and the Data
31.2
Point Estimation
31.3
Interval Estimation
31.4
Testing
31.5
The Bayesian Answer
31.6
Looking Back and Looking Forward
Preparing for the Exam
A
Background from Probability
A.1
Expectation, Variance and Independence
A.2
Standard Distributions
A.3
Sampling from a Normal Population
A.4
Some Inequalities
A.5
Conditional Expectation
A.6
Convergence and Limit Theorems
A.7
Transformation of a Random Variable
A.8
Bayes’ Theorem
B
Introduction to R
B.1
Getting Started
B.2
Vectors and Arithmetic
B.3
Lists
B.4
Random Numbers and Distributions
B.5
Writing Your Own Code
B.6
Estimation by Simulation
B.7
Plotting, and Getting Results into a Report
B.8
Numerical Maximum Likelihood
C
Solutions to the Exercises
C.1
Statistical Models and the Estimation Problem
C.2
Bias, Mean Squared Error and Consistency
C.3
Likelihood and the Method of Moments
C.4
Maximum Likelihood Estimation
C.5
Exponential Families
C.6
Sufficiency
C.7
Rao–Blackwell and Lehmann–Scheffe
C.8
Fisher Information
C.9
The Cramer–Rao Inequality
C.10
Multiparameter Fisher Information
C.11
Consistency of the MLE
C.12
Asymptotic Normality of the MLE
C.13
The EM Algorithm
C.14
Hypothesis Testing
C.15
The Neyman–Pearson Lemma
C.16
Likelihood-Ratio Tests
C.17
Confidence Intervals
C.18
Bayesian Inference: Priors and Posteriors
C.19
Credible Intervals and Bayesian Testing
C.20
Standard Tests Revisited
C.21
The Exponential Model: From Start to Finish
Bibliography
Index