Preface xix

Preface to the First Edition xxi

1 Preliminaries 1

1.1 Probability and Bayes' Theorem 1

1.1.1 Notation 1

1.1.2 Axioms for probability 2

1.1.3 'Unconditional' probability 5

1.1.4 Odds 6

1.1.5 Independence 7

1.1.6 Some simple consequences of the axioms; Bayes' Theorem 7

1.2 Examples on Bayes' Theorem 9

1.2.1 The Biology of Twins 9

1.2.2 A political example 10

1.2.3 A warning 10

1.3 Random variables 12

1.3.1 Discrete random variables 12

1.3.2 The binomial distribution 13

1.3.3 Continuous random variables 14

1.3.4 The normal distribution 16

1.3.5 Mixed random variables 17

1.4 Several random variables 17

1.4.1 Two discrete random variables 17

1.4.2 Two continuous random variables 18

1.4.3 Bayes' Theorem for random variables 20

1.4.4 Example 21

1.4.5 One discrete variable and one continuous variable 21

1.4.6 Independent random variables 22

1.5 Means and variances 23

1.5.1 Expectations 23

1.5.2 The expectation of a sum and of a product 24

1.5.3 Variance, precision and standard deviation 25

1.5.4 Examples 25

1.5.5 Variance of a sum; covariance and correlation 27

1.5.6 Approximations to the mean and variance of a function of a random variable 28

1.5.7 Conditional expectations and variances 29

1.5.8 Medians and modes 31

1.6 Exercises on Chapter 1 31

2 Bayesian inference for the normal distribution 36

2.1 Nature of Bayesian inference 36

2.1.1 Preliminary remarks 36

2.1.2 Post is prior times likelihood 36

2.1.3 Likelihood can be multiplied by any constant 38

2.1.4 Sequential use of Bayes' Theorem 38

2.1.5 The predictive distribution 39

2.1.6 A warning 39

2.2 Normal prior and likelihood 40

2.2.1 Posterior from a normal prior and likelihood 40

2.2.2 Example 42

2.2.3 Predictive distribution 43

2.2.4 The nature of the assumptions made 44

2.3 Several normal observations with a normal prior 44

2.3.1 Posterior distribution 44

2.3.2 Example 46

2.3.3 Predictive distribution 47

2.3.4 Robustness 47

2.4 Dominant likelihoods 48

2.4.1 Improper priors 48

2.4.2 Approximation of proper priors by improper priors 49

2.5 Locally uniform priors 50

2.5.1 Bayes' postulate 50

2.5.2 Data translated likelihoods 52

2.5.3 Transformation of unknown parameters 52

2.6 Highest density regions 54

2.6.1 Need for summaries of posterior information 54

2.6.2 Relation to classical statistics 55

2.7 Normal variance 55

2.7.1 A suitable prior for the normal variance 55

2.7.2 Reference prior for the normal variance 58

2.8 HDRs for the normal variance 59

2.8.1 What distribution should we be considering? 59

2.8.2 Example 59

2.9 The role of sufficiency 60

2.9.1 Definition of sufficiency 60

2.9.2 Neyman's factorization theorem 61

2.9.3 Sufficiency principle 63

2.9.4 Examples 63

2.9.5 Order statistics and minimal sufficient statistics 65

2.9.6 Examples on minimal sufficiency 66

2.10 Conjugate prior distributions 67

2.10.1 Definition and difficulties 67

2.10.2 Examples 68

2.10.3 Mixtures of conjugate densities 69

2.10.4 Is your prior really conjugate? 71

2.11 The exponential family 71

2.11.1 Definition 71

2.11.2 Examples 72

2.11.3 Conjugate densities 72

2.11.4 Two-parameter exponential family 73

2.12 Normal mean and variance both unknown 73

2.12.1 Formulation of the problem 73

2.12.2 Marginal distribution of the mean 75

2.12.3 Example of the posterior density for the mean 76

2.12.4 Marginal distribution of the variance 77

2.12.5 Example of the posterior density of the variance 77

2.12.6 Conditional density of the mean for given variance 77

2.13 Conjugate joint prior for the normal distribution 78

2.13.1 The form of the conjugate prior 78

2.13.2 Derivation of the posterior 80

2.13.3 Example 81

2.13.4 Concluding remarks 82

2.14 Exercises on Chapter 2 82

3 Some other common distributions 85

3.1 The binomial distribution 85

3.1.1 Conjugate prior 85

3.1.2 Odds and log-odds 88

3.1.3 Highest density regions 90

3.1.4 Example 91

3.1.5 Predictive distribution 92

3.2 Reference prior for the binomial likelihood 92

3.2.1 Bayes' postulate 92

3.2.2 Haldane's prior 93

3.2.3 The arc-sine distribution 94

3.2.4 Conclusion 95

3.3 Jeffreys' rule 96

3.3.1 Fisher's information 96

3.3.2 The information from several observations 97

3.3.3 Jeffreys' prior 98

3.3.4 Examples 98

3.3.5 Warning 100

3.3.6 Several unknown parameters 100

3.3.7 Example 101

3.4 The Poisson distribution 102

3.4.1 Conjugate prior 102

3.4.2 Reference prior 103

3.4.3 Example 104

3.4.4 Predictive distribution 104

3.5 The uniform distribution 106

3.5.1 Preliminary definitions 106

3.5.2 Uniform distribution with a fixed lower endpoint 107

3.5.3 The general uniform distribution 108

3.5.4 Examples 110

3.6 Reference prior for the uniform distribution 110

3.6.1 Lower limit of the interval fixed 110

3.6.2 Example 111

3.6.3 Both limits unknown 111

3.7 The tramcar problem 113

3.7.1 The discrete uniform distribution 113

3.8 The first digit problem; invariant priors 114

3.8.1 A prior in search of an explanation 114

3.8.2 The problem 114

3.8.3 A solution 115

3.8.4 Haar priors 117

3.9 The circular normal distribution 117

3.9.1 Distributions on the circle 117

3.9.2 Example 119

3.9.3 Construction of an HDR by numerical integration 120

3.9.4 Remarks 122

3.10 Approximations based on the likelihood 122

3.10.1 Maximum likelihood 122

3.10.2 Iterative methods 123

3.10.3 Approximation to the posterior density 123

3.10.4 Examples 124

3.10.5 Extension to more than one parameter 126

3.10.6 Example 127

3.11 Reference posterior distributions 128

3.11.1 The information provided by an experiment 128

3.11.2 Reference priors under asymptotic normality 130

3.11.3 Uniform distribution of unit length 131

3.11.4 Normal mean and variance 132

3.11.5 Technical complications 134

3.12 Exercises on Chapter 3 134

4 Hypothesis testing 138

4.1 Hypothesis testing 138

4.1.1 Introduction 138

4.1.2 Classical hypothesis testing 138

4.1.3 Difficulties with the classical approach 139

4.1.4 The Bayesian approach 140

4.1.5 Example 142

4.1.6 Comment 143

4.2 One-sided hypothesis tests 143

4.2.1 Definition 143

4.2.2 P-values 144

4.3 Lindley's method 145

4.3.1 A compromise with classical statistics 145

4.3.2 Example 145

4.3.3 Discussion 146

4.4 Point (or sharp) null hypotheses with prior information 146

4.4.1 When are point null hypotheses reasonable? 146

4.4.2 A case of nearly constant likelihood 147

4.4.3 The Bayesian method for point null hypotheses 148

4.4.4 Sufficient statistics 149

4.5 Point null hypotheses for the normal distribution 150

4.5.1 Calculation of the Bayes' factor 150

4.5.2 Numerical examples 151

4.5.3 Lindley's paradox 152

4.5.4 A bound which does not depend on the prior distribution 154

4.5.5 The case of an unknown variance 155

4.6 The Doogian philosophy 157

4.6.1 Description of the method 157

4.6.2 Numerical example 157

4.7 Exercises on Chapter 4 158

5 Two-sample problems 162

5.1 Two-sample problems - both variances unknown 162

5.1.1 The problem of two normal samples 162

5.1.2 Paired comparisons 162

5.1.3 Example of a paired comparison problem 163

5.1.4 The case where both variances are known 163

5.1.5 Example 164

5.1.6 Non-trivial prior information 165

5.2 Variances unknown but equal 165

5.2.1 Solution using reference priors 165

5.2.2 Example 167

5.2.3 Non-trivial prior information 167

5.3 Variances unknown and unequal (Behrens-Fisher problem) 168

5.3.1 Formulation of the problem 168

5.3.2 Patil's approximation 169

5.3.3 Example 170

5.3.4 Substantial prior information 170

5.4 The Behrens-Fisher controversy 171

5.4.1 The Behrens-Fisher problem from a classical standpoint 171

5.4.2 Example 172

5.4.3 The controversy 173

5.5 Inferences concerning a variance ratio 173

5.5.1 Statement of the problem 173

5.5.2 Derivation of the F distribution 174

5.5.3 Example 175

5.6 Comparison of two proportions; the 2 × 2 table 176

5.6.1 Methods based on the log-odds ratio 176

5.6.2 Example 177

5.6.3 The inverse root-sine transformation 178

5.6.4 Other methods 178

5.7 Exercises on Chapter 5 179

6 Correlation, regression and the analysis of variance 182

6.1 Theory of the correlation coefficient 182

6.1.1 Definitions 182

6.1.2 Approximate posterior distribution of the correlation coefficient 184

6.1.3 The hyperbolic tangent substitution 186

6.1.4 Reference prior 188

6.1.5 Incorporation of prior information 189

6.2 Examples on the use of the correlation coefficient 189

6.2.1 Use of the hyperbolic tangent transformation 189

6.2.2...