Statistical Pattern Recognition -- Contents -- Preface -- Notation -- 1 Introduction to Statistical Pattern Recognition -- 1.1 Statistical Pattern Recognition -- 1.1.1 Introduction -- 1.1.2 The Basic Model -- 1.2 Stages in a Pattern Recognition Problem -- 1.3 Issues -- 1.4 Approaches to Statistical Pattern Recognition -- 1.5 Elementary Decision Theory -- 1.5.1 Bayes' Decision Rule for Minimum Error -- 1.5.2 Bayes' Decision Rule for Minimum Error - Reject Option -- 1.5.3 Bayes' Decision Rule for Minimum Risk -- 1.5.4 Bayes' Decision Rule for Minimum Risk - Reject Option -- 1.5.5 Neyman-Pearson Decision Rule -- 1.5.6 Minimax Criterion -- 1.5.7 Discussion -- 1.6 Discriminant Functions -- 1.6.1 Introduction -- 1.6.2 Linear Discriminant Functions -- 1.6.3 Piecewise Linear Discriminant Functions -- 1.6.4 Generalised Linear Discriminant Function -- 1.6.5 Summary -- 1.7 Multiple Regression -- 1.8 Outline of Book -- 1.9 Notes and References -- Exercises -- 2 Density Estimation - Parametric -- 2.1 Introduction -- 2.2 Estimating the Parameters of the Distributions -- 2.2.1 Estimative Approach -- 2.2.2 Predictive Approach -- 2.3 The Gaussian Classifier -- 2.3.1 Specification -- 2.3.2 Derivation of the Gaussian Classifier Plug-In Estimates -- 2.3.3 Example Application Study -- 2.4 Dealing with Singularities in the Gaussian Classifier -- 2.4.1 Introduction -- 2.4.2 Naı̈ve Bayes -- 2.4.3 Projection onto a Subspace -- 2.4.4 Linear Discriminant Function -- 2.4.5 Regularised Discriminant Analysis -- 2.4.6 Example Application Study -- 2.4.7 Further Developments -- 2.4.8 Summary -- 2.5 Finite Mixture Models -- 2.5.1 Introduction -- 2.5.2 Mixture Models for Discrimination -- 2.5.3 Parameter Estimation for Normal Mixture Models -- 2.5.4 Normal Mixture Model Covariance Matrix Constraints -- 2.5.5 How Many Components? -- 2.5.6 Maximum Likelihood Estimation via EM.

2.5.7 Example Application Study -- 2.5.8 Further Developments -- 2.5.9 Summary -- 2.6 Application Studies -- 2.7 Summary and Discussion -- 2.8 Recommendations -- 2.9 Notes and References -- Exercises -- 3 Density Estimation - Bayesian -- 3.1 Introduction -- 3.1.1 Basics -- 3.1.2 Recursive Calculation -- 3.1.3 Proportionality -- 3.2 Analytic Solutions -- 3.2.1 Conjugate Priors -- 3.2.2 Estimating the Mean of a Normal Distribution with Known Variance -- 3.2.3 Estimating the Mean and the Covariance Matrix of a Multivariate Normal Distribution -- 3.2.4 Unknown Prior Class Probabilities -- 3.2.5 Summary -- 3.3 Bayesian Sampling Schemes -- 3.3.1 Introduction -- 3.3.2 Summarisation -- 3.3.3 Sampling Version of the Bayesian Classifier -- 3.3.4 Rejection Sampling -- 3.3.5 Ratio of Uniforms -- 3.3.6 Importance Sampling -- 3.4 Markov Chain Monte Carlo Methods -- 3.4.1 Introduction -- 3.4.2 The Gibbs Sampler -- 3.4.3 Metropolis-Hastings Algorithm -- 3.4.4 Data Augmentation -- 3.4.5 Reversible Jump Markov Chain Monte Carlo -- 3.4.6 Slice Sampling -- 3.4.7 MCMC Example - Estimation of Noisy Sinusoids -- 3.4.8 Summary -- 3.4.9 Notes and References -- 3.5 Bayesian Approaches to Discrimination -- 3.5.1 Labelled Training Data -- 3.5.2 Unlabelled Training Data -- 3.6 Sequential Monte Carlo Samplers -- 3.6.1 Introduction -- 3.6.2 Basic Methodology -- 3.6.3 Summary -- 3.7 Variational Bayes -- 3.7.1 Introduction -- 3.7.2 Description -- 3.7.3 Factorised Variational Approximation -- 3.7.4 Simple Example -- 3.7.5 Use of the Procedure for Model Selection -- 3.7.6 Further Developments and Applications -- 3.7.7 Summary -- 3.8 Approximate Bayesian Computation -- 3.8.1 Introduction -- 3.8.2 ABC Rejection Sampling -- 3.8.3 ABC MCMC Sampling -- 3.8.4 ABC Population Monte Carlo Sampling -- 3.8.5 Model Selection -- 3.8.6 Summary -- 3.9 Example Application Study.

3.10 Application Studies -- 3.11 Summary and Discussion -- 3.12 Recommendations -- 3.13 Notes and References -- Exercises -- 4 Density Estimation - Nonparametric -- 4.1 Introduction -- 4.1.1 Basic Properties of Density Estimators -- 4.2 k-Nearest-Neighbour Method -- 4.2.1 k-Nearest-Neighbour Classifier -- 4.2.2 Derivation -- 4.2.3 Choice of Distance Metric -- 4.2.4 Properties of the Nearest-Neighbour Rule -- 4.2.5 Linear Approximating and Eliminating Search Algorithm -- 4.2.6 Branch and Bound Search Algorithms: kd-Trees -- 4.2.7 Branch and Bound Search Algorithms: Ball-Trees -- 4.2.8 Editing Techniques -- 4.2.9 Example Application Study -- 4.2.10 Further Developments -- 4.2.11 Summary -- 4.3 Histogram Method -- 4.3.1 Data Adaptive Histograms -- 4.3.2 Independence Assumption (Naı̈ve Bayes) -- 4.3.3 Lancaster Models -- 4.3.4 Maximum Weight Dependence Trees -- 4.3.5 Bayesian Networks -- 4.3.6 Example Application Study - Naı̈ve Bayes Text Classification -- 4.3.7 Summary -- 4.4 Kernel Methods -- 4.4.1 Biasedness -- 4.4.2 Multivariate Extension -- 4.4.3 Choice of Smoothing Parameter -- 4.4.4 Choice of Kernel -- 4.4.5 Example Application Study -- 4.4.6 Further Developments -- 4.4.7 Summary -- 4.5 Expansion by Basis Functions -- 4.6 Copulas -- 4.6.1 Introduction -- 4.6.2 Mathematical Basis -- 4.6.3 Copula Functions -- 4.6.4 Estimating Copula Probability Density Functions -- 4.6.5 Simple Example -- 4.6.6 Summary -- 4.7 Application Studies -- 4.7.1 Comparative Studies -- 4.8 Summary and Discussion -- 4.9 Recommendations -- 4.10 Notes and References -- Exercises -- 5 Linear Discriminant Analysis -- 5.1 Introduction -- 5.2 Two-Class Algorithms -- 5.2.1 General Ideas -- 5.2.2 Perceptron Criterion -- 5.2.3 Fisher's Criterion -- 5.2.4 Least Mean-Squared-Error Procedures -- 5.2.5 Further Developments -- 5.2.6 Summary -- 5.3 Multiclass Algorithms.

5.3.1 General Ideas -- 5.3.2 Error-Correction Procedure -- 5.3.3 Fisher's Criterion - Linear Discriminant Analysis -- 5.3.4 Least Mean-Squared-Error Procedures -- 5.3.5 Regularisation -- 5.3.6 Example Application Study -- 5.3.7 Further Developments -- 5.3.8 Summary -- 5.4 Support Vector Machines -- 5.4.1 Introduction -- 5.4.2 Linearly Separable Two-Class Data -- 5.4.3 Linearly Nonseparable Two-Class Data -- 5.4.4 Multiclass SVMs -- 5.4.5 SVMs for Regression -- 5.4.6 Implementation -- 5.4.7 Example Application Study -- 5.4.8 Summary -- 5.5 Logistic Discrimination -- 5.5.1 Two-Class Case -- 5.5.2 Maximum Likelihood Estimation -- 5.5.3 Multiclass Logistic Discrimination -- 5.5.4 Example Application Study -- 5.5.5 Further Developments -- 5.5.6 Summary -- 5.6 Application Studies -- 5.7 Summary and Discussion -- 5.8 Recommendations -- 5.9 Notes and References -- Exercises -- 6 Nonlinear Discriminant Analysis - Kernel and Projection Methods -- 6.1 Introduction -- 6.2 Radial Basis Functions -- 6.2.1 Introduction -- 6.2.2 Specifying the Model -- 6.2.3 Specifying the Functional Form -- 6.2.4 The Positions of the Centres -- 6.2.5 Smoothing Parameters -- 6.2.6 Calculation of the Weights -- 6.2.7 Model Order Selection -- 6.2.8 Simple RBF -- 6.2.9 Motivation -- 6.2.10 RBF Properties -- 6.2.11 Example Application Study -- 6.2.12 Further Developments -- 6.2.13 Summary -- 6.3 Nonlinear Support Vector Machines -- 6.3.1 Introduction -- 6.3.2 Binary Classification -- 6.3.3 Types of Kernel -- 6.3.4 Model Selection -- 6.3.5 Multiclass SVMs -- 6.3.6 Probability Estimates -- 6.3.7 Nonlinear Regression -- 6.3.8 Example Application Study -- 6.3.9 Further Developments -- 6.3.10 Summary -- 6.4 The Multilayer Perceptron -- 6.4.1 Introduction -- 6.4.2 Specifying the MLP Structure -- 6.4.3 Determining the MLP Weights -- 6.4.4 Modelling Capacity of the MLP.

6.4.5 Logistic Classification -- 6.4.6 Example Application Study -- 6.4.7 Bayesian MLP Networks -- 6.4.8 Projection Pursuit -- 6.4.9 Summary -- 6.5 Application Studies -- 6.6 Summary and Discussion -- 6.7 Recommendations -- 6.8 Notes and References -- Exercises -- 7 Rule and Decision Tree Induction -- 7.1 Introduction -- 7.2 Decision Trees -- 7.2.1 Introduction -- 7.2.2 Decision Tree Construction -- 7.2.3 Selection of the Splitting Rule -- 7.2.4 Terminating the Splitting Procedure -- 7.2.5 Assigning Class Labels to Terminal Nodes -- 7.2.6 Decision Tree Pruning - Worked Example -- 7.2.7 Decision Tree Construction Methods -- 7.2.8 Other Issues -- 7.2.9 Example Application Study -- 7.2.10 Further Developments -- 7.2.11 Summary -- 7.3 Rule Induction -- 7.3.1 Introduction -- 7.3.2 Generating Rules from a Decision Tree -- 7.3.3 Rule Induction Using a Sequential Covering Algorithm -- 7.3.4 Example Application Study -- 7.3.5 Further Developments -- 7.3.6 Summary -- 7.4 Multivariate Adaptive Regression Splines -- 7.4.1 Introduction -- 7.4.2 Recursive Partitioning Model -- 7.4.3 Example Application Study -- 7.4.4 Further Developments -- 7.4.5 Summary -- 7.5 Application Studies -- 7.6 Summary and Discussion -- 7.7 Recommendations -- 7.8 Notes and References -- Exercises -- 8 Ensemble Methods -- 8.1 Introduction -- 8.2 Characterising a Classifier Combination Scheme -- 8.2.1 Feature Space -- 8.2.2 Level -- 8.2.3 Degree of Training -- 8.2.4 Form of Component Classifiers -- 8.2.5 Structure -- 8.2.6 Optimisation -- 8.3 Data Fusion -- 8.3.1 Architectures -- 8.3.2 Bayesian Approaches -- 8.3.3 Neyman-Pearson Formulation -- 8.3.4 Trainable Rules -- 8.3.5 Fixed Rules -- 8.4 Classifier Combination Methods -- 8.4.1 Product Rule -- 8.4.2 Sum Rule -- 8.4.3 Min, Max and Median Combiners -- 8.4.4 Majority Vote -- 8.4.5 Borda Count.

8.4.6 Combiners Trained on Class Predictions.