Optimal Learning -- CONTENTS -- Preface -- Acknowledgments -- 1 The Challenges of Learning -- 1.1 Learning the Best Path -- 1.2 Areas of Application -- 1.3 Major Problem Classes -- 1.4 The Different Types of Learning -- 1.5 Learning from Different Communities -- 1.6 Information Collection Using Decision Trees -- 1.6.1 A Basic Decision Tree -- 1.6.2 Decision Tree for Offline Learning -- 1.6.3 Decision Tree for Online Learning -- 1.6.4 Discussion -- 1.7 Website and Downloadable Software -- 1.8 Goals of this Book -- Problems -- 2 Adaptive Learning -- 2.1 The Frequentist View -- 2.2 The Bayesian View -- 2.2.1 The Updating Equations for Independent Beliefs -- 2.2.2 The Expected Value of Information -- 2.2.3 Updating for Correlated Normal Priors -- 2.2.4 Bayesian Updating with an Uninformative Prior -- 2.3 Updating for Non-Gaussian Priors -- 2.3.1 The Gamma-Exponential Model -- 2.3.2 The Gamma-Poisson Model -- 2.3.3 The Pareto-Uniform Model -- 2.3.4 Models for Learning Probabilities* -- 2.3.5 Learning an Unknown Variance* -- 2.4 Monte Carlo Simulation -- 2.5 Why Does It Work?* -- 2.5.1 Derivation of σ -- 2.5.2 Derivation of Bayesian Updating Equations for Independent Beliefs -- 2.6 Bibliographic Notes -- Problems -- 3 The Economics of Information -- 3.1 An Elementary Information Problem -- 3.2 The Marginal Value of Information -- 3.3 An information Acquisition Problem -- 3.4 Bibliographic Notes -- Problems -- 4 Ranking and Selection -- 4.1 The Model -- 4.2 Measurement Policies -- 4.2.1 Deterministic Versus Sequential Policies -- 4.2.2 Optimal Sequential Policies -- 4.2.3 Heuristic Policies -- 4.3 Evaluating Policies -- 4.4 More Advanced Topics* -- 4.4.1 An Alternative Representation of the Probability Space -- 4.4.2 Equivalence of Using True Means and Sample Estimates -- 4.5 Bibliographic Notes -- Problems -- 5 The Knowledge Gradient.

5.1 The Knowledge Gradient for Independent Beliefs -- 5.1.1 Computation -- 5.1.2 Some Properties of the Knowledge Gradient -- 5.1.3 The Four Distributions of Learning -- 5.2 The Value of Information and the S-Curve Effect -- 5.3 Knowledge Gradient for Correlated Beliefs -- 5.4 Anticipatory Versus Experiential Learning -- 5.5 The Knowledge Gradient for Some Non-Gaussian Distributions -- 5.5.1 The Gamma-Exponential Model -- 5.5.2 The Gamma-Poisson Model -- 5.5.3 The Pareto-Uniform Model -- 5.5.4 The Beta-Bernoulli Model -- 5.5.5 Discussion -- 5.6 Relatives of the Knowledge Gradient -- 5.6.1 Expected Improvement -- 5.6.2 Linear Loss* -- 5.7 The Problem of Priors -- 5.8 Discussion -- 5.9 Why Does It Work?* -- 5.9.1 Derivation of the Knowledge Gradient Formula -- 5.10 Bibliographic Notes -- Problems -- 6 Bandit Problems -- 6.1 The Theory and Practice of Gittins Indices -- 6.1.1 Gittins Indices in the Beta-Bernoulli Model -- 6.1.2 Gittins Indices in the Normal-Normal Model -- 6.1.3 Approximating Gittins Indices -- 6.2 Variations of Bandit Problems -- 6.3 Upper Confidence Bounding -- 6.4 The Knowledge Gradient for Bandit Problems -- 6.4.1 The Basic Idea -- 6.4.2 Some Experimental Comparisons -- 6.4.3 Non-Normal Models -- 6.5 Bibliographic Notes -- Problems -- 7 Elements of a Learning Problem -- 7.1 The States of our System -- 7.2 Types of Decisions -- 7.3 Exogenous Information -- 7.4 Transition Functions -- 7.5 Objective Functions -- 7.5.1 Designing Versus Controlling -- 7.5.2 Measurement Costs -- 7.5.3 Objectives -- 7.6 Evaluating Policies -- 7.7 Discussion -- 7.8 Bibliographic Notes -- Problems -- 8 Linear Belief Models -- 8.1 Applications -- 8.1.1 Maximizing Ad Clicks -- 8.1.2 Dynamic Pricing -- 8.1.3 Housing Loans -- 8.1.4 Optimizing Dose Response -- 8.2 A Brief Review of Linear Regression -- 8.2.1 The Normal Equations -- 8.2.2 Recursive Least Squares.

8.2.3 A Bayesian Interpretation -- 8.2.4 Generating a Prior -- 8.3 The Knowledge Gradient for a Linear Model -- 8.4 Application to Drug Discovery -- 8.5 Application to Dynamic Pricing -- 8.6 Bibliographic Notes -- Problems -- 9 Subset Selection Problems -- 9.1 Applications -- 9.2 Choosing a Subset Using Ranking and Selection -- 9.2.1 Setting Prior Means and Variances -- 9.2.2 Two Strategies for Setting Prior Covariances -- 9.3 Larger Sets -- 9.3.1 Using Simulation to Reduce the Problem Size -- 9.3.2 Computational Issues -- 9.3.3 Experiments -- 9.4 Very Large Sets -- 9.5 Bibliographic Notes -- Problems -- 10 Optimizing a Scalar Function -- 10.1 Deterministic Measurements -- 10.2 Stochastic Measurements -- 10.2.1 The Model -- 10.2.2 Finding the Posterior Distribution -- 10.2.3 Choosing the Measurement -- 10.2.4 Discussion -- 10.3 Bibliographic Notes -- Problems -- 11 Optimal Bidding -- 11.1 Modeling Customer Demand -- 11.1.1 Some Valuation Models -- 11.1.2 The Logit Model -- 11.2 Bayesian Modeling for Dynamic Pricing -- 11.2.1 A Conjugate Prior for Choosing Between Two Demand Curves -- 11.2.2 Moment Matching for Nonconjugate Problems -- 11.2.3 An Approximation for the Logit Model -- 11.3 Bidding Strategies -- 11.3.1 An Idea From Multi-Armed Bandits -- 11.3.2 Bayes-Greedy Bidding -- 11.3.3 Numerical Illustrations -- 11.4 Why Does It Work?* -- 11.4.1 Moment Matching for Pareto Prior -- 11.4.2 Approximating the Logistic Expectation -- 11.5 Bibliographic Notes -- Problems -- 12 Stopping Problems -- 12.1 Sequential Probability Ratio Test -- 12.2 The Secretary Problem -- 12.2.1 Setup -- 12.2.2 Solution -- 12.3 Bibliographic Notes -- Problems -- 13 Active Learning in Statistics -- 13.1 Deterministic Policies -- 13.2 Sequential Policies for Classification -- 13.2.1 Uncertainty Sampling -- 13.2.2 Query by Committee -- 13.2.3 Expected Error Reduction.

13.3 A Variance-Minimizing Policy -- 13.4 Mixtures of Gaussians -- 13.4.1 Estimating Parameters -- 13.4.2 Active Learning -- 13.5 Bibliographic Notes -- 14 Simulation Optimization -- 14.1 Indifference Zone Selection -- 14.1.1 Batch Procedures -- 14.1.2 Sequential Procedures -- 14.1.3 The 0-1 Procedure: Connection to Linear Loss -- 14.2 Optimal Computing Budget Allocation -- 14.2.1 Indifference-Zone Version -- 14.2.2 Linear Loss Version -- 14.2.3 When Does It Work? -- 14.3 Model-Based Simulated Annealing -- 14.4 Other Areas of Simulation Optimization -- 14.5 Bibliographic Notes -- 15 Learning in Mathematical Programming -- 15.1 Applications -- 15.1.1 Piloting a Hot Air Balloon -- 15.1.2 Optimizing a Portfolio -- 15.1.3 Network Problems -- 15.2 Learning on Graphs -- 15.3 Alternative Edge Selection Policies -- 15.4 Learning Costs for Linear Programs* -- 15.5 Bibliographic Notes -- 16 Optimizing Over Continuous Measurements -- 16.1 The Belief Model -- 16.1.1 Updating Equations -- 16.1.2 Parameter Estimation -- 16.2 Sequential Kriging Optimization -- 16.3 The Knowledge Gradient for Continuous Parameters* -- 16.3.1 Maximizing the Knowledge Gradient -- 16.3.2 Approximating the Knowledge Gradient -- 16.3.3 The Gradient of the Knowledge Gradient -- 16.3.4 Maximizing the Knowledge Gradient -- 16.3.5 The KGCP Policy -- 16.4 Efficient Global Optimization -- 16.5 Experiments -- 16.6 Extension to Higher-Dimensional Problems -- 16.7 Bibliographic Notes -- 17 Learning With a Physical State -- 17.1 Introduction to Dynamic Programming -- 17.1.1 Approximate Dynamic Programming -- 17.1.2 The Exploration vs. Exploitation Problem -- 17.1.3 Discussion -- 17.2 Some Heuristic Learning Policies -- 17.3 The Local Bandit Approximation -- 17.4 The Knowledge Gradient in Dynamic Programming -- 17.4.1 Generalized Learning Using Basis Functions -- 17.4.2 The Knowledge Gradient.

17.4.3 Experiments -- 17.5 An Expected Improvement Policy -- 17.6 Bibliographic Notes -- Index.