Cover -- Title Page -- Copyright -- Contents -- Preface -- Chapter 1 Optimization problem tasks and how they arise -- 1.1 The general optimization problem -- 1.2 Why the general problem is generally uninteresting -- 1.3 (Non-)Linearity -- 1.4 Objective function properties -- 1.4.1 Sums of squares -- 1.4.2 Minimax approximation -- 1.4.3 Problems with multiple minima -- 1.4.4 Objectives that can only be imprecisely computed -- 1.5 Constraint types -- 1.6 Solving sets of equations -- 1.7 Conditions for optimality -- 1.8 Other classifications -- References -- Chapter 2 Optimization algorithms-an overview -- 2.1 Methods that use the gradient -- 2.2 Newton-like methods -- 2.3 The promise of Newton's method -- 2.4 Caution: convergence versus termination -- 2.5 Difficulties with Newton's method -- 2.6 Least squares: Gauss-Newton methods -- 2.7 Quasi-Newton or variable metric method -- 2.8 Conjugate gradient and related methods -- 2.9 Other gradient methods -- 2.10 Derivative-free methods -- 2.10.1 Numerical approximation of gradients -- 2.10.2 Approximate and descend -- 2.10.3 Heuristic search -- 2.11 Stochastic methods -- 2.12 Constraint-based methods-mathematical programming -- References -- Chapter 3 Software structure and interfaces -- 3.1 Perspective -- 3.2 Issues of choice -- 3.3 Software issues -- 3.4 Specifying the objective and constraints to the optimizer -- 3.5 Communicating exogenous data to problem definition functions -- 3.5.1 Use of "global'' data and variables -- 3.6 Masked (temporarily fixed) optimization parameters -- 3.7 Dealing with inadmissible results -- 3.8 Providing derivatives for functions -- 3.9 Derivative approximations when there are constraints -- 3.10 Scaling of parameters and function -- 3.11 Normal ending of computations -- 3.12 Termination tests-abnormal ending.

3.13 Output to monitor progress of calculations -- 3.14 Output of the optimization results -- 3.15 Controls for the optimizer -- 3.16 Default control settings -- 3.17 Measuring performance -- 3.18 The optimization interface -- References -- Chapter 4 One-parameter root-finding problems -- 4.1 Roots -- 4.2 Equations in one variable -- 4.3 Some examples -- 4.3.1 Exponentially speaking -- 4.3.2 A normal concern -- 4.3.3 Little Polly Nomial -- 4.3.4 A hypothequial question -- 4.4 Approaches to solving 1D root-finding problems -- 4.5 What can go wrong? -- 4.6 Being a smart user of root-finding programs -- 4.7 Conclusions and extensions -- References -- Chapter 5 One-parameter minimization problems -- 5.1 The optimize() function -- 5.2 Using a root-finder -- 5.3 But where is the minimum? -- 5.4 Ideas for 1D minimizers -- 5.5 The line-search subproblem -- References -- Chapter 6 Nonlinear least squares -- 6.1 nls() from package stats -- 6.1.1 A simple example -- 6.1.2 Regression versus least squares -- 6.2 A more difficult case -- 6.3 The structure of the nls() solution -- 6.4 Concerns with nls() -- 6.4.1 Small residuals -- 6.4.2 Robustness-"singular gradient'' woes -- 6.4.3 Bounds with nls() -- 6.5 Some ancillary tools for nonlinear least squares -- 6.5.1 Starting values and self-starting problems -- 6.5.2 Converting model expressions to sum-of-squares functions -- 6.5.3 Help for nonlinear regression -- 6.6 Minimizing R,functions that compute sums of squares -- 6.7 Choosing an approach -- 6.8 Separable sums of squares problems -- 6.9 Strategies for nonlinear least squares -- References -- Chapter 7 Nonlinear equations -- 7.1 Packages and methods for nonlinear equations -- 7.1.1 BB -- 7.1.2 nleqslv -- 7.1.3 Using nonlinear least squares -- 7.1.4 Using function minimization methods.

7.2 A simple example to compare approaches -- 7.3 A statistical example -- References -- Chapter 8 Function minimization tools in the base R system -- 8.1 optim() -- 8.2 nlm() -- 8.3 nlminb() -- 8.4 Using the base optimization tools -- References -- Chapter 9 Add-in function minimization packages for R -- 9.1 Package optimx -- 9.1.1 Optimizers in optimx -- 9.1.2 Example use of optimx() -- 9.2 Some other function minimization packages -- 9.2.1 nloptr and nloptwrap -- 9.2.2 trust and trustOptim -- 9.3 Should we replace optim() routines? -- References -- Chapter 10 Calculating and using derivatives -- 10.1 Why and how -- 10.2 Analytic derivatives-by hand -- 10.3 Analytic derivatives-tools -- 10.4 Examples of use of R tools for differentiation -- 10.5 Simple numerical derivatives -- 10.6 Improved numerical derivative approximations -- 10.6.1 The Richardson extrapolation -- 10.6.2 Complex-step derivative approximations -- 10.7 Strategy and tactics for derivatives -- References -- Chapter 11 Bounds constraints -- 11.1 Single bound: use of a logarithmic transformation -- 11.2 Interval bounds: Use of a hyperbolic transformation -- 11.2.1 Example of the tanh transformation -- 11.2.2 A fly in the ointment -- 11.3 Setting the objective large when bounds are violated -- 11.4 An active set approach -- 11.5 Checking bounds -- 11.6 The importance of using bounds intelligently -- 11.6.1 Difficulties in applying bounds constraints -- 11.7 Post-solution information for bounded problems -- Appendix 11.A Function transfinite -- References -- Chapter 12 Using masks -- 12.1 An example -- 12.2 Specifying the objective -- 12.3 Masks for nonlinear least squares -- 12.4 Other approaches to masks -- References -- Chapter 13 Handling general constraints -- 13.1 Equality constraints -- 13.1.1 Parameter elimination.

13.1.2 Which parameter to eliminate? -- 13.1.3 Scaling and centering? -- 13.1.4 Nonlinear programming packages -- 13.1.5 Sequential application of an increasing penalty -- 13.2 Sumscale problems -- 13.2.1 Using a projection -- 13.3 Inequality constraints -- 13.4 A perspective on penalty function ideas -- 13.5 Assessment -- References -- Chapter 14 Applications of mathematical programming -- 14.1 Statistical applications of math programming -- 14.2 R packages for math programming -- 14.3 Example problem: L1 regression -- 14.4 Example problem: minimax regression -- 14.5 Nonlinear quantile regression -- 14.6 Polynomial approximation -- References -- Chapter 15 Global optimization and stochastic methods -- 15.1 Panorama of methods -- 15.2 R packages for global and stochastic optimization -- 15.3 An example problem -- 15.3.1 Method SANN from optim() -- 15.3.2 Package GenSA -- 15.3.3 Packages DEoptim and RcppDE -- 15.3.4 Package smco -- 15.3.5 Package soma -- 15.3.6 Package Rmalschains -- 15.3.7 Package rgenoud -- 15.3.8 Package GA -- 15.3.9 Package gaoptim -- 15.4 Multiple starting values -- References -- Chapter 16 Scaling and reparameterization -- 16.1 Why scale or reparameterize? -- 16.2 Formalities of scaling and reparameterization -- 16.3 Hobbs' weed infestation example -- 16.4 The KKT conditions and scaling -- 16.5 Reparameterization of the weeds problem -- 16.6 Scale change across the parameter space -- 16.7 Robustness of methods to starting points -- 16.7.1 Robustness of optimization techniques -- 16.7.2 Robustness of nonlinear least squares methods -- 16.8 Strategies for scaling -- References -- Chapter 17 Finding the right solution -- 17.1 Particular requirements -- 17.1.1 A few integer parameters -- 17.2 Starting values for iterative methods -- 17.3 KKT conditions -- 17.3.1 Unconstrained problems.

17.3.2 Constrained problems -- 17.4 Search tests -- References -- Chapter 18 Tuning and terminating methods -- 18.1 Timing and profiling -- 18.1.1 rbenchmark -- 18.1.2 microbenchmark -- 18.1.3 Calibrating our timings -- 18.2 Profiling -- 18.2.1 Trying possible improvements -- 18.3 More speedups of R computations -- 18.3.1 Byte-code compiled functions -- 18.3.2 Avoiding loops -- 18.3.3 Package upgrades - an example -- 18.3.4 Specializing codes -- 18.4 External language compiled functions -- 18.4.1 Building an R function using Fortran -- 18.4.2 Summary of Rayleigh quotient timings -- 18.5 Deciding when we are finished -- 18.5.1 Tests for things gone wrong -- References -- Chapter 19 Linking R to external optimization tools -- 19.1 Mechanisms to link R to external software -- 19.1.1 R functions to call external (sub)programs -- 19.1.2 File and system call methods -- 19.1.3 Thin client methods -- 19.2 Prepackaged links to external optimization tools -- 19.2.1 NEOS -- 19.2.2 Automatic Differentiation Model Builder (ADMB) -- 19.2.3 NLopt -- 19.2.4 BUGS and related tools -- 19.3 Strategy for using external tools -- References -- Chapter 20 Differential equation models -- 20.1 The model -- 20.2 Background -- 20.3 The likelihood function -- 20.4 A first try at minimization -- 20.5 Attempts with optimx -- 20.6 Using nonlinear least squares -- 20.7 Commentary -- Reference -- Chapter 21 Miscellaneous nonlinear estimation tools for R -- 21.1 Maximum likelihood -- 21.2 Generalized nonlinear models -- 21.3 Systems of equations -- 21.4 Additional nonlinear least squares tools -- 21.5 Nonnegative least squares -- 21.6 Noisy objective functions -- 21.7 Moving forward -- References -- Index.