Statistical Methods for Testing and Evaluating Defense Systems -- Copyright -- Contents -- Preface -- Executive Summary -- KEY ISSUES -- Experimental Design -- Testing of Software-Intensive Systems -- System Reliability, Availability, and Maintainability -- Use of Modeling and Simulation in Operational Testing -- TOPICS FOR FURTHER STUDY -- 1 Introduction -- STUDY CONTEXT -- The Defense Acquisition Process -- Operational Testing as Part of the Acquisition Process -- PANEL OBJECTIVES -- STATISTICS AND INFORMATION MANAGEMENT IN DEFENSE TESTING -- Operational Testing of Complex Systems -- Overview -- A Continuum of Information Gathering -- Complex Testing Conditions -- Effects of Constrained Test Resources -- Testing and Evaluation in Nonmilitary Applications -- Conclusion -- THIS REPORT AND FUTURE WORK -- 2 Use of Experimental Design in Operational Testing -- CASE STUDY #1: APACHE LONGBOW HELICOPTER -- Some Issues and Concerns -- Choice of Test Scenarios -- Uncertain Scoring Rules, Measurement Inefficiencies -- Testing Only "Inside the Envelope" -- CASE STUDY #2: THE ATACMS/BAT SYSTEM -- Plans for Operational Testing -- Important Test Factors and Conditions -- Possible Statistical Methods and Aspects for Further Consideration -- FUTURE WORK -- 3 Testing of Software-Intensive Systems -- ROLE FOR STATISTICAL METHODS -- ACTIVITIES TO DATE -- FUTURE WORK -- 4 System Reliability, Availability, and Maintainability -- RELIABILITY, AVAILABILITY, AND MAINTAINABILITY TESTING AND EVALUATION IN THE MILITARY SERVICES -- VARIABILITY IN RELIABILITY, AVAILABILITY, AND MAINTAINABILITY POLICY AND PRACTICE -- INDUSTRIAL (NONMILITARY) STANDARDS -- FUTURE WORK -- 5 Use of Modeling and Simulation in Operational Testing -- SCOPE, PROCEDURES, AND PROGRESS TO DATE -- CONCERNS -- Rigorous Validation of Simulations Is Infrequent.

Little Evidence Is Seen for Use of Statistical Methods in Simulations -- DoD Literature on Use of Simulations Often Lacks Statistical Content -- Distributed Interactive Simulation Raises Additional Concerns -- Simulations Cannot Identify the "Unknown Unknowns" -- FUTURE WORK -- 6 Efforts Toward a Taxonomic Structure of DoD Systems for Operational Testing -- PRELIMINARY WORK TOWARD A TAXONOMIC STRUCTURE -- FUTURE WORK -- Appendices -- APPENDIX A -- The Organizational Structure of Defense Acquisition -- APPENDIX B -- A Short History of Experimental Design, with Commentary for Operational Testing -- THE VALUE OF CONTROLS, BLOCKING, AND RANDOMIZATION -- VARYING MORE THAN ONE FACTOR AT A TIME -- OPTIMAL EXPERIMENTAL DESIGNS -- RESPONSE SURFACE DESIGNS -- BAYESIAN AND SEQUENTIAL EXPERIMENTAL DESIGNS -- APPENDIX C -- Selecting a Small Number of Operational Test Environments -- DUBIN'S CHALLENGE AS A SAMPLING PROBLEM -- AN APPROACH TO MODELING THE SCENARIO STRUCTURE -- ISSUES AND ALTERNATIVES -- APPENDIX D -- Individuals Consulted -- SPONSORS -- GENERAL -- MODELING AND SIMULATION -- RELIABILITY, AVAILABILITY, AND MAINTAINABILITY -- SOFTWARE -- EXPERIMENTAL DESIGN -- ORGANIZATIONAL CONTEXT -- APPENDIX E -- DoD and the Army Test and Evaluation Organization -- References -- Biographical Sketches of Panel Members and Staff.