The optimality of a design depends on the
statistical model and is assessed with respect to a statistical criterion, which is related to the variance-matrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding of
statistical theory and practical knowledge with
designing experiments.
Optimal designs reduce the costs of experimentation by allowing
statistical models to be estimated with fewer experimental runs.
Optimal designs can accommodate multiple types of factors, such as process, mixture, and discrete factors.
Designs can be optimized when the design-space is constrained, for example, when the mathematical process-space contains factor-settings that are practically infeasible (e.g. due to safety concerns).
Minimizing the variance of estimators
Experimental designs are evaluated using statistical criteria.[6]
When the
statistical model has several
parameters, however, the
mean of the parameter-estimator is a
vector and its
variance is a
matrix. The
inverse matrix of the variance-matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using
statistical theory, statisticians compress the information-matrix using real-valued
summary statistics; being real-valued functions, these "information criteria" can be maximized.[8] The traditional optimality-criteria are
invariants of the
information matrix; algebraically, the traditional optimality-criteria are
functionals of the
eigenvalues of the information matrix.
A-optimality ("average" or trace)
One criterion is A-optimality, which seeks to minimize the
trace of the
inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.
C-optimality
This criterion minimizes the variance of a
best linear unbiased estimator of a predetermined linear combination of model parameters.
D-optimality (determinant)
A popular criterion is D-optimality, which seeks to minimize |(X'X)−1|, or equivalently maximize the
determinant of the
information matrix X'X of the design. This criterion results in maximizing the
differential Shannon information content of the parameter estimates.
E-optimality (eigenvalue)
Another design is E-optimality, which maximizes the minimum
eigenvalue of the information matrix.
This criterion maximizes a quantity measuring the mutual column orthogonality of X and the
determinant of the information matrix.
T-optimality
This criterion maximizes the discrepancy between two proposed models at the design locations.[10]
Other optimality-criteria are concerned with the variance of
predictions:
G-optimality
A popular criterion is G-optimality, which seeks to minimize the maximum entry in the
diagonal of the
hat matrix X(X'X)−1X'. This has the effect of minimizing the maximum variance of the predicted values.
I-optimality (integrated)
A second criterion on prediction variance is I-optimality, which seeks to minimize the average prediction variance over the design space.
V-optimality (variance)
A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance over a set of m specific points.[11]
Catalogs of optimal designs occur in books and in software libraries.
In addition, major
statistical systems like
SAS and
R have procedures for optimizing a design according to a user's specification. The experimenter must specify a
model for the design and an optimality-criterion before the method can compute an optimal design.[13]
Practical considerations
Some advanced topics in optimal design require more
statistical theory and practical knowledge in designing experiments.
Model dependence and robustness
Since the optimality criterion of most optimal designs is based on some function of the information matrix, the 'optimality' of a given design is model dependent: While an optimal design is best for that
model, its performance may deteriorate on other
models. On other
models, an optimal design can be either better or worse than a non-optimal design.[14] Therefore, it is important to
benchmark the performance of designs under alternative
models.[15]
Choosing an optimality criterion and robustness
The choice of an appropriate optimality criterion requires some thought, and it is useful to benchmark the performance of designs with respect to several optimality criteria. Cornell writes that
since the [traditional optimality] criteria . . . are variance-minimizing criteria, . . . a design that is optimal for a given model using one of the . . . criteria is usually near-optimal for the same model with respect to the other criteria.
Indeed, there are several classes of designs for which all the traditional optimality-criteria agree, according to the theory of "universal optimality" of
Kiefer.[17] The experience of practitioners like Cornell and the "universal optimality" theory of Kiefer suggest that robustness with respect to changes in the optimality-criterion is much greater than is robustness with respect to changes in the model.
Flexible optimality criteria and convex analysis
High-quality statistical software provide a combination of libraries of optimal designs or iterative methods for constructing approximately optimal designs, depending on the model specified and the optimality criterion. Users may use a standard optimality-criterion or may program a custom-made criterion.
When scientists wish to test several theories, then a statistician can design an experiment that allows optimal tests between specified models. Such "discrimination experiments" are especially important in the
biostatistics supporting
pharmacokinetics and
pharmacodynamics, following the work of
Cox and Atkinson.[21]
The use of a
Bayesian design does not force statisticians to use
Bayesian methods to analyze the data, however. Indeed, the "Bayesian" label for probability-based experimental-designs is disliked by some researchers.[23] Alternative terminology for "Bayesian" optimality includes "on-average" optimality or "population" optimality.
Iterative experimentation
Scientific experimentation is an iterative process, and statisticians have developed several approaches to the optimal design of sequential experiments.
Optimal designs for
response-surface models are discussed in the textbook by Atkinson, Donev and Tobias, and in the survey of Gaffke and Heiligers and in the mathematical text of Pukelsheim. The
blocking of optimal designs is discussed in the textbook of Atkinson, Donev and Tobias and also in the monograph by Goos.
The earliest optimal designs were developed to estimate the parameters of regression models with continuous variables, for example, by
J. D. Gergonne in 1815 (Stigler). In English, two early contributions were made by
Charles S. Peirce and
Kirstine Smith.
Pioneering designs for multivariate
response-surfaces were proposed by
George E. P. Box. However, Box's designs have few optimality properties. Indeed, the
Box–Behnken design requires excessive experimental runs when the number of variables exceeds three.[28]
Box's
"central-composite" designs require more experimental runs than do the optimal designs of Kôno.[29]
System identification and stochastic approximation
There are several methods of finding an optimal design, given an a priori restriction on the number of experimental runs or replications. Some of these methods are discussed by Atkinson, Donev and Tobias and in the paper by Hardin and
Sloane. Of course, fixing the number of experimental runs a priori would be impractical. Prudent statisticians examine the other optimal designs, whose number of experimental runs differ.
Discretizing probability-measure designs
In the mathematical theory on optimal experiments, an optimal design can be a
probability measure that is
supported on an infinite set of observation-locations. Such optimal probability-measure designs solve a mathematical problem that neglected to specify the cost of observations and experimental runs. Nonetheless, such optimal probability-measure designs can be
discretized to furnish
approximately optimal designs.[32]
In some cases, a finite set of observation-locations suffices to
support an optimal design. Such a result was proved by Kôno and
Kiefer in their works on
response-surface designs for quadratic models. The Kôno–Kiefer analysis explains why optimal designs for response-surfaces can have discrete supports, which are very similar as do the less efficient designs that have been traditional in
response surface methodology.[33]
Charles S. Peirce proposed an economic theory of scientific experimentation in 1876, which sought to maximize the precision of the estimates. Peirce's optimal allocation immediately improved the accuracy of gravitational experiments and was used for decades by Peirce and his colleagues. In his 1882 published lecture at
Johns Hopkins University, Peirce introduced experimental design with these words:
Logic will not undertake to inform you what kind of experiments you ought to make in order best to determine the acceleration of gravity, or the value of the Ohm; but it will tell you how to proceed to form a plan of experimentation.
[....] Unfortunately practice generally precedes theory, and it is the usual fate of mankind to get things done in some boggling way first, and find out afterward how they could have been done much more easily and perfectly.[34]
Kirstine Smith proposed optimal designs for polynomial models in 1918. (Kirstine Smith had been a student of the Danish statistician
Thorvald N. Thiele and was working with
Karl Pearson in London.)
^The adjective "optimum" (and not "optimal") "is the slightly older form in English and avoids the construction 'optim(um) + al´—there is no 'optimalis' in Latin" (page x in Optimum Experimental Designs, with SAS, by Atkinson, Donev, and Tobias).
^Traditionally, statisticians have evaluated estimators and designs by considering some
summary statistic of the covariance matrix (of a
mean-
unbiased estimator), usually with positive real values (like the
determinant or
matrix trace). Working with positive real-numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).
For several parameters, the covariance-matrices and information-matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a
partiallyordered vector space, under the
Loewner (Löwner) order. This cone is closed under matrix-matrix addition, under matrix-inversion, and under the multiplication of positive real-numbers and matrices.
An exposition of matrix theory and the Loewner-order appears in Pukelsheim.
^Shin, Yeonjong; Xiu, Dongbin (2016). "Nonadaptive quasi-optimal points selection for least squares linear regression". SIAM Journal on Scientific Computing. 38 (1): A385–A411.
Bibcode:
2016SJSC...38A.385S.
doi:
10.1137/15M1015868.
^The above optimality-criteria are convex functions on domains of
symmetric positive-semidefinite matrices: See an on-line textbook for practitioners, which has many illustrations and statistical applications:
Boyd, Stephen P.; Vandenberghe, Lieven (2004).
Convex Optimization(PDF). Cambridge University Press.
ISBN978-0-521-83378-3. Retrieved October 15, 2011. (book in pdf)
Boyd and Vandenberghe discuss optimal experimental designs on pages 384–396.
^Iterative methods and approximation algorithms are surveyed in the textbook by Atkinson, Donev and Tobias and in the monographs of Fedorov (historical) and Pukelsheim, and in the survey article by Gaffke and Heiligers.
^Such benchmarking is discussed in the textbook by Atkinson et al. and in the papers of Kiefer. Model-
robust designs (including "Bayesian" designs) are surveyed by Chang and Notz.
^Cornell, John (2002). Experiments with Mixtures: Designs, Models, and the Analysis of Mixture Data (third ed.). Wiley.
ISBN978-0-471-07916-3. (Pages 400-401)
^An introduction to "universal optimality" appears in the textbook of Atkinson, Donev, and Tobias. More detailed expositions occur in the advanced textbook of Pukelsheim and the papers of Kiefer.
^Computational methods are discussed by Pukelsheim and by Gaffke and Heiligers.
^Bayesian designs are discussed in Chapter 18 of the textbook by Atkinson, Donev, and Tobias. More advanced discussions occur in the monograph by Fedorov and Hackl, and the articles by Chaloner and Verdinelli and by DasGupta.
Bayesian designs and other aspects of "model-robust" designs are discussed by Chang and Notz.
^As an alternative to "Bayesian optimality", "on-average optimality" is advocated in Fedorov and Hackl.
^Chernoff, H. (1972) Sequential Analysis and Optimal Design, SIAM Monograph.
^Zacks, S. (1996) "Adaptive Designs for Parametric Models". In: Ghosh, S. and Rao, C. R., (Eds) (1996). Design and Analysis of Experiments, Handbook of Statistics, Volume 13. North-Holland.
ISBN0-444-82061-2. (pages 151–180)
^Henry P. Wynn wrote, "the modern theory of optimum design has its roots in the decision theory school of U.S. statistics founded by
Abraham Wald" in his introduction "Jack Kiefer's Contributions to Experimental Design", which is pages xvii–xxiv in the following volume:
Kiefer acknowledges Wald's influence and results on many pages – 273 (page 55 in the reprinted volume), 280 (62), 289-291 (71-73), 294 (76), 297 (79), 315 (97) 319 (101) – in this article:
Kiefer, J. (1959). "Optimum Experimental Designs". Journal of the Royal Statistical Society, Series B. 21: 272–319.
^Some step-size rules for of Judin & Nemirovskii and of
PolyakArchived 2007-10-31 at the
Wayback Machine are explained in the textbook by Kushner and Yin:
^The
discretization of optimal probability-measure designs to provide
approximately optimal designs is discussed by Atkinson, Donev, and Tobias and by Pukelsheim (especially Chapter 12).
^Regarding designs for quadratic
response-surfaces, the results of Kôno and
Kiefer are discussed in Atkinson, Donev, and Tobias.
Mathematically, such results are associated with
Chebyshev polynomials, "Markov systems", and "moment spaces": See
^Peirce, C. S. (1882), "Introductory Lecture on the Study of Logic" delivered September 1882, published in Johns Hopkins University Circulars, v. 2, n. 19, pp. 11–12, November 1882, see p. 11, Google BooksEprint. Reprinted in Collected Papers v. 7, paragraphs 59–76, see 59, 63, Writings of Charles S. Peirce v. 4, pp. 378–82, see 378, 379, and The Essential Peirce v. 1, pp. 210–14, see 210–1, also lower down on 211.
Logothetis, N.;
Wynn, H. P. (1989). Quality through design: Experimental design, off-line quality control, and Taguchi's contributions. Oxford U. P. pp. 464+xi.
ISBN978-0-19-851993-5.
Shah, Kirti R. & Sinha, Bikas K. (1989). Theory of Optimal Designs. Lecture Notes in Statistics. Vol. 54. Springer-Verlag. pp. 171+viii.
ISBN978-0-387-96991-6.
Further reading
Textbooks for practitioners and students
Textbooks emphasizing regression and response-surface methodology
The textbook by Atkinson, Donev and Tobias has been used for short courses for industrial practitioners as well as university courses.
Logothetis, N.;
Wynn, H. P. (1989). Quality through design: Experimental design, off-line quality control, and Taguchi's contributions. Oxford U. P. pp. 464+xi.
ISBN978-0-19-851993-5.
Textbooks emphasizing block designs
Optimal
block designs are discussed by Bailey and by Bapat. The first chapter of Bapat's book reviews the
linear algebra used by Bailey (or the advanced books below). Bailey's exercises and discussion of
randomization both emphasize statistical concepts (rather than algebraic computations).
Ghosh, S.;
Rao, C. R., eds. (1996). Design and Analysis of Experiments. Handbook of Statistics. Vol. 13. North-Holland.
ISBN978-0-444-82061-7.
"
ModelRobust Designs". Design and Analysis of Experiments. Handbook of Statistics. pp. 1055–1099.
Cheng, C.-S. "Optimal Design: Exact Theory". Design and Analysis of Experiments. Handbook of Statistics. pp. 977–1006.
DasGupta, A. "Review of Optimal
Bayesian Designs". Design and Analysis of Experiments. Handbook of Statistics. pp. 1099–1148.
Gaffke, N. & Heiligers, B. "Approximate Designs for
Polynomial Regression:
Invariance,
Admissibility, and Optimality". Design and Analysis of Experiments. Handbook of Statistics. pp. 1149–1199.
Majumdar, D. "Optimal and Efficient Treatment-Control Designs". Design and Analysis of Experiments. Handbook of Statistics. pp. 1007–1054.
Stufken, J. "Optimal
Crossover Designs". Design and Analysis of Experiments. Handbook of Statistics. pp. 63–90.
Zacks, S. "Adaptive Designs for Parametric Models". Design and Analysis of Experiments. Handbook of Statistics. pp. 151–180.