A note on research methodology and benchmarking optimization algorithms

A pervasive problem in the field of optimization algorithms is the lack of meaningful and consistent algorithm benchmarking methodology. This includes but is not limited to issues of the selection of problem instances, the selection of algorithm specifications the algorithm configuration parameters and interpretation of results. The intention of this paper is to summarize the literature related to benchmarking optimization algorithms with a focus on benchmarking in the face of the 'no free lunch' theorem and useful statistical tools for interpreting results. This context for this review is biologically inspired optimization algorithms applied to continuous function optimization although the principles extend beyond these themes.