Source-linked AI summary
COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, Dimo Brockhoff
TL;DR
Benchmarking black-box optimizers is tedious and can produce results that are difficult to interpret consistently. COCO provides an open-source framework that standardizes solver evaluation, automates data processing, and supports comparisons across deterministic and stochastic solvers in single- and multiobjective settings. The paper presents the platform, its benchmarking methodology, and guidance on interpreting aggregated performance results.
Problem
Benchmarking optimization solvers involves many validity-critical choices, while common aggregated views can obscure relevant performance differences.
Method
COCO combines solver interfaces, test suites, randomized function instances, logging, target-based evaluation, function-call runtime, and automated post-processing in one framework.
Results
COCO supports comparisons of deterministic and stochastic solvers across single- and multiobjective problems, with more than 300 datasets available as of November 2019.
Takeaways & Limitations
Open benchmarking data and standardized processing make results easier to reproduce and compare, while scalable testbeds support performance aggregation across target values.
Takeaways & Limitations
Aggregated results can be strongly biased by unbalanced test suites and indiscriminate aggregation over functions, potentially disconnecting benchmark performance from real-world performance.
Abstract
from arXiv · showhide
We introduce COCO, an open source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same framework deterministic and stochastic solvers for both single and multiobjective optimization. We present the rationales behind the (decade-long) development of the platform as a general proposition for guidelines towards better benchmarking. We detail underlying fundamental concepts of COCO such as the definition of a problem as a function instance, the underlying idea of instances, the use of target values, and runtime defined by the number of function calls as the central performance measure. Finally, we give a quick overview of the basic code structure and the currently available test suites.
1. Introduction
COCO addresses the tedious, decision-heavy process of benchmarking continuous black-box optimizers by providing an automated framework, methodology, and reusable comparison data. It supports standardized evaluation across solver types and optimization settings while emphasizing interpretable performance assessment.
- Benchmarking setting: In zero-order black-box optimization, solvers access the search domain and boundaries but learn about f only by querying f(x), with runtime measured by function calls.This derivative-free setting is the central interface between solver and problem.
- Motivation: Benchmarking requires coordinated choices about functions, instances, experimental design, performance measures, data collection, and result presentation.The paper stresses that each step affects the validity and interpretability of the outcome.
- Motivation: COCO implements conceptual benchmarking guidelines intended to address these experimental and analytical challenges.The framework is presented as a practical realization of a broader methodology for better benchmarking.
- Platform: COCO provides solver interfaces, test-problem suites with randomized instances, logging, automated post-processing, plots, and tables.Supported interfaces include C/C++, Java, Matlab/Octave, and Python.
- Platform: COCO supplies automatically generated experiment outputs, including function evaluations needed to reach target precisions, and data from more than 200 prior experiments.Its software environment does not provide the solvers or a server to execute experiments.
- Platform: The framework compares deterministic and stochastic solvers for single- and multiobjective problems using shared benchmark data.More than 200 solvers have been benchmarked by dozens of researchers, with collected data openly accessible for comparison.
2. Why COCO?
COCO addresses the tedious, error-prone nature of benchmarking by combining conceptual guidelines with an automated framework. Its design emphasizes comprehensible, scalable benchmark suites, robust instances, budget-free anytime assessment, and evaluation through function-evaluation runtime.
- COCO aims to reduce the time burden, pitfalls, bugs, and omissions of repetitive benchmarking while providing guidelines and an archive of comparative data.The framework has supported benchmarking more than 200 solvers, with openly accessible data usable in COCO post-processing.
- Benchmark suites should be comprehensible, difficult to exploit through artificial regularities or symmetries, and scalable with input dimension.The functions are used as black boxes by solvers while remaining explicitly known to the scientific community.
- COCO uses pseudo-randomized instances and models problem difficulties such as ill-conditioning, multimodality, and ruggedness.Solver input parameters may depend on the black-box interface, dimension, and search domain, but not on specific benchmark functions.
- The budget-free procedure supports anytime assessment, comparing results up to the smallest shared budget rather than only at a fixed evaluation budget.Runtime is measured by the number of function evaluations, a platform-independent measure that is quantitative and interpretable without expert domain knowledge.
- During solver design, fewer tailored functions and measuring tools, plus informal rapid iteration, are preferable to a formal benchmarking framework.COCO is positioned for formal validation and regression testing after the design outcome, rather than as the sole tool during interactive design.
3. Benchmarking methodology
COCO formalizes benchmarking around function instances, target-hitting runtimes, and repeated or randomized experiments. Its budget-free, anytime design supports runtime-based comparison, while aggregation methods require care because sorted distributions can hide problem-specific performance.
- Functions, instances, and problems: COCO defines a problem by fixing a function, dimension, and instance, while varying instance parameters represents randomized versions of the same function.Instance variation can encode translations, rotations, and other parameter settings, enabling repetitions for deterministic solvers and reducing overfitting or exploitability.
- Runtime and target values: Targets are linked to individual problem instances and are chosen from precision values relative to a known or estimated optimum reference.In multiobjective settings, COCO uses a more intricate quality indicator, such as a hypervolume variant, and estimates the optimal value experimentally.
- Runtime and target values: Runtime is the number of function evaluations required to reach or surpass a prescribed target value for the first time.For each run, COCO records runtimes across around a hundred target values; unsuccessful runs have undefined runtimes bounded below by their evaluation count.
- Experimental procedure: The experimental procedure is budget-free and anytime: performance is assessed over the entire run, with larger budgets yielding more runtime measurements.Budget-based targets can instead be selected from expected runtimes of a solver set, choosing the easiest target whose expected runtime exceeds the given budget.
- Aggregation: Sorting runtimes across problems can make one ECDF appear dominant even when each solver is faster on a different problem.The paper’s example gives Solver A runtimes of 50 and 500 versus Solver B’s 1000 and 100, despite A dominating in the sorted profiles.
- Aggregation: ERT divides total evaluations from successful and unsuccessful runs by the number of runs that reached the target.The Q-measure or SP1 uses only successful runs and is not currently used within COCO.
4. Test Suites
COCO’s extensible test-suite design combines diverse functions, dimensions, instances, and target levels to support meaningful black-box benchmarking.
- COCO can add new test suites with comparatively little effort while retaining the established benchmarking setup.
- The bbob suite contains 24 functions across dimensions 2, 3, 5, 10, 20, and 40, organized into five function subgroups.
- bbob-noisy contains 30 noisy functions across the same six dimensions and three subgroups with three noise models.
- Test-suite composition determines the anticipated purpose of a benchmark, including whether it emphasizes difficult or easy problems.
- Test suites use transformations in x- and f-space to make functions less regular, less symmetrical, and less exploitable through simple formulas.
- New suites should provide multidimensional functions, pseudo-randomized instances, common target precision levels, and suitable noisy-function distributions.
5. Usage and Output Examples
COCO combines experiment execution, data logging, Python post-processing, and browser-based reporting to compare solvers across functions, targets, dimensions, and suites.
- COCO’s implementation has experiments and post-processing parts: experiments define suites and collect data, while post-processing produces benchmarking results.
- A Python script can benchmark scipy.optimize.fmin on bbob, writing raw data to exdata and generated figures and tables to ppdata.
- The cocopp module is accessible from the command line, Python/IPython, or Jupyter notebooks and can process one or multiple experiments.
- COCO’s main displays are runtime ECDFs based on function evaluations, aggregating problems across target precisions with simulated restarts.
- The platform supports ECDFs for individual functions, function groups, complete suites, and multiobjective solver comparisons on bbob-biobj.
- Absolute and budget-based targets can yield different runtime-distribution views, with budget-based targets extending to budgets up to 50 times dimension evaluations.
- Single-solver analyses include fixed-target runtime ECDFs and precision ECDFs within specified budgets, while scaling plots show average runtime over dimension.
6. Usage Statistics
COCO has supported BBOB workshops since 2009 and has increasingly been used beyond that series, with visibility summarized through citation data.
- Since 2009, COCO has facilitated submissions to BBOB workshops at ACM GECCO and has increasingly been used outside the workshop series.
- Table 1 summarizes COCO’s visibility and user base, including citations to its documentation as of November 19, 2019.
7. Extensions under Development
COCO’s development roadmap expands suite authoring, external evaluation, benchmark coverage, and recommendations for noisy-function assessment, but these features were not yet released.
- Planned interfaces would enable Python-based suite implementation and external evaluator communication through sockets.
- Under-development benchmark suites cover constrained problems, three-objective multiobjective problems, and game-based real-world problems.
- Recommendations are being developed to address a usually small bias in noisy-function performance evaluation.
- These features had not been released and could undergo relevant changes before release.
8. Summary and Discussion
COCO is an open-source platform that automates benchmarking for numerical optimization and embeds a methodology designed to make comparisons more reproducible, interpretable, and broadly applicable. Its framework uses instances, scalable test suites, target values, and aggregation choices to compare solver behavior while warning against biased performance summaries.
- Platform and methodology: COCO automatically benchmarks numerical optimization algorithms through multilingual solver interfaces, test functions, data logging, post-processing, and open-access results.As of November 2019, more than 300 datasets were available, and users could implement new test suites.
- Platform and methodology: Function instances enable deterministic and stochastic solvers to be compared in one framework while reducing exploitation of specific instance properties.Instances typically vary optima, function-value shifts, and rotations, under the assumption that instances of one function have similar difficulty.
- Platform and methodology: COCO uses scalable test functions and avoids aggregating results over dimension because dimension is known to the solver and can inform solver selection.Results may instead be aggregated over multiple target values, and the methodology extends to multiobjective problems through quality indicators and targets.
- Cautions and practical implications: Unbalanced test suites and aggregation across all functions can strongly bias empirical runtime distributions, data profiles, and performance profiles.Such biases may disconnect benchmark performance from real-world performance and misguide optimization research.
- Cautions and practical implications: Test suites should emphasize challenging but solvable problems and prevent aggregated measures from over-emphasizing unimportant classes such as easy-to-solve problems.This recommendation is framed as part of designing software intended to address real-world difficulties.
Appendix A. How We Chose Test Functions
The bbob test suite was constructed from comprehensible, scalable functions chosen to represent important, plausible, or basic continuous-optimization difficulties. Functions were modified to reduce simple exploits and paired to isolate effects of specific changes while keeping repeated experiments feasible.
- Selection criteria: bbob functions were selected to represent practical difficulties, plausibly relevant difficulties, or simple topologies that search algorithms should handle.Examples include multimodality, ill-conditioning, ridge-like topology, ruggedness, and linear slopes.
- Design requirements: The functions were required to be comprehensible and scalable with the input dimension to support interpretation and experiments across dimensions.Comprehensibility was treated as important for understanding solver behavior rather than presenting functions as opaque competition targets.
- Design requirements: Functions were slightly modified to make them less amenable to simple exploits and paired to clarify the effect of particular changes on algorithm performance.The suite was limited to roughly two dozen functions to permit repeated experiments over multiple dimensions.
Appendix B. COCO Versus Competitive Testing
COCO’s benchmarking objective is to characterize solver performance and behavior on comprehensible, stable test functions, whereas competitions primarily reward performance and therefore require stronger protection against exploitation. The paper distinguishes these goals while acknowledging that experimental precautions do not prevent intentional protocol violations.
- Benchmarking versus competition: COCO seeks comprehensive solver-performance profiles that explain why algorithms perform differently across functions.Understanding solver behavior requires the underlying functions to be comprehensible and sufficiently stable for comparison with earlier data.
- Benchmarking versus competition: Competition testing prioritizes performing well, so designers should hide functions from both solvers and researchers and change them frequently to limit overtuning.This contrasts with COCO’s preference for comprehensible functions that support behavioral analysis.
- Benchmarking versus competition: COCO makes trivial exploitation unlikely by restricting access to function optima and optimal function values through function definitions and the experimental setup.These precautions target unintended exploits rather than deliberate violations of the prescribed setup.
- Benchmarking versus competition: Intentional exploitation and neglect of the prescribed experimental setup are not prevented by COCO’s design precautions.The limitation concerns deliberate behavior, not the platform’s attempt to reduce accidental exploitation.
Appendix C. Details on the Used Biobjective Performance Measure
COCO evaluates biobjective algorithms with a normalized hypervolume-based quality indicator applied to the nondominated solutions found during a run. The indicator switches from negative distance to positive hypervolume when the nadir point becomes dominated, and runtimes are recorded for target indicator values.
- Quality indicator: COCO’s biobjective quality measure is based on the hypervolume indicator, which represents the size of the space covered by the nondominated solutions found so far.The indicator depends on the solution set after t function evaluations and on whether the nadir point has been dominated.
- Normalization: The objective space is normalized so that indicator values can be compared across functions, dimensions, and instances.The normalized ideal and nadir points correspond to (0, 0) and (1, 1), respectively; algorithms still see raw objective values.
- Indicator cases: If the nadir point is dominated, the quality indicator equals the hypervolume of the solution set using (1, 1) as reference point.This is the positive-quality regime of the indicator.
- Indicator cases: If the nadir point is not dominated, the indicator uses the negative distance of the closest normalized objective vector to the region dominating the nadir point.Both regimes meet at zero, with positive values after domination and negative values before it.
- Runtime targets: COCO records the number of function evaluations needed to reach target indicator values, including targets such as 10^-5, zero, and several negative values.Negative and zero targets account for the fact that the best-known Pareto-set approximation is only an estimate of the true Pareto set.