Source-linked AI summary
IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics
Hao Wang, Diederick Vermetten, Furong Ye, Carola Doerr, Thomas Bäck
TL;DR
Complex IOH behavior makes empirical evaluation important, but benchmarking requires laborious setup and analysis. IOHanalyzer offers an interactive platform for configurable performance and algorithm-state analyses. It provides fixed-target and fixed-budget statistics, supports aggregation across problems, and includes a limitation in estimating values when duplicates are numerous.
Problem
IOHs’ complex, interdependent behavior makes performance difficult to predict, while empirical benchmarking requires laborious choices about instances, setup, indicators, visualizations, and statistics.
Source: §1 · p. 2Method
IOHanalyzer is an interactive platform for assessing, comparing, and visualizing IOH performance data, with user-controlled performance measures, ranges, and precision.
Source: §1.1 · pp. 2, 4Results
IOHanalyzer provides fixed-target and fixed-budget performance analyses, cross-problem aggregation, and statistics on evolving algorithm components.
Source: §1.1 · p. 4Takeaways & Limitations
The platform supports tailored analysis of IOH performance data and evolving algorithm components, including data from external benchmarking environments.
Source: §1.1 · p. 4Estimations become imprecise when values are duplicated frequently, a situation considered likely in discrete optimization examples.
Source: §3.2.1 · p. 17
Abstract
from arXiv · showhide
Benchmarking and performance analysis play an important role in understanding the behaviour of iterative optimization heuristics (IOHs) such as local search algorithms, genetic and evolutionary algorithms, Bayesian optimization algorithms, etc. This task, however, involves manual setup, execution, and analysis of the experiment on an individual basis, which is laborious and can be mitigated by a generic and well-designed platform. For this purpose, we propose IOHanalyzer, a new user-friendly tool for the analysis, comparison, and visualization of performance data of IOHs. Implemented in R and C++, IOHanalyzer is fully open source. It is available on CRAN and GitHub. IOHanalyzer provides detailed statistics about fixed-target running times and about fixed-budget performance of the benchmarked algorithms with a real-valued codomain, single-objective optimization tasks. Performance aggregation over several benchmark problems is possible, for example in the form of empirical cumulative distribution functions. Key advantages of IOHanalyzer over other performance analysis packages are its highly interactive design, which allows users to specify the performance measures, ranges, and granularity that are most useful for their experiments, and the possibility to analyze not only performance traces, but also the evolution of dynamic state parameters. IOHanalyzer can directly process performance data from the main benchmarking platforms, including the COCO platform, Nevergrad, the SOS platform, and IOHexperimenter. An R programming interface is provided for users preferring to have a finer control over the implemented functionalities.
1 Introduction
Iterative optimization heuristics are complex and difficult to evaluate manually, while theoretical results often cover only simple, unrepresentative cases. IOHanalyzer provides an interactive platform for analyzing IOH performance data and complements existing benchmarking environments with flexible analyses.
1 Introduction
- IOHs’ randomized, interdependent behavior makes performance difficult to predict, and theoretical results are often limited to simple, unrepresentative algorithms or problems.Empirical benchmarking is therefore used to understand realistic solvers and applications, but experiment design and analysis are laborious.Source: §1 · p. 2
1.1 IOHanalyzer: Overview and Availability
- IOHanalyzer is an interactive platform for assessing, comparing, and visualizing IOH performance data, with analyses tailored through user-selected measures, ranges, and precision.It accepts data from IOHexperimenter, COCO, Nevergrad, or users’ own experimentation platforms.Source: §1.1 · pp. 2–3
- IOHanalyzer supports fixed-target and fixed-budget perspectives, performance aggregation across problems, and statistics on evolving algorithm components such as self-adjusting hyperparameters.Its R interface offers fine control, while a web-based GUI supports users less familiar with R.Source: §1.1 · p. 4
1.2 Related Benchmarking Environments
- COCO performance files can be analyzed in IOHanalyzer, whose interactive interface lets users vary performance measures, ranges, and granularity rather than relying only on standardized analyses.The paper describes COCO as generating detailed performance traces and automatically producing a fixed set of standardized analyses.Source: §1.2 · pp. 4–5
- Nevergrad data can be visualized by IOHanalyzer, but Nevergrad stores final recommended-point quality rather than search trajectories; its tracking extension was under construction.Nevergrad’s performance evaluation therefore takes a fixed-budget perspective.Source: §1.2 · p. 5
2 Background
This section sets the scope of the optimization tasks and iterative optimization heuristics considered, then introduces performance indicators for comparing their behavior across evaluation budgets and targets.
2.1 Iterative Optimization Heuristics
- IOHanalyzer’s setting is single-objective optimization with real-valued objectives; search spaces may be discrete or continuous, constrained or unconstrained, and black-box or partly known.The analysis applies to sampling-based traces based on evaluated candidates, even when the objective is explicitly known.Source: §2.1 · p. 6
- IOHs sample candidate solutions and use their evaluated function values to guide search, covering methods such as local search, evolutionary algorithms, and Bayesian optimization.The section allows intermediate gray-box settings and does not require the objective function to be unknown.Source: §2.1 · pp. 6–7
2.2 Selected Performance Indicators
- Performance is commonly measured in function evaluations: fixed-target analysis counts evaluations to reach a quality threshold, while fixed-budget analysis measures the best quality found within a budget.Unlike CPU time, evaluation counts are machine-independent and less sensitive to implementation.Source: §2.2 · p. 7
- PAR-c averages target-hitting evaluations while assigning unsuccessful runs a penalty of cB; the section also describes quantiles, including the median, and standard deviations.IOHanalyzer typically studies PAR-1 and refers to it as the mean.Source: §2.2 · p. 8
- Success rate is the fraction of runs reaching a target within budget, while ERT estimates running time under independent restarts and is infinite if no run succeeds.ERT accounts for unsuccessful runs by restarting them after the allocated budget.Source: §2.2 · pp. 8–9
- IOHanalyzer estimates distributions and ECDFs, aggregating performance over targets and problems; users can change targets, which may be generated linearly, log-linearly, or using COCO defaults.For fixed-budget function values it estimates density with KDE; for fixed-target running times it applies KDE to estimate a probability mass function.Source: §2.2 · pp. 9–10
3 Graphical User Interface
IOHanalyzer provides a GUI for loading optimization performance data and exploring fixed-target, fixed-budget, and position-based analyses. Users can interactively select analysis settings and export plots and tables.
3 Graphical User Interface
- The GUI supports interactive plots with adjustable views and downloadable figures and tables.Plots can be zoomed, and curves hidden or shown; figures and most tables can be downloaded in multiple formats.Source: §3 · p. 12
3.1 The “Upload Data” Section
- Users can upload local performance data or select repository datasets, then review a summary of the loaded data.For large datasets, an option uniformly downsamples the data to speed uploading and plotting.Source: §3.1 · p. 12
3.2 The “Fixed-Target Results” Section
- Fixed-target analysis measures evaluations until a solution first meets a selected quality criterion, with results for single or multiple functions.The single-function section groups results into data summaries, expected runtime, probability distributions, cumulative distributions, algorithm parameters, and statistical tests.Source: §3.2 · p. 12, §3.2.1 · p. 12
3.2.1 The “Single Function” Subsection
- Within 24 evaluations, RLS succeeds on about 76% of target-run pairs across targets 4–16, versus 53% for (1, λ) GA.The example aggregates ECDFs over evenly spaced target values.Source: §3.2.1 · p. 17, Figure 8 · p. 19
3.2.2 The “Multiple Functions” Subsection
- Across multiple functions, users can compare ERTs, aggregate running-time ECDFs and AUCs, or rank algorithms using Glicko-2.The ranking is based on repeated simulated pairwise games using sampled running times.Source: §3.2.2 · pp. 20–21
3.3 The “Fixed-Budget Results” Section
- Fixed-budget analysis evaluates the best solution found within a given number of evaluations, but its multiple-function cumulative-distribution feature was still under development.The paper describes this as the state of the tool at the time of writing.Source: §3.3 · p. 21
3.4 The “Position Information” Section
- Position Information uses parallel coordinates to display each run’s final search point, but the described functionality supports only SOS-framework data.Extensions to other data sources and analysis types were in progress.Source: §3.4 · p. 21
3.5 Command-Line interface
- The R command-line interface provides fine-grained control and access to all functionality described in the paper.Source: §3.5 · p. 21
4 Discussion and Outlook
IOHanalyzer is presented as a versatile environment for evaluating IOH performance data, and the IOHprofiler project is under continuous development. Planned extensions target broader platform compatibility, benchmark collections, statistical methods, and performance aggregation.
4 Discussion and Outlook
- IOHanalyzer is presented as a versatile environment for evaluating IOH performance data, with further IOHprofiler development planned.Near-term extensions include increased compatibility with additional benchmarking environments and platforms.Source: §4 · p. 21
- Nevergrad data visualization is already supported, while additional interfaces are being developed to connect Nevergrad and IOHprofiler logging and benchmark problems.The planned interfaces would let Nevergrad users access IOHprofiler logging and problems, and IOHprofiler users access Nevergrad benchmarks.Source: §4 · pp. 21, 23
- ParadisEO and a modular CMA-ES framework are integrated; adding other modular frameworks and automated configuration tools is intended to support a more generic optimization research environment.The proposed integrations include irace, SMAC, hyperband, and MIP-EGO; feature-extraction techniques such as those in flacco are also considered for supervised learning.Source: §4 · p. 23
- Existing benchmark support includes PBO, Wmodel, and BBOB, while interfaces to other benchmark collections and generators are in development.Multi-objective and noisy optimization are also being considered.Source: §4 · p. 23
- The authors are considering additional statistical procedures, including rank-based Bayesian inference and DSCtool, and an integration for empirical attainment functions.Empirical attainment functions could help visualize IOH time-quality-robustness trade-offs.Source: §4 · p. 23
- IOHanalyzer is implementing aggregation across test problems and considering additional anytime performance measures, including performance profiles based on normalized performance values.Performance profiles are described as empirical cumulative distributions of normalized performance across problems.Source: §4 · p. 23
A Supported Data Format
IOHanalyzer accepts several benchmark data formats and can usually detect the format automatically. Its processing relies on metadata files and raw-data records, with conventions for organizing runs, tracking parameters, and validating records.
A Supported Data Format
- Supported inputs include IOHprofiler, COCO, Nevergrad, SOS, and a simplified two-column format, with automatic format detection in the programming and graphical interfaces.The two-column format is a simplified IOHprofiler format; Nevergrad uses pure CSV files.Source: A Supported Data Format · pp. 29–30
- Raw-data files contain performance information indexed by running time and are organized in problem subfolders; the different data-file types share a common record format adapted from CSV.Metadata names identify problem files, while raw-data filenames can also identify dimension.Source: A Supported Data Format · pp. 30–31
- Metadata files with the .info suffix summarize performance by problem and can include multiple dimensions; their three-line structure records experiment metadata, comments, and data-file paths with instance information.The first line requires case-sensitive funcId, DIM, and algId fields; the second line begins with a percent sign, and the third gives the relative raw-data path and instance metadata.Source: A Supported Data Format · pp. 30–31
- Each separator line marks a new independent run, and only function evaluation and best-so-far f(x) are mandatory raw-data columns.Current f(x) records the observed value at an evaluation, while best-so-far f(x) tracks the best value since the run began.Source: A Supported Data Format · p. 31
- For improvement-based records, each run block must end with its final evaluation so IOHanalyzer can recover the used budget; incomplete data lines are dropped.Separators must retain double quotation marks, and columns must be separated consistently using either spaces or tabs.Source: A Supported Data Format · p. 32
- Customizable parameter columns can record dynamic internal-parameter states during search, and users can choose which parameters to track through IOHexperimenter.A single quotation mark is used if a parameter name needs a quotation mark.Source: A Supported Data Format · p. 32
- The examples show pairwise KS-test p-values and hypothesis rejections for 12 algorithms, ERT comparisons for RLS and (1, λ) GA on four PBO problems, and an aggregated ECDF across functions and targets.In the radar plot, greater span indicates better performance; RLS is superior on F01, F02, and F23, while (1, λ) GA is superior on F19.Source: Figure 9 · p. 33, Figure 10 · p. 34, Figure 11 · p. 35