PaperlayerView exact revision on arXiv

Source-linked AI summary

IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics

Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, Thomas Bäck

arXiv:1810.05281v1cs.NE

TL;DR

Comparing optimization heuristics across problems involves choices about algorithms, benchmarks, measures, and other properties, alongside technical differences in implementation. IOHprofiler combines experimental evaluation with interactive post-processing and parameter profiling. It reports fixed-target and fixed-budget performance distributions, while its current problem setting assumes static, noise-free objectives.

  • Problem

    Benchmarking must address how to compare algorithms across problems, choose benchmark problems and performance measures, and assess properties beyond performance.

    Source: §1 · p. 4

  • Method

    IOHprofiler combines COCO-based experiments on discrete-alphabet problems with post-processing that evaluates performance and tracks user-selected algorithm parameters.

    Source: §1 · p. 4, §1.2 · p. 5, §1.4 · p. 6

  • Results

    IOHprofiler reports distributions of fixed-target running times and fixed-budget function values, alongside statistics on algorithm parameters at fixed budgets or target values.

    Source: §1.3 · p. 5, §1.4 · p. 6

  • Takeaways & Limitations

    Interactive performance evaluation and parameter profiling provide information relevant to comparing algorithms and designing suitable optimization heuristics.

    Source: §1.3 · p. 5, §1.4 · p. 6, §4 · p. 11

    The current problem setting assumes static, noise-free objectives; extensions to dynamic, noisy, and multi-objective problems are under consideration.

    Source: §1.3 · p. 5

Abstract

from arXiv · show

IOHprofiler is a new tool for analyzing and comparing iterative optimization heuristics. Given as input algorithms and problems written in C or Python, it provides as output a statistical evaluation of the algorithms' performance by means of the distribution on the fixed-target running time and the fixed-budget function values. In addition, IOHprofiler also allows to track the evolution of algorithm parameters, making our tool particularly useful for the analysis, comparison, and design of (self-)adaptive algorithms. IOHprofiler is a ready-to-use software. It consists of two parts: an experimental part, which generates the running time data, and a post-processing part, which produces the summarizing comparisons and statistical evaluations. The experimental part is build on the COCO software, which has been adjusted to cope with optimization problems that are formulated as functions $f:\mathcal{S}^n \to \R$ with $\mathcal{S}$ being a discrete alphabet of integers. The post-processing part is our own work. It can be used as a stand-alone tool for the evaluation of running time data of arbitrary benchmark problems. It accepts as input files not only the output files of IOHprofiler, but also original COCO data files. The post-processing tool is designed for an interactive evaluation, allowing the user to chose the ranges and the precision of the displayed data according to his/her needs. IOHprofiler is available on GitHub at \url{https://github.com/IOHprofiler}.

1 Introduction

IOHprofiler addresses the challenge of comparing iterative optimization heuristics across problems while also examining how their parameters evolve. It combines experimental data generation with interactive post-processing for performance evaluation and profiling.

1 Introduction

  • Benchmarking provides empirical comparisons across optimization problems, while algorithm, benchmark, performance-measure, and profiling choices remain difficult and partly subjective.Technical differences in programming languages and execution platforms also need to be addressed.Source: §1 · p. 4

1.1 Iterative Optimization Heuristics

  • IOH are iterative search algorithms that evaluate candidate solutions and use their values to update how later candidates are generated.The class includes local and global search heuristics; the experimental focus is on discrete-alphabet problems, including pseudo-Boolean optimization.Source: §1.1 · p. 4, §1.2 · p. 5

1.2 Real-Valued Optimization

  • The post-processing tool accepts arbitrary real-valued optimization problems and original COCO data, although the experimental part focuses on discrete search spaces.The experimental part assumes maximization, while post-processing can detect whether an experiment used minimization or maximization.Source: §1.2 · p. 5

1.3 Performance Measures

  • IOHprofiler reports distributions of fixed-target running times and fixed-budget function values, with interactive control over display granularity, ranges, and precision.Evaluation counts are used as performance measures under the assumption that candidate evaluations dominate running time; they reflect CPU time accurately only when that assumption holds.Source: §1.3 · p. 5
  • The current problem setting assumes static, noise-free objectives; extensions to dynamic, noisy, and multi-objective problems are under consideration.The tool is available through GitHub, and the paper provides a manual for the experimental and post-processing components.Source: §1.3 · p. 5, §1.5 · p. 6, §1.9 · p. 7

1.4 Tracking Additional Information

  • Users can track algorithm parameters over time and obtain statistics at fixed budgets or fixed target values, supporting analysis relevant to heuristic design.The experimental component is built on COCO with modifications, while the post-processing component is original work.Source: §1.4 · p. 6, §1.8 · p. 6

2 Summary of Algorithm and Problem Requirements

IOHprofiler targets iterative heuristics whose performance is assessed through function evaluations. Its experimental setup supports discrete-alphabet problems and transformations of search points and objective values, subject to stated scope assumptions.

2.1 Iterative Optimization Heuristics

  • IOHprofiler focuses on iterative heuristics that use function evaluations to search, and measures performance by evaluation counts in fixed-target or fixed-budget terms.Fixed-target measures evaluations needed to reach a quality; fixed-budget measures the best quality found within an evaluation budget.Source: §2.1 · p. 7
  • The original focus is on general-purpose black-box algorithms, but IOHprofiler can also compare and profile problem-aware heuristics.In the black-box setting, algorithms learn about an instance through solution evaluations.Source: §2.1 · p. 8

2.2 Admissible Benchmark Problems

  • These transformations let users test whether an algorithm is invariant to changes in problem representation or absolute function values.Users can choose whether to apply transformations; without one, the algorithm receives the original function values.Source: §2.1 · p. 8, §2.2 · pp. 8–9
  • The experimental part assumes maximization over discrete integer alphabets, while post-processing accepts real-valued problems and either minimization or maximization.Both parts assume static, noise-free problems.Source: §2.2 · p. 8
  • Problem instances can transform objective values with multiplicative and additive shifts and transform search points with XOR shifts or permutations.The XOR shift is defined for pseudo-Boolean problems and can be extended to certain bounded integer search spaces.Source: §2.2 · p. 8

3 Precision of the Analysis and Data Format

IOHprofiler lets users choose how densely experimental data are recorded, including which algorithm parameters to track. Its output files organize per-run data and metadata for post-processing.

3 Precision of the Analysis and Data Format

  • Standard output tracks evaluated search points and current and best-so-far function values; users can also select algorithm parameters such as mutation rate for tracking.Tracked parameter statistics support analysis at fixed budgets and fixed targets.Source: §3 · p. 9
  • Four tracking modes are available: every evaluation (*.cdat), every τ evaluations (*.idat), each best-so-far improvement (*.dat), or user-set running-time budgets (*.tdat); *.dat is always generated.Time-based budgets are evenly spaced on a log-10 scale, with v and t set by the user.Source: §3 · p. 9
  • Figure 2 shows experimental log files containing current and best-so-far function values and possibly algorithm parameters; these files feed IOHprofiler’s post-processing part.Source: Figure 2 · p. 10
  • Each algorithm has a separate folder with an .info file for each benchmark function; performance data are stored in function subfolders, with different dimensions sharing a folder.When multiple dimensions are tested, their corresponding information is listed one below another in the .info file.Source: §3 · pp. 10–11
  • The first line of each .info file records the benchmark suite and function, dimension, algorithm name, and IOHprofiler version; a second line can hold user-specified experiment notes.The notes are specified in the configuration file.Source: §3 · p. 10
  • For each run, the .info file records the runtime-data path, number of stored data lines, and final best-so-far value; the example reports 12,503 data points and a best value of 100.Source: §3 · p. 11

4 Supported Performance Analyses

IOHprofiler provides fixed-target and fixed-budget performance analyses alongside tools for tracking algorithm-parameter evolution. Its outputs summarize run distributions and support interactive inspection at user-selected ranges and granularity.

4 Supported Performance Analyses

  • IOHprofiler combines fixed-target and fixed-budget benchmarking with tracking the evolution of algorithm parameters.These outputs support statistical comparisons of iterative heuristics and analysis of algorithm states relevant to optimizer design.Source: §4 · p. 11

4.1 Notation and Basic Terminology

  • Across independent runs, T records evaluations to reach a target, V records the best value within a budget, and averages and quantiles summarize their distributions.The tool also computes empirical cumulative distribution functions (ECDFs) for fixed-target and fixed-budget perspectives.Source: §4.1 · p. 12

4.2 Linking the Data Files

  • Users link folders containing performance data and can select IOHprofiler or COCO format; efficient mode speeds computation at the cost of precision.The upload interface identifies the processed data, including run counts and problem dimension.Source: §4.2 · p. 13

4.3.1 Fixed-Target: ‘Data Summary’

  • Fixed-target data summaries report run counts, mean hitting times, and quantiles, and provide sorted first-hitting-time samples for individual runs.Users select target-value range and granularity; the raw samples can be downloaded as CSV.Source: §4.3.1 · p. 14, Figure 5 · p. 15

4.3 Fixed-Target Results

  • Fixed-target plots show average, median, and standard deviation against best-so-far values, while histograms and approximated probability mass functions describe hitting-time distributions.The approximation treats running times as continuous and can be imprecise when many values are duplicated.Source: §4.3.2 · p. 15, §4.3.3 · pp. 15–16, Figure 6 · p. 16

4.3.4 Fixed-Target: ‘Cumulative Distribution’

  • Fixed-target ECDFs show the fraction of run–target pairs reached by each budget; their areas are normalized against a theoretical algorithm that reaches fmax on its first evaluation.Users can select target ranges and steps, view individual-target curves, and inspect normalized areas.Source: §4.3.4 · pp. 16, 18, Figure 9 · p. 18

4.4 Fixed-Budget Results

  • Fixed-budget outputs include data summaries, expected target-value plots, histograms and approximate probability mass functions, and ECDFs for selected budgets.These plots are described as similar to the fixed-target displays.Source: §4.4 · p. 18

4.5 Parameter Evolution

  • Parameter-evolution displays summarize tracked values at selected target values, with averages, quantiles, and standard deviations available; corresponding fixed-budget results were planned for later release.The example contrasts a static population size of 50 with a dynamic population size averaging around 163 from function value 80.Source: §4.5 · p. 20, Figure 12 · p. 20

5 Conclusions and Possible Extensions

The work leaves benchmark-problem selection outside its scope while outlining future extensions to IOHprofiler’s performance measures and supported problem types.

5 Conclusions and Possible Extensions

  • Benchmark-problem selection is outside this work; for discrete benchmarks, the authors invite discussion of which problem features to represent and aggregate across.The proposed discussion starts from those design questions rather than prescribing a standardized set.Source: §5 · p. 20
  • IOHprofiler’s performance indicators count function evaluations; future time-based comparisons with non-iterative methods would face the challenge of system-independent measurement.The authors note that outputs would need adjustment for time-based indicators.Source: §5 · p. 21
  • Planned extensions include noisy, dynamic, constrained, and multi-objective problems, as well as other search domains such as permutation-based problems.Source: §5 · p. 21
  • Short-term plans include additional performance measures, comparisons across dimensions, standard statistical tests, and simpler creation of problem suites.Source: §5 · p. 21

A Manual for the Experimental Part: Data Generation

This section describes IOHprofiler’s experimental part and provides an illustrated manual for conducting user-chosen experiments.

A Manual for the Experimental Part: Data Generation

A.1 Preparation

Preparation requires a Python environment and C compiler, and the experimental and post-processing parts are obtained by downloading and extracting their archives.

A.1 Preparation

  • Running IOHprofiler requires a working Python environment and a C compiler; problems and algorithms can be supplied in Python or C.Source: §A.1 · p. 22
  • The software had been tested with Python 2.7.12 and gcc 5.4.1.Source: §A.1 · p. 22
  • The experimental and post-processing parts are downloadable from the IOHprofiler GitHub page, and the experimental archive must be extracted after download.Source: §A.1 · p. 22

A.2 Overview of the Main Steps

The data-generation workflow selects and configures a benchmark suite, sets up an algorithm, and runs the experiment to generate performance data.

A.2 Overview of the Main Steps

  • The C and Python interface files are located in their respective code-experiments/build directories.Source: §A.2 · p. 22

Benchmark Selection and Configuration

  • Benchmark selection and configuration define the problem suite, data-storage granularity, and results location in configuration.ini.Source: §A.2 · p. 22

Algorithm Setup

  • Algorithm setup replaces the example user algorithm function in user algorithm.c or user algorithm.py with the algorithm being evaluated.Source: §A.2 · p. 22

Data Generation

  • Running the C or Python command from the corresponding build directory generates performance data in the current path.Source: §A.2 · p. 22

A.3 The Configuration File

The configuration file selects benchmark suites, problems, instances, and dimensions, and controls output files and data granularity. It also records algorithm metadata and specifies which parameters to track.

A.3 The Configuration File

  • Configuration variables are grouped into [suite], [observer], and [triggers] sections.Source: §A.3 · p. 23
  • The [suite] section selects the suite, function IDs, instances, and problem dimensions for the experiment.The pre-installed PBO suite includes OneMax, LeadingOnes, a linear function with random weights, and a jump function with gap size k = 1; the authors recommend using PBO and adding benchmark functions to it.Source: §A.3 · p. 23
  • The [observer] section specifies the observer and output-file settings, with PBO suggested for current use.The configuration also names the result folder and algorithm, and can store additional algorithm information and selected parameter values in the output.Source: §A.3 · pp. 23–24
  • The [triggers] section sets performance-data granularity and determines which output files are generated.Complete triggers enable *.cdat output; number interval triggers set the function-evaluation step for *.idat files, while number target triggers and base evaluation triggers control *.tdat data.Source: §A.3 · p. 24

A.4 Adding New Benchmark Problems

Adding a user-defined benchmark problem requires defining it in a problem file and including it in a suite. The example workflow adapts the OneMax implementation and updates the suite to register the new problem.

A.4 Adding New Benchmark Problems

  • A new benchmark problem requires both a problem definition file and inclusion in a problem suite.Source: §A.4 · p. 24
  • The suggested starting point is to copy the OneMax example, replace its raw function implementation, and rename the relevant occurrences.If different instances are needed, create them in the new problem's allocation function.Source: §A.4 · p. 24
  • To register the problem in PBO, add its file to the suite header, call its allocator from PBO get problem, and update problem numbers in PBO initialize.Source: §A.4 · p. 24

A.5 Problem Instances

This section describes IOHprofiler’s problem-instance support and related resources, including transformed instances for testing algorithm sensitivity, configurable examples, and files for defining and configuring user algorithms.

A.5 Problem Instances

  • Users can combine objective scaling or shifting with search-point XOR or permutation transformations to test algorithm sensitivity across instances of one problem.For OneMax, an example composes XOR, scaling, and shifting, with a in [1/5, 5] and b in [−1000, 1000].Source: §A.5 · p. 25

A.6 Examples

  • The examples provide code and results for pure random search and a (1 + λ) evolutionary algorithm.They are located in /example/example1/ and /example/example2/, respectively.Source: §A.6 · p. 26

A.6.1 Example 1: Pure Random Search

  • The random-search example evaluates problems 1–4 at dimension 100 across instances 1–100, with configurable independent runs and interval, target, and evaluation triggers.Its configuration sets the result folder to EXP and stores the parameter named “evaluation”; output files record iteration, trigger, and improvement data.Source: §A.6.1 · pp. 26–27

A.6.2 Example 2: A (1 + λ) EA

  • The (1 + λ) EA example runs problems 1–4 at dimensions 100, 500, and 1000 on untransformed instance 1 while tracking mutation rate and flipped-bit count.Each run has a budget of 50 times the dimension and performs ten independent runs; each generation creates λ mutated offspring and evaluates them.Source: §A.6.2 · pp. 28, 30

A.7 Overview of the Different Files

  • The experimental codebase includes problem implementations, public interfaces, observers, transformation modules, and user-editable algorithm and configuration files.The overview identifies the user algorithm and configuration files as files that users need to edit.Source: §A.7 · pp. 30–31
  • The user files invoke the algorithm and define it, while configuration.ini holds configuration settings.user experiment.py invokes the user algorithm, user algorithm.py contains its definition, and configuration.ini is the configuration file.Source: §A.7 · p. 31
Loading 1810.05281v1…