Source-linked AI summary
An Easy-to-use Real-world Multi-objective Optimization Problem Suite
Ryoji Tanabe, Hisao Ishibuchi
TL;DR
Synthetic test problems can contain unrealistic properties, motivating broader evaluation on accessible real-world multi-objective problems. The paper presents the 16-problem RE suite, supporting materials, approximated Pareto-front analyses, and comparisons of six algorithms. The suite varies in objectives, front shapes, and variable types, while four problems are mixed-integer; the paper also presents eight constrained problems.
Problem
Synthetic test problems may contain unrealistic properties that can distort evolutionary multi-objective optimization performance assessment, while accessible real-world benchmark suites are limited.
Method
The paper constructs 16 bound-constrained real-world problems using simple equations or surrogate models, provides source code, examines approximated Pareto fronts, and compares six evolutionary algorithms.
Results
The RE suite spans varied objective counts, Pareto-front shapes, and design-variable types; four problems are mixed-integer, and six algorithms are compared across the suite.
Takeaways & Limitations
The RE suite is intended as an off-the-shelf set of standard real-world benchmark problems for evaluating evolutionary multi-objective algorithms.
Takeaways & Limitations
Four surrogate-based RE problems use response-surface parameters from simulation-sampled data and therefore are not exactly their original problems, potentially making them “artificial.”
Abstract
from arXiv · showhide
Although synthetic test problems are widely used for the performance assessment of evolutionary multi-objective optimization algorithms, they are likely to include unrealistic properties which may lead to overestimation/underestimation. To address this issue, we present a multi-objective optimization problem suite consisting of 16 bound-constrained real-world problems. The problem suite includes various problems in terms of the number of objectives, the shape of the Pareto front, and the type of design variables. 4 out of the 16 problems are multi-objective mixed-integer optimization problems. We provide Java, C, and Matlab source codes of the 16 problems so that they are available in an off-the-shelf manner. We examine an approximated Pareto front of each test problem. We also analyze the performance of six representative evolutionary multi-objective optimization algorithms on the 16 problems. In addition to the 16 problems, we present 8 constrained multi-objective real-world problems.
1. Introduction
Synthetic problems are convenient for benchmarking evolutionary multi-objective optimization algorithms but often contain unrealistic properties that can distort performance assessment. The paper introduces an easy-to-use suite of real-world problems to broaden empirical evaluation.
- Evolutionary multi-objective optimization algorithms are usually assessed empirically because their stochastic nature makes theoretical performance evaluation difficult.
- Synthetic test problems are easy to implement and evaluate, with relatively clear properties such as Pareto-front shapes.
- Unrealistic properties in synthetic problems can overrate algorithms that exploit those properties, including decomposition-based methods on triangular Pareto fronts.
- Real-world problems are needed for more reliable evaluation, but existing problems are difficult to use and rarely provide broad benchmarking coverage.
- The RE suite provides 16 bound-constrained real-world problems defined by simple equations or surrogate models, covering varied objectives, Pareto-front shapes, and design variables.
- The paper contributes problem details, source code in Java, C, and Matlab, comparisons of six evolutionary algorithms, and eight additional constrained problems.
2. RE problem suite
The RE suite provides 16 easy-to-use bound-constrained real-world multi-objective problems spanning objectives, variables, and Pareto-front structures, plus eight constrained counterparts. Its approximated fronts reveal irregular and disconnected structures, while the suite remains bounded by surrogate-based, historical, and low-dimensional scope constraints.
- RE problem suite: Four RE problems use response-surface parameters from simulation data and therefore are not exactly their corresponding original problems.The authors caution that these surrogate-based problems may be viewed as artificial despite their real-world application data.
- Constraint handling: Constraint violations are reformulated as objective functions for most RE problems, while eight CRE problems preserve the original constrained formulations.The sum of constraint violation values is added as an objective except for RE9-7-1; CRE problems correspond to the original versions.
- Scope: The suite focuses on unconstrained problems as a first step, although the authors provide eight CRE problems without analyzing them in this paper.The CRE problems are presented as potential benchmarks for constrained multi-objective optimizers.
- Variable types: Five RE problems use integer or discrete variables, including four mixed-integer problems and one integer problem, but none include categorical variables.The suite is intended to support research on multi-objective mixed-integer optimization, for which suitable benchmark problems are described as scarce.
- Pareto-front analysis: Approximated fronts show continuous, disconnected, mixed, locally concave or convex, and sharp-knee structures across the two-objective RE problems.RE2-4-3 and RE2-3-5 have disconnected fronts; RE2-2-4 has a convex front with a sharp knee, and RE2-3-5 combines linear and convex regions.
- Pareto-front analysis: For problems with three or more objectives, front shapes are generally difficult to visualize clearly because most RE problems have complicated Pareto fronts.Scatter matrices provide clear shapes only for RE3-4-6 among the listed cases, while parallel coordinates are also described as insufficient for front shape.
3. Experimental settings
The experiments assess representative EMOAs on the RE problem suite using HV and performance scores, with normalization and settings tailored to differing objectives and variable types.
- 3. Experimental settings: HV was selected because real-world Pareto fronts and optimal reference vectors are unavailable.The indicator captures convergence and diversity without requiring reference objective vectors.
- 3. Experimental settings: Each problem’s objectives were normalized using approximated ideal and nadir points before HV calculation.The ideal and nadir points were obtained from L-SHADE and five EMOAs using 31 independent runs and 100,000 evaluations.
- 3. Experimental settings: Performance scores rank algorithms by how many competitors significantly outperform them under the Wilcoxon rank-sum test.A small score indicates that an algorithm is not worse than many others.
- 3. Experimental settings: Five representative EMOAs were evaluated using the jMetal 4.5 framework.The algorithms cover dominance, decomposition, indicator, and reference-vector approaches.
- 3. Experimental settings: All EMOAs used SBX crossover and polynomial mutation, with population sizes varying by objective count and rounding for discrete variables.The crossover probability was pc = 1, mutation probability pm = 1/D, and both distribution indices were 20.
- 3. Experimental settings: MOEA/D-TCH and MOEA/D-PBI used normalization to handle RE problems whose objective values have different scales.For MOEA/D-PBI, θ = 5 balances convergence distance d1 and diversity-related distance d2.
4. Experimental results
The results show no universal winner across two-objective problems, while IBEA often performs best on problems with three or more objectives. NSGA-II outperforms NSGA-III on most problems by performance-score rank, and MOEA/D-PBI performs poorly overall.
- 4. Experimental results: There is no clear winner across all two-objective RE problems.IBEA, MOEA/D-TCH, NSGA-II, and SMS-EMOA each obtain the best HV on different two-objective problems.
- 4. Experimental results: IBEA obtains the best HV on 6 out of 10 RE problems with three or more objectives.The IBEA implementation uses the additive epsilon indicator rather than directly optimizing HV.
- 4. Experimental results: MOEA/D-PBI shows the worst performance on 10 out of 16 RE problems.This contrasts with its strong performance on many-objective problems reported in previous studies.
- 4. Experimental results: NSGA-II outperforms NSGA-III on 10 out of 16 RE problems by performance-score rank.This comparison uses performance-score ranks rather than raw HV values.
- 4. Experimental results: Irregular Pareto-front shapes and a small evaluation budget are identified as two reasons for weak NSGA-III and MOEA/D-PBI performance.The study notes that some algorithms require larger evaluation budgets, although real-world simulations may make such budgets impractical.
5. Conclusion
The RE problem suite provides 16 real-world multi-objective optimization problems with source code for standard benchmarking, and compares six EMOAs on them. The paper argues that the suite can address undesirable features of synthetic test problems while motivating further benchmark and problem analyses.
- Conclusion: The RE problem suite contains 16 real-world multi-objective optimization problems, with Java, C, and Matlab source code supplied as supplementary materials.The suite is intended for use as a standard benchmark problem set.
- Conclusion: The study compares six evolutionary multi-objective optimization algorithms on the RE problems.
- Conclusion: The authors identify further analysis of each RE problem and the design of other benchmark sets, including dynamic real-world problems, as future work.Suggested analyses include feasible-region shape, distance from initial solutions to the Pareto front, and convergence and diversification difficulty.
- Conclusion: Figure 4 summarizes progress on multi-objective continuous test problem sets.