PaperlayerView exact revision on arXiv

Source-linked AI summary

Influence of Initialization on the Performance of Metaheuristic Optimizers

Qian Li, San-Yang Liu, Xin-She Yang

arXiv:2003.03789v1cs.NEcs.LGmath.OC

TL;DR

Metaheuristic optimizers commonly initialize randomly, but initialization can affect their performance and lacks systematic comparison. The paper compares 22 initialization methods across five optimizers and 19 benchmark functions. It finds algorithm-specific sensitivity, distinct population-size needs, and useful distribution choices.

  • Problem

    The paper addresses the limited systematic evidence on how initialization distributions affect metaheuristic optimizer performance.

  • Method

    The study compares 22 initialization methods across DE, PSO, CS, ABC, and GA using 19 diverse benchmark functions and statistical analyses.

  • Results

    DE is relatively insensitive to initialization, whereas PSO, CS, and ABC are more sensitive, with suitable distributions varying by algorithm and problem.

  • Takeaways & Limitations

    Initialization should be selected with the optimizer and problem in mind, while population size and iteration count should be tuned under the evaluation budget.

  • Takeaways & Limitations

    The conclusions are preliminary because later selection effects on population diversity and nonlinear-constraint handling were not studied.

Abstract

from arXiv · show

All metaheuristic optimization algorithms require some initialization, and the initialization for such optimizers is usually carried out randomly. However, initialization can have some significant influence on the performance of such algorithms. This paper presents a systematic comparison of 22 different initialization methods on the convergence and accuracy of five optimizers: differential evolution (DE), particle swarm optimization (PSO), cuckoo search (CS), artificial bee colony (ABC) algorithm and genetic algorithm (GA). We have used 19 different test functions with different properties and modalities to compare the possible effects of initialization, population sizes and the numbers of iterations. Rigorous statistical ranking tests indicate that 43.37\% of the functions using the DE algorithm show significant differences for different initialization methods, while 73.68\% of the functions using both PSO and CS algorithms are significantly affected by different initialization methods. The simulations show that DE is less sensitive to initialization, while both PSO and CS are more sensitive to initialization. In addition, under the condition of the same maximum number of function evaluations (FEs), the population size can also have a strong effect. Particle swarm optimization usually requires a larger population, while the cuckoo search needs only a small population size. Differential evolution depends more heavily on the number of iterations, a relatively small population with more iterations can lead to better results. Furthermore, ABC is more sensitive to initialization, while such initialization has little effect on GA. Some probability distributions such as the beta distribution, exponential distribution and Rayleigh distribution can usually lead to better performance. The implications of this study and further research topics are also discussed in detail.

Acronyms

The paper uses abbreviations for five optimizers and several initialization or algorithm variants.

  • CS denotes Cuckoo Search, DE denotes Differential Evolution, GA denotes Genetic Algorithm, and PSO denotes Particle Swarm Optimization.
  • LHS denotes Latin Hypercube Sampling, while DE-a and PSO-w denote an adaptive DE variant and PSO with an inertia weight.

1 Introduction

The paper addresses the underexamined dependence of metaheuristic optimization on initialization, population size, and iteration count through a systematic comparison across representative algorithms and benchmark problems.

  • Motivation: Complex, nonlinear, multimodal, and nonsmooth optimization problems can make traditional methods difficult to apply, motivating metaheuristic alternatives.
  • Motivation: Most optimization algorithms depend on their initial settings, especially for large-scale and multimodal problems, yet systematic evidence about initialization distributions remains limited.
  • Prior approaches: Specialized initialization schemes may improve particular applications but can require sophisticated point allocation, increase computational costs, and lack generality across problems or algorithms.
  • Study design: The study compares 22 probability-distribution-based initialization methods across five representative optimizers: DE, PSO, CS, ABC, and GA.
  • Study design: Statistical ranking and correlation tests are used to identify suitable initialization methods for benchmark functions, alongside recommendations for population sizes and iteration counts.
  • Main findings: Initialization significantly affects some algorithms more than others: DE is relatively insensitive, whereas PSO is more sensitive.

2 Metaheuristic Optimizers

The paper reviews five metaheuristic optimizers, emphasizing their distinct search mechanisms and update operations that shape how they explore and exploit the search space.

  • Algorithms: The study uses DE, PSO, CS, ABC, and GA because they represent evolutionary, swarm-intelligence, Lévy-flight, bee-based, and cornerstone evolutionary mechanisms.
  • Differential Evolution: DE represents each candidate as a point in a D-dimensional space and generates mutation vectors by modifying current solution vectors.
  • Differential Evolution: DE crossover modifies solution components according to the crossover parameter CR or a randomly selected index, producing an updated vector.
  • Differential Evolution: Its simplified adaptive variant selects mutation strategies and parameters from predefined sets when better solutions are not found.
  • Particle Swarm Optimization: PSO updates particles using velocity, position, individual-best, and population-best information, with learning parameters and random coefficients.
  • Particle Swarm Optimization: A larger PSO inertia weight can accelerate convergence but also increase premature convergence; the study therefore uses a dynamically adjusted PSO-w variant.
  • Cuckoo Search: CS combines local and global search, using Lévy flights whose occasional long jumps improve exploration and help escape local optima.

3 Initialization Methods

The paper investigates probability distributions as alternative initialization strategies and evaluates how they affect algorithm performance.

  • The main objective is to examine different probability distributions for initialization and their effects on the performance of the selected algorithms.

3.1 Motivations of this work

Initialization can strongly affect optimizer convergence and accuracy, with the better distribution depending on the objective function. These examples motivate a systematic comparison of initialization methods.

  • A favorable initial population can reduce search effort and increase the chance of finding the true optimum.The location of the optimum is unknown, so initialization is commonly chosen without that information.
  • These contrasting examples show that uniform distributions are not consistently the best initialization method across functions.The preferred initialization changes between the Bukin and Michalewicz examples.
  • On the Bukin function, uniform initialization lets PSO-w converge near the optimum after five iterations, whereas beta initialization performs poorly.The supplied figure descriptions show the uniform run near the true optimum and the beta run far from it.
  • On the Michalewicz function, uniform initialization causes premature convergence, while beta initialization reaches the global optimum after five iterations.The two-dimensional optimum is reported as fmin = -1.801 at [2.20319,1.57049].
  • The study therefore evaluates diverse initialization methods, including Latin hypercube sampling and multiple probability distributions.The stated aim is to investigate initialization effects systematically rather than rely on isolated examples.

3.2 Details of initialization methods

The paper introduces initialization methods designed to distribute candidate solutions through the search space. Latin hypercube sampling provides a spatial-filling approach, while beta distributions add controllable shape parameters.

  • The paper first outlines the main initialization methods before conducting detailed simulations.
  • Latin hypercube sampling: Latin hypercube sampling divides each search-space dimension into equal intervals and randomly places samples within them.Ancillary variables ensure that each variable is represented in a fully stratified feature space.
  • Latin hypercube sampling: For three samples in two dimensions, LHS permits four possible location scenarios while maintaining stratified coverage.The samples can also be scattered in the diagonal subspace of the two-dimensional search space.
  • Latin hypercube sampling: LHS is intended to spread samples sparsely and avoid excessive aggregation in the search space.Prior studies report better spread than uniform distributions, but no distinct advantage for higher-dimensional problems.
  • Beta distribution: The beta distribution is defined on (0,1), with shape parameters a > 0 and b > 0 controlling its form.Its expected value and variance are determined by these parameters.

3.2.3 Uniform distribution

Uniform distributions are widely used for initialization and are defined over an interval whose limits determine their mean and variance. The passage also contrasts this with Gaussian distributions, which are widely used generally but not usually for initialization.

  • Uniform distributions are widely used in initialization over a bounded interval [a, b].
  • The interval limits a and b determine the uniform distribution’s expectation and variance.
  • Gaussian distributions are widely used in applications but are not usually used for initialization.
  • For a Gaussian distribution N(µ, σ2), µ determines the central location and σ determines the spread around the mean.

3.2.5 Logarithmic normal distribution

The logarithmic normal distribution is presented as an asymmetric alternative to the normal distribution. Its logarithm is normally distributed, and its mean and variance depend on μ and σ².

  • Logarithmic normal distribution: The logarithmic normal distribution is asymmetric, unlike the normal distribution.
  • Logarithmic normal distribution: A random variable X is lognormally distributed when ln X follows N(μ,σ²).The passage also gives expressions for its expectation and variance.

3.2.6 Exponential distribution

This section introduces exponential and Rayleigh probability distributions used among the initialization methods. The exponential distribution is characterized by a positive rate parameter and an asymmetric long tail.

  • The exponential distribution is asymmetric and has a long tail.
  • The Rayleigh distribution is introduced through its probability density function for use in the numerical experiments.

3.2.8 Weibull distribution

This section introduces the Weibull distribution as an initialization distribution and describes how its parameters connect it to exponential and Rayleigh distributions.

  • The Weibull distribution is defined by a probability density function with scale parameter λ and shape parameter k.
  • The Weibull distribution generalizes several other distributions through its shape parameter.
  • When k = 1, the Weibull distribution corresponds to an exponential distribution.
  • When k = 2, the Weibull distribution leads to the Rayleigh distribution.
  • The paper uses these probability distributions in subsequent numerical experiments.

4 Numerical Experiments

The experiments compare initialization methods across benchmark functions and metaheuristic algorithms under controlled evaluation budgets. Results show that initialization sensitivity varies substantially by algorithm, with DE generally less sensitive than PSO and CS.

  • Experimental design: The study tests five algorithms on benchmark functions with different modal shapes and numbers of local optima.
  • Experimental design: The experiments use a common maximum of 600000 function evaluations and 20 independent runs per initialization method.
  • Population and iterations: Under fixed evaluations, PSO-w performs best with a larger population, whereas CS performs better with a small population and repeated iterations.
  • DE-a: The DE-a Friedman test gives p = 0.617, indicating little overall sensitivity to initialization across the nine benchmark functions.
  • PSO-w: The PSO-w Friedman test gives p = 0.001, with Random, Bep(2.5, 2.5) and LHS ranked as its three best initialization methods.
  • CS: The CS Friedman test gives p = 0.00276, with Beta, Rayleigh and Uniform ranked as the leading initialization methods.
  • Overall sensitivity: 43.37% of functions show significant initialization differences for DE-a, compared with 73.68% for both PSO-w and CS.

5 Discussions

The analyses examine how initialization distributions, population settings, and initial distance relate to optimizer performance. Sensitivity varies across algorithms and problem types, with DE relatively robust, PSO and CS more affected, and ABC more affected than GA on simpler problems.

  • Initialization–performance relationship: The study tests whether the initial population’s average distance from the optimum predicts final-solution quality across repeated runs.Each initialization method and algorithm was run independently 20 times.
  • Initialization–performance relationship: The average initial distance and final-solution distance show no significant correlation in the reported tests.The CS analysis excludes an extreme log-transformed value as an outlier before correlation testing.
  • Algorithm sensitivity: DE shows little sensitivity to initialization, whereas PSO and CS are more affected; CS is the most sensitive among the three.The reported sensitivity ordering is CS > PSO > DE.
  • ABC and GA: ABC is greatly affected by initialization, while GA is little affected on basic functions but becomes sensitive on complex CEC functions.For ABC, beta, Rayleigh, and LHS variants are reported as better-performing methods; GA recommendations include beta, LHS, random, and uniform distributions.
  • Implications: The findings recommend algorithm-specific initialization studies, including beta, exponential, and Rayleigh distributions, while emphasizing that conclusions may depend on the objective landscape.The authors also propose testing hybrid distributions and validating them with rigorous statistical methods.
  • Population size and iterations: Initialization sensitivity and population requirements vary by algorithm: PSO generally needs larger populations, DE benefits from more iterations with smaller populations, and CS needs very small populations.The conclusions also note that sufficiently diverse populations and long runs can allow the algorithms to find optimal solutions.
  • Limitations: The study does not examine how selection mechanisms affect later population diversity or how nonlinear constraint handling changes the findings.These are identified as topics for further research.

6 Conclusions

The paper systematically compares initialization methods, population sizes, and iteration budgets across five optimizers and 19 diverse benchmark functions. Its conclusions show algorithm-dependent sensitivity and population requirements, while stressing that the findings remain preliminary and bounded by the benchmark setting.

  • Study scope: The study compares 22 initialization methods across five algorithms and 19 diverse benchmark functions.The comparison evaluates initialization effects on algorithm performance.
  • Population and iteration settings: Under equal function-evaluation budgets, DE generally favors smaller populations with more iterations, CS needs fewer than 100 individuals, and PSO requires larger populations.Population size and maximum iterations are both closely related to algorithm accuracy.
  • Initialization sensitivity: DE is relatively robust to initialization, while PSO, CS, and ABC are sensitive; GA is less sensitive on easy smooth functions but more sensitive on complex ones.Recommended methods differ by algorithm and problem complexity, including random, beta, LHS, Rayleigh, exponential, and uniform distributions.
  • Distance and final quality: The initial population’s average distance from the optimum does not significantly correlate with final-solution quality.The paper instead emphasizes maintaining population diversity and allowing enough iterations.
  • Scope and implications: The conclusions are preliminary because performance depends on both the algorithm and the problem.The paper calls for identifying suitable initialization methods for specific algorithm–problem combinations.
  • Future directions: Hybrid distribution methods are proposed for complex problems to increase population diversity, but their effects require rigorous statistical testing.The proposed testing concerns population statistical properties across iteration stages.
  • Scope and limitations: The experiments use benchmark problems with simple bounds on regular domains, so nonlinear constraints and irregular search domains remain untested.The authors suggest evaluating such settings using real-world problems.
  • Future directions: The paper proposes automatic or self-adaptive selection of initialization methods for a given algorithm and problem type.This is presented as a direction for future research.
Loading 2003.03789v1…