PaperlayerView exact revision on arXiv

Source-linked AI summary

Metaheuristic Design of Feedforward Neural Networks: A Review of Two Decades of Research

Varun Kumar Ojha, Ajith Abraham, Václav Snášel

arXiv:1705.05584v1cs.NEcs.LG

TL;DR

The paper addresses how FNN optimization can improve generalization despite limitations of gradient-based methods and the absence of a universal optimizer. It reviews conventional, metaheuristic, hybrid, multiobjective, ensemble, and evolving-FNN approaches, concluding that metaheuristics support a wide spectrum of FNN adaptations while leaving data quality, dynamic training, and computational cost as challenges.

  • Problem

    Gradient-based FNN optimization has limitations, while no single optimization method solves every problem, motivating broader approaches to obtain generalized FNNs.

  • Method

    The article comprehensively reviews conventional, metaheuristic, hybrid, multiobjective, ensemble, and component-combination approaches for FNN optimization.

  • Results

    The review identifies a wide spectrum of FNN optimization and adaptation through metaheuristics, including evolving, cooperative-coevolution, and hybrid designs.

  • Takeaways & Limitations

    Metaheuristic treatment can support generalized FNN design through simultaneous component adaptation, multiobjective training, and ensembles of candidate networks.

  • Takeaways & Limitations

    Metaheuristic training time grows with population size, while FNN training remains sensitive to data quality and dynamic training remains an open problem.

Abstract

from arXiv · show

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era.

1 Introduction

FNN optimization seeks suitable network structures and weights to improve generalization, moving beyond gradient-based methods toward metaheuristic and ensemble approaches. The review surveys conventional, metaheuristic, multiobjective, ensemble, and future research directions.

  • FNN optimization searches for an appropriate network structure and weights, including neurons, layer arrangements, and connection parameters.
  • Gradient-based techniques were initially popular, but their limitations motivated the adoption of metaheuristic optimization methods.
  • Metaheuristics formulate FNN components as optimization variables and can address approximation error, generalization, and model simplification simultaneously.
  • An ensemble of candidates from a metaheuristic final population may provide a more general solution than selecting only the best FNN.
  • The review covers FNN fundamentals, conventional optimization, metaheuristic and hybrid methods, multiobjective approaches, ensembles, challenges, and future research directions.

2 Feedforward neural networks

FNNs are layered ANN models used for function approximation, classification, regression, and prediction. The section distinguishes them from recurrent networks and notes that the article focuses specifically on FNNs.

  • An FNN arranges neurons in layers with forward connections from each layer to the previous one, giving the model a structural network representation.
  • MLPs extend single-layer perceptrons with hidden layers, enabling solutions to nonlinearly separable patterns.
  • FNNs can approximate any continuous function under the stated single-hidden-layer and continuous sigmoidal-activation conditions.
  • Radial basis function networks and support vector machines are described as special three-layer FNN classes for supervised regression and classification.
  • Recurrent networks use feedback connections for sequence, temporal-association, memory, and nonlinear-dynamic-system tasks, but the review is limited to FNNs.

2.1 Components of FNNs

FNNs represent functions through layered neurons, weights, activation functions, and biases. Their optimization involves discovering both an appropriate architecture and an appropriate weight vector.

  • FNNs consist of neurons connected by weights and arranged layer by layer, with each layer receiving forward connections from the previous layer.
  • A node processes weighted inputs and uses an activation function to limit its output amplitude to a certain range.
  • The FNN phenotype represents a function f(x, w) parameterized by an input vector x and a real-valued weight vector w.
  • Architecture optimization selects activation functions, node counts, layer counts, and connection arrangements.
  • FNN optimization also includes learning algorithms and learning environments alongside weights, architecture, and node choices.

2.2 Influencing factors in FNN optimization

FNN optimization is shaped by training objectives, learning paradigms, error surfaces, and generalization requirements. Practical conditions can make global optimization difficult and make performance sensitive to data quality and changing environments.

  • FNN training supplies N input–output pairs and compares the network’s predicted output vectors with desired output vectors.
  • Supervised learning minimizes the difference between desired and predicted outputs through a cost function, with different functions used for regression and classification.
  • Squared-error cost functions are described as inconsistent for classification, where accuracy and misclassification rate are used instead.
  • Gradient-based learning can fall into local minima or plateaus, and conditions proposed to avoid them may be unlikely in real-world problems.
  • Generalization requires balancing bias and variance and avoiding underfitting and overfitting, with training patterns sufficiently exceeding network connections.
  • Standard generalization strategies include architecture selection, early stopping, regularization, and adding noise to training data.

2.3 Conventional optimization approaches

Conventional FNN optimization methods update weights using gradient-based search, including first-order BP and second-order approaches. Their variants differ in update direction, curvature information, and parameter sensitivity.

  • Gradient-based methods: Gradient-descent optimization starts from an initial weight vector and iteratively updates weights to reduce the cost function.The steepest-descent update uses the negative gradient scaled by the learning rate.
  • Second-order methods: Second-order methods use curvature information through Hessian-based updates, while conjugate gradient searches directions conjugate to previous steps.Conjugate-gradient steps are described as non-interfering because later steps do not undo earlier minimization.
  • Gradient-based methods: BP propagates output error backward through hidden layers using forward and backward computation phases.Its weight update depends on the previous-layer inputs, learning rate, and momentum factor.
  • Gradient-based methods: BP is sensitive to learning rate and momentum: excessive values can cause zigzagging, overshooting, or nonconvergence, while small values slow learning.The classical algorithm is slow and tends to fall into local minima.
  • Improved conventional methods: Quickpro, learning-rate acceleration techniques, and Rprop were proposed to improve BP training and its parameter sensitivity.Rprop enlarges weight changes when gradient signs persist and reduces them when signs change; it is reported faster than BP but slower than Quickpro.
  • Other conventional methods: Other conventional approaches include Quasi-Newton, Gauss-Newton, Levenberg-Marquardt, Kalman-filter, and recursive least-squares methods.Gauss-Newton frames FNN optimization as nonlinear least squares, and LM is reported to outperform BP, CG, and Quasi-Newton in cited studies.

2.4 Comments on conventional approaches

Conventional algorithms are fast and support online or batch training, but their local search and weight-focused formulation limit broader FNN optimization. Metaheuristics address structure and complexity more directly, although they introduce different trade-offs.

  • Advantages: Gradient-based methods operate on a single weight vector, making them computationally faster than methods that evaluate multiple solution vectors.BP and Online BFGS support both stochastic and batch FNN training.
  • Advantages: 10 benefits of stochastic or online training include redundancy handling, dynamic incorporation of currently absent data, and faster training than batch mode.The supplied passage lists these as basic advantages of online training.
  • Trade-offs: Batch training can guarantee a local minimum under a simple condition and may be faster than stochastic training for larger datasets.This contrasts with online training, which offers dynamic learning and can be faster in other settings.
  • Limitations: Conventional methods tend to fall into local minima and are primarily limited to optimizing FNN weights.Their cost-function requirements also constrain methods such as Gauss-Newton and Levenberg-Marquardt.
  • Limitations: Metaheuristics can directly support automatic FNN structure determination and complexity reduction, unlike conventional methods focused mainly on weights.The motivation is tied to improving generalization through fewer weights and less complex architectures.

3 Metaheuristic approaches

Metaheuristics provide alternatives to local gradient-based FNN optimization by balancing exploration and exploitation and handling broader objective-function classes. Their effectiveness remains problem dependent under the no-free-lunch principle.

  • Rationale: Metaheuristics combine exploration of unknown search regions with exploitation of discovered solutions to seek near-global FNN optima.These strategies are described as contradictory, requiring a trade-off in search design.
  • Capabilities: Metaheuristic algorithms can address complex, nonlinear, and non-differentiable problems that conventional methods requiring continuous differentiable objectives cannot.This capability is presented as an alternative to conventional FNN optimization.
  • Problem dependence: The no-free-lunch theorem asks whether a general-purpose optimization algorithm exists and compares problem sets where competing algorithms perform better.The paper uses this theorem to frame algorithm specialization by problem structure.
  • Problem dependence: A universal optimization strategy is impossible; one strategy can outperform another when specialized to the specific problem structure.The paper notes that most real-world problems are not closed under permutation, motivating continued FNN-specific algorithm development.

3.1 Metaheuristic algorithms

FNN metaheuristics include single-solution, population-based, evolutionary, swarm, physics-inspired, and hybrid methods. Hybridization combines complementary search behaviors, while the literature also faces classification and novelty concerns.

  • 3.1.1 Single solution based algorithms: Single-solution metaheuristics improve one candidate through neighborhood search, including simulated annealing, tabu search, variable neighborhood search, and GRAP.They are also characterized as local-search methods because they exploit the current solution.
  • 3.1.1 Single solution based algorithms: Simulated annealing uses probabilistic acceptance inspired by metallurgical cooling, while tabu search discourages revisiting previously explored solutions.Tabu search adds restrictions to acceptance and is described as improving upon simulated annealing.
  • 3.1.2 Population based algorithms: Population-based algorithms operate on multiple candidates and generally provide greater exploration or global-search ability than single-solution methods.The paper groups evolutionary, swarm, and other nature- or phenomenon-inspired algorithms within this family.
  • 3.1.2 Population based algorithms: Evolutionary algorithms use natural-selection dynamics and operators such as selection, crossover, and mutation to search for near-optimal solutions.GA, EP, ES, GP, and differential evolution differ in operators and whether they evolve vectors, parameters, or program structures.
  • 3.1.2 Population based algorithms: Swarm intelligence algorithms model collective self-organization, with PSO, ACO, and ABC using particles, pheromone communication, or bee roles to search solutions.PSO represents particles as weight vectors with location and velocity; ACO uses pheromone-mediated path selection, and ABC assigns employed, onlooker, and scout bees distinct roles.
  • Other metaheuristics and concerns: The expanding metaheuristic literature has prompted criticism that many algorithms share similar mechanisms and lack groundbreaking optimization differences.The paper acknowledges this criticism while continuing to discuss numerous algorithm families and combinations.
  • 3.1.3 Hybrid and memetic algorithms: Hybrid and memetic algorithms combine conventional and metaheuristic methods or multiple metaheuristics to exploit complementary heuristics.Examples include GA with BP, PSO with BP, ABC with LM, and GA with PSO; Figure 2 depicts reciprocal use of methods for initial weights and later optimization.

3.2 Metaheuristic formulation of the FNN components

Metaheuristic FNN optimization formulates network components as genotypes and searches them singly, jointly, or hierarchically. The review covers weight representations, population-based exploration, component co-adaptation, and learning-rule optimization.

  • Metaheuristics search near-optimal FNN solutions without guaranteeing global optimality, particularly for complex problems with incomplete or imprecise information.
  • FNN performance depends on jointly choosing weights, architecture, activation functions, learning parameters, and training environment, not weights alone.
  • 3.2 Metaheuristic formulation of the FNN components: The spectrum ranges from weight-only optimization to increasingly complex combinations involving architecture, activation functions, and learning-rule parameters.
  • 3.2 Metaheuristic formulation of the FNN components: Components can be optimized separately or simultaneously by encoding one component, or a combination of components, into a vector representation.
  • 3.2.1 Weight optimization: Population-based evolutionary and swarm methods explore multiple agents, giving them greater search-space exploration ability than single-solution methods such as SA, TS, and BP.
  • 3.2.1 Weight optimization: Real-coded chromosomes avoid binary weight encoding’s precision and computational costs, while GA and DE efficiently optimize real-valued weight vectors.
  • 3.2.1 Weight optimization: Reported studies found CCPSO better than BP and GA for fuzzy neural-network optimization, FF better than BP, GA, and ABC for weight vectors, and metaheuristics superior to conventional methods in several training studies.
  • 3.2.6 Combination of FNN components optimization: Evolutionary neural networks and neuroevolution extend adaptation beyond learning, while simultaneous architecture-and-weight optimization was reported to improve generalization.

3.3 Comments on metaheuristics approaches

Metaheuristics broaden FNN optimization beyond gradient-based methods and can support generalized architectures, but their population-based search increases training time and lacks universal convergence guarantees.

  • Metaheuristics enable FNN optimization across multiple components and can facilitate architecture simplification linked to generalization.Their population-based search evolves network components toward generalized solutions.
  • Metaheuristic training time grows with population size because many candidate solutions are evaluated during optimization.Population-based optimization processes multiple FNN candidates rather than a single solution.
  • Both conventional and metaheuristic FNN training require substantially more time than extreme learning machines.ELM randomly assigns input-to-hidden weights and updates hidden-to-output weights in one least-squares step.
  • No metaheuristic is expected to solve every problem, and its convergence speed or attainment of a satisfactory solution must be established empirically.The paper attributes this difficulty to problem-specific behavior and the absence of theoretical convergence guarantees.

4 Multiobjective metaheuristic approaches

The review presents multiobjective FNN optimization as a way to balance training error with generalization-related objectives such as validation error, regularization, complexity, and input reduction. It contrasts scalarized objectives with Pareto-based approaches that retain trade-off solutions for generalized networks.

  • Why multiobjective optimization is needed: Multiobjective FNN optimization simultaneously addresses training error and additional objectives because low training error alone does not ensure generalization.Candidate additional objectives include validation error, regularization, network complexity, and input-dimension reduction.
  • Approaches in the literature: FNN studies apply aggregated or Pareto-based objectives to combinations of training error, validation error, regularization, complexity, architecture, and activation functions.Examples include scalarized formulations and evolutionary or hybrid approaches that optimize several network properties together.
  • Scalarized objectives: Scalarizing conflicting objectives requires a hyperparameter and produces one model without revealing how the trade-off was handled.The paper identifies hyperparameter selection and single-solution output as disadvantages of scalarized generalization objectives.
  • Pareto-based optimization: Multiobjective algorithms guide populations toward the Pareto front, estimate solution density, and maintain distribution across trade-offs.The described operators include fitness assignment and density estimation, alongside an additional diversity-preserving operator.
  • Approaches in the literature: The review identifies Pareto-based treatment as an efficient choice because aggregated objectives make control-parameter selection challenging.The cited discussion reports Pareto-based methods being used to obtain generalized networks in real-life applications.
  • Pareto-based optimization: Pareto-based optimization evaluates nondominated solutions while preserving alternatives across competing FNN objectives.Pareto-optimal solutions cannot improve one objective without worsening at least one other objective.

5 Ensemble of feedforward neural networks

The review describes ensembles that combine diverse FNN candidates from metaheuristic populations, using voting or averaging and increasingly selective procedures. It emphasizes that ensemble quality depends on both accuracy and diversity, not simply population size or individual fitness.

  • Constructing ensembles: Metaheuristic final populations provide diverse FNN candidates differing in weights, structures, and training subsets for ensemble construction.The review identifies parametric, structural, and training-set diversity among final-population solutions.
  • Constructing ensembles: Candidate decisions can be combined through majority voting, arithmetic means, rank-based combinations, or linear combinations.Majority voting is used for classification, while arithmetic means are used for regression.
  • Selection and diversity: Ensemble design must determine its size and preserve diversity, motivating methods such as negative correlation learning and cooperative construction.Some procedures automatically select ensemble members or size while considering accuracy and diversity during training.
  • Selection and diversity: Training fitness alone is insufficient for selection because it does not capture each candidate’s role or influence within the ensemble.A GA-based selective ensemble evaluates candidate subsets and strengths, and the selected subset outperformed the whole population in the cited study.
  • Training-data and input-space partitioning: A few bootstrapped FNNs can outperform larger ensembles, showing that ensemble size alone does not determine performance.Partitioning training data allows different FNNs to learn different samples or input-space regions.
  • Training-data and input-space partitioning: Clustering and coevolution create FNNs specialized for different input-space regions while reducing input nodes and learning runtime.The approach partitions the input space and uses divide-and-conquer coevolution.
  • Selection and diversity: Accuracy and diversity are contradictory ensemble objectives, so multiobjective methods can evolve populations that optimize both simultaneously.The review cites multiobjective regularized negative correlation learning as an approach that maximizes performance and negative correlation.

6 Challenges and future scopes

FNN research faces challenges from data quality, high-dimensional and streaming data, heterogeneous views, and the need for simpler models. The paper identifies these as open directions for improving FNN training and adaptation.

  • FNN training depends on data quality across accuracy, reliability, timeliness, relevance, completeness, currency, consistency, flexibility, and precision.
  • Insufficient data can be addressed by generating virtual samples, but efficient FNN training on such data remains an open question.
  • Deep learning and encoding or decoding methods help process or reduce high-dimensional data, giving FNNs a role in feature reduction.
  • Dynamic FNN training for non-stationary data streams remains an open problem despite retraining-based online learning.
  • Future models must accommodate heterogeneous, evolving data and new knowledge while remaining simple enough for IoT applications.

7 Conclusions

The conclusion contrasts gradient-based local search with metaheuristic FNN optimization, which supports broader exploration and simultaneous adaptation of FNN components. It highlights multiobjective training, ensembles, and future work on data quality and big-data challenges.

  • Conventional FNN optimizers such as backpropagation usually find local minima because their local-search procedures lack exploration ability.
  • Metaheuristics combine exploitation and exploration and can simultaneously adapt multiple FNN components, motivating hybrid and dynamic designs.
  • Metaheuristic FNN optimization represents network phenotypes as vectors or other manipulable forms, enabling a wide spectrum of adaptation methods.
  • Multiobjective training addresses approximation error, network complexity, and input dimension, while ensembles combine candidates to support accuracy and diversity.
  • Future FNN paradigms are expected to address data-quality problems and challenges posed by big-data information processing.
Loading 1705.05584v1…