The limitation of neural nets for approximation and optimization

1 Introduction

• The study evaluates feedforward neural networks as surrogate models against interpolation and regression models on nonlinear optimization test problems.It considers objective functions with varied linearity, convexity, and separability, assessing both local derivatives and broader function-value behavior.Source: §1 · p. 2
• SiLU provides the best performance among the tested activation functions for approximating nonlinear optimization objectives.Source: §1 · p. 2
• Neural networks are competitive for zero- and first-order approximations but underperform interpolation/regression models for second-order approximations, despite high training cost.Source: §1 · p. 2
• Combining an activation function with the natural quadratic basis can remove cross terms and reduce the number of parameters to determine.Source: §1 · p. 2
• Approximating gradients with the tested surrogate models hardly improves the performance of the state-of-the-art derivative-free optimization algorithm.The objective-function derivatives are evaluated for approximation accuracy but are unavailable when solving the optimization problems.Source: §1 · p. 3

2.2 Numerical experiments

• The experiments compare ReLU, ELU, SiLU, Sigmoid, and Tanh using neural networks trained to minimize mean squared error on objective-function samples.Testing datasets measure performance on points not used during training.Source: §2.2 · p. 5
• Performance profiles rank methods by relative convergence effort, with ρs(1) measuring efficiency and large-α values measuring robustness.Source: §2.2 · p. 5
• The common architecture uses two hidden layers with 4n neurons per layer, 300 Adam epochs, dynamic learning-rate reduction, and normalized samples from a unit ball around the initial point.Training uses (n + 1)(n + 2)/2 points, while testing uses 0.2(n + 1)(n + 2)/2 points.Source: §2.2 · p. 6
• SiLU achieves the best testing-dataset performance for both tolerances, followed by ELU and ReLU, across the two CUTEst problem sets.ReLU performs best on the 38-problem training set, whereas SiLU and ELU outperform it on the 53-problem training set.Source: Figure 1 · p. 7, §2.2 · p. 8, Figure 2 · p. 8

2.1 Review of popular activation functions in machine learning

• ReLU, ELU, and SiLU outperform Sigmoid and Tanh in efficiency and robustness, while sigmoidal activations suffer from flat regions and vanishing gradients.Source: §2.1 · p. 4, §2.2 · p. 8

3.1 Review of polynomial interpolation and regression

• Polynomial interpolation and regression use sample sets whose size determines whether models are underdetermined, determined, or overdetermined, with quadratics capturing curvature beyond linear models.Quadratic interpolation is determined at (n + 1)(n + 2)/2 samples, while regression uses more samples and underdetermined interpolation uses fewer.Source: §3.1 · pp. 10–11

3.2 The use of an activation function as a basis function

• Activation-function bases that remove quadratic cross terms achieved the most accurate function, gradient, and Hessian approximations on 38 problems, while radial basis functions were slightly worse for Hessians.The reduced basis has 3n + 1 degrees of freedom, fewer than the (n + 1)(n + 2)/2 terms required for determined quadratic interpolation.Source: §3.2 · pp. 12–13, Figure 5 · p. 14
• Across 53 problems, reduced activation-function bases performed similarly to the natural and cross-term bases, suggesting that quadratic cross terms may be unnecessary even with negative curvature.The conclusion was unchanged when evaluations used points between the origin and the initial point.Source: Figure 6 · p. 14, §3.2 · p. 15

3.3 The use of neural networks for approximation

• Neural networks were evaluated on matched datasets using five activations and the same function-value, gradient, and Hessian error metrics as interpolation/regression models.Training used (n + 1)(n + 2)/2 randomly generated points in a unit ball around each CUTEst initial point.Source: §3.3 · p. 15, Figure 7 · p. 16, Figure 8 · p. 16
• Neural networks approximated function values and gradients most accurately with ReLU variants, but no activation captured Hessian information accurately except that SiLU performed best.Sigmoidal activations often produced nearly zero gradients because of vanishing gradients, whereas ReLU’s curvature limitation follows from its piecewise-linear form.Source: §3.3 · pp. 15, 17, §3.2 · p. 15

4.1 The use of a surrogate model for optimizing without derivatives

• The surrogate-based method replaces finite-difference gradients requiring n additional evaluations with gradients from interpolation, regression, or neural-network models trained on collected evaluations.The model is built from the current dataset, whose accuracy can improve as more points are collected.Source: §4.1 · pp. 18–19
• Line-search samples can be poorly distributed for interpolation or regression models, whereas neural-network surrogates may require additional points to approximate the objective accurately.A direct-search step can collect more evenly distributed points to address both limitations.Source: §4.1 · p. 19

4.2 Numerical experiments

• The experiments compare FLE against FLE-S using polynomial and radial-basis surrogates plus ReLU and SiLU neural networks across two test-problem sets.Neural networks use all collected points for training, while interpolation or regression models use selected local samples; SiLU was chosen because it captured curvature information best among the tested neural activations.Source: §4.2 · pp. 20–21
• FLE-S did not outperform the default FLE method with any tested surrogate, and neural-network performance deteriorated under the stricter convergence test.For larger problems with τ = 10^-2, neural-network FLE-S sometimes became more efficient but still lacked FLE's robustness; with τ = 10^-5, efficiency and robustness significantly deteriorated.Source: §4.2 · p. 21

• SiLU delivered the best activation-function performance on both test sets, with superior testing efficiency and robustness.The evaluation used performance profiles on sets of 38 and 53 nonlinear optimization problems.Source: §5 · p. 21
• Activation-function composition with the natural quadratic basis improved function-value and gradient approximations without requiring cross terms.This produced models with fewer parameters to determine.Source: §5 · p. 21
• Neural networks produced highly inaccurate Hessian approximations for most activations, except SiLU.ReLU and its variants provided better function-value and gradient approximations, but Hessian accuracy remained poor.Source: §5 · pp. 21, 23
• Surrogate-based gradient approximations did not significantly improve the default FLE method, while neural-network surrogates were less robust but more efficient on larger problems.Experiments replacing the objective function in line searches or adding surrogate minimization also did not perform well.Source: §5 · p. 23

• The first CUTEst collection contains 38 nonlinear unconstrained problems with user-defined dimensions.Source: A Appendix: Test Problems from CUTEst · p. 29, Table 1 · p. 29
• The second CUTEst collection contains 53 nonlinear unconstrained problems with corresponding dimensions.Source: A Appendix: Test Problems from CUTEst · p. 29

• Directional direct-search methods use polling directions and objective-function values without constructing gradient or objective models.They can converge on nonsmooth objectives.Source: B Appendix: Full-Low Evaluation Methods · p. 29
• Model-based methods build local objective models from sampled function values, but dense linear algebra, ill-conditioning, noise, and nonsmoothness limit their performance.Source: B Appendix: Full-Low Evaluation Methods · p. 30
• FLE combines directional and model-based ideas and had superior overall performance to interpolation-based methods and related direct-search combinations.Source: B Appendix: Full-Low Evaluation Methods · p. 30
• An LE iteration accepts a polling step when it satisfies a sufficient-decrease condition, using a direction and its negative on the unit sphere.The method can switch between LE and FE iterations based on unsuccessful LE iterations and prior line-search backtracks.Source: B Appendix: Full-Low Evaluation Methods · pp. 30–31
• The practical FLE setup uses a uniformly generated unit-sphere direction and its negative as the polling set.Source: B Appendix: Full-Low Evaluation Methods · p. 31
• The algorithm takes an iterate and stepsize as input, generates nonzero polling directions, updates the iterate and stepsize after sufficient decrease, and outputs the new state.Source: B Appendix: Full-Low Evaluation Methods · p. 31