PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks

Asif Ameer, Maryam Bashir, Irfan Younas, Muhammad Fayyaz

arXiv:2609.02460v1cs.NE

TL;DR

Neural architecture search remains costly because evaluating candidate networks and training surrogate models can require substantial computation. MFSPNet uses a model-free surrogate within particle swarm optimization and achieves comparable accuracy across datasets while reducing search time by approximately 45%.

  • Problem

    Neural architecture search lacks consistently efficient principles for evaluating novel designs, making architecture exploration computationally expensive.

    Source: §1 · p. 5

  • Method

    MFSPNet combines a model-free surrogate performance predictor with particle swarm optimization and block-based dense connections for evolving and stacking CNN architectures.

    Source: §1 · p. 5, §4.6 · p. 23

  • Results

    The surrogate integration reduced search time by approximately 45% while maintaining comparable validation accuracy within ±0.3%.

    Source: §6.8 · p. 40

  • Takeaways & Limitations

    MFSPNet supports resource-efficient architecture discovery with reported generalization and transferability across datasets.

    Source: §7 · p. 40

    The method depends on CIFAR-10 as a proxy dataset, which may not adequately represent larger or more diverse imagery domains.

    Source: §7 · p. 41

Abstract

from arXiv · show

Neural Architecture Search (NAS) has emerged as a powerful paradigm for automatically designing deep neural networks; however, its practical adoption is often limited by substantial computational cost. To alleviate expensive full-training evaluations, surrogate-based methods have been introduced to estimate network performance efficiently. Nevertheless, existing approaches-particularly model-based surrogates-require training many candidate architectures and involve additional optimization overhead. In this work, we propose a Model-Free Surrogate PSO Network (MFSPNet) for evolving convolutional neural network architectures. The proposed method integrates a lightweight model-free surrogate predictor within a particle swarm optimization (PSO) framework, eliminating the need for pre-trained surrogate models. Specifically, MFSPNet introduces two key contributions: (1) a validation-loss-driven exponential moving average estimator (VLE-EMA) that captures early generalization behavior for reliable architecture ranking; and (2) a block-based dense connection strategy that enables effective stacking of evolved blocks while mitigating vanishing-gradient issues. This design also facilitates transferability of learned blocks across datasets. Extensive experiments demonstrate that MFSPNet achieves competitive performance with reduced computational cost. Under a consistent training protocol with ten independent runs, the proposed method attains error rates of 3.91%, 17.68%, and 1.91% on CIFAR-10, CIFAR-100, and SVHN, respectively, along with top-1/top-5 error rates of 28.29%/12.82% on ImageNet, while requiring less than three GPU days for architecture search. Due to computational constraints, the ImageNet result is based on a single run and should be interpreted as indicative of scalability. Overall, MFSPNet provides an efficient and reliable framework for cost-aware neural architecture search.

Highlights · Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks

The work presents a PSO-based neural architecture search strategy that evolves dense blocks using a model-free surrogate predictor and validation-loss fitness estimation. It also introduces block-based skip connections and evaluates evolved blocks across large-scale datasets.

Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks

1. Introduction

Neural architecture search is motivated by the high computational cost and uncertain benefits of manually designing increasingly deep CNNs. MFSPNet addresses this challenge with a model-free surrogate integrated into PSO-based dense-block evolution, including limited-epoch validation-loss ranking, variable growth rates, block stacking, and transferability assessment.

1. Introduction

  • Deeper CNNs improve feature extraction but remain difficult to train because architectural changes lack consistent training principles and their performance effects can be ambiguous.These challenges motivate automatic discovery of effective architectural parameters.Source: §1 · pp. 4–5
  • MFSPNet uses a model-free surrogate performance predictor to assist evolutionary architecture search, aiming to reduce GPU hours and computational costs while enabling resource-efficient discovery of high-performing CNN architectures.The approach is intended for computationally limited scenarios, including UAV-based monitoring, agricultural imaging, and embedded vision systems.Source: §1 · p. 5
  • MFSPNet evolves individual dense blocks with PSO rather than complete CNNs, reducing search cost and enabling investigation of transferring learned blocks across datasets.The block-evolution strategy is inspired by EffPNet, while PSO is selected for simplicity, computational efficiency, and effectiveness across diverse functions.Source: §1 · p. 6
  • The method explores layer-specific growth rates instead of DenseNet’s fixed growth rate, providing more flexible optimization of dense-block architectures.DenseNet provides the skip-connection inspiration for this design.Source: §1 · p. 6

1.1. Goal

  • The surrogate ranks candidate CNNs from validation losses accumulated during limited-epoch SGD training, eliminating prior training of large datasets of fully trained architectures.Later validation-loss dynamics receive greater weight, contrasting with model-based surrogate approaches such as EffPNet.Source: §1 · p. 6, §1.1 · p. 7
  • Block stacking uses block-based skip connections to increase DenseNet depth while mitigating training difficulties, and the study evaluates the evolved block’s transferability on other datasets.The research also analyzes surrogate efficacy, removes underperforming candidates after a few SGD epochs, and examines evolutionary convergence.Source: §1.1 · p. 7

2. Background and Preliminaries

This section introduces dense blocks, particle swarm optimization (PSO), and surrogate-assisted NAS as foundations for efficient architecture search. Dense blocks reuse preceding features through concatenation, while PSO and model-free surrogates address optimization and evaluation costs.

2.1. Dense Block Architecture

  • Dense blocks concatenate outputs from all preceding layers with the current input, using BN-ReLU-Conv composite layers and skip-connections to promote feature reuse and gradient flow.A four-layer dense block is illustrated, and its layer inputs combine the original feature maps with outputs from layers 0 through l−1.Source: §2.1 · p. 8, Figure 1 · p. 9
  • The DenseNet growth rate r determines the feature maps generated by each layer, with block size expressed as N_l+1 = k_0 + r∗l.Here, k_0 is the block input and k_l is the lth-layer output.Source: §2.1 · p. 9

2.2. Particle Swarm Optimization (PSO)

  • PSO updates particles toward both personal-best pbest and global-best gbest positions, combining swarm exploration with information sharing during optimization.Particle velocity updates include inertia, cognitive, and social weights, with uniformly sampled random factors.Source: §2.2 · pp. 9–10
  • PSO is selected for surrogate-assisted variable-length CNN block search because it offers fast convergence, fewer control parameters, and collective information sharing under noisy early surrogate estimates.The passage contrasts PSO with GA, ACO, and DE, whose encoding complexity, overhead, or parameter sensitivity can increase search cost.Source: §2.2 · p. 10

2.3. Surrogate Models in NAS

  • Surrogate-assisted NAS reduces costly candidate evaluations by estimating architecture performance from limited data, while model-free predictors avoid explicit training on pre-trained candidate architectures.A validation-loss-landscape proxy can estimate architectural changes, framing architecture search as validation-loss minimization.Source: §2.3 · p. 11

3. Related Work

Related NAS research spans reinforcement-learning, evolutionary, gradient-based, and training-free methods, but existing approaches face substantial computational costs, instability, or limited ranking reliability. MFSPNet addresses these limitations with a lightweight model-free VLE-EMA predictor integrated into evolutionary search.

Reinforcement-Learning NAS

  • RL-based NAS can achieve strong performance but suffers slow convergence, sample inefficiency, and extreme computational demands, reaching 22,400 GPU-days for NAS [32].Controller networks also require careful hyperparameter tuning, adding to the computational burden.Source: §3 · pp. 11–12

Evolutionary and Gradient-Based NAS

  • Evolutionary NAS preserves population-based search, while methods such as AmoebaNet, AECNN, and EffPNet reduce search costs through regularization, fewer candidates, or surrogate assistance with accuracy trade-offs.Gradient-based methods including ProxylessNAS and DARTS reduce the discrete search space for faster optimization but can suffer instability or cross-dataset performance loss from approximation errors.Source: §3 · p. 12

Training-Free NAS

  • Training-free proxies such as NAS-WOT [40] and SynFlow [41] have negligible cost but may rank architectures inconsistently for deeper, stacked, or cross-dataset configurations because they ignore learning dynamics.Their initialization-dependent signals do not capture evolving feature representations, which can matter when transferring architectures from CIFAR to ImageNet.Source: §3 · pp. 12–13

Proposed Surrogate

  • VLE-EMA remains model-free while training for a small budget such as c = 15 epochs and using validation-loss dynamics to capture early generalization for more robust transfer-oriented ranking.It avoids learned regression or classification surrogate models while costing substantially less than full training.Source: §3 · p. 13

Motivation

  • Table 4 summarizes the performance–cost trade-off across representative NAS methods, motivating an EC-based search guided by a model-free surrogate to reduce overhead while retaining competitive accuracy.The EC strategy is selected for its fast convergence and lower computational complexity relative to alternative search paradigms.Source: §3 · p. 13

4. Proposed Methodology

MFSPNet combines model-free validation-loss surrogate evaluation, variable-length dense-block evolution, proxy data, and weighted block stacking to reduce NAS cost and support cross-dataset transfer. Its methodology covers particle encoding, diverse initialization, surrogate-assisted PSO evolution, and final architecture construction.

4. Proposed Methodology

  • MFSPNet uses a low-resolution proxy dataset and model-free surrogate fitness evaluation instead of expensive full SGD training during evolutionary search.The evolved block is subsequently stacked, evaluated, and transferred across datasets including CIFAR-100, SVHN, and ImageNet.Source: §4 · p. 14, Figure 2 · p. 14

4.1. Particle Encoding

  • PSO particles encode dense blocks with a fixed maximum length, per-layer growth rates, and a special flag that disables layers to create variable-length architectures.The growth-rate range is 12–32, the disabled-layer flag is 11, and the vector length is 16.Source: §4.1 · pp. 15–16

4.2. Population Initialization

  • Latin hypercube sampling combined with standard-deviation ordering initializes a diverse population intended to improve PSO exploration and convergence toward the global optimum.Samples are generated across the search dimensions, ranked by standard deviation, and then added to the initial population.Source: §4.2 · p. 17

4.3. Surrogate Performance Predictor

  • VLE-EMA ranks candidate blocks from few-epoch validation losses, emphasizing stable late-epoch generalization signals while avoiding pretrained surrogate models.The decay factor γ = 0.78 was selected through sensitivity analysis over γ ∈[0.6, 0.9].Source: §4.3 · pp. 19–20

4.4. Proxy Data Set

  • Downscaled proxy images reduce fitness-evaluation memory and processing requirements, with downscaling factor n = 4 selected after values from 2 to 8 showed little proxy-accuracy impact.Downscaling from w × h to w/n × h/n reduces the feature-map size to 1/n × 1/n of the original.Source: §4.4 · p. 21

4.5. Evolutionary Process

  • The evolutionary process updates PSO positions and retains candidate solutions when their VLE-EMA fitness improves over personal bests, eliminating full candidate training each generation.Fitness is computed after a validation cutoff of c epochs, and the global best is returned after the specified generations.Source: §4.5 · pp. 22–23

4.6. Block Stacking and Domain Transfer

  • Weighted residual aggregation projects preceding block outputs and decays their influence with distance, enabling variable-depth stacking, feature reuse, gradient stability, and domain transfer.The maximum stack depth Tmax is adjusted according to dataset complexity, and candidate stacks are evaluated to select the highest-accuracy architecture.Source: §4.6 · pp. 23–24

5. Design of Experiments

MFSPNet is evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet using grouped peer comparisons, constrained experimental settings, and explicitly reported search costs. The design uses CIFAR-10 as a lightweight proxy while assessing transferability and reproducibility across datasets and compute budgets.

5.1. Benchmark Data Sets

  • CIFAR-10 serves as both the proxy-search and evaluation dataset, while CIFAR-100, SVHN, and ImageNet test transferability across increasing difficulty, domains, and scale.This choice introduces acknowledged benchmark-selection bias, but transferability is empirically assessed on more diverse datasets.Source: §5 · p. 25, §5.1 · p. 26

5.2. Peer Comparison

  • The experiments compare MFSPNet with manually designed CNNs, reinforcement-learning methods, and evolutionary-computation approaches selected by metric availability and methodological relevance.More relevant transfer comparisons include WideResNet, ResNet, DenseNet, CiCNet, Deeply Supervised Net, FractalNet, Network in Network, and EffPNet.Source: §5.2 · p. 27

5.3. Experimental Settings

  • Experimental settings combine standard PSO parameters, sensitivity-based surrogate parameters, and hardware-constrained architecture ranges to balance convergence, ranking reliability, memory use, and model capacity.The dense block starts at 16 layers, growth rates range from 12 to 32, and the settings include cutoffepochs 15, downscaling factor 2, Tmax 8, and ψ 0.88.Source: §5.3 · pp. 27–28

5.4. Compute Budget and Search Cost Accounting

  • Each candidate uses 8×8 proxy inputs, batch size 256, Adam at learning rate 0.001, and 15 epochs within a 20-particle, 55-generation PSO search.The protocol evaluates 1100 candidates, with approximately 3.5 hours per run and 1.46 GPU-days across ten independent runs.Source: §5.4 · p. 29
  • Under 5.2 GPU-days, MFSPNet completes the search while maintaining competitive performance and enabling transparent comparisons across hardware and training pipelines.The accounting reports single-GPU FP32 experiments and emphasizes strict reproducibility.Source: §5.4 · pp. 28–29

6. Results and Analysis

MFSPNet achieves competitive accuracy across benchmark datasets with substantially lower search cost, while its surrogate-guided PSO converges, preserves architecture rankings, and supports transferable block stacking. Analyses further show layerwise growth-rate adaptation, scalable complexity, and complementary benefits from the surrogate predictor and stacking mechanism.

6.1.1. Block Assessment on CIFAR10 Dataset

  • 3.76% error with fewer than 3 GPU-days positions MFSPNet as a competitive, cost-efficient alternative to RL-based NAS methods requiring up to 22,400 GPU-days.MFSPNet ranks fourth among 19 competing methods, with comparable mean error and reduced variance under 10 independent runs and the same GPU-day constraint.Source: §6.1.1 · pp. 29–30

6.1.2. Performance on CIFAR-100, SVHN and ImageNet Dataset

  • MFSPNet’s transferred block generalizes across CIFAR-100, SVHN, and ImageNet, with statistically supported improvement on CIFAR-100 but no statistical superiority claim on SVHN.On CIFAR-100, MWW gives p = 0.008 and Cliff’s δ = 0.65, whereas SVHN gives p = 0.18; ImageNet uses a single run and is indicative rather than statistically validated.Source: §6.1.2 · pp. 31–32

6.5. Block Stacking Analysis

  • Transferred evolved blocks support adaptation across datasets by stacking selected feature-extracting blocks at different depths and model sizes.CIFAR-10 proxy rankings guide search, while retraining on full target datasets supports fair final evaluation; transferability is demonstrated across increasingly difficult image-classification benchmarks, not unconditionally across all domains.Source: §6.1.2 · p. 32, §6.5 · pp. 36–37

6.2. PSO Convergence

  • PCA trajectories fluctuate during early exploration, stabilize after several generations, and nearly flatten later, indicating PSO convergence from exploration toward exploitation.Inertia weight, acceleration factors, and population size affect this balance; swarms below 20 lack variety, whereas those above 60 increase computation without considerable accuracy gains.Source: §6.2 · pp. 32–33, Figure 4 · p. 33

6.3.1. Analysis of Loss Landscape

  • The surrogate’s best-particle validation loss decreases across generations, indicating that VLE-EMA fitness guidance assists evolutionary search toward a global optimum.Fitness uses validation loss from 15 epochs to evaluate candidate CNN architectures.Source: §6.3.1 · p. 33, Figure 5 · p. 34

6.3.2. Performance with Different Epochs

  • 15 cutoffepochs provide a more robust VLE-EMA performance estimate than 10 cutoffepochs by increasing variation among candidate fitness values.Fewer epochs enable rapid exploration but may mislead search because of insufficient training and poor generalization.Source: §6.3.2 · pp. 34–35, Figure 6 · p. 35

6.4. Ranking Consistency of the Surrogate Estimator

  • ρ = 0.87 and τ = 0.71 show that VLE-EMA preserves candidate architecture rankings better than TSE-EMA and low-epoch training under identical evaluation conditions.The analysis samples N = 50 architectures, evaluates VLE-EMA at c = 15 epochs, and compares surrogate rankings with fully trained validation accuracy.Source: §6.4 · p. 36

6.8. Ablation Study

  • 45% lower search time with comparable validation accuracy demonstrates the surrogate’s benefit, while deeper stacking improves accuracy and nearly constant complexity avoids excessive overhead.The ablation compares surrogate-assisted PSO with baseline PSO and attributes stable stacking cost to fixed channel dimensionality and lightweight 1×1 projection.Source: §6.8 · p. 40

7. Conclusion and Future Work … Direct comparison and interpretation

MFSPNet offers efficient, transferable surrogate-assisted architecture search but remains constrained by its DenseNet-inspired space, CIFAR-10 proxy dependence, hyperparameter sensitivity, and hardware limits. Appendix analysis supports VLE-EMA over TSE-EMA through validation alignment, late-epoch stability, and Monte Carlo evidence.

7. Conclusion and Future Work

  • MFSPNet efficiently searches dense block structures with robust generalization and transferability, achieving comparable accuracy across CIFAR-100, SVHN, and ImageNet while reducing computational cost.The conclusion attributes this efficiency partly to using a downscaled proxy dataset and frames the method as scalable in resource-limited settings.Source: §7 · p. 40
  • The approach is limited by its DenseNet-inspired search space, CIFAR-10 proxy dependence, hyperparameter sensitivity, and hardware constraints; future work targets CNN–Transformer spaces, alternative metaheuristics, and broader domains.Planned applications include medical imaging, remote sensing, and other high-resolution visual tasks.Source: §7 · pp. 40–41

Appendix A. Signal–Noise model and comparison of TSE-EMA

VLE-EMA (validation-loss EMA)

TSE-EMA (training-loss EMA)

Direct comparison and interpretation

  • VLE-EMA is projected to correlate more strongly with final performance and produce smaller prediction error by aligning with validation loss and emphasizing later, more stable epochs.Under standard training/validation gaps, its bias is qualitatively no greater than TSE-EMA, while stated noise conditions imply lower or comparable variance.Source: Direct comparison and interpretation · pp. 42–43
  • −0.01 versus −0.045 bias and 0.0026 versus 0.0041 variance show VLE-EMA is empirically more accurate and stable than TSE-EMA across 1,000 Monte Carlo replications.The experiment used signal–noise trajectories with epoch-dependent noise variances.Source: Direct comparison and interpretation · p. 43

Appendix B. Sensitivity Analysis of downscaling factor (n)

The sensitivity analysis finds that downscaling factors n ∈ {2, 4, 8} minimally affect proxy accuracy and preserve candidate-architecture rankings across resolutions. Accordingly, n = 4 is selected as the best efficiency–fidelity compromise.

Sensitivity Analysis of downscaling factor (n)

  • Across n ∈ {2, 4, 8}, changing the downscaling factor had minimal impact on proxy accuracy.Source: Appendix B · p. 43
  • n = 4 was selected as the optimum compromise between computational efficiency and proxy fidelity.The proxy evaluation significantly lowers computational expenses while providing valid guidelines for evolutionary search.Source: Appendix B · p. 44
  • Candidate-architecture rankings remained consistent across the tested resolutions.Source: Appendix B · p. 44

Appendix C. PSO Parameter Sensitivity Analysis

This appendix outlines the effect of PSO parameters on the proposed strategy.

Appendix C. PSO Parameter Sensitivity Analysis

  • The effects of PSO parameters on the proposed strategy are presented in Table C.9.Source: Appendix C · p. 44

Appendix D. Proxy vs Target Dataset Correlation

The appendix assesses the correlation between proxy- and target-dataset performance, with results illustrated in Table D.10.

Appendix D. Proxy vs Target Dataset Correlation

  • Correlation between proxy and target dataset performances is assessed and illustrated in Table D.10.Source: Appendix D · p. 44

Appendix E. Ranking Ability of VLE-EMA vs. Common Proxy Methods

This appendix evaluates VLE-EMA’s ability to rank candidate architectures by correlating its estimates with final validation accuracy and comparing it with common proxy methods.

Ranking Ability of VLE-EMA vs. Common Proxy Methods

  • VLE-EMA is assessed using Spearman’s ρ and Kendall’s τ rank correlations against final validation accuracy, with comparisons to TSE-EMA and low-epoch validation; higher values indicate more consistent ranking.Table 6 reports these correlation measures for sampled candidate architectures.Source: Appendix E · p. 44

Related papers

Loading 2609.02460v1…