PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks

Asif Ameer, Maryam Bashir, Irfan Younas, Muhammad Fayyaz

arXiv:2609.02460v1cs.NE

TL;DR

NAS is constrained by the cost of evaluating many candidate architectures and training model-based surrogates. MFSPNet combines PSO-based dense-block evolution with a model-free validation-loss estimator and block stacking, achieving competitive results while reducing search cost. Its ImageNet result is indicative because it uses a single run, and the method remains bounded by its DenseNet-inspired search space and CIFAR-10 proxy.

  • Problem

    NAS requires costly candidate-architecture evaluations, while model-based surrogates require training many architectures and additional optimization overhead.

  • Method

    MFSPNet integrates PSO-based dense-block evolution with a model-free validation-loss surrogate and block-based skip-connections for stacking.

  • Results

    The surrogate reduced search time by approximately 45% while maintaining comparable validation accuracy within ±0.3% on the CIFAR-10 proxy dataset.

  • Takeaways & Limitations

    MFSPNet provides an efficient and scalable surrogate-assisted NAS framework with empirically demonstrated transferability across datasets.

  • Takeaways & Limitations

    ImageNet evaluation used a single run, and the method depends on a CIFAR-10 proxy within a DenseNet-inspired search space.

Abstract

from arXiv · show

Neural Architecture Search (NAS) has emerged as a powerful paradigm for automatically designing deep neural networks; however, its practical adoption is often limited by substantial computational cost. To alleviate expensive full-training evaluations, surrogate-based methods have been introduced to estimate network performance efficiently. Nevertheless, existing approaches-particularly model-based surrogates-require training many candidate architectures and involve additional optimization overhead. In this work, we propose a Model-Free Surrogate PSO Network (MFSPNet) for evolving convolutional neural network architectures. The proposed method integrates a lightweight model-free surrogate predictor within a particle swarm optimization (PSO) framework, eliminating the need for pre-trained surrogate models. Specifically, MFSPNet introduces two key contributions: (1) a validation-loss-driven exponential moving average estimator (VLE-EMA) that captures early generalization behavior for reliable architecture ranking; and (2) a block-based dense connection strategy that enables effective stacking of evolved blocks while mitigating vanishing-gradient issues. This design also facilitates transferability of learned blocks across datasets. Extensive experiments demonstrate that MFSPNet achieves competitive performance with reduced computational cost. Under a consistent training protocol with ten independent runs, the proposed method attains error rates of 3.91%, 17.68%, and 1.91% on CIFAR-10, CIFAR-100, and SVHN, respectively, along with top-1/top-5 error rates of 28.29%/12.82% on ImageNet, while requiring less than three GPU days for architecture search. Due to computational constraints, the ImageNet result is based on a single run and should be interpreted as indicative of scalability. Overall, MFSPNet provides an efficient and reliable framework for cost-aware neural architecture search.

Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks

MFSPNet combines PSO-based dense-block search with a model-free surrogate predictor, surrogate validation-loss estimation, and block-based skip-connections for efficient architecture evolution and transfer analysis.

  • MFSPNet uses PSO to search dense blocks for image classification.
  • A model-free surrogate performance predictor estimates candidate fitness during evolutionary search.
  • The surrogate dataset supports validation-loss-based fitness estimation during evolution.
  • The framework analyzes evolved-block efficacy across different large-scale datasets.
  • MFSPNet introduces block-based skip-connections for stacking evolved dense blocks.

1. Introduction

The introduction identifies computationally expensive CNN architecture search as a central challenge and motivates MFSPNet's model-free surrogate-assisted evolutionary approach with transferable block search.

  • Deep and complex CNN designs remain difficult to train because foundational principles for novel architectures are inconsistent.
  • Many evolutionary and reinforcement-learning NAS methods evaluate vast numbers of architectures, incurring substantial computational costs.
  • MFSPNet uses a model-free surrogate predictor to reduce architecture-search cost in resource-limited settings.
  • Unlike model-based surrogates, the proposed estimator requires no prior surrogate training and uses validation-loss dynamics with greater weight on later epochs.
  • The method evolves individual dense blocks and transfers them across datasets using block-based skip-connections.

2. Background and Preliminaries

The background defines dense blocks and PSO, emphasizing feature reuse through concatenation and particle updates toward personal and global best solutions for architecture search.

  • 2.1. Dense Block Architecture: A dense block concatenates outputs from all preceding layers with the current layer's input.
  • 2.1. Dense Block Architecture: Each dense-block composite layer contains batch normalization, ReLU, and a 3x3 convolution.
  • 2.1. Dense Block Architecture: DenseNet's growth rate determines the feature maps generated by each layer and is fixed across a standard dense block.
  • 2.2. Particle Swarm Optimization (PSO): PSO updates particles toward both their personal-best and global-best positions while searching for an optimum.
  • 2.2. Particle Swarm Optimization (PSO): PSO updates each particle's velocity and position across iterations using inertia, cognitive, social, and random components.
  • 2.2. Particle Swarm Optimization (PSO): PSO is selected for its simplicity, computational efficiency, and suitability for optimizing diverse functions in variable-growth-rate block search.

3. Related Work

Related NAS methods trade computational cost, sample efficiency, and accuracy, while MFSPNet uses a model-free validation-loss estimator to rank architectures without pretraining a surrogate model.

  • RL-based NAS methods commonly suffer slow convergence and sample inefficiency because candidates require training to generate rewards.
  • Evolutionary NAS preserves population-based search but existing methods often trade classification accuracy against computational cost.
  • Zero-cost proxies incur negligible computation but may rank deeper, stacked, or transferred architectures inconsistently with final validation accuracy.
  • VLE-EMA performs limited actual training and uses validation-loss dynamics to capture early generalization behavior for architecture ranking.
  • Existing NAS methods face a significant performance-versus-computational-expense trade-off, limiting practical applicability.

4. Proposed Methodology

MFSPNet reduces evolutionary NAS cost by searching reusable variable-length dense blocks with PSO and a model-free validation-loss surrogate. Its VLE-EMA estimator ranks candidates from early generalization behavior, while weighted residual block stacking supports deeper architectures.

  • Surrogate-Assisted Search: MFSPNet uses a model-free surrogate and proxy dataset to evaluate particle fitness without fully training candidate CNNs during each evolutionary generation.The proxy dataset reduces evaluation cost, and the surrogate avoids training a separate model-based predictor.
  • Block-Based Search: The search evolves compact dense blocks rather than entire CNNs, reducing search complexity and enabling block transfer across datasets through stacking.The framework includes population initialization, proxy-data fitness evaluation, PSO evolution, and cross-dataset transfer evaluation.
  • Particle Encoding: Variable-length dense blocks encode layer growth rates in a vector, using a special flag to disable selected layers.The representation specifies maximum block length, growth-rate bounds, and a disabled-layer value.
  • Surrogate Performance Predictor: VLE-EMA estimates candidate performance from validation losses after cutoff-epoch training and emphasizes stable late-epoch generalization signals.The exponential decay factor γ = 0.78 was selected through sensitivity analysis over γ ∈[0.6, 0.9].
  • Evolutionary Process: The evolutionary process updates a candidate only when its surrogate-estimated fitness outperforms its personal best, avoiding full training in every generation.This personal-best comparison is used to reduce computational cost during search.
  • Block Stacking and Domain Transfer: Weighted residual aggregation projects and adaptively fuses outputs from preceding dense blocks, supporting feature reuse and gradient propagation in variable-depth stacks.The final architecture can contain a variable number of dense blocks, with maximum stack depth adjusted to dataset complexity.

5. Design of Experiments

MFSPNet is evaluated across four benchmark datasets, with CIFAR-10 serving as both the proxy search dataset and an evaluation benchmark. The experiments compare evolved blocks with diverse competitors under explicitly constrained computational settings.

  • Benchmark datasets: CIFAR-10, CIFAR-100, SVHN, and ImageNet are used as benchmark datasets for evaluating the proposed method.
  • Evaluation strategy: The evolved block is evaluated first on CIFAR-10 and then on CIFAR-100, SVHN, and ImageNet to assess performance and transferability.
  • Proxy dataset: CIFAR-10 provides the proxy dataset because it offers varied image instances, low CNN-training cost, and few classes for faster fitness evaluation.
  • Proxy dataset: CIFAR-10 contains 60,000 RGB images across 10 classes, including 50,000 training images and 10,000 test images.
  • Comparison methods: Competitors are grouped into manually designed CNNs, reinforcement-learning methods, and evolutionary-computation methods, with the final group most relevant to MFSPNet.
  • Experimental settings: The experiments use a 16-layer initial block, growth rates from 12 to 32, and parameters selected through prior settings, sensitivity analysis, preliminary experiments, and hardware constraints.
  • Experimental settings: Each candidate uses 8×8 proxy resolution, batch size 256, Adam at learning rate 0.001, and 15 training epochs within a 20-particle, 55-generation PSO search.
  • Compute budget: Under the reported accounting, MFSPNet completes the search in under 5.2 GPU-days while maintaining competitive performance.

6. Results and Analysis

MFSPNet achieves competitive accuracy under a substantially smaller search budget, with evidence of convergence, surrogate-assisted ranking, transferability, and scalable block design. Results support cross-dataset performance, while statistical and ImageNet findings require qualified interpretation.

  • Benchmark performance: MFSPNet achieves comparable mean error with reduced variance under a search budget below 3 GPU-days across 10 independent runs.The reported comparison is made under the same GPU-day constraint.
  • Benchmark performance: 3.76% error rate is achieved in fewer than 3 GPU-days, offering a favorable accuracy-to-search-cost trade-off against RL-based NAS methods.NASNet-A requires approximately 2,000 GPU-days, NAS v3 22,400 GPU-days, and BlockQNN 96 GPU-days.
  • Statistical analysis: p = 0.008 and Cliff’s δ = 0.65 indicate a statistically significant, medium-to-large CIFAR-100 difference versus DenseNet, but the imbalanced sample sizes limit conclusiveness.MFSPNet used 10 runs, whereas reproduced baselines used 3 runs.
  • Statistical analysis: p = 0.18 on SVHN does not indicate statistical significance, so the study makes no claim of statistical superiority for that dataset.The result supports competitive performance and reasonable generalization rather than superiority.
  • Transferability and architecture analysis: The evolved block shows transfer potential across similar dataset domains, while deeper stacking and layer-specific growth rates support feature reuse and gradient-flow stability.Growth rates trend toward larger means with less variation at greater depth, and intermediate layers have comparatively high median values.
  • Surrogate and convergence analysis: Validation-loss fitness from the model-free surrogate assists evolutionary search, with 15 cutoff epochs providing a more robust performance estimate than 10 epochs.The surrogate uses 15 epochs to estimate candidate CNN performance across generations.

7. Conclusion and Future Work

MFSPNet efficiently searches dense block structures using a downscaled proxy dataset, while VLE-EMA offers a validation-aligned, stable surrogate estimate. The study also identifies dataset and proxy-scaling limitations that motivate broader validation.

  • Conclusion: MFSPNet efficiently searches dense block structures with generalization and transferability across datasets.The method uses a downscaled proxy dataset while maintaining comparable accuracy on CIFAR-100, SVHN, and ImageNet.
  • Conclusion: VLE-EMA aligns surrogate estimates with final validation performance and reduces estimate variance by emphasizing later, more stable epochs.Its rationale combines lower bias from validation-loss alignment with lower variance from late-epoch weighting.
  • Conclusion: Monte Carlo validation compared TSE-EMA and VLE-EMA using 1,000 replications of signal–noise trajectories.The experiment was designed to test their theoretical distinction empirically.

Appendix B. Sensitivity Analysis of downscaling factor (n)

The sensitivity analysis examines proxy-dataset downscaling factors and finds that candidate rankings remain consistent across the tested resolutions. It selects n = 4 as the efficiency–fidelity compromise.

  • Downscaling factor: Experiments tested downscaling factors n ∈{2, 4, 8} for the proxy dataset.The study assessed how proxy resolution affected accuracy.
  • Downscaling factor: Changing n had minimal impact on proxy accuracy, suggesting consistent candidate-architecture rankings across resolutions.The proxy evaluation therefore remained useful for evolutionary search.
  • Downscaling factor: n = 4 was selected as the optimum compromise between efficiency and fidelity.This choice followed the sensitivity analysis of proxy evaluation.

Appendix C. PSO Parameter Sensitivity Analysis

Appendix C reports the effect of PSO parameters on the proposed strategy in Table C.9.

  • PSO parameter sensitivity: The effect of PSO parameters on the proposed strategy is outlined in Table C.9.The supplied passage identifies the sensitivity-analysis table but does not report its findings.
  • PSO parameter sensitivity: Table C.9 is the stated source for analyzing PSO-parameter effects.No parameter-specific outcome is provided in the passage.
  • PSO parameter sensitivity: The passage establishes that PSO-parameter sensitivity was examined, without specifying a preferred configuration.Interpretation beyond the table reference is unsupported here.

Appendix D. Proxy vs Target Dataset Correlation

Appendix D assesses the correlation between proxy-dataset and target-dataset performance, with the results illustrated in Table D.10.

  • Proxy–target correlation: Correlation between proxy and target dataset performances is assessed.The passage identifies the analysis but does not provide a correlation value.
  • Proxy–target correlation: The proxy–target performance relationship is illustrated in Table D.10.No specific trend or strength is stated in the supplied passage.
  • Proxy–target correlation: The passage reports an evaluation of proxy transferability without specifying its numerical outcome.The table contents are not described further here.

Appendix E. Ranking Ability of VLE-EMA vs. Common Proxy Methods

VLE-EMA is evaluated by measuring how well its rankings of sampled architectures agree with final validation accuracy, using rank correlations against common proxy methods.

  • Spearman’s ρ and Kendall’s τ quantify agreement between proxy-based rankings and final validation accuracy.Higher correlation values indicate more consistent architecture ranking.
  • The evaluation compares VLE-EMA with TSE-EMA and low-epoch validation as commonly used proxy methods.
Loading 2609.02460v1…