PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discovery

Yutao Lai, Kezhao Lai, Hai-Lin Liu

arXiv:2608.21927v1cs.LGcs.AI

TL;DR

Zero-cost proxy rankings vary across search spaces, while automated ensemble discovery must jointly handle aggregation structure and continuous coefficients. Bi-EZP separates these decisions by using an LLM for executable program generation and CMA-ES for fixed-program calibration. Under the evaluated protocols, it achieves higher rank correlation than compared baselines on NATS-Bench and NDS and supports downstream DARTS search.

  • Problem

    Zero-cost proxies capture different properties and can rank architectures inconsistently across search spaces, while coupled structure-and-parameter optimization complicates structural comparison.

  • Method

    Bi-EZP uses an LLM to generate executable aggregation programs with parameter bounds, then CMA-ES calibrates each fixed program before validation-based structural selection.

  • Results

    Bi-EZP achieves higher rank correlation than compared baselines on NATS-Bench and NDS and can serve as an evaluation signal in downstream DARTS search.

  • Takeaways & Limitations

    Separating program discovery from numerical calibration provides an effective approach to automated ensemble zero-cost proxy construction.

  • Takeaways & Limitations

    Offline discovery incurs nontrivial LLM token costs and latency, and proxy quality depends on the pre-selected base metrics and prompt alignment.

Abstract

from arXiv · show

Zero-cost proxies enable neural architecture search (NAS) to rank candidate networks from statistics computed at initialization, avoiding repeated training. However, different proxies capture different properties and often produce inconsistent rankings across search spaces. Ensemble proxies can combine complementary signals, but automated discovery must optimize both discrete aggregation structures and their continuous coefficients, making structural quality difficult to separate from parameter calibration. We propose Bi-EZP, a bilevel framework that decouples these decisions. At the upper level, a large language model generates executable aggregation programs over four complementary base proxies with program-specific parameter bounds. At the lower level, covariance matrix adaptation evolution strategy (CMA-ES) optimizes the continuous parameters of each fixed program on an inner training split. The calibrated programs are then evaluated using Kendall's rank correlation on a disjoint validation split, enabling evolutionary selection to favor structures that generalize beyond their calibration data. Experiments on NATS-Bench and Network Design Spaces evaluate ranking performance across heterogeneous search spaces, and DARTS experiments assess downstream architecture search. Results show that separating program discovery from numerical calibration provides an effective approach to automated ensemble zero-cost proxy construction. The source code is available at: https://anonymous.4open.science/r/Bi-EZP-318D

I. INTRODUCTION

NAS evaluation is expensive, and heterogeneous zero-cost proxies can rank architectures inconsistently across search spaces. Bi-EZP addresses this by separating executable structure discovery from numerical calibration in a bilevel, LLM-guided framework.

  • Architecture evaluation dominates NAS cost because training every candidate is prohibitively expensive, while weight sharing can distort candidate rankings.
  • Different zero-cost proxies encode distinct inductive biases, so a proxy informative in one search space may be weak or misleading in another.
  • Bi-EZP decouples symbolic aggregation-structure discovery from structure-dependent continuous parameter calibration through bilevel optimization.
  • An LLM generates executable proxy programs and program-specific parameter constraints, which are validated before CMA-ES optimizes their continuous parameters.
  • Bi-EZP evaluates competitive rank correlation across NATS-Bench and NDS and supports downstream architecture search experiments on DARTS.

B. Zero-Cost Proxies

Bi-EZP builds a reusable ensemble proxy from four heterogeneous zero-cost signals while separating program discovery from parameter calibration. Its executable candidates are validated, calibrated with CMA-ES, and selected using validation performance.

  • Ensemble proxies combine complementary signals, but existing parameterizations can restrict the interactions their aggregation functions express.
  • Bi-EZP retains NASWOT, MeCo, Sweet-SNIP, and SZiCo while making aggregation programs and parameter bounds outer-level variables.
  • The upper level uses an LLM to generate, recombine, and mutate symbolic aggregation programs, while CMA-ES optimizes parameters for each fixed program.
  • A code gate checks syntax, interface, and finite bounds before calibration; invalid candidates receive correction attempts or a fixed fallback.
  • The fixed four-proxy basis provides activation-, feature-, parameter-, and gradient-level views of initialized architectures.
  • Bi-EZP studies transformations and interactions among a fixed proxy set rather than searching for an optimal subset of proxies.

1) NASWOT (snw):

The proxy basis captures complementary properties of initialized networks, spanning activation patterns, feature geometry, parameter saliency, and gradient consistency.

  • NASWOT: NASWOT measures network expressivity through the diversity of binary activation patterns induced by a mini-batch.
  • NASWOT: NASWOT depends only on activation patterns, providing a gradient-free view that complements gradient- and feature-related signals.
  • MeCo: MeCo quantifies redundancy in intermediate representations using the minimum eigenvalue of a Pearson correlation matrix.
  • MeCo: A larger MeCo minimum eigenvalue indicates less-degenerate sampled feature directions and complements activation-pattern information.
  • Sweet-SNIP: Sweet-SNIP measures initialized-parameter importance through the product of parameter magnitude and loss gradient.
  • SZiCo: SZiCo evaluates gradient consistency by comparing mean gradient magnitude with variation across input batches, capturing a distinct trainability-related signal.

C. Parameterized Proxy Representation

Bi-EZP represents each LLM-generated candidate as executable aggregation code with candidate-specific finite parameter bounds. Validation makes the generated object evaluable before CMA-ES calibration and evolutionary comparison.

  • A candidate program maps a matrix of four-proxy vectors to one scalar score for each architecture.
  • The representation stores generated rationale H, executable Python code G, and a feasible parameter set Θ(G).
  • Each generated response must provide an Idea section, a Code section, a literal BOUNDS list, and one synchronous aggregate function.
  • The aggregate interface receives four one-dimensional proxy arrays and CMA-ES parameter values, returning a finite score vector of matching length.
  • The parser checks syntax, function signature, argument restrictions, and finite numerical bounds before evaluation.
  • The operational search space lacks an operator whitelist, maximum expression depth, and explicit complexity penalty.

D. Bilevel Discovery Objective

Bi-EZP separates structural program evolution from numerical calibration by evaluating each candidate after optimizing its parameters on inner-training data. Validation Kendall’s τ then determines structural fitness and evolutionary survival.

  • Bilevel objective: Each proposed program is independently calibrated on Dtrain before receiving fitness on the disjoint Dval split.The calibrated candidate, rather than an unevaluated parameterization, enters structural comparison.
  • Bilevel objective: Kendall’s τ evaluates ranking agreement at both levels, but inner calibration and outer structural comparison use different data and variables.This prevents the outer loop from favoring programs whose initial coefficients happen to be favorable.
  • Population evaluation: The outer search maintains a population of executable programs whose fitness is their validation Kendall correlation after lower-level calibration.Initialization and later offspring follow the same nested evaluation protocol.
  • Structural evolution: Tournament selection chooses valid parents, while LLM crossover or mutation proposes executable offspring with compatible interfaces and bounds.Fitness guides generation indirectly through parent selection rather than appearing directly in prompts.
  • Structural evolution: Elitist (P + P) replacement merges parents and offspring, retaining the best P validated programs while continuing exploration through new code.The highest-fitness calibrated candidate is returned after a fixed number of outer cycles.

F. Lower-Level Parameter Adaptation via CMA-ES

CMA-ES calibrates the continuous parameters of each fixed aggregation program within bounds supplied by the LLM. It optimizes rank correlation on inner-training data while penalizing numerically invalid proxy evaluations.

  • Parameter optimization: CMA-ES optimizes coefficients within the fixed program’s bounded parameter manifold because the rank-correlation objective can be rugged, non-differentiable, and ill-conditioned.Its derivative-free covariance adaptation suits this lower-level objective.
  • Initialization: The search distribution is initialized geometrically within the LLM-specified bounds, with step size scaled to the average boundary range.The parameter domain is represented as a Cartesian product of per-parameter intervals.
  • Evolutionary adaptation: At each generation, CMA-ES samples λ parameter vectors from a multivariate normal distribution defined by the current mean and covariance.The distribution is iteratively adapted using selected candidate vectors.
  • Objective and stability: Valid candidates are scored by proxy rank correlation on Dtrain, while NaN or Inf outputs receive a large penalty.This penalty steers optimization away from mathematically unstable parameter regions.

G. Nested Search Procedure

The nested procedure validates LLM-generated programs, calibrates their parameters on Dtrain, and compares frozen calibrated programs on Dval. Evolutionary selection uses only validation fitness, while later proxy use scores networks from initialization statistics.

  • Nested evaluation: Every generated program is checked for syntax and interface validity before CMA-ES calibrates its own bounded parameters on Dtrain.The resulting parameters are frozen before outer evaluation on Dval.
  • Nested evaluation: Validation fitness alone drives tournament selection and elitist replacement, preventing validation information from entering CMA-ES’s numerical objective.Fallback candidates follow the same lower- and upper-level evaluation path.
  • Frozen-proxy evaluation: After discovery, the frozen program scores candidate networks using initialization-time statistics rather than training each candidate to convergence.The discovery phase itself uses benchmark accuracies as supervision and is not entirely training-free.
  • Algorithm flow: Algorithm 1 initializes candidates with the LLM, validates or corrects them, evolves offspring through crossover or mutation, and returns the highest-fitness population member.The procedure uses Dtrain and Dval with population size P, outer cycles T, and crossover probability Pc.
  • Frozen-proxy evaluation: In DARTS experiments, Bi-EZP supplies an evaluation signal to the existing architecture-search procedure rather than replacing it.Discovery cost is incurred once before the frozen proxy scores many architectures.

I. Computational Characteristics and Scope

The discovery cost scales with candidate-program evaluations, LLM correction attempts, and CMA-ES aggregation evaluations. The method’s empirical scope is bounded by its four-proxy basis, validation choices, and flexible but incompletely characterized program space.

  • Computational characteristics: P(T + 1) candidate programs are evaluated for population size P and T outer cycles when each cycle produces a full offspring population.This count includes the initial population and one full offspring population per cycle.
  • Lower-level procedure: CMA-ES receives G, B, Dtrain, and Emax, initializes within Cartesian-product bounds, clips sampled parameters, and ranks them by 1 − τ.Invalid scores receive a penalty before population updates.
  • Lower-level procedure: CMA-ES updates the mean from the top μ vectors and adapts the global step size during each lower-level run.The procedure sorts candidates by loss before applying these updates.
  • Computational characteristics: At most rP(T + 1) LLM requests and EmaxP(T + 1) aggregation evaluations occur under the stated correction and CMA-ES budgets.Total discovery cost also includes base-proxy extraction and validation evaluation.
  • Scope and assumptions: The framework assumes its four fixed base proxies contain complementary information, but does not establish that this pool is optimal or explicitly penalize program complexity.Reproducibility requires reporting model, decoding, seed, population, cycle, correction, budget, partition, and final-program details.
  • Scope and assumptions: Empirical claims are restricted to the supplied four-proxy basis, reported benchmarks, and evaluated discovery and transfer protocols.The implementation validates syntax, interface shape, and finite bounds but permits an open operator space and semantically duplicate programs.

IV. EXPERIMENTAL STUDIES

Bi-EZP is evaluated across NAS benchmarks, datasets, search spaces, and downstream DARTS search using Kendall’s τ as the primary ranking metric. Under the reported protocols, it achieves higher rank correlation than compared methods, while the experiments do not isolate which program-design factor causes the gains.

  • Evaluation scope: The evaluation spans NATS-Bench, NDS, and DARTS search spaces, with CIFAR-10, CIFAR-100, and ImageNet used across proxy extraction, calibration, transfer, and final evaluation.
  • Rank-correlation evaluation: Kendall’s τ measures consistency between proxy scores and ground-truth test accuracies on NATS-Bench topology-based and size-based search spaces.
  • NATS-Bench results: Bi-EZP increases NATS-Bench TSS τ over ECP from 0.782 to 0.809 on CIFAR-10, from 0.771 to 0.791 on CIFAR-100, and from 0.740 to 0.775 on ImageNet-16-120.
  • NATS-Bench results: On NATS-Bench SSS, Bi-EZP increases τ over ECP from 0.771 to 0.792 on CIFAR-10 and from 0.806 to 0.826 on ImageNet-16-120.
  • Interpretation boundary: The experiments do not establish whether the observed gains arise from program representation, operator choice, or expression size.
  • Baseline comparisons: Bi-EZP achieves higher rank correlations than the reported AZ-NAS and EZNAS-A comparisons under the reported evaluation protocol.

2) Experimental Result in Rank Correlation Across NDS Benchmarks:

Bi-EZP achieves strong rank correlation across heterogeneous NDS architectural families and retains measurable cross-space reuse when its discovered proxy is frozen. Downstream DARTS and ImageNet experiments further evaluate practical transfer, while reported search costs exclude one-time proxy discovery.

  • Bi-EZP obtains the highest rank correlation among compared methods across five NDS architectural families.
  • 0.621 versus 0.568 on DARTS, 0.582 versus 0.512 on ENAS, and 0.511 versus 0.406 on Amoeba relative to ECP.
  • Cross-dataset transfer: Frozen proxies transfer across datasets with mean Kendall’s τ of 0.775 versus 0.792 for matched settings, a decrease of 0.017.
  • Cross-search-space transfer: Off-diagonal NDS correlations range from 0.429 to 0.627, averaging 0.515 versus 0.557 on the diagonal.
  • Cross-search-space transfer: The PNAS-discovered proxy reaches the highest DARTS correlation, 0.627, slightly above the DARTS-discovered proxy at 0.621.
  • Downstream search: Bi-EZP-guided DARTS search reaches test errors of 2.47% on CIFAR10 and 16.10% on CIFAR-100, while ImageNet transfer reaches 24.7% Top-1 test error with 5.3M parameters.

D. Ablation Study and Hyperparameter Analysis

Ablations indicate that bilevel separation and LLM-based structural proposals contribute to Bi-EZP’s reported correlation. Sensitivity analyses and LLM comparisons examine operating choices, while the reported downstream search cost excludes offline discovery.

  • Hyperparameter analysis: The ablation study evaluates Bi-EZP mechanisms on NATS-Bench and analyzes sensitivity to population size and evolutionary generations up to 50.
  • Ablation study: Removing bilevel structure reduces TSS CIFAR-10 correlation from 0.809 to 0.783.
  • Ablation study: Replacing LLM proposals with ECP aggregation plus CMA-ES lowers TSS CIFAR-10 correlation from 0.809 to 0.784.

2) Comparison of Lower-Level Optimizers:

Bi-EZP compares lower-level continuous optimizers and evolutionary hyperparameter settings while separating program discovery from parameter calibration. CMA-ES performs best among the tested alternatives, and population and generation budgets show non-monotonic sensitivity.

  • Comparison of Lower-Level Optimizers: CMA-ES achieves higher rank correlation than GA, DE, and PSO variants across the evaluated search spaces.The comparison motivates CMA-ES because covariance adaptation handles dependencies among continuous parameters and the objective is non-differentiable.
  • Impact of Evolutionary Hyperparameters: Population performance generally improves from P = 5 to P = 20 but does not improve further at P = 50.Observed τ reaches approximately 0.809 on CIFAR-10, 0.791 on CIFAR-100, and 0.775 on ImageNet-16-120 at P = 20; P = 50 lowers two of these values.
  • Impact of Evolutionary Hyperparameters: The generation sweep is non-monotonic, with the strongest balanced observed result at Tmax = 20 rather than 50 generations.Increasing from 5 to 20 generations raises correlation to 0.809/0.791/0.775 on CIFAR-10, CIFAR-100, and ImageNet-16-120, while 50 generations produce mixed results.
  • Performance Sensitivity under Different LLMs: GLM-4.7-Flash produces higher correlations than DeepSeek-V4-Flash in several evaluated settings, including ImageNet-16-120 under TSS and SSS.The comparison shows operation with either tested backbone but does not establish invariance across providers, model scales, or decoding configurations.
  • Evolutionary Trajectory and Search Dynamics: The best-so-far validation correlation reaches 0.8255 at Generation 17 after an early increase to 0.8115 during Generations 1-5.The trajectory then plateaus through Generations 5-16 and remains unchanged through the recorded budget; it does not establish global convergence.
  • Overall Findings: Across the evaluated protocols, Bi-EZP achieves higher rank correlation than compared baselines on NATS-Bench and NDS and supports downstream DARTS search.The framework separates LLM-based program proposal from CMA-ES-based continuous parameter calibration.
Loading 2608.21927v1…