PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-EZP: LLM-Guided Bilevel Program Evolution for Ensemble Zero-Cost Proxy Discovery

Yutao Lai, Kezhao Lai, Hai-Lin Liu

arXiv:2608.21927v1cs.LGcs.AI

TL;DR

Zero-cost proxy rankings vary across search spaces, while automated ensemble discovery must jointly handle aggregation structure and continuous coefficients. Bi-EZP separates these decisions by using an LLM for executable program generation and CMA-ES for fixed-program calibration. Under the evaluated protocols, it achieves higher rank correlation than compared baselines on NATS-Bench and NDS and supports downstream DARTS search.

  • Problem

    Zero-cost proxies capture different properties and can rank architectures inconsistently across search spaces, while coupled structure-and-parameter optimization complicates structural comparison.

    Source: §I · p. 1

  • Method

    Bi-EZP uses an LLM to generate executable aggregation programs with parameter bounds, then CMA-ES calibrates each fixed program before validation-based structural selection.

    Source: §I · p. 2, E. LLM-Guided Structural Evolution · p. 5

  • Results

    Bi-EZP achieves higher rank correlation than compared baselines on NATS-Bench and NDS and can serve as an evaluation signal in downstream DARTS search.

    Source: 1) Experimental Result in Rank… · p. 8, 2) Experimental Result in Rank… · p. 9, §V · p. 12

  • Takeaways & Limitations

    Separating program discovery from numerical calibration provides an effective approach to automated ensemble zero-cost proxy construction.

    Source: §V · p. 12

    Offline discovery incurs nontrivial LLM token costs and latency, and proxy quality depends on the pre-selected base metrics and prompt alignment.

    Source: §V · p. 12

Abstract

from arXiv · show

Zero-cost proxies enable neural architecture search (NAS) to rank candidate networks from statistics computed at initialization, avoiding repeated training. However, different proxies capture different properties and often produce inconsistent rankings across search spaces. Ensemble proxies can combine complementary signals, but automated discovery must optimize both discrete aggregation structures and their continuous coefficients, making structural quality difficult to separate from parameter calibration. We propose Bi-EZP, a bilevel framework that decouples these decisions. At the upper level, a large language model generates executable aggregation programs over four complementary base proxies with program-specific parameter bounds. At the lower level, covariance matrix adaptation evolution strategy (CMA-ES) optimizes the continuous parameters of each fixed program on an inner training split. The calibrated programs are then evaluated using Kendall's rank correlation on a disjoint validation split, enabling evolutionary selection to favor structures that generalize beyond their calibration data. Experiments on NATS-Bench and Network Design Spaces evaluate ranking performance across heterogeneous search spaces, and DARTS experiments assess downstream architecture search. Results show that separating program discovery from numerical calibration provides an effective approach to automated ensemble zero-cost proxy construction. The source code is available at: https://anonymous.4open.science/r/Bi-EZP-318D

I. INTRODUCTION

NAS evaluation is expensive, and heterogeneous zero-cost proxies can rank architectures inconsistently across search spaces. Bi-EZP addresses this by separating executable structure discovery from numerical calibration in a bilevel, LLM-guided framework.

  • Architecture evaluation dominates NAS cost because training every candidate is prohibitively expensive, while weight sharing can distort candidate rankings.Source: §I · p. 1
  • Different zero-cost proxies encode distinct inductive biases, so a proxy informative in one search space may be weak or misleading in another.Source: §I · p. 1
  • Bi-EZP decouples symbolic aggregation-structure discovery from structure-dependent continuous parameter calibration through bilevel optimization.Source: §I · pp. 1–2
  • An LLM generates executable proxy programs and program-specific parameter constraints, which are validated before CMA-ES optimizes their continuous parameters.Source: §I · p. 2
  • Bi-EZP evaluates competitive rank correlation across NATS-Bench and NDS and supports downstream architecture search experiments on DARTS.Source: §I · p. 2

B. Zero-Cost Proxies

Bi-EZP builds a reusable ensemble proxy from four heterogeneous zero-cost signals while separating program discovery from parameter calibration. Its executable candidates are validated, calibrated with CMA-ES, and selected using validation performance.

  • Ensemble proxies combine complementary signals, but existing parameterizations can restrict the interactions their aggregation functions express.Source: C. Automated and Ensemble Proxy… · p. 2
  • Bi-EZP retains NASWOT, MeCo, Sweet-SNIP, and SZiCo while making aggregation programs and parameter bounds outer-level variables.Source: C. Automated and Ensemble Proxy… · p. 3, B. Base Proxy Space · p. 3
  • The upper level uses an LLM to generate, recombine, and mutate symbolic aggregation programs, while CMA-ES optimizes parameters for each fixed program.Source: A. Overview · p. 3
  • A code gate checks syntax, interface, and finite bounds before calibration; invalid candidates receive correction attempts or a fixed fallback.Source: A. Overview · p. 3
  • The fixed four-proxy basis provides activation-, feature-, parameter-, and gradient-level views of initialized architectures.Source: B. Base Proxy Space · p. 3
  • Bi-EZP studies transformations and interactions among a fixed proxy set rather than searching for an optimal subset of proxies.Source: B. Base Proxy Space · p. 3

1) NASWOT (snw):

The proxy basis captures complementary properties of initialized networks, spanning activation patterns, feature geometry, parameter saliency, and gradient consistency.

NASWOT

  • NASWOT measures network expressivity through the diversity of binary activation patterns induced by a mini-batch.Source: 1) NASWOT (snw) · p. 3
  • NASWOT depends only on activation patterns, providing a gradient-free view that complements gradient- and feature-related signals.Source: 1) NASWOT (snw) · p. 3

MeCo

  • MeCo quantifies redundancy in intermediate representations using the minimum eigenvalue of a Pearson correlation matrix.Source: 2) MeCo (smc) · p. 3
  • A larger MeCo minimum eigenvalue indicates less-degenerate sampled feature directions and complements activation-pattern information.Source: 2) MeCo (smc) · pp. 3–4

Sweet-SNIP

  • Sweet-SNIP measures initialized-parameter importance through the product of parameter magnitude and loss gradient.Source: 3) Sweet-SNIP (sss) · p. 4

SZiCo

  • SZiCo evaluates gradient consistency by comparing mean gradient magnitude with variation across input batches, capturing a distinct trainability-related signal.Source: 4) SZiCo (ssz) · p. 4

C. Parameterized Proxy Representation

Bi-EZP represents each LLM-generated candidate as executable aggregation code with candidate-specific finite parameter bounds. Validation makes the generated object evaluable before CMA-ES calibration and evolutionary comparison.

D. Bilevel Discovery Objective

Bi-EZP separates structural program evolution from numerical calibration by evaluating each candidate after optimizing its parameters on inner-training data. Validation Kendall’s τ then determines structural fitness and evolutionary survival.

Bilevel objective

  • Each proposed program is independently calibrated on Dtrain before receiving fitness on the disjoint Dval split.The calibrated candidate, rather than an unevaluated parameterization, enters structural comparison.Source: D. Bilevel Discovery Objective · p. 5, E. LLM-Guided Structural Evolution · p. 5
  • Kendall’s τ evaluates ranking agreement at both levels, but inner calibration and outer structural comparison use different data and variables.This prevents the outer loop from favoring programs whose initial coefficients happen to be favorable.Source: D. Bilevel Discovery Objective · p. 5

Population evaluation

  • The outer search maintains a population of executable programs whose fitness is their validation Kendall correlation after lower-level calibration.Initialization and later offspring follow the same nested evaluation protocol.Source: E. LLM-Guided Structural Evolution · p. 5

Structural evolution

  • Tournament selection chooses valid parents, while LLM crossover or mutation proposes executable offspring with compatible interfaces and bounds.Fitness guides generation indirectly through parent selection rather than appearing directly in prompts.Source: E. LLM-Guided Structural Evolution · p. 5
  • Elitist (P + P) replacement merges parents and offspring, retaining the best P validated programs while continuing exploration through new code.The highest-fitness calibrated candidate is returned after a fixed number of outer cycles.Source: E. LLM-Guided Structural Evolution · p. 5

F. Lower-Level Parameter Adaptation via CMA-ES

CMA-ES calibrates the continuous parameters of each fixed aggregation program within bounds supplied by the LLM. It optimizes rank correlation on inner-training data while penalizing numerically invalid proxy evaluations.

Parameter optimization

  • CMA-ES optimizes coefficients within the fixed program’s bounded parameter manifold because the rank-correlation objective can be rugged, non-differentiable, and ill-conditioned.Its derivative-free covariance adaptation suits this lower-level objective.Source: F. Lower-Level Parameter Adaptation… · p. 5

Initialization

  • The search distribution is initialized geometrically within the LLM-specified bounds, with step size scaled to the average boundary range.The parameter domain is represented as a Cartesian product of per-parameter intervals.Source: F. Lower-Level Parameter Adaptation… · pp. 5–6

Evolutionary adaptation

  • At each generation, CMA-ES samples λ parameter vectors from a multivariate normal distribution defined by the current mean and covariance.The distribution is iteratively adapted using selected candidate vectors.Source: F. Lower-Level Parameter Adaptation… · p. 6

Objective and stability

  • Valid candidates are scored by proxy rank correlation on Dtrain, while NaN or Inf outputs receive a large penalty.This penalty steers optimization away from mathematically unstable parameter regions.Source: F. Lower-Level Parameter Adaptation… · p. 6

G. Nested Search Procedure

The nested procedure validates LLM-generated programs, calibrates their parameters on Dtrain, and compares frozen calibrated programs on Dval. Evolutionary selection uses only validation fitness, while later proxy use scores networks from initialization statistics.

Nested evaluation

  • Every generated program is checked for syntax and interface validity before CMA-ES calibrates its own bounded parameters on Dtrain.The resulting parameters are frozen before outer evaluation on Dval.Source: G. Nested Search Procedure · p. 6
  • Validation fitness alone drives tournament selection and elitist replacement, preventing validation information from entering CMA-ES’s numerical objective.Fallback candidates follow the same lower- and upper-level evaluation path.Source: G. Nested Search Procedure · p. 6

Frozen-proxy evaluation

  • After discovery, the frozen program scores candidate networks using initialization-time statistics rather than training each candidate to convergence.The discovery phase itself uses benchmark accuracies as supervision and is not entirely training-free.Source: H. Discovery Protocol and Frozen-Proxy… · p. 6

Algorithm flow

  • Algorithm 1 initializes candidates with the LLM, validates or corrects them, evolves offspring through crossover or mutation, and returns the highest-fitness population member.The procedure uses Dtrain and Dval with population size P, outer cycles T, and crossover probability Pc.Source: H. Discovery Protocol and Frozen-Proxy… · p. 6

Frozen-proxy evaluation

  • In DARTS experiments, Bi-EZP supplies an evaluation signal to the existing architecture-search procedure rather than replacing it.Discovery cost is incurred once before the frozen proxy scores many architectures.Source: H. Discovery Protocol and Frozen-Proxy… · p. 6

I. Computational Characteristics and Scope

The discovery cost scales with candidate-program evaluations, LLM correction attempts, and CMA-ES aggregation evaluations. The method’s empirical scope is bounded by its four-proxy basis, validation choices, and flexible but incompletely characterized program space.

Computational characteristics

  • P(T + 1) candidate programs are evaluated for population size P and T outer cycles when each cycle produces a full offspring population.This count includes the initial population and one full offspring population per cycle.Source: §I · p. 6

Lower-level procedure

  • CMA-ES receives G, B, Dtrain, and Emax, initializes within Cartesian-product bounds, clips sampled parameters, and ranks them by 1 − τ.Invalid scores receive a penalty before population updates.Source: §I · p. 7
  • CMA-ES updates the mean from the top μ vectors and adapts the global step size during each lower-level run.The procedure sorts candidates by loss before applying these updates.Source: §I · p. 7

Computational characteristics

  • At most rP(T + 1) LLM requests and EmaxP(T + 1) aggregation evaluations occur under the stated correction and CMA-ES budgets.Total discovery cost also includes base-proxy extraction and validation evaluation.Source: §I · p. 7

Scope and assumptions

  • The framework assumes its four fixed base proxies contain complementary information, but does not establish that this pool is optimal or explicitly penalize program complexity.Reproducibility requires reporting model, decoding, seed, population, cycle, correction, budget, partition, and final-program details.Source: §I · p. 7
  • Empirical claims are restricted to the supplied four-proxy basis, reported benchmarks, and evaluated discovery and transfer protocols.The implementation validates syntax, interface shape, and finite bounds but permits an open operator space and semantically duplicate programs.Source: §I · p. 7

IV. EXPERIMENTAL STUDIES

Bi-EZP is evaluated across NAS benchmarks, datasets, search spaces, and downstream DARTS search using Kendall’s τ as the primary ranking metric. Under the reported protocols, it achieves higher rank correlation than compared methods, while the experiments do not isolate which program-design factor causes the gains.

Evaluation scope

  • The evaluation spans NATS-Bench, NDS, and DARTS search spaces, with CIFAR-10, CIFAR-100, and ImageNet used across proxy extraction, calibration, transfer, and final evaluation.Source: A. Datasets and Search Space · p. 7

Rank-correlation evaluation

NATS-Bench results

Interpretation boundary

Baseline comparisons

2) Experimental Result in Rank Correlation Across NDS Benchmarks:

Bi-EZP achieves strong rank correlation across heterogeneous NDS architectural families and retains measurable cross-space reuse when its discovered proxy is frozen. Downstream DARTS and ImageNet experiments further evaluate practical transfer, while reported search costs exclude one-time proxy discovery.

Cross-dataset transfer

  • Frozen proxies transfer across datasets with mean Kendall’s τ of 0.775 versus 0.792 for matched settings, a decrease of 0.017.Source: 3) Cross-Dataset and… · p. 9

Cross-search-space transfer

Downstream search

D. Ablation Study and Hyperparameter Analysis

Ablations indicate that bilevel separation and LLM-based structural proposals contribute to Bi-EZP’s reported correlation. Sensitivity analyses and LLM comparisons examine operating choices, while the reported downstream search cost excludes offline discovery.

Hyperparameter analysis

Ablation study

2) Comparison of Lower-Level Optimizers:

Bi-EZP compares lower-level continuous optimizers and evolutionary hyperparameter settings while separating program discovery from parameter calibration. CMA-ES performs best among the tested alternatives, and population and generation budgets show non-monotonic sensitivity.

Comparison of Lower-Level Optimizers

  • CMA-ES achieves higher rank correlation than GA, DE, and PSO variants across the evaluated search spaces.The comparison motivates CMA-ES because covariance adaptation handles dependencies among continuous parameters and the objective is non-differentiable.Source: 2) Comparison of Lower-Level Optimizers · p. 11, Table VII · p. 12

Impact of Evolutionary Hyperparameters

  • Population performance generally improves from P = 5 to P = 20 but does not improve further at P = 50.Observed τ reaches approximately 0.809 on CIFAR-10, 0.791 on CIFAR-100, and 0.775 on ImageNet-16-120 at P = 20; P = 50 lowers two of these values.Source: 3) Impact of Evolutionary… · p. 12
  • The generation sweep is non-monotonic, with the strongest balanced observed result at Tmax = 20 rather than 50 generations.Increasing from 5 to 20 generations raises correlation to 0.809/0.791/0.775 on CIFAR-10, CIFAR-100, and ImageNet-16-120, while 50 generations produce mixed results.Source: 3) Impact of Evolutionary… · p. 12

Performance Sensitivity under Different LLMs

  • GLM-4.7-Flash produces higher correlations than DeepSeek-V4-Flash in several evaluated settings, including ImageNet-16-120 under TSS and SSS.The comparison shows operation with either tested backbone but does not establish invariance across providers, model scales, or decoding configurations.Source: 4) Performance Sensitivity under… · p. 12

Evolutionary Trajectory and Search Dynamics

  • The best-so-far validation correlation reaches 0.8255 at Generation 17 after an early increase to 0.8115 during Generations 1-5.The trajectory then plateaus through Generations 5-16 and remains unchanged through the recorded budget; it does not establish global convergence.Source: E. Evolutionary Trajectory and Search… · p. 12

Overall Findings

  • Across the evaluated protocols, Bi-EZP achieves higher rank correlation than compared baselines on NATS-Bench and NDS and supports downstream DARTS search.The framework separates LLM-based program proposal from CMA-ES-based continuous parameter calibration.Source: §V · p. 12

Related papers

Loading 2608.21927v1…