PaperlayerView exact revision on arXiv

Source-linked AI summary

CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search

Yifan Yang, Zhaoyan Wang, Zheng Gao, Xiaoyu Li, Jiaojiao Jiang

arXiv:2609.11884v1cs.LGcs.CV

TL;DR

Zero-cost proxies are cheap but unreliable across NAS search spaces, motivating a method that combines static ranking with limited dynamic evidence. CoRA-NAS uses a gated consensus prior and anchor-based residual refinement, achieving robust cross-space rankings at low cost while remaining tied with the strongest capacity proxies on pure size spaces.

  • Problem

    Zero-cost proxies rank NAS architectures cheaply, but their reliability varies across search spaces, making robust performance estimation difficult.

  • Method

    CoRA-NAS combines an equal-weight, target-free consensus of capacity and structure proxies with anchor-based residual correction from approximately 1% early-training curves.

  • Results

    Across four vision search spaces, CoRA-Refine has no weak regime and achieves Spearman correlations of 0.946, 0.715, 0.786, and 0.894, with the highest worst-space correlation among compared methods.

  • Takeaways & Limitations

    CoRA-NAS provides cross-space ranking robustness from one configuration while using low-cost dynamic refinement for architecture selection.

  • Takeaways & Limitations

    On pure size spaces, refinement improves ranking but only ties the strongest capacity proxies for best-found accuracy, while non-vision NAS-Bench-NLP is an honest weak regime.

Abstract

from arXiv · show

Zero-cost proxies rank architectures cheaply, but their reliability varies across search spaces. We introduce CoRA-NAS (COarse Ranking + Anchor-residual), a two-stage framework combining a static ranking prior with low-cost learning-curve refinement. CoRA-Rank aggregates capacity and structure-at-initialization proxies through an equal-weight log-rank consensus and a target-free consensus gate. CoRA-Refine samples anchors across this prior, extrapolates their early validation curves, and propagates a learned residual correction with an ExtraTrees model. The refinement uses approximately 1% of the cost of fully training the candidate set. Fully trained architecture-accuracy labels are not used to fit the ranker. One configuration is used across spaces, with space-specific architecture encodings. Across NAS-Bench-201, NAS-Bench-101, TransNAS-Bench-101, and NATS-SSS, CoRA-Refine achieves mean Spearman correlations of 0.946, 0.715, 0.786, and 0.894, respectively. Its worst-space correlation of 0.715 is the highest among the compared methods. On NAS-Bench-201/CIFAR-100, its selected architecture reaches 73.32% accuracy, near the reported ground-truth best of 73.37%. On the pure size space, refinement recovers the static prior's shortfall relative to parameter count, while remaining tied with the strongest capacity proxies within noise. The resulting framework combines cross-space ranking robustness with low-cost architecture selection.

1 Introduction

CoRA-NAS addresses weak cross-space reliability in zero-cost architecture ranking with a label-free Rank+Refine pipeline that combines complementary static proxies with a small dynamic curve-residual correction. Across four vision spaces, it improves robustness while using approximately 1% refinement cost and a single configuration.

  • Motivation: Zero-cost proxies face a cross-over: capacity-driven methods falter on topology-varying spaces, whereas structure-driven methods falter on size-varying spaces.This leaves no single simple baseline reliably superior to #Params/FLOPs across both regimes.
  • CoRA-NAS: CoRA-Rank combines capacity and structure-at-initialization proxies through an equal-weight rank consensus and a target-free gate that adapts the bank without accuracy labels.The gate removes proxies based on inter-proxy agreement rather than target accuracy.
  • CoRA-NAS: CoRA-Refine samples anchors, extrapolates their early learning curves, and propagates a learned residual correction with a tree ensemble before local smoothing.The residual corrects the static prior rather than predicting final accuracy from scratch.
  • Findings: Approximately 1% refinement cost reaches near-oracle ranking quality on NAS-Bench-201, raising mean Spearman from CoRA-Rank’s 0.927 to 0.946.CoRA-Rank remains on the cost–quality Pareto frontier despite not being the cheapest zero-cost proxy.
  • Findings: 0.715 worst-space Spearman is the highest floor among compared methods, while CoRA-Refine reports 0.946/0.715/0.786/0.894 across four vision benchmarks.The benchmarks span NAS-Bench-201, NAS-Bench-101, TransNAS-Bench-101, and NATS-SSS.
  • Findings: On real DARTS cell space, refinement beats the diluted prior by +0.046 and matches #Params within noise using real 15-epoch curves.A scramble control confirms the lift.

2 Related Work

Related work spans zero-cost proxies, proxy aggregation, supervised performance prediction, multi-fidelity search, and one-shot methods. CoRA-NAS differs chiefly by combining complementary static proxies with a label-free curve residual rather than using target accuracy or retraining a supernet per space.

  • 2.1 Zero-cost (training-free) proxies: Zero-cost proxies estimate architecture quality from one or a few initialization-time forward/backward passes, but no single proxy robustly beats #Params across structure and size spaces.The related-work discussion covers gradient-based and gradient-free proxy families.
  • 2.2 Aggregating and combining proxies: Proxy-combination methods differ in supervision: RoBoT queries ground-truth evaluations, LIBRA-NAS selects proxies using validation accuracy, whereas CoRA uses label-free consensus and gating.CoRA also reuses published signals rather than introducing a new proxy.
  • 2.3 Performance prediction: Supervised performance predictors and learning-curve extrapolators can improve accuracy over individual proxies, but typically require ground-truth labels and a fresh fit or transfer for each search space.CoRA instead forms a label-free residual on a static prior.
  • 2.4 Multi-fidelity methods: Multi-fidelity methods spend adaptive training budgets, whereas CoRA uses a small tunable budget on fixed anchors and propagates a label-free curve residual without online ground-truth comparisons.This places CoRA in a similar cost–quality regime with a different supervision mechanism.
  • 2.5 One-shot methods: One-shot and supernet methods amortize training but introduce supernet-induced ranking bias and require retraining per search space, unlike CoRA’s proxy-agnostic Rank+Refine design.CoRA is not a one-shot method.

3 Method

CoRA-NAS decomposes architecture quality into static capacity, initialization-time structure, and learning dynamics, then estimates them with a label-free prior and low-cost residual refinement. Its two-stage pipeline combines gated zero-cost consensus ranking with anchor-based early-curve extrapolation and learned correction.

  • 3 Method: CoRA-NAS targets architecture ranking without training the full search space or querying ground-truth accuracy.
  • 3 Method: The method combines static capacity and initialization-time structure in a prior, then injects learning dynamics as a residual correction.The three axes are presented as complementary, with the prior covering the two static axes and refinement adding the dynamic axis.
  • 3.2 Stage 1: Rank with a zero-cost coarse prior: Stage 1 ranks five off-the-shelf proxies across capacity and structure-at-initialization, using a target-free consensus gate before aggregation.The bank includes #Params, Synflow, jacov, and AZ-NAS expressivity and trainability views; undefined proxies are removed before gating.
  • 3.2 Stage 1: Rank with a zero-cost coarse prior: The kept proxies form a static prior by equal-weight log-rank consensus, with no learned weights or ground-truth tuning across spaces.The gate retains the full bank on most evaluated spaces and removes jacov on NB101, where it anti-correlates with the consensus.
  • 3.3 Stage 2: Refine with an anchor-based dynamic residual: Stage 2 samples anchors across the prior, trains them for 5–30 epochs, extrapolates early validation curves, and fits an ExtraTrees residual model.The anchor and prefix fractions use approximately 1% of full-training cost, and the residual is propagated to all architectures before smoothing.
  • 3.4 Why these choices: a clean prior keeps the residual learnable: Residual correction is intended to be learnable because static-ranking errors vary systematically with architecture operations and topology encodings.Scrambling curve-to-anchor assignments collapses refinement toward the prior, providing a control for the role of genuine curve signal.

4 Experiments

Across four search spaces, CoRA-Refine combines a label-free static prior with low-cost learning-curve residuals to improve ranking robustness and architecture selection under one configuration. Experiments also identify boundaries: refinement helps most when the prior is weak, while surrogate-based and non-vision evaluations constrain generalization claims.

  • NAS-Bench-201 selection: 73.32% CIFAR-100 accuracy nearly matches the 73.37% ground-truth best, while refinement improves mean best-found accuracy across all three NB201 datasets.The Stage 2 result uses approximately 1% of full-training cost and is evaluated separately from zero-cost Stage 1 ranking.
  • Cross-space generalization: 0.715 worst-space Spearman is the highest among compared methods, while CoRA-Refine reaches 0.946, 0.786, and 0.894 on NB201, TransNAS, and NATS-SSS.Across the four spaces, refinement improves the static prior from 0.927 to 0.946, 0.750 to 0.786, and 0.747 to 0.894, with NB101 reaching the 0.715 floor.
  • Size-space boundary: 0.894 on NATS-SSS recovers the static prior’s 0.747 shortfall against #Params, but selection remains roughly tied with #Params-driven selection.The size-space gain is primarily ranking robustness rather than a single-pick selection advantage.
  • Prior ablation: A clean consensus prior reaches 0.951 after refinement, whereas opaque AZ-NAS reaches 0.929 despite standalone ranking of 0.896.Residual learnability matters alongside starting rank: params-only has CV-R2 0.46 but loses because its starting rank is weak at 0.724.
  • Real-space search: On DARTS, CoRA-Refine reaches 2.81±0.05% test error, beating AZ-NAS, CoRA-Rank, and random but trailing gradient or training-free references at 2.50–2.76%.The comparison uses a shared 100k-genotype pool, five shared seeds, and a surrogate-based ranking reference that is not directly comparable to tabular true-accuracy correlations.
  • Transformer transfer: ViT-Bench-101 shows bounded transfer: CoRA-Rank is approximately 0.03 below label-using L-SWAG on mean Spearman, although it leads on CIFAR-100-KD.#Params remains stronger on the capacity-leaning space, and CoRA-Rank’s worst-case correlation falls to 0.002.

5 Limitations

The method’s claims are bounded by space, encoding, consensus, and cost assumptions: it is not universally robust, uses space-specific encodings, can suffer consensus dilution, and Refine is neither data-free nor zero-cost.

  • 5 Limitations: On NATS-SSS, CoRA-Refine reaches 0.894 from 0.747, beats #Params by 0.07, but ties Synflow and MeCo within corrected noise.Its size-space claim is robustness and baseline clearance, not superiority over the strongest capacity proxies.
  • 5 Limitations: The algorithm and hyperparameters are shared across spaces, but Stage 2 requires space-specific operation, topology, and locality encodings.“One configuration” therefore does not mean a space-agnostic architecture representation.
  • 5 Limitations: The consensus gate detects anti-correlation, not dilution, so uninformative proxies can lower consensus quality on capacity-dominated spaces.On DARTS, expressivity alone reaches 0.578 versus 0.470 for the consensus; importance weighting is left for future work.
  • 5 Limitations: CoRA is label-free with respect to fully trained accuracy labels, but jacov uses a small unlabeled real-data minibatch and Refine costs approximately 1% of training.Stage 1 is the zero-cost operating point; Refine is positioned on a cost–quality frontier.

6 Conclusion

CoRA-NAS is a label-free, single-configuration Rank+Refine framework that combines static proxy consensus with early-curve residual correction. Across four vision benchmarks it avoids the weak regime seen in every baseline, although its benefit depends on informative priors and encodings.

  • 6 Conclusion: Stage 1 combines capacity and structure-at-initialization proxies with a target-free anti-correlation gate, while Stage 2 propagates residuals from approximately 1% early training curves.The same algorithmic configuration is used across spaces, with architecture encodings defined per space.
  • 6 Conclusion: CoRA-Refine has no weak regime across four vision benchmarks, while every baseline weakens on at least one space and its worst space is the highest among methods.The four spaces include topology, cross-task, and size variation; the recurrent-space result scopes this claim to vision.
  • 6 Conclusion: CoRA’s benefit scales with the informativeness of its prior and architecture encoding, producing an honest weak regime when both are weak.On ViT-Bench-101, the label-free consensus is competitive with L-SWAG and ahead of AZ-NAS.

A Reproduction notes (Table 2 baselines)

The reproduction appendix documents unified baseline reruns, cross-machine checks, and provenance for discrepancies rather than introducing new empirical claims.

  • A Reproduction notes (Table 2 baselines): Nine GPU-dependent proxies agree with an independent RTX-4090 reproduction within |ΔSPR| ≤0.009 across all 27 corresponding cells.#Params and FLOPs are deterministic, while AZ-NAS is absent from the 4090 comparison.
  • A Reproduction notes (Table 2 baselines): Table 13 records eight methods with published-value differences exceeding 0.02 and documents the cause for each discrepancy.The appendix states that none of these discrepancies is a port error.
  • A Reproduction notes (Table 2 baselines): T-CET is evaluated using KT, and AZ-NAS reports a faithfully reproduced ImageNet16-120 value of 0.877 when its published figure was not reproducible.Cross-space provenance and reproduction caveats for several baselines are documented in Table 3 footnotes.

B Cross-space table: per-cell sourcing and reproduction caveats

The cross-space table combines live, sampled, benchmark-sourced, and paper-cited values, with explicit caveats for data availability, target-aware comparisons, and non-reproducing baselines.

  • B Cross-space table: per-cell sourcing and reproduction caveats: Table 3 mixes live computations, benchmark-sourced values, and originating-paper citations, with unchanged values documented cell by cell.The appendix distinguishes provenance rather than treating every entry as a fresh live reproduction.
  • B Cross-space table: per-cell sourcing and reproduction caveats: NB201 uses all 15,625 architectures, whereas NB101, TransNAS, and NATS-SSS live results use fixed 3,000-architecture samples.Refine adds approximately 1% training, using 1,000 anchors on NB201 and 192 stratified anchors in the other three columns.
  • B Cross-space table: per-cell sourcing and reproduction caveats: TransNAS anchor curves come from recorded benchmark trajectories, while NB201, NB101, and NATS-SSS anchors are live-trained.Jacov is omitted from NB101 ExtraTrees features through the consensus gate.
  • B Cross-space table: per-cell sourcing and reproduction caveats: TransNAS classic baselines rely on Suite-Zero scores because required Taskonomy images are non-redistributable, while CoRA and unavailable proxies are computed live.Fisher’s stored pipeline value is 0.44 versus 0.21 from the live adapter.
  • B Cross-space table: per-cell sourcing and reproduction caveats: LIBRA-NAS is a soft upper bound because it selects proxies using ground-truth accuracy labels, unlike CoRA’s early-curve refinement.CoRA’s reported correlations exceed LIBRA-NAS on NB201, TransNAS, and NATS-SSS.
  • B Cross-space table: per-cell sourcing and reproduction caveats: Dextr and MeCo are anti-correlated on NATS-SSS, so the table reports absolute Spearman values while preserving their negative raw sign.Their scores grow with channel capacity, which saturates against test accuracy.
  • B Cross-space table: per-cell sourcing and reproduction caveats: SWAP’s published NB101 value of 0.77 does not reproduce: the raw score reaches 0.44 and the size-regularized score 0.40.The same regularizer improves NATS-SSS from 0.54 to 0.70, and the discrepancy is not a sample-count artifact.
  • B Cross-space table: per-cell sourcing and reproduction caveats: T-CET’s TransNAS result uses one available real-image class_object task as a surrogate, with the other task images unavailable.Its absolute Spearman is compared with the published Object Kendall-τ.
Loading 2609.11884v1…