PaperlayerView exact revision on arXiv

Source-linked AI summary

RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search

Yifan Yang, Zhaoyan Wang, Zheng Gao, Xiaoyu Li, Jiaojiao Jiang

arXiv:2609.12418v1cs.LGcs.CV

TL;DR

Neural architecture search is costly when enough candidates must be fully trained to rank an entire space, while cheap proxies and learned predictors have limitations. RiPPLE trains a small coverage set to early prefixes, extrapolates their curves into labels, and propagates those labels over label-free features; it ranks near the oracle across twelve cells under a fractional training budget.

  • Problem

    Ranking an entire architecture space is costly because fully training candidates is expensive, while zero-cost proxies vary in ranking quality and learned predictors often require full labels or per-candidate partial-training features.

    Source: §1 · pp. 1–2

  • Method

    RiPPLE prefix-trains a small set of coverage anchors, extrapolates their learning curves into surrogate labels, and propagates those labels from label-free features to rank the whole space.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Under a ceiling of about 150 full-training equivalents, one unchanged pipeline ranks near the oracle in twelve cells from four families; on NAS-Bench-201 it reaches Kendall τ 0.85 at 127 equivalents.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    RiPPLE’s anchor-label design decouples training cost from ranking reach, allowing one unchanged pipeline to rank tabular benchmarks and the DARTS space without retuning.

    Source: §1 · p. 2, §5 · p. 12

    RiPPLE does not beat dedicated multi-fidelity search on the single architecture returned; their best-found accuracies are comparable under reported protocols, without establishing statistical equivalence.

    Source: §5 · p. 12

Abstract

from arXiv · show

Neural architecture search (NAS) evaluates candidate networks, but fully training enough architectures to rank an entire space is expensive. Zero-cost proxies score architectures at initialization, yet their ranking quality varies across search spaces. Learned predictors reduce evaluation cost but typically require fully trained labels or partial-training features for individual candidates. We introduce $\textbf{RiPPLE}$, $\underline{\textbf{R}}$anking v$\underline{\textbf{i}}$a $\underline{\textbf{P}}$refix-$\underline{\textbf{P}}$ropagated $\underline{\textbf{L}}$abel $\underline{\textbf{E}}$xtrapolation, which treats partial training as a source of labels for a small coverage set of anchors. RiPPLE trains these anchors to an early prefix, extrapolates their learning curves to surrogate labels, and propagates the labels over label-free architecture features. The early-training signal remains a label on the anchors rather than a per-candidate feature. Feature, readout, and encoding rules are selected without held-out accuracy and reused across search spaces. We evaluate the method on twelve benchmark cells from four search-space families and on the larger DARTS space. The results examine ranking quality, label efficiency, architecture selection, and the roles of readout, coverage, and propagation. RiPPLE provides a whole-space ranking from a fractional anchor-training budget, with comparisons interpreted under their respective evaluation and cost protocols.

1 Introduction

RiPPLE addresses the cost of ranking an entire architecture space by using partial training to label a small set of anchors, then propagating those labels across the space. Across twelve cells from four search-space families, one unchanged pipeline ranks near the oracle under a ceiling of about 150 full-training equivalents.

1 Introduction

  • Figure 1’s budget-quality curves are descriptive comparisons, not evidence of strictly matched evaluation sets and total costs.FlowerFormer’s reported label budget and independent-test protocol are not harmonized with RiPPLE’s FTE accounting.Source: Figure 1 · p. 2
  • Whole-space ranking at a fractional budget remains challenging because zero-cost proxies are weak across spaces, while supervised predictors pay for a full training per label.NAS training is expensive, and many evaluated architectures are later discarded.Source: §1 · pp. 1–2
  • RiPPLE confines partial training to a few coverage anchors, extrapolates their learning curves into surrogate labels, and propagates those labels using zero-cost proxies and architecture encodings.The early-training signal is a label on anchors rather than a per-candidate feature, separating training cost from whole-space ranking reach.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Under a ceiling of about 150 full-training equivalents, one unchanged pipeline ranks near the oracle in all twelve cells; on NAS-Bench-201 it reaches Kendall τ 0.85 at 127 equivalents.Its top-ranked networks lie within 0.08 on CIFAR-10 and 0.26 on CIFAR-100 of the optima, and it is competitive with trained search on DARTS.Source: §1 · pp. 2–3
  • The contributions characterize RiPPLE as matching supervised predictors with fewer labels while selecting near-optimal architectures.The paper also reports that zero-cost proxies do not hold across the evaluated spaces and that graph- or sequence-based predictors do not apply to some of them.Source: §1 · p. 3

2 Related Work

Prior NAS predictors trade off whole-space reach, ranking quality, and the cost or applicability of their signals. RiPPLE differs by propagating early-curve surrogate labels from coverage anchors rather than using curves as per-candidate features or a corrected proxy prior.

2 Related Work

  • Zero-cost proxies score architectures from an untrained forward pass, but their agreement is capped near Spearman 0.9 on NAS-Bench-201 and does not transfer across spaces.Supervised predictors use fully trained examples; graph and sequence predictors can be accurate but are label-hungry and may lack encodings for size or recurrent spaces.Source: §2 · pp. 3–4
  • Curve-extrapolation, multi-fidelity, and feature-augmented methods partially train candidates they score, while weight-sharing methods bind to one space or rank weakly.OMNI uses a learning-curve statistic as a per-candidate feature; RiPPLE instead confines prefix training to anchors and propagates labels across the space.Source: §2 · p. 4
  • CoRA-NAS also uses early learning curves on anchors, but combines a coarse zero-cost ranking prior with residual correction; RiPPLE directly propagates prefix-derived labels from coverage anchors.RiPPLE uses zero-cost proxies as input features rather than as a separately corrected ranking prior.Source: §2 · p. 4

3 Method

RiPPLE ranks a search space by training a small set of coverage anchors to an early prefix, converting their learning curves into surrogate labels, and propagating those labels over label-free architecture features. Its training budget is measured in full-training equivalents, and only the anchors incur training.

3.1 Problem setup and notation

  • FTE = nπ measures the cost of training n architectures to prefix π of a full training schedule.One FTE is the cost of training one architecture to the end of its schedule.Source: §3.1 · p. 5
  • RiPPLE uses structural encodings and zero-cost proxies as label-free features, applying the standard proxy bank and available architecture encodings without per-space selection.The propagator can down-weight uninformative feature coordinates.Source: §3.1 · p. 5

3.2 The RiPPLE framework

  • Farthest-point sampling selects anchors in encoding space so they spread across the search space and the propagator interpolates within covered support.The anchor order is nested, allowing a single list to supply different budgets by truncation.Source: §3.2 · p. 5
  • Each anchor is trained for a prefix, and a reader converts its partial training-loss curve into a surrogate label using recent loss, an exponential moving average, or finite-horizon extrapolation.The log-linear extrapolation horizon and reader branch are fixed per space before training.Source: §3.2 · p. 6
  • Extremely randomized trees propagate the noisy anchor labels over label-free features, and scoring the untrained architectures requires only one inference pass.The method uses the ranking induced by its score, not a score calibrated to true performance.Source: §3.1 · p. 5, §3.2 · p. 6
  • RiPPLE produces a whole-space ranking by fitting a propagator to anchor labels and scoring every architecture from precomputed features, with training limited to the anchors.The partial-training signal is stored in the labels rather than used as a per-candidate feature.Source: §3 · p. 5, §3.2 · p. 6

4 Experiments

Experiments test RiPPLE across twelve cells from four search-space families and the DARTS space, measuring held-out ranking, selection, and ablations. Under the reported protocols, RiPPLE-prefix ranks well across families, and the ablations examine the roles of anchors, propagation, and feature blocks.

4.1 Experimental settings

  • The evaluation covers twelve cells from four families, with ranking measured on architectures whose training curves were not read and a common 150-FTE ceiling for the main comparison.The cells span topological, size, recurrent, and transfer families; DARTS is evaluated separately, with NAS-Bench-301 surrogate ground truth for correlations.Source: §4.1 · pp. 6–7, §4.2 · p. 11

4.2 Results

  • RiPPLE-prefix beats every external baseline on eleven of twelve cells under a 150-FTE ceiling, while RiPPLE-full and RiPPLE-prefix rank well on every cell.GP-WL edges RiPPLE-prefix by 0.001 on NATS-SSS ImageNet16-120, where RiPPLE-full is best.Source: Table 1 · p. 8, §4.2 · p. 8
  • RiPPLE-prefix first reaches Kendall τ≥0.85 at 127 FTE on NAS-Bench-201 CIFAR-10, the first crossing in its tested grid rather than a proven continuous minimum.External predictor results use their own protocols, so this is not a controlled common-cost comparison.Source: Table 2 · p. 9, §4.2 · p. 9
  • Under the predictor papers’ protocol, RiPPLE-full is strongest at every standard budget, while RiPPLE-prefix exceeds the best prior method at the smallest budget.This comparison uses a proxy-augmented feature set and tree propagator against learned representations, not a controlled test of the label mechanism.Source: §4.2 · p. 9, Table 3 · p. 10
  • RiPPLE-prefix selects the strongest architecture among equal-budget methods on the three NAS-Bench-201 datasets and lands within 0.08 to 0.83 points of the optimum.On NAS-Bench-101, methods agree within noise; the top of these spaces is described as flat.Source: §4.2 · p. 10, Table 4 · p. 10
  • On DARTS, the unchanged pipeline ranks above fidelity baselines at every budget, and its selected architecture reaches 2.61% test error under real training.The ranking correlations use NAS-Bench-301 surrogate ground truth and are bounded by its fidelity; the selection result does not depend on that surrogate.Source: §4.2 · p. 11

4.3 Ablation study

  • Across the ablations, anchors and propagation each help, both feature blocks together are best on eleven of twelve cells, and label-free rules recover a mean correlation of 0.902.Without anchors, mean held-out correlation is 0.593; reading curves without propagation covers at most 16% of a space. Proxies lead on topological cells, while encodings lead on size cells.Source: §4.3 · pp. 11–12, Table 5 · p. 11, Table 6 · p. 12

5 Conclusion

RiPPLE ranks architecture spaces from a fractional training budget by using partial training as labels for coverage anchors. It performs strongly across twelve cells while leaving limits around single-architecture search and the coverage guarantee explicit.

5 Conclusion

  • Across twelve cells from four search-space families, RiPPLE is the strongest ranker under a common budget ceiling and selects near-optimal NAS-Bench-201 networks.It is competitive with its full-training control while paying a fraction of the label cost; selected networks are within a fraction of a point of the held-out evaluation-pool optimum on CIFAR-10 and CIFAR-100.Source: §5 · p. 12
  • RiPPLE uses partial-training signals as labels on coverage anchors, decoupling training cost from whole-space ranking reach and enabling transfer to about 10^18 DARTS architectures without retuning.The same pipeline ranks tabular benchmarks and DARTS.Source: §5 · p. 12
  • RiPPLE does not outperform dedicated multi-fidelity search on its single returned architecture, and its coverage analysis bounds a conservative nearest-neighbor propagator rather than the deployed regressor.The two search approaches achieve comparable best-found accuracy under the reported protocols, without establishing statistical equivalence; adaptive anchors and a learned curve reader remain future work.Source: §5 · pp. 12–13

Supplementary Material

The supplement collects deferred deployment, theory, cost, selection, ablation, and protocol-comparison material. It also specifies the fixed, transferable per-space configuration used by RiPPLE.

Supplementary Material

  • The supplement covers deployment configuration, theoretical analysis, cost and FTE fairness, horizon selection, ablations, selection and scaling, and protocol comparisons.These topics are deferred from the main paper.Source: Supplementary Material · p. 17

A Deployed configuration

  • RiPPLE propagates prefix-derived surrogate labels directly from coverage-selected anchors, unlike CoRA-NAS, which refines a zero-cost ranking prior with anchor residuals.RiPPLE uses zero-cost proxies as input features rather than as a separately corrected prior.Source: A Deployed configuration · p. 17
  • The deployed pipeline fixes its tree propagator, zero-cost proxy bank, architecture encodings, and schedule-shape reader rules before any benchmark is seen.The configuration uses 400 extremely randomized trees; Table 7 gives three per-space rules, and Table 8 lists each cell’s anchor count, prefix epochs, and FTE budget.Source: A Deployed configuration · p. 17

B Theoretical analysis

The analysis decomposes RiPPLE’s ranking into readout and propagation, then studies budget allocation, prefix-label error, and anchor coverage. Its guarantees apply under stated assumptions, with empirical checks reported separately.

B Theoretical analysis

  • End-to-end Kendall correlation is bounded by the readout and propagation correlations, but their product is only the leading term because an exact identity also includes covariance.The identity applies on the stated common set; Figure 3 is a qualitative diagnostic, not a numerical verification on paired runs.Source: B Theoretical analysis · pp. 17–18
  • Below Bcrit, the compute-optimal allocation is interior, so neither fully training a few anchors nor trivially training many anchors is optimal.The empirical budget-quality frontier rises and saturates at a knee; the deployed point sits there, with steep-frontier spaces reaching it within tens of FTE and NAS-Bench-101 needing more.Source: B Theoretical analysis · pp. 18–19
  • On SSS-C10, the Figure 4 allocation uses 22 epochs, about 41 FTE, versus about 43 FTE for the current 23-epoch deployment.Figure 4 plots held-out correlation against budget in full-training equivalents on a log scale.Source: Figure 4 · p. 19
  • The fitted far-horizon exponent is −1.575, close to the predicted −3/2; TNB-Object’s −2.44 fit falls outside the assumed far-horizon regime.The required prefix falls with the cube root of the noise-to-tolerance ratio and with schedule length; the steeper TNB-Object slope reflects its probe horizon equaling store length.Source: B Theoretical analysis · p. 19, Figure 5 · p. 20
  • The coverage certificate bounds nearest-neighbor label propagation under a Lipschitz target and bounded anchor-label error; sufficiently separated true-value pairs are ordered correctly.The nested farthest-point order has coverage radius at most twice the optimal covering radius.Source: B Theoretical analysis · p. 19
  • The certificate is for conservative nearest-neighbor propagation, not the deployed tree, which also uses proxy coordinates and ranks better on every cell.The authors treat the bound-to-tree gap as a measured gain rather than a lemma claim.Source: B Theoretical analysis · p. 19
  • Farthest-point anchors have smaller coverage radii than random anchors on eleven of twelve cells; the deployed tree is at least as good on eight of ten non-degenerate cells.Mean Spearman gains are +0.0005 for the tree and +0.0115 for nearest-neighbor propagation; NAS-Bench-101 is degenerate, while the tree loses on TNB-Object and TNB-Jigsaw.Source: B Theoretical analysis · p. 20

C Cost and FTE fairness

The cost analysis distinguishes training-equivalent budgets from wall-clock cost and measures feature and predictor overhead. Matched FLOPs distributions support the fairness of the matched-FTE comparison in expectation.

C Cost and FTE fairness

  • The cost measurements include proxy-bank time, one training epoch, and architecture-encoding time on a pinned L40S.The proxy bank takes a median 0.64 seconds against 1.75 seconds per NAS-Bench-201 architecture, and 3.4 seconds against 9.6 seconds on NAS-Bench-101.Source: C Cost and FTE fairness · p. 20
  • FTE is not a wall-clock proxy by itself because per-architecture epoch time varies 28–194× within a space.Matched-FTE comparisons are fair in expected wall-clock terms because RiPPLE anchors and sampled baselines have matching FLOPs distributions.Source: C Cost and FTE fairness · p. 20
  • RiPPLE alone pays a zero-cost proxy-bank cost of about 0.6 s per architecture, roughly 0.4 of one training epoch, totaling about 19% of the 150-FTE anchor budget.Source: C Cost and FTE fairness · pp. 20–21
  • RiPPLE’s CPU tree ensemble fits in 1.6 seconds, versus 10.8 seconds for the GCN and 32.7 seconds for SemiNAS, while GP-WL fits in 0.09 seconds.RiPPLE is not the cheapest predictor, and its proxy-bank feature cost accompanies higher ranking quality than the kernel and graph predictors.Source: C Cost and FTE fairness · pp. 20–21

D Label-free selection of the reader horizon

The reader horizon is selected by backcasting the observed prefix without true accuracy labels, and the deployed horizon stays near the correlation peak. Extended ablations show that the proxy bank, tree propagator, and reader choice each contribute to the pipeline’s cross-cell performance.

D Label-free selection of the reader horizon

  • A label-free backcast selects the true-label oracle’s horizon on nine of twelve cells, and the deployed horizon is at or near each cell’s peak.Reading at or beyond the store length can sharply reduce correlation, including a drop from 0.893 at the peak to 0.541 at T on TNB-Scene.Source: D Label-free selection of the reader… · pp. 21–22, Figure 7 · p. 22

E.1 Proxy bank

  • The full available proxy bank with architecture encoding matches the per-cell leakage oracle within 0.001 in aggregate, while the universal set trails by up to 0.022 on NAS-Bench-101.The proxy bank is determined by what each space exposes, and the resulting twelve-cell mean is 0.902.Source: §E.1 · pp. 22–23, Table 10 · p. 23, Table 11 · p. 23

E.2 Propagator

  • The deployed ExtraTrees propagator ranks first at a twelve-cell mean of 0.902, ahead of histogram gradient boosting at 0.896 and the Gaussian process at 0.877.The wide feature block penalizes distance- and norm-based regressors, while tree ensembles can down-weight uninformative columns.Source: §E.2 · pp. 23–24, Table 12 · p. 24

E.3 Reading rule

  • On NAS-Bench-101, fit-based extrapolators and the ensemble range from 0.30 to 0.64, whereas TSE-EMA reaches 0.8539.The stated conclusion is that fit-based extrapolators fail on this cell and a non-extrapolating reader is required; TSE-EMA’s 0.0086 margin over the next robust reader is within noise at n=1.Source: §E.3 · p. 24

E.4 Architecture encoding

Concatenating encodings avoids selecting one fixed representation, while the top of the predicted ranking remains less discriminative than its global rank correlation suggests.

E.4 Architecture encoding

  • 0.9023 is the twelve-cell mean for concat-all encodings, matching the per-cell leakage oracle at 0.9011 and exceeding the fixed-single encoding mean of 0.8872.No single fixed encoding is best across the suite; concat-all lets the propagator use multiple informative encodings without label-based selection.Source: §E.4 · pp. 24–25, Table 14 · p. 25

E.5 Discrimination at the top of the ranking

  • Global Spearman ranges from 0.70 to 0.98, but top-1% tier overlap ranges from 0.00 to 0.67 and precision at the top ten is low.On classification cells, the predicted best is within 0.08 to 1.94 accuracy points of the true best; top-10% overlap ranges from 0.25 to 0.85.Source: §E.5 · pp. 25–26, Table 15 · p. 26

F Selection and scaling

RiPPLE transfers to the large DARTS space and selects strong architectures under the zero-cost sampling protocol. Against multi-fidelity search, best-found accuracy is similar, while RiPPLE’s distinct contribution is a whole-space ranking.

Scaling to a large space

  • On DARTS, held-out Spearman reaches 0.775 at FTE=50 and 0.808 at FTE=100, above the strongest baseline’s 0.68 and 0.77.RiPPLE is applied without retuning to the roughly 10^18-architecture space, using NAS-Bench-301 surrogate ground truth.Source: F Selection and scaling · p. 26

Selection at equal sample budget

  • Under the N=3000 zero-cost protocol, RiPPLE selects the strongest architecture on all three NAS-Bench-201 datasets, reaching 94.00, 71.95, and 45.67 against optima of 94.37, 73.50, and 47.31.The ranker is built once at about 150 full-training equivalents and then scores each candidate for free.Source: F Selection and scaling · pp. 26–27, Table 17 · p. 27

Comparison with multi-fidelity search

  • ASHA, BOHB, and RiPPLE’s best-found architecture trade places within roughly one accuracy point, so this comparison is not a RiPPLE win.The truth sources differ by about 0.4 points per cell, and NAS-Bench-101, NLP, and TransNAS cells are omitted because their per-epoch tables are unavailable.Source: F Selection and scaling · pp. 26–27

G RiPPLE under the predictor-lane protocols

Under the predictor-lane protocol, RiPPLE-prefix exceeds the strongest baseline at the 1% budget, while its ranking saturates near the extrapolated-label readout ceiling. The NAS-Bench-101 comparison is indicative rather than like-for-like because RiPPLE’s features and test sets do not cover the full benchmark in the same way as the baselines.

Search-cost comparison

  • The search-method comparison does not establish identical actual costs or statistical equivalence: nominal caps may be exceeded after a rung, and RiPPLE’s truth source differs from search’s test accuracy.For example, SSS-C10 retains a 41-FTE search cap while RiPPLE’s 23-epoch deployment costs 42.68 FTE.Source: G RiPPLE under the predictor-lane… · p. 28

Predictor-lane protocol on NAS-Bench-201

  • At the 1% budget on NAS-Bench-201, RiPPLE-prefix reaches Kendall’s τ of 0.821 against the strongest baseline’s 0.804, using about 30 full-training equivalents versus 156 full trainings.The true-label RiPPLE-full variant is strongest at every budget column.Source: G RiPPLE under the predictor-lane… · pp. 27–28, Table 19 · p. 28
  • As anchor count grows, RiPPLE-prefix saturates near 0.856, matching the extrapolated-label readout ceiling of approximately 0.85.Adding anchors closes the propagation gap but cannot lift the prefix label past its ceiling, making the true-label variant stronger once the budget is spent.Source: G RiPPLE under the predictor-lane… · pp. 28–29

NAS-Bench-101 comparison

  • On NAS-Bench-101, the encoding-only and proxy-augmented RiPPLE rows use different restricted test sets from each other and from the baselines, so they are indicative, not like-for-like comparisons.The deployed encoder covers only 7-node cells, and the proxy bank is available only on a 9,781-architecture sample rather than the full 423,624-architecture space.Source: G RiPPLE under the predictor-lane… · p. 29, Table 20 · p. 29
Loading 2609.12418v1…