PaperlayerView exact revision on arXiv

Source-linked AI summary

QuaNAS: Uncertainty-Aware Reuse of Earlier Searches for Query-Efficient Neural Architecture Search

Brama Seta Putra Satria, Dwi Sunaryono, Misbakhul Munir Irfan Subakti, Irzal Ahmad Sabilla, Laras Suciningtyas

arXiv:2610.31251v1cs.LG

TL;DR

With short-training budgets, NAS must choose evaluations before candidate quality is known, while prior approaches often start from scratch and proxies are unreliable across tasks. QuaNAS builds target-label-free priors from pooled main effects of earlier searches and zero-cost proxies, then uses a Gaussian process that reweights them from each evaluation. On held-out datasets, every library-using method had lower mean regret than every pre-specified comparator; QuaNAS matched RGPE (mean difference 0.000, 95% CI [−0.012, 0.013]) with no negative transfer observed.

  • Problem

    With short-training budgets, NAS must choose evaluations before candidate quality is known, while prior approaches often start from scratch and proxies are unreliable across tasks.

    Source: §1 · p. 2

  • Method

    QuaNAS builds target-label-free priors from pooled main effects of earlier searches and zero-cost proxies, then uses a Gaussian process that reweights them from each evaluation.

    Source: §2.2 · p. 5

  • Results

    On held-out datasets, every library-using method had lower mean regret than every pre-specified comparator; QuaNAS matched RGPE (mean difference 0.000, 95% CI [−0.012, 0.013]) with no negative transfer observed.

    Source: §5 · p. 21, §6 · p. 23

  • Takeaways & Limitations

    Pooled main effects work better on weakly transferable tabular tasks, while per-task surrogates work better on image, time-series, and text tasks and with small or mismatched libraries.

    Source: §5 · p. 22, §6 · p. 23

    QuaNAS was not fastest on every task, and its 914 s per-search full-budget runtime exceeded random search, local search, and REA.

    Source: §4.3 · p. 15

Abstract

from arXiv · show

Neural architecture search (NAS) on a new dataset often affords only tens of short training runs, so what the search knows before its first evaluation decides its efficiency. Yet most NAS methods start from scratch, guided at best by zero-cost proxies that are unreliable across tasks. QuaNAS treats a library of earlier searches in the same search space as explicit knowledge of uncertain reliability. Main effects of design decisions learned from this library and, secondarily, zero-cost proxies become priors over candidate architectures without any label of the new dataset. A Gaussian process starts at the prior optimum and re-estimates after every evaluation how far each prior is trusted. Under a protocol frozen before any confirmation data were seen, QuaNAS was evaluated once on 12 new image, time-series, text and tabular datasets. It lowered anytime regret on the short-training objective 2.7- to 3.9-fold relative to five pre-specified NAS baselines that start from scratch (Holm-adjusted p = 0.0012 each). Its final models matched (non-inferiority margin 0.5 pp) or exceeded matched-recipe reference models on nine datasets, as did the prior optimum, selected without a single evaluation. Post-hoc comparisons with established transfer methods given the same library trace the gain to the library: without it, the same Gaussian process had over twice the regret. They also show where each transfer mechanism works best: the pooled main effects of QuaNAS for weakly transferable tabular tasks, per-task surrogates for image, time-series and text tasks and for small or mismatched libraries. Finally, the main effects make the transferred knowledge inspectable.

1 Introduction

With only a few dozen to a hundred short training runs, what NAS knows before its first evaluation can determine search efficiency. QuaNAS reuses earlier searches as inspectable, uncertain priors and evaluates this approach under a frozen protocol across held-out datasets.

1 Introduction

  • With budgets of a few dozen to a hundred short runs, evaluation order matters, while zero-cost proxies are unreliable across tasks.On 35 development pools, 19 contained a proxy with negative τ, and on 15 even the best proxy had τ < 0.3.Source: §1 · pp. 1–2, Figure 1 · p. 2

1.1 Gap Analysis and Motivation

  • Prior work leaves open whether small libraries can be checked before evaluation, made inspectable, reweighted online, and supplemented when uninformative.The literature also relies on large collections for transfer, often reuses design and evaluation benchmarks, and has limited evidence beyond vision.Source: §1.1 · p. 3

1.2 Contributions

  • QuaNAS pools ridge main-effect models of earlier searches and weights them against zero-cost proxies without labels of the new task.The resulting main effects expose which design choices carried over and how consistently.Source: §1.2 · p. 3
  • A Gaussian process starts at the prior optimum and updates the weights of prior knowledge with every new-task evaluation.A reliability switch discards a transferred prior that the library itself shows to be uninformative.Source: §1.2 · p. 3
  • On 12 held-out datasets, QuaNAS lowered anytime regret 2.7- to 3.9-fold versus five pre-specified scratch baselines and matched or exceeded reference models on 9.The paper also releases 47 candidate pools with real short-training accuracies, proxy costs, full-training results, code, and per-run results.Source: §1.2 · p. 4

2 Method

QuaNAS formulates NAS as selecting architectures from a candidate pool under a limited query budget, then reuses earlier searches and zero-cost proxies as target-label-free priors. A Gaussian process updates its trust in those priors as evaluations accumulate, and a small full-training shortlist selects the reported model.

2.1 Problem Formulation

  • The task is a candidate pool in a shared search space, where each query short-trains one architecture and returns its validation accuracy and measured cost.The search is limited to B distinct candidates; the task’s own labels are available only through its queries.Source: §2.1 · p. 4

2.2 Design Overview: QuaNAS

  • QuaNAS builds target-label-free priors, searches with a Gaussian process that updates their weights after each evaluation, then selects the reported model from a small full-training shortlist.The priors use earlier tasks in the same search space and zero-cost proxies; no new-task accuracy is used to construct them.Source: §2.2 · p. 5, §2.3 · p. 6, Figure 2 · p. 6

2.3 Target-Label-Free Priors from Earlier Searches and Zero-Cost Proxies

  • Earlier-task ridge main-effect models and screened zero-cost proxies form the architecture and proxy priors, with their weights estimated by leave-one-task-out stacking on the library.The architecture prior uses additive effects over design choices; proxies are computed only on the top 20% of candidates ranked by that prior.Source: §2.3 · pp. 6–7

2.4 Search with Uncertain Prior Weights

  • The Gaussian process treats prior weights as random coefficients, so new evaluations can update trust in each prior while a Hamming kernel captures structure and interactions.The priors depend on accuracies through average ranks, making the search invariant to strictly increasing accuracy transformations.Source: §2.4 · pp. 7–8
  • The first query selects the prior optimum, and every later query maximizes expected improvement over unqueried candidates.There is no random initial design.Source: §2.4 · p. 8

2.5 Selection of the Final Model

  • The three highest short-training-accuracy candidates among the first 50 queries are fully trained, and the one with the highest validation accuracy is selected for final evaluation.The selection step does not change the search or its regret.Source: §2.5 · p. 8

2.6 Computational Complexity

  • For B = 100 and N ≤3456, measured algorithmic overhead averaged 0.28 s per run, compared with 6.0 s for one short training.The reported complexity of the t-th query is O(t^3 + Nt^2), after precomputing the Hamming matrix.Source: §2.6 · p. 8

3 Experimental Protocol

The evaluation used earlier development searches to design QuaNAS, then tested the frozen method on 12 previously unseen confirmation datasets under a precommitted protocol. Methods shared query and proxy budgets, while training recipes and statistical tests were specified in advance.

3.1 Development and Confirmation Datasets

  • The confirmation set comprised four image, two time-series, three text, and three tabular classification tasks selected without using their accuracy.Datasets had not been loaded in earlier project cycles; substitutions followed download or quality-control outcomes alone.Source: §3.1 · p. 9

3.2 Search Spaces, Pools and Quality Control

  • Confirmation pools used samples or whole search spaces, and convolutional and text architectures were generally disjoint from development pools; each member had proxies and low-fidelity validation measurements.Pool checks required accuracy spread above 0.5 pp and rank stability of at least rs = 0.5, with one rebuild and fallback if needed.Source: §3.2 · p. 10

3.3 Leakage Prevention and Frozen Protocol

  • The frozen protocol tested QuaNAS once on 12 new datasets after committing its configuration, seeds, endpoints, tests, and criteria; confirmation accuracy was unavailable before the tag.Test splits were locked until all selections were complete and read once for final evaluation.Source: §3.3 · p. 10, Figure 3 · p. 11

3.4 Compared Methods

  • All methods received 100 distinct queries, up to 2,000 proxy evaluations, 30 seeds, and common random numbers for initial designs; five pre-specified methods were primary comparators.Four post-hoc transfer comparators received QuaNAS’s library and Gaussian process, while differing in how they used the library.Source: §3.4 · pp. 10–11

3.5 Training, Reference Models and Hardware

  • Modality-specific training recipes were fixed before pool construction, and final models were compared with matched-recipe references trained with 10 seeds.The reference models were not state-of-the-art; the full-training plan used 454 runs and 19.3 GPU-hours, below the 40-hour cap.Source: §3.5 · pp. 11–12

3.6 Metrics and Statistical Analysis

  • Search methods were compared over 12 task means, with Holm-adjusted tests for the five primary comparators; final models used one-sided Welch tests with a 0.5 pp non-inferiority margin.The task, not each seed, was the unit of replication because QuaNAS was deterministic up to tie-breaking.Source: §3.6 · p. 12

4 Results

QuaNAS achieved lower anytime regret than the five pre-specified scratch baselines across the 12 held-out tasks, with the advantage visible early in the search. Evaluation checks also found weak proxy correlations on several tasks and unreliable short-training rankings among top candidates.

4.1 Evaluation Integrity and Dataset Characteristics

  • The best single proxy’s Kendall correlation with short-training accuracy ranged from 0.06 to 0.55, and on five tasks even the best proxy reached at most 0.30.Transferred architecture-prior weights were positive across all three spaces, while proxy-prior weights were small.Source: §4.1 · p. 12

4.2 Search Efficiency on the Held-Out Datasets

  • 0.058 mean anytime regret for QuaNAS versus 0.155–0.226 for the five primary comparators, a 2.7- to 3.9-fold reduction; Holm-adjusted p = 0.0012 against each.QuaNAS ranked first on 11 of 12 tasks, with mean rank 1.08 versus 6.33 for the next method, QIEA-ZC.Source: §4.2 · p. 13
  • After 25 queries, QuaNAS had mean normalized regret 0.054 versus 0.182–0.250 for the primary comparators and had found the best pool member on seven tasks.After 100 queries, its regret remained lowest at 0.022 versus 0.048–0.137; its confirmation-minus-development regret gap was −0.016.Source: §4.2 · p. 13

4.3 Search Cost and Time to a Near-Optimal Architecture

QuaNAS reached a near-optimal low-fidelity result with less average compute than the primary baselines, though it was not the cheapest method when all 100 queries were used. Comparisons using the same library show that the library contributes substantially, while transfer methods differ by task type and trade-offs.

4.3 Search Cost and Time to a Near-Optimal Architecture

  • 197 s average to reach normalized regret 0.05 for QuaNAS versus 589–1068 s for the primary comparators, or 3.0–5.4 times faster on this low-fidelity objective.Within 100 queries, 88.6% of QuaNAS runs reached the target versus 52.8–69.4% for primary comparators; two CPU-loaded pools lengthened some recorded times.Source: §4.3 · pp. 13–14, Table 5 · p. 14
  • Over the full budget of 100 queries, QuaNAS spent 914 s per search—more than random search (602 s), local search (660 s), and REA (695 s), but less than QIEA-ZC (981 s), RoBoT-style (1255 s), and PTK-MR (1356 s).QuaNAS was not fastest on every task: local search, REA, or RoBoT-style led on Chess KRK, Grid Stability, and FetalECG.Source: §4.3 · p. 15
  • Over 100 queries, QuaNAS averaged 914 s per search, more than random search, local search, and REA; its payoff is when a good architecture is needed early.Proxy evaluations averaged 294 per search, while the Gaussian process itself took 0.28 s.Source: §4.3 · p. 15

4.4 Comparison with Transfer-Based Bayesian Optimization

  • Without transfer, the same Gaussian process had mean AUC 0.130, more than twice QuaNAS’s regret; nearest-task ranking without feedback reached 0.108.The no-transfer comparison favored QuaNAS on 11 tasks and tied on one, with Holm-adjusted p = 0.0020.Source: §4.4 · p. 15
  • QuaNAS and RGPE matched in mean regret at 0.058, but RGPE had lower median regret; RGPE did better on image, time-series, and text pools, while QuaNAS did better on tabular pools.RGPE’s per-task Gaussian processes can express decision interactions that QuaNAS’s additive main-effect prior cannot.Source: §4.4 · p. 15
  • QuaNAS had lower CVaR0.2 than RGPE on held-out tasks, 0.151 versus 0.174, but the 12-task difference was unresolved; RGPE reached the near-optimal target sooner on average, 146 versus 197 s.No negative transfer was observed for QuaNAS on held-out datasets; the count was one each for RGPE and WS-GP and two for NT.Source: §4.4 · p. 15

4.5 Which Components Matter?

The prior-driven start, Hamming kernel, and transferred architecture prior were necessary for QuaNAS’s gains on the confirmation tasks. Online re-weighting and a reliability switch instead guarded against rare failures, while tested defaults outperformed alternative settings overall.

4.5 Which Components Matter?

  • Replacing the prior-driven start with ten random queries raised regret on all 12 confirmation tasks by 0.050; removing the Hamming kernel raised it by 0.051, and removing the architecture prior by 0.041.The respective Holm-adjusted p-values were 0.0015, 0.0049, and 0.019; losses reached 0.18 on Imagenette-32.Source: §4.5 · p. 16
  • Removing the proxy prior raised mean confirmation regret by 0.021 (p = 0.13), while fixing meta-learned weights left it essentially unchanged at −0.002.Fixing weights raised development regret by up to 0.27 on Adult, showing the value of online re-weighting when some weights are wrong.Source: §4.5 · p. 16
  • On development pools, fixing meta-learned weights increased regret by up to 0.27 on Adult, while the reliability switch prevented losses of up to 0.12 on three pools.The switch was inactive on confirmation datasets, so its observed benefit was limited to development pools.Source: §4.5 · p. 16
  • The transferred priors and GP search introduced in D3 brought the main development gain; D3’s mean confirmation AUC was 0.073, with 3 wins, 7 ties, and 2 losses.The final version’s confirmation mean was 0.058 versus 0.056 for fixed weights, a non-significant difference; no removal improved QuaNAS significantly (all Holm p = 1).Source: §4.5 · pp. 16–17
  • Expected improvement yielded lower confirmation regret than greedy posterior-mean acquisition, 0.077 (p = 0.018), and tested screening and kernel/noise alternatives raised mean regret.Across tested alternatives, confirmation regret rose to 0.065–0.086 and development regret to 0.080–0.091; defaults were fixed during development.Source: §4.5 · p. 17

4.6 Robustness: Library Composition, Manipulated Libraries and a Full Search Space

QuaNAS retained lower regret than primary comparators under reduced libraries and on a full NAS-Bench-201 search, but transfer quality depended on library composition and task structure. Per-task surrogates performed better than pooled main effects when benchmark tasks shared candidates and rankings.

4.6 Robustness: Library Composition, Manipulated Libraries and a Full Search Space

  • With leave-family-out and stricter reduced libraries, QuaNAS still had lower regret than each primary comparator (Holm p ≤ 0.019 and p ≤ 0.0024), though mean AUC rose to 0.081 and 0.078.Image, time-series, and text tasks were almost unaffected; small tabular libraries lost most, including Credit Default (0.357 versus 0.216) and Chess KRK (0.168 versus 0.090).Source: §4.6 · p. 17
  • With scrambled library accuracies, QuaNAS had higher mean AUC than RGPE (0.120 versus 0.164), but also did worse with one or two searches or a library from another data type.The leave-one-task-out check detected scrambled libraries before evaluation, but consistent libraries from another data type passed it.Source: §4.6 · p. 17
  • On NAS-Bench-201, QuaNAS lowered mean AUC 2.1- to 5.8-fold versus methods without transfer (0.049 versus 0.105–0.287), but transfer comparators achieved 0.002–0.010.The three datasets share 15,625 candidates and nearly identical rankings (Kendall τ = 0.81–0.85); per-task surrogates can exploit this, whereas additive main effects cannot represent edge-operation interactions.Source: §4.6 · p. 17, Table 8 · p. 18

4.7 Final Models: Accuracy, Size and Latency

QuaNAS’s final models matched or exceeded matched-recipe references on 9 of 12 datasets, and its prior optimum did so without task evaluations. The main effects reveal which architecture choices transfer, while light-model benefits were concentrated on image and time-series tasks.

4.7 Final Models: Accuracy, Size and Latency

  • QuaNAS was superior to matched-recipe references on 7 datasets and non-inferior within 0.5 pp on 2 more, meeting the frozen criterion on 9 of 12.It was more accurate than the strongest matched baseline on 7 datasets and within 0.5 pp on 9; on 8 of those 9, it was also smaller.Source: §4.7 · p. 18
  • Against references, QuaNAS gained +2.47 pp on Chess KRK, +1.54 pp on FetalECG, and +1.51 pp on Imagenette-32; image and time-series models were 3–29% of reference size.Those models ran up to 4.7 times faster on one CPU thread; Credit Default was more accurate but larger than CatBoost.Source: §4.7 · p. 18, Figure 6 · p. 19

4.8 Knowledge Transfer without Queries and Light Models

  • Without any task evaluation, the transferred prior optimum was non-inferior to references on 9 of 12 datasets and not worse than RoBoT-style transfer on 11.It was more accurate than RoBoT-style transfer on 8 datasets, with mean difference to reference of −0.70 versus −0.99 pp.Source: Table 10 · p. 19, §4.8 · p. 19
  • Library main effects favored deeper convolutional stages and selected text-space choices, with effect signs shared by 80% of tasks in those spaces versus 65% in tabular.In tabular tasks, large effects such as numeric embedding changed sign; the main effects show both transferred choices and where transfer is weak.Source: §4.8 · pp. 19–20, Figure 7 · p. 20
  • Under parameter caps, QuaNAS remained the most efficient search (AUC 0.068 versus 0.134–0.185) and met all light-model conditions on 5 datasets, all image or time-series.Those models were 3–18% of reference size and 1.7–3.7 times lower latency; the criterion did not hold where fastText or CatBoost references were already small or fast.Source: §4.8 · p. 20

4.9 Error Analysis and Failure Cases

Error analyses identify low-fidelity ranking as a shared constraint for text tasks, validation/test shift on OCTMNIST, and weak proxy signals on flat tabular landscapes. The frozen-protocol assessment supported four hypotheses, partly supported two, and met the modality-detector check.

4.9 Error Analysis and Failure Cases

  • On Yahoo Answers and 20 Newsgroups, QuaNAS’s final models were 3.4 and 4.2 pp less accurate than fastText, although QuaNAS had the lowest regret on both tasks.After short training, fastText-family architectures ranked at best 166th and 282nd of 600, behind recurrent and convolutional encoders; fully trained fastText models were most accurate.Source: §4.9 · p. 20

4.10 Summary of the Frozen Hypotheses

  • Of seven hypotheses fixed before confirmation, four were supported, two partly supported, and the modality-detector operational check was met.The hypotheses were evaluated once on the 12 confirmation datasets.Source: Table 11 · p. 21, §4.10 · p. 21

4.9 Error Analysis and Failure Cases

  • On OCTMNIST, ResNet-18 scored 94.2% on validation but 78.5% on test, and QuaNAS was 0.89 pp below the reference.Every method selected by validation accuracy, which cannot correct the marked validation/test shift.Source: §4.9 · p. 21
  • Credit Default and Grid Stability had AUCs of 0.216 and 0.146, among QuaNAS’s hardest tasks, with uninformative proxies and low stability coefficients.QuaNAS still had the lowest regret among primary comparators; SynFlow alone did better on Credit Default (0.194 versus 0.216), despite being worst overall by mean AUC (0.355).Source: §4.9 · p. 21

5 Discussion

The discussion attributes small-budget NAS efficiency chiefly to earlier searches, while distinguishing pooled main effects from per-task surrogates and identifying limits in transfer and evidence scope.

5 Discussion

  • Every method using the library had lower mean regret than every pre-specified comparator, while the same GP without the library had over twice QuaNAS’s mean regret.The prior optimum alone matched or exceeded reference models on 9 of 12 datasets.Source: §5 · p. 21
  • Zero-cost proxies contributed little on average, with weak, task-dependent correlations and computation costs before the first query.Source: §5 · pp. 21–22
  • Per-task surrogates suit close matches and small libraries, whereas pooled main effects degrade more gracefully on weakly transferable tabular tasks and expose which design options transfer.On tabular tasks, QuaNAS was better on 9 of 13 pools.Source: §5 · p. 22
  • With an existing library, transfer-based search reached a near-optimal architecture after 2.4 to 3.3 minutes on average, versus 10 to 18 minutes for proxy-based and evolutionary comparators.On image and time-series data, final QuaNAS models were smaller than standard references and, except on FetalECG, faster on a CPU; fastText was better for latency-critical text applications.Source: §5 · p. 22
  • Evidence is bounded by fixed candidate pools and a short-training objective, with full training limited to shortlists and selected architectures; 12 confirmation datasets leave limited power for small task-level differences.The protocol was frozen in private version control, and transfer across search spaces was outside the study’s scope.Source: §5 · pp. 22–23

6 Conclusion

QuaNAS reuses earlier searches as target-label-free priors and updates their trust during search. It improved regret and reference-model outcomes on held-out datasets; the paper also makes its materials publicly available.

6 Conclusion

  • QuaNAS lowered anytime regret 2.7- to 3.9-fold against five pre-specified baselines and matched or exceeded matched-recipe references on nine of 12 new datasets.Even its prior alone selected architectures matching or exceeding references on nine datasets.Source: §6 · p. 23
  • Pooled main effects transferred better on weakly transferable tabular tasks, while per-task surrogates did better on image, time-series, and text tasks, small or mismatched libraries, and NAS-Bench-201.The conclusions are scoped to candidate pools within shared search spaces and budgets of about one hundred short trainings.Source: §6 · p. 23

Data and Code Availability

  • The paper archives code, configurations, frozen-protocol documents, candidate-pool data, run results, and scripts for its tables and figures on Zenodo.Source: Data and Code Availability · p. 23

A Library Stress Tests

Library stress tests compare QuaNAS and RGPE under small, mismatched, and uninformative libraries. QuaNAS’s reliability switch helps when the library contains no information, but cannot detect a coherent library from another data type.

A Library Stress Tests

  • AUC is mean normalized anytime regret over 100 queries, with lower values better; negative transfer means library use raises AUC by more than 0.01 over GP-BO.Source: A Library Stress Tests · p. 29
  • With one earlier search, QuaNAS’s leave-one-task-out check could not compare tasks, so it fell back to proxy-guided search and had mean AUC 0.103 versus RGPE’s 0.080.With two earlier searches it used the prior in most runs but remained behind: 0.073 versus 0.055 (two-sided p = 0.005).Source: A Library Stress Tests · p. 29
  • For a library from the other data type, QuaNAS’s switch did not fire and its mean AUC was 0.108, with negative transfer on 6 of 21 pools; RGPE had mean AUC 0.061 and one such pool.The cross-type library remained consistent within itself, so the leave-one-task-out check did not identify the mismatch.Source: A Library Stress Tests · p. 29
  • When library accuracies were randomly permuted, QuaNAS’s switch fired on all 21 pools and it had mean AUC 0.120 with five negative-transfer pools, versus RGPE’s 0.164 with 12 pools.RGPE could not recognize an uninformative library before evaluating the new task.Source: A Library Stress Tests · p. 29

B Combinations of the Two Transfer Mechanisms

The paper tested combinations of per-task surrogates and QuaNAS’s pooled prior, but none met the pre-set dominance rule. Their different task-level strengths remained complementary rather than combining into one superior method.

B Combinations of the Two Transfer Mechanisms

Loading 2610.31251v1…