PaperlayerView exact revision on arXiv

Source-linked AI summary

Searching with Sparse Histories: Cross-Task Architecture Search from Incomplete Search Logs and Zero-Cost Proxies

Yuefeng Xu, Jun Yu

arXiv:2610.32179v1cs.LGcs.AIcs.CVcs.NE

TL;DR

Cross-task NAS can reuse prior search knowledge, but task-specific rankings and sparse histories complicate transfer. SHINE combines sparse source logs with zero-cost proxies through champion-based initialization and joint pairwise-ranker pretraining. Across pooled targets, it improves on a proxy-only ranker in mean and worst-case task-level performance while mitigating severe failures.

  • Problem

    Cross-task NAS can reuse prior search knowledge, but architecture rankings shift across tasks and existing approaches often assume dense source tables rather than sparse, weakly overlapping histories.

    Source: §1 · p. 1

  • Method

    SHINE initializes search with a portfolio of imputed source champions and jointly pretrains a pairwise ranker on sparse source tasks and zero-cost proxies treated as pseudo-tasks.

    Source: §1 · p. 2, §4.2 · p. 3

  • Results

    Across 13 pooled targets, SHINE significantly improves on the proxy-only ranker in both mean and worst-case task-level performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SHINE is a risk-robust strategy that mitigates severe failures of either knowledge source while retaining competitive average performance.

    Source: §6 · p. 8

    SHINE assumes past tasks share the new task's search space; transfer across different search spaces remains future work.

    Source: §6 · p. 8

Abstract

from arXiv · show

Neural architecture search (NAS) is expensive, but cross-task transfer can reduce costs by reusing knowledge from previous tasks. However, existing methods struggle with sparse search histories and task-specific transfer failures. We propose a method, a risk-robust strategy that combines sparse source logs with zero-cost proxies through champion-based initialization and joint pairwise-ranker pretraining. On TransNAS-Bench-101, the proposal mitigates severe worst-case failures, reducing worst-10% ranks from 290 to 43 for Micro autoencoding and from 465 to 45 for Macro room layout. Across 13 cell-space targets, it outperforms a proxy-only ranker on 10 targets in both mean and worst-case performance, while maintaining competitive average performance.

1. Introduction

Cross-task NAS can reuse knowledge from earlier searches, but architecture rankings shift by task and practitioners often have only sparse, weakly overlapping logs. SHINE combines sparse source histories with zero-cost proxies to mitigate transfer failures while retaining competitive typical performance.

1. Introduction

  • Sparse histories leave existing cross-task methods without their assumed dense source tables, despite practitioners often having only a few dozen to a few hundred evaluations per task.The histories may have little overlap across tasks.Source: §1 · p. 1
  • When sources contain only m=50 evaluations, champion coverage loses its dense-table advantage, while zero-cost proxies are unreliable on dense-prediction tasks.These weaknesses motivate combining the two sources of knowledge.Source: §1 · pp. 1–2
  • SHINE jointly pretrains a pairwise ranker on sparse source tasks and zero-cost proxies, then initializes search with an imputed source-champion portfolio.It requires neither dense source tables nor task embeddings, and its overhead is negligible relative to one architecture evaluation.Source: §1 · p. 2
  • Across 13 targets, SHINE significantly improves on a proxy-only ranker at the task level in both mean and worst-case performance, while typical performance remains on par with the better single source.The pooled analysis includes seven targets trained from scratch on new datasets.Source: §1 · p. 2

2. Related Work

Prior work ranks candidates with predictors, transfers knowledge across tasks, uses zero-cost proxies, or warm-starts search from previous tasks. SHINE combines sparse source histories with proxies and adapts a portfolio-based initialization to sparse sources.

Predictor-based NAS

  • Predictor-based NAS ranks candidates before evaluation; pairwise and listwise ranking losses are preferred to regression because search needs architecture order, not predicted values.BANANAS uses path-based encodings, while weakNAS iteratively fits weak predictors and narrows the sampling region.Source: §2 · p. 2

Cross-task and transferable NAS

  • Cross-task methods include task-context meta-learning, architecture generation from dataset embeddings, simultaneous task knowledge exchange, and predictor transfer across tasks or devices.Arch-Graph uses one source task and target fine-tuning; TRP pretrains on zero-cost proxy votes.Source: §2 · p. 2

Zero-cost proxies

  • Zero-cost proxies score untrained networks from one minibatch, but their reliability varies across tasks and search spaces; SHINE uses them as auxiliary pretraining tasks rather than as a ranking.The paper confirms proxy unreliability on dense-prediction tasks.Source: §2 · p. 2

Warm-starting from previous tasks

  • Warm-starting reuses strong configurations from prior datasets, often through a greedy portfolio; SHINE extends this portfolio idea to sparse sources by imputation.Source: §2 · p. 2

3. Cross-Task NAS from Sparse Search Histories

The setting combines sparse evaluations from several source tasks with cheap proxy scores, then searches a new target under a fixed evaluation budget. Dense-table heuristics deteriorate when histories are sparse.

3. Cross-Task NAS from Sparse Search Histories

  • Each of S source tasks provides observed quality values for only m much smaller than N architectures, while K zero-cost proxies score every architecture; the target permits B sequential evaluations.The target's quality is otherwise unavailable.Source: §3 · pp. 2–3
  • Performance is measured by the returned architecture's global rank, averaged over target tasks and seeds, with each target held out in turn as the others serve as sources.A second rank measure uses the global rank of the returned architecture's average metric value for comparison with published tables.Source: §3 · p. 3

3.1. Why naive transfer fails with sparse histories

  • With dense source tables, the champion portfolio reaches average ranks of 11.0 on Macro and 15.2 on Micro after 50 target evaluations, while consensus reaches 90.5 and 44.2.Top architectures are task-specific, so averaging source rankings hides them.Source: §3.1 · p. 3
  • With only m=50 random evaluations per source, the portfolio degrades to average ranks of 24.3 on Macro and 44.3 on Micro, and cross-source disagreement selection collapses to 122.6 and 516.2.The results motivate generalization beyond observed source architectures and warn against relying on sources alone.Source: §3.1 · p. 3

4. Method

SHINE initializes search from imputed source champions, jointly pretrains a pairwise ranker on sparse sources and proxy pseudo-tasks, and adapts the ranker during target search. Its target-side loop narrows candidate sampling before a final exploitation phase.

4. Method

  • SHINE has three components: an imputed champion portfolio, joint pairwise-ranker pretraining on sparse sources and zero-cost pseudo-tasks, and target-side ranker adaptation during search.Source: §4 · p. 3, Figure 1 · p. 3

4.1. Champion portfolio from imputed sources

  • For each source, a ridge regressor with pairwise feature interactions imputes a complete ranking while preserving the observed architecture order; the initial portfolio greedily minimizes summed normalized regret.Its first S selections visit each source's imputed champion, then refine source-specific top regions rather than relying on a consensus prior.Source: §4.1 · p. 3

4.2. Multi-task pairwise ranker pretraining

  • A shared encoder and task-specific heads are pretrained jointly on sparse source labels and proxy rankings, treating each proxy as a pseudo-task.Proxy tasks provide dense but noisy signals, while source tasks provide accurate but incomplete rankings; pairwise ranking avoids calibrating metric scales.Source: §4.2 · p. 3

4.3. Target adaptation and search

  • Target search repeatedly fine-tunes the ranker and samples batches from a geometrically shrinking predicted top-k region, then uses the final F evaluations for exploitation.The regularizer keeps the ranker close to transferable structure when target labels are few; the full search takes about one GPU-minute.Source: §4.3 · pp. 3–4

5. Experiments

Experiments compare sparse-history transfer with proxy-only and other NAS baselines across TransNAS-Bench-101, MedMNIST-NAS-Bench, and real-training targets. SHINE combines sparse source logs and zero-cost proxies, with its strongest overall evidence in reduced worst-case failures and wins over proxy-only ranking across 13 Micro targets.

5.1. Setup

  • At B=50 evaluations and m=50 revealed architectures per source task, most results average 20 seeds (10 for source-density and budget sweeps), comparing published NAS methods with source-only, proxy-only, tree-based, and ensemble variants.Source: §5.1 · p. 4

5.2. Main results

  • SHINE reduces proxy-only worst-10% mean rank from 34.8 to 20.9 on Macro and from 152.1 to 88.9 on Micro, although typical-case results are often similar.On Micro, the single worst run ranks 545th of 4,096 for proxy-only versus 185th for SHINE.Source: §5.2 · p. 5, Table 1 · p. 5

5.3. Complementarity of histories and proxies

  • On Macro, proxies perform well on coarse skeleton choices, while sparse histories correct proxy failures on Micro autoencoding and scene classification; combining them performs best on Macro room layout.The respective reported mean ranks are 43.2 and 20.2 for proxy-only on Micro autoencoding and scene classification, and 74.4 versus 15.5 for sources-only versus SHINE on Macro room layout.Source: §5.3 · pp. 5–6, Table 2 · p. 6

5.4. Ablations

  • Macro ablations consistently favor retaining ranker pretraining and the portfolio, but task-level differences are not significant after Holm correction; Micro effects vary by target.No variant has lighter tails than SHINE on both spaces, and removing either knowledge source sharply worsens worst-case runs.Source: §5.4 · p. 6, Table 3 · p. 6

5.5. Target budget and source density

  • At B=25, SHINE has better Micro six-target mean rank than proxy-only (39.9 vs. 57.3), while proxy-only is better on Macro (12.6 vs. 23.2).SHINE is robust to source density m=20–200; its best Micro result is 14.7 at m=200, with no consistent Macro trend.Source: §5.5 · p. 6, Figure 2 · p. 7

5.6. Independent validation on MedMNIST-NAS-Bench

  • On MedMNIST Tiny, m=50 sparse-source pretraining improves over target-only ranking on three of four tasks; on Small, m=200 histories reduce test regret a further 24%.MedMNIST-NAS-Bench was not used during development and has no zero-cost proxies.Source: §5.6 · pp. 6–7, Table 4 · p. 7

5.7. Validation with real training and pooled analysis

  • On 13 Micro targets, SHINE beats proxy-only ranking on 10/13 in both mean and worst-case performance, while retaining competitive average performance.The pooled task-level Holm-corrected p-values are 0.012 and 0.016 for the two comparisons.Source: Table 5 · p. 7, §5.7 · p. 8
  • On four real-training targets run after the analysis plan was fixed, SHINE beats proxy-only in mean and worst-case performance on all four, while histories alone have lighter tails.The broader real-training and TransNAS pooled analysis also finds SHINE better than proxy-only on seven real-training targets alone (6/7).Source: §5.7 · pp. 7–8, Table 5 · p. 7

6. Limitations and Conclusion

SHINE combines sparse histories and zero-cost proxies as a risk-robust transfer strategy, mitigating severe failures from either source while retaining competitive average performance. The evidence is bounded by the evaluated benchmarks and by a same-search-space assumption.

Limitations

  • Experiments cover few, partly dependent targets, and real-training validation uses pools of 300 architectures, short schedules, one seed, and 64×64 inputs.Hyperparameters were developed on TransNAS-Bench-101, SHINE trails reported TRP results on Micro, and its real-training Macro pools do not discriminate between methods.Source: §6 · p. 8

Limitations and Conclusion

  • SHINE assumes source and target tasks share a search space; transfer across different spaces remains future work.The conclusion identifies adaptive control of how much to trust each knowledge source as a promising direction.Source: §6 · p. 8

Conclusion

  • SHINE substantially mitigates the severe failures of either knowledge source, reduces tail risk relative to proxies, and retains competitive average performance.Its advantage over proxies alone is small on six TransNAS targets but significant across 13 cell-space targets; histories alone are on par with it on cell spaces but fail severely on Macro.Source: §6 · p. 8

A. Implementation details

The implementation uses distinct architecture encodings for Micro and Macro, combines proxy scores with imputed sparse-source rankings, and keeps search compute modest.

Encodings

  • Micro uses one-hot edge operations and path features, while Macro uses per-layer types, down-sampling and widening indicators, cumulative features, and depth.Both feature sets are standardised.Source: A. Implementation details · p. 10

Zero-cost proxies

  • Twelve NAS-Bench-Suite-Zero proxies are computed on object classification and rank-normalised to [0, 1].The proxies are fisher, flops, grad-norm, grasp, jacov, l2-norm, nwot, params, plain, snip, synflow, and zen.Source: A. Implementation details · p. 10

Imputation

  • Each sparse source is imputed with ridge regression (α=10) on first-order encoding features and all pairwise products, fitted to observed architectures’ percentile ranks.Source: A. Implementation details · p. 10

Compute

  • A full search with B=50 takes about 77 seconds on an RTX 5090 shared by 12 processes; the ZC-only ranker takes 54 seconds and the tree variant under one second.Source: A. Implementation details · p. 10

B. Exploratory: an online gate between pretrained and target-only rankers

The exploratory online gate did not reliably distinguish pretrained from target-only rankers under the small evaluation budget. A separate comparison found no significant difference between trajectory and uniform source logs.

B. Exploratory: an online gate between pretrained and target-only rankers

  • The gate’s predeclared comparison criterion was within 1 rank, or 0.01 regret, of the better single model.Source: Table 6 · p. 10
  • With 50 evaluations, the exploratory gate missed its criterion on both TransNAS spaces and MedMNIST Tiny, while its average weights remained between 0.4 and 0.6 across targets.The fixed ensemble had the best metric-matched TransNAS numbers (2.00 / 8.00), but was not adopted because it was not a pre-declared candidate.Source: B. Exploratory: an online gate between… · p. 10

C. Search-trajectory source logs

  • Trajectory logs comprise 50 architectures evaluated by an Extra-Trees weak-predictor search per source; the accompanying tables report mean-rank and worst-10% comparisons, including pooled CVaR.The pooled CVaR uses a 95% hierarchical bootstrap interval over targets and then seeds.Source: Table 7 · p. 10, Table 8 · p. 11
  • SHINE with trajectory logs did not differ significantly from SHINE with uniform logs or from the sources-only ranker with trajectory logs in paired target-seed tests.The reported p-values were 0.56 and 0.40 for the first comparison, and 0.81 and 0.30 for the second, on Macro and Micro respectively.Source: C. Search-trajectory source logs · p. 10

E. Per-target real-training results

The table reports real-training per-target test ranks for architectures selected from pools of 300.

E. Per-target real-training results

  • The real-training table reports mean test rank over 30 seeds, with worst-10% ranks in parentheses, across classification, reconstruction, denoising, inpainting, and segmentation metrics.The caption does not provide per-target result values.Source: Table 9 · p. 11

F. Sensitivity of the pooled analysis

This subsection reports a sensitivity analysis of the pooled comparison, which is not part of the fixed plan.

F. Sensitivity of the pooled analysis

  • The table reports how many targets SHINE beats and uncorrected task-level Wilcoxon p-values for mean rank and CVaR10 (% of pool).Source: Table 10 · p. 11

Related papers

Loading 2610.32179v1…