PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, Mi Zhang

arXiv:2006.06936v2cs.CVcs.LG

TL;DR

NAS faces discrete encodings that scale poorly and supervised continuous representations that jointly learn from search-selected accuracies. This paper pre-trains architecture representations without accuracy labels, then applies downstream search strategies. It reports consistent advantages over discrete and supervised alternatives across three search spaces, with smoother latent-space performance structure.

  • Problem

    Discrete NAS encodings scale poorly, while supervised continuous methods jointly learn representations and search using selected architectures’ accuracies; architecture representations in NAS remain poorly understood.

    Source: §1 · p. 1

  • Method

    arch2vec pre-trains a variational graph isomorphism autoencoder on neural architectures without accuracy labels, then uses the embeddings for downstream search.

    Source: §1 · p. 2

  • Results

    With the same downstream strategy, arch2vec consistently outperforms discrete encodings and supervised representation learning across NAS-Bench-101, NASBench-201, and DARTS.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that unsupervised architecture representations produce a more smoothly changing performance surface and benefit different downstream search strategies in three NAS search spaces.

    Source: §5 · p. 9

    The training objective assumes adjacency and operation matrices are conditionally independent given the latent variable.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.

1 Introduction

The introduction frames a trade-off between discrete NAS encodings that scale poorly and supervised continuous representations that can bias search. It proposes arch2vec, which learns architecture representations without accuracy labels, and reports benefits across search spaces and strategies.

1 Introduction

  • Unsupervised representation learning is motivated as learning data representations independent of a specific downstream task, a pattern studied across several domains.The introduction applies this idea to NAS search spaces and downstream search strategies.Source: §1 · p. 1
  • Discrete adjacency-matrix encodings grow quadratically with search-space size, while supervised continuous encoders can favor early-search weight-free operations and lower final accuracy.The supervised methods jointly optimize representations and search using selected architectures’ accuracies.Source: §1 · p. 1
  • arch2vec decouples representation learning from search by pre-training a variational graph isomorphism autoencoder on architectures without their accuracies.The learned embeddings capture local structure and cluster architectures with similar structure more smoothly in latent space.Source: §1 · p. 2
  • The learned representations preserve topology and operation similarities, helping architectures with similar accuracy cluster together.The paper says this latent-space organization facilitates downstream architecture search.Source: §1 · p. 2
  • With matched downstream strategies, arch2vec outperforms discrete encodings and supervised representation learning across NAS-Bench-101, NASBench-201, and DARTS.The reported contributions also describe downstream search as more efficient and robust across three search spaces, two strategies, and two datasets.Source: §1 · p. 2

2 Related Work

Related work covers graph representation learning, regularized autoencoders, and neural architecture search. The paper distinguishes its approach by encoding neural architectures into continuous embeddings with an architecture-reconstruction objective.

Unsupervised Representation Learning of Graphs

  • Graph representation methods use random-walk statistics, matrix factorization, edge reconstruction, or mutual-information objectives; this work instead uses GINs to encode architecture structure.The cited work also studies graph expressiveness and asynchronous message passing for DAGs.Source: §2 · p. 3

Regularized Autoencoders

  • The paper uses a variational autoencoder with KL-divergence regularization to map finite discrete architectures into unique continuous latent regions.Its stated goals are to group architectures with similar performance and make latent-space transitions relatively smooth.Source: §2 · p. 3

Neural Architecture Search (NAS)

  • Prior NAS methods move from discrete encodings toward continuous encoders, but commonly jointly optimize representations and search direction using accuracy supervision.The paper instead uses architecture reconstruction to preserve local structural relationships in the latent space.Source: §2 · p. 3

3 arch2vec

arch2vec represents cell-based neural architectures as continuous embeddings using a variational graph isomorphism autoencoder. Its reconstruction-based training is followed by reinforcement-learning and Bayesian-optimization search strategies.

3 arch2vec

  • The arch2vec section describes the representation-learning method followed by two downstream architecture-search strategies.Those strategies are reinforcement learning and Bayesian optimization.Source: §3 · p. 3

3.1.1 Preliminaries

  • Each cell is a labeled DAG with operation-labeled nodes, represented by an adjacency matrix and a one-hot operation matrix; these encodings are non-unique under node permutations.The permutation equivalence is graph isomorphism.Source: §3.1.1 · p. 3

3.1.2 Encoder

  • The encoder symmetrizes adjacency and uses GIN layers to produce node embeddings, then derives posterior mean and variance and sums node vectors into an architecture representation.The GIN supports representations invariant to isomorphic graphs; summing node vectors yields the architecture embedding.Source: §3.1.2 · p. 4

3.1.3 Decoder

  • The decoder reconstructs adjacency and operation matrices from latent variables, using sigmoid edge predictions and row-wise softmax operation predictions.The selected node operation is represented by an operation index from the predefined set.Source: §3.1.3 · p. 4

3.2 Training Objective

  • Training maximizes a variational lower bound combining reconstruction likelihood with KL-divergence regularization.The objective assumes adjacency and operation matrices are conditionally independent given the latent variable; it uses a Gaussian prior and reparameterization.Source: §3.2 · p. 4

3.3 Architecture Search Strategies

  • The downstream evaluation uses reinforcement learning and Bayesian optimization as representative search algorithms.Source: §3.3 · p. 4

3.3.1 Reinforcement Learning (RL)

  • In RL, a Policy LSTM samples actions from pre-trained embeddings and retrieves the next valid architecture by nearest-neighbor distance, using accuracy as reward.The search uses REINFORCE and L2 distance for retrieval.Source: §3.3.1 · p. 5

3.3.2 Bayesian Optimization (BO)

  • In BO, DNGO with expected improvement selects the top five architectures each round, adds them to the pool, and retrains the network.The process continues until the maximum estimated wall-clock time is reached.Source: §3.3.2 · p. 5

4 Experimental Results

The experiments assess representation quality and downstream NAS performance across three search spaces, comparing arch2vec with supervised and discrete encodings.

4 Experimental Results

  • The evaluation covers NAS-Bench-101, NASBench-201, and DARTS; arch2vec embeddings are pre-trained by reconstructing architecture inputs before downstream search.The NAS-Bench-101 set has 423,624 architectures, while the DARTS setting uses 600,000 sampled architectures.Source: §4 · pp. 5–6

4.1 Pre-training Performance

  • Arch2vec outperforms GAE and VGAE in reconstruction accuracy, validity, and uniqueness across all three search spaces.The authors attribute reconstruction accuracy to GIN neighbor aggregation and generative performance to KL-term regularization.Source: §4.1 · p. 6, Table 1 · p. 6
  • Arch2vec’s latent-space L2 distance grows monotonically with architecture edit distance, whereas supervised representations do not capture this closeness well.The comparison measures architecture pairs sampled along a long random walk.Source: §4.1 · p. 6, Figure 3 · p. 7
  • Arch2vec clusters architectures with similar accuracies and decodes neighboring architectures with a 2.9 average edit distance, versus 8.9 for supervised representations.The visualization shows a smoother accuracy surface for arch2vec; the decoded sequences begin from the same architecture.Source: §4.1 · pp. 6–7, Figure 4 · p. 7, Figure 5 · p. 7
  • For NAS-Bench-101, arch2vec’s RMSE is 0.018±0.001 and Pearson’s r is 0.67±0.02, versus 0.038±0.025 and 0.53±0.09 for supervised representations.The Gaussian Process evaluation uses 250 sampled architectures and reports predictive performance across 10 seeds.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 Neural Architecture Search (NAS) Performance

  • After 5 × 10^4 wall-clock seconds, arch2vec considerably outperforms supervised and adjacency-matrix methods and robustly achieves the lowest final test regret after 1 × 10^6 seconds across 500 runs.Figure 6 reports mean test regret and the empirical cumulative distribution of final regret.Source: Figure 6 · p. 8, §4.2 · p. 8, Table 2 · p. 8
  • On NAS-Bench-201, arch2vec yields better validation and test accuracy with reduced variability across CIFAR-10, CIFAR-100, and ImageNet-16-120 runs.CIFAR-10 searches use a 1.2 × 10^4-second budget; the other datasets use budgets corresponding to the same number of queries.Source: §4.2 · p. 9, Table 3 · p. 9
  • On DARTS, arch2vec generally achieves competitive search performance among cell-based NAS methods with comparable model parameters.The comparison uses a budget of 100 queries and reports CIFAR-10 results averaged over five seeds.Source: §4.2 · p. 9, Table 4 · p. 9

5 Conclusion

The conclusion presents arch2vec as an unsupervised NAS method that learns representations without architecture accuracies and benefits downstream search across three search spaces.

5 Conclusion

  • arch2vec learns architecture representations without accuracies, producing a more smoothly changing latent performance surface than supervised representation learning.The authors demonstrate benefits for different downstream search strategies in three NAS search spaces.Source: §5 · p. 9
  • The conclusion recommends closer study of architecture representation learning for neural architecture search.Source: §5 · p. 9
  • The authors suggest that a better continuous-space search strategy combined with arch2vec could produce better results.Source: §5 · p. 9

Broader Impact

The paper examines unsupervised architecture representation learning through prediction and search-efficiency experiments, and identifies larger search spaces as a possible setting for its benefits.

Broader Impact

  • The authors compare supervised and unsupervised representations on predictive performance and on NAS efficiency and robustness.They report that unsupervised architecture representation learning performs reasonably well in both experiments.Source: Broader Impact · p. 10
  • The paper identifies search spaces containing billions of architectures, with few labeled accuracies, as a setting where unsupervised representations may benefit downstream applications.The authors note that supervised optimization in such large search spaces might be less effective.Source: Broader Impact · p. 10
  • The authors call for future study of unsupervised NAS with meaningful pretext tasks on larger search spaces.They also identify graph-neural-network-based pretraining as a promising direction.Source: Broader Impact · p. 10

A Pre-training and search details on each search space

The appendix describes pretraining and search details for the evaluated spaces, including a shared adjacency- and operation-matrix input encoding and NAS-Bench-101-specific setup.

A Pre-training and search details on each search space

A.1 NAS-Bench-101

  • NAS-Bench-101 reinforcement-learning search uses REINFORCE with a one-layer, 128-hidden-unit LSTM controller, while Bayesian optimization uses DNGO with a 128-unit fully connected network.Both configurations use a 1 × 10^6-second maximum estimated wall-clock budget.Source: §A.1 · p. 14

A.2 NAS-Bench-201

The appendix describes how NAS-Bench-201 architectures are transformed for encoding and searched, then details the DARTS cell structure, encoding, and evaluation budget.

A.2 NAS-Bench-201

  • NAS-Bench-201 DAGs are transformed to operation-node, data-flow-edge graphs and encoded with the NAS-Bench-101 adjacency- and operation-matrix scheme.The accompanying figure describes original and transformed representations alongside their matrices.Source: §A.2 · p. 14, Figure 8 · p. 15
  • NAS-Bench-201 reinforcement-learning search uses dataset-specific time budgets, while Bayesian optimization begins with 16 samples and adds the best-performing architecture each iteration.The Bayesian optimization expected-improvement value is 1.0 for all datasets.Source: §A.2 · p. 15

A.3 DARTS Search Space

  • In the DARTS cell, two inputs come from previous cells and each intermediate node receives two operation-associated connections.The output concatenates all intermediate nodes; the encoding uses an 11 × 11 edge matrix and an 11 × 11 operation matrix.Source: §A.3 · p. 15
  • The DARTS experiments sample 600,000 unique architectures and use a budget of 100 queries, training each sampled architecture for 50 epochs.Source: §A.3 · p. 15

B More details on pre-training evaluation metrics

Pre-training evaluation measures reconstruction, generative validity, and uniqueness, using held-out architectures and search-space-specific validity checks. Figure 9 illustrates a cell alongside its adjacency and operation matrices.

More details on pre-training evaluation metrics

  • Reconstruction Accuracy measures errorless recovery, Validity measures valid architectures generated from the prior, and Uniqueness measures distinct architectures among valid generations.Uniqueness is described as measuring smoothness and diversity of generated samples.Source: B More details on pre-training… · pp. 15–16
  • Arch2vec pre-training uses a 90% training split and a 10% held-out test split, with reconstruction accuracy computed as the proportion of held-out inputs decoded identically.The evaluation metrics are suggested by [32].Source: B More details on pre-training… · pp. 15–16
  • Figure 9 shows a DARTS cell with its corresponding adjacency matrix and operation matrix.Source: Figure 9 · p. 16
  • Validity is measured by decoding 10,000 prior samples scaled and shifted using the training embeddings, then reporting the proportion that are valid architectures.Uniqueness is the proportion of unique architectures among valid decoded architectures.Source: B More details on pre-training… · p. 16
  • For DARTS, validity requires the designated input and output nodes, connected intermediate nodes with two incoming edges, and an upper-triangular binary DAG matrix.NAS-Bench-101 and NAS-Bench-201 validity are checked with their official APIs.Source: B More details on pre-training… · p. 16

C Best found cells and transfer learning results on ImageNet

Arch2vec’s best cells from RL- and BO-based search are evaluated on ImageNet after CIFAR-10 search. Under a mobile compute limit, both outperform DARTS and SNAS in the DARTS search space and remain competitive among cell-based methods.

Best found cells and transfer learning results on ImageNet

D More visualization results of each search space

Across NAS-Bench-101 and NAS-Bench-201, visualizations contrast arch2vec with supervised representations; the DARTS example shows local structural similarity preserved in an arch2vec sequence.

NAS-Bench-101

  • On NAS-Bench-101, arch2vec better captures topology and operator similarity, clustering straight networks separately from skip-connected networks unlike supervised representations.The visualized sequences are randomly selected pairs beginning from the same point and proceeding to the closest unvisited architecture.Source: D More visualization results of each… · p. 17, Figure 11 · p. 18

NAS-Bench-201

DARTS Search Space

  • In the DARTS example, arch2vec’s decoded sequence gradually changes edge operations while retaining connections, indicating preserved local structural similarity.The sequence starts from a random point and repeatedly selects the closest unvisited latent-space point; the authors report no architecture accuracies for this search space.Source: D More visualization results of each… · p. 17, Figure 13 · p. 20

Related papers

Loading 2006.06936v2…