PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Task-Aware Architecture Inference

Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gabor Bartok, Jesse Berent

arXiv:1902.05781v1cs.LGstat.ML

TL;DR

Neural architecture search is promising but costly because each new task can require substantial computation and model training. The paper amortizes search across related tasks with a deep value network that predicts architecture performance and gradient ascent over continuous architecture parameters. Experiments show performance relatively close to an expensive NAS baseline, while the online phase requires no child-model training.

  • Problem

    Neural architecture search remains computationally costly for each new task, motivating efficient search under resource constraints.

    Source: §1 · p. 1

  • Method

    A deep value network uses task information and prior cross-task architecture experiments to predict performance, then gradient ascent infers a continuous architecture for a new task.

    Source: §1 · p. 2, §3 · p. 3, §3.4 · p. 5

  • Results

    The found architecture achieves performance relatively close to the expensive NAS baseline without child-model training during online inference.

    Source: §5.3 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Architecture-search cost can be amortized across related tasks while retaining efficient inference for a new task.

    Source: §1 · p. 2, §6 · p. 8

    Continuous architectures can make inference expensive because computations cover all possible design choices, motivating future pruning work.

    Source: §5.3 · p. 6

Abstract

from arXiv · show

Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite costly. To this point, we propose a novel gradient-based framework for efficient architecture search by sharing information across several tasks. We start by training many model architectures on several related (training) tasks. When a new unseen task is presented, the framework performs architecture inference in order to quickly identify a good candidate architecture, before any model is trained on the new task. At the core of our framework lies a deep value network that can predict the performance of input architectures on a task by utilizing task meta-features and the previous model training experiments performed on related tasks. We adopt a continuous parametrization of the model architecture which allows for efficient gradient-based optimization. Given a new task, an effective architecture is quickly identified by maximizing the estimated performance with respect to the model architecture parameters with simple gradient ascent. It is key to point out that our goal is to achieve reasonable performance at the lowest cost. We provide experimental results showing the effectiveness of the framework despite its high computational efficiency.

1 Introduction

Neural architecture search can automate model design but remains costly for each new task. This paper proposes amortized, cross-task architecture inference using a deep value network and gradient-based optimization.

1 Introduction

  • The framework amortizes architecture-search cost across related tasks by transferring information about architecture performance between datasets.It leverages prior model-training experiments rather than searching independently from scratch for every new task.Source: §1 · pp. 1–2
  • It uses a deep value network to estimate an architecture’s task performance from task meta-features and the architecture parameters.The network provides a differentiable performance estimate for candidate architectures.Source: §1 · p. 2
  • For a new task, gradient ascent identifies a candidate architecture before any model training is performed.The online phase optimizes the estimated performance rather than training child models on the new task.Source: Figure 1 · p. 1, §1 · p. 2
  • The proposed contributions combine efficient gradient-based architecture search, learned task meta-features, and cross-learning across many task datasets.The paper reports experimental results demonstrating the framework’s potential.Source: §1 · p. 2
  • The framework supports learning task meta-features directly from raw task samples, rather than relying only on pre-computed features.This is presented as a contribution alongside cross-task learning and gradient-based architecture optimization.Source: §1 · p. 2

2 Problem formulation

The problem is to identify an effective architecture quickly for an unseen task while amortizing search costs across related datasets. The desired framework should scale across tasks and propose an architecture without training a model on the new dataset.

2 Problem formulation

  • The framework seeks an effective architecture for a new, unseen dataset before any model is trained.It transfers knowledge from prior model-training experiments on related tasks.Source: §2 · p. 2
  • Architecture search should learn collectively across task datasets instead of operating independently on each new dataset.The formulation aims to leverage the accumulated information from model-training experiments.Source: §2 · p. 2
  • The proposed solution is intended to scale in computational resources while preserving cross-task learning and rapid architecture proposal.These properties define the framework’s target behavior.Source: §2 · p. 2

3 Proposed framework

The framework transfers architecture-performance knowledge across tasks through a deep value network, enabling gradient-based inference of candidate architectures before training on a new task. It represents architectures continuously so task-conditioned performance estimates can be optimized efficiently.

3 Proposed framework

  • Cross-task architecture search uses a deep value network to estimate how candidate architectures perform for task data described by meta-features.The network acts as a transferable meta-model that learns which architectures suit particular data types.Source: §3 · pp. 2–3

3.1 The meta-features of a task

  • Task meta-features describe dataset characteristics, while learned embeddings can be extracted jointly from raw samples and labels.The framework can use precomputed statistics or learn permutation-invariant task representations directly from the dataset.Source: §3.1 · p. 3

3.2 Off-line training phase

  • Offline training builds a lifelong database by training multiple child architectures on each task and using their architecture, task, and validation-performance records to train the value network.The value network improves as additional tasks and model-training experiments are added.Source: §3.2 · pp. 3–4

3.3 Online inference phase

  • Online inference extracts task meta-features, then applies gradient ascent to architecture parameters and selects the best result across multiple initializations.The architecture parameters remain continuous because gradient-based inference is sensitive to initialization and requires a continuous search space.Source: §3.3 · p. 4, §3.2 · p. 4

3.4 Architecture parametrization

  • Continuous parametrization replaces categorical choices with softmax-weighted mixtures of base layers, allowing architecture parameters to be optimized by gradients.The child-network encoding includes layer-mixture weights, layer-presence controls, and embedding-selection parameters, u := {{α}, {β}, {γ}}.Source: §3.4 · pp. 4–5

4 Related work

Related work spans Bayesian optimization, reinforcement learning, evolutionary and morphing methods, and performance prediction for automated model building. The proposed framework extends performance prediction with task-aware meta-features and differentiable gradient-based search.

  • Automated model-building methods include Bayesian optimization, reinforcement learning, evolutionary search, morphing, and performance prediction.Bayesian and performance-prediction methods model architecture performance, while reinforcement-learning, evolutionary, and morphing methods iteratively evaluate or modify architectures.Source: §4 · p. 5
  • Performance-prediction methods forecast candidate architecture performance without training and typically rely on databases of previous architecture trainings.Source: §4 · p. 5
  • The proposed framework is task-aware: its deep value network combines architecture information with task meta-features, including features learned from raw task samples.Source: §4 · p. 5
  • Compared with SMAC, the framework replaces expensive Bayesian optimization with efficient gradient-based architecture search enabled by a structured-prediction formulation.Source: §4 · p. 5
  • Unlike task-embedding reinforcement learning [20], the framework derives task information directly from data samples and requires no child-model training on the test task.Source: §4 · p. 5

5 Experiments

Experiments use leave-one-out evaluation across NLP classification tasks, training a deep value network to predict architecture performance and guide architecture search. Predictions are generally strong, the method approaches NAS accuracy without online child-model training, and continuous architectures remain costly to execute.

5 Experiments

  • Experiments use publicly available NLP datasets in leave-one-out experiments, with each task held out while the remaining tasks train the deep value network.Source: §5 · p. 5

5.1 Setup

  • The deep value network is trained from a database containing about 500 randomly generated child-model architectures per task.Source: §5.1 · p. 6

5.2 Predicting the model performance

  • Spearman rank correlations lie around 0.8 in most leave-one-out tasks despite using no child-model trainings on the test task.Predictive performance is evaluated with Spearman correlation and R2 over ten repetitions.Source: §5.2 · p. 6
  • Meta-features improve both predictive metrics by providing task-specific information for estimating relative architecture performance.Source: §5.2 · p. 6

5.3 Architecture search

  • The proposed method achieves test accuracy not too far from NAS while requiring no child-model training during online inference.NAS is evaluated on the same child models, whereas the proposed method uses predicted performance to select architectures efficiently.Source: §5.3 · p. 6
  • Continuous child-model architectures may make inference expensive because computations cover all possible design choices.The authors suggest pruning architectures to speed inference while preserving model quality, leaving this for future work.Source: §5.3 · p. 6

6 Conclusions and future work

The framework enables efficient architecture inference across tasks using a deep value network, and its test accuracy is close to NAS without training child models on the test task.

6 Conclusions and future work

  • The value network’s predictive performance is evaluated with Spearman’s rank correlation and R2 across tasks, with meta-features helping in the majority of tasks.Tables 2 and 3 report task-level breakdowns for these metrics.Source: Table 2 · p. 7, Table 3 · p. 7, §6 · p. 8
  • The proposed method achieves test accuracy close to NAS in the majority of cases without training any child model on the test task.NAS reports the number of child models trained to reach its selected accuracy, whereas the proposed method performs no child-model training on the test task.Source: Table 4 · p. 7
  • A deep value network predicts candidate-architecture performance from learned task meta-features, enabling cross-task architecture inference with gradient ascent.The method infers a candidate architecture for a new task by maximizing predicted performance.Source: §6 · pp. 6, 8

Supplementary Material

The supplied supplementary-material passage contains an arXiv version date from 15 February 2019.

Supplementary Material

1 Adding more training tasks improves performance

Adding more training tasks improves the deep value network’s average predictive performance.

1 Adding more training tasks improves performance

  • Adding more training tasks improves the deep value network’s average Spearman correlation and R2 performance.The experiment averages results over 100 task-order randomizations, training the value network 10 times per ordering.Source: §1 · p. 10

2 Meta-features visualizations

Task embeddings remain stable across different batches from the same task, as shown through tSNE and PCA visualizations.

2 Meta-features visualizations

  • Different batch realizations from the same task produce close-by embeddings in the learned meta-feature space.The visualizations use tSNE and PCA for 10 random batches from training and test tasks, including the test task ‘airline.’Source: §2 · pp. 10, 12, 13
  • Figure 1 plots Spearman correlation and R2 against the number of training tasks for the ‘airline’ test task.The caption states that similar behavior holds for the other tasks.Source: §2 · p. 11

3 Text input embedding modules

This section lists TensorFlow Hub embedding modules for text input, including language, dataset size, embedding dimension, and module handles.

  • The table catalogs TensorFlow Hub embedding modules for text input.Source: §3 · p. 14
  • The table provides TensorFlow Hub handles through a shared prefix for the listed modules.Source: §3 · p. 14
  • English modules span embedding dimensions from 50 to 512 and include Wiki-words, nnlm, and universal-sentence-encoder handles.Source: §3 · p. 14
  • The entries report dataset sizes ranging from 4B to 200B where specified, while some modules have unspecified sizes.Source: §3 · p. 14

Related papers

Loading 1902.05781v1…