PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Neural Attentive Meta-Learner

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel

arXiv:1707.03141v3cs.AIcs.LGcs.NEstat.ML

TL;DR

Deep networks struggle with scarce data and rapid task changes, motivating meta-learning across related tasks; many existing methods nevertheless hard-code specialized architectures or algorithms. The paper introduces SNAIL, which combines temporal convolutions with causal attention to learn generic strategies, and reports significant state-of-the-art performance across widely benchmarked supervised and reinforcement-learning tasks. The authors also identify broader sequence-to-sequence applications and longer-term memory as future directions.

  • Problem

    Existing meta-learning methods often rely on hand-designed, application-specific architectures or hard-coded algorithms, limiting how generic learners can solve related tasks.

  • Method

    SNAIL combines temporal convolutions for past-context access with causal attention for selecting specific information in a simple, generic meta-learner architecture.

  • Results

    SNAIL achieves state-of-the-art performance by significant margins on widely benchmarked meta-learning tasks in both supervised and reinforcement learning.

  • Takeaways & Limitations

    SNAIL demonstrates that a generic architecture without application-specific components or algorithmic priors can perform strongly across supervised and reinforcement-learning meta-learning tasks.

  • Takeaways & Limitations

    The reported SNAIL agent attends over only a few recent episodes rather than its entire lifetime of experience, leaving lifelong memory as future work.

Abstract

from arXiv · show

Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on a distribution of similar tasks, in the hopes of generalization to novel but related tasks by learning a high-level strategy that captures the essence of the problem it is asked to solve. However, many recent meta-learning approaches are extensively hand-designed, either using architectures specialized to a particular application, or hard-coding algorithmic components that constrain how the meta-learner solves the task. We propose a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information. In the most extensive set of meta-learning experiments to date, we evaluate the resulting Simple Neural AttentIve Learner (or SNAIL) on several heavily-benchmarked tasks. On all tasks, in both supervised and reinforcement learning, SNAIL attains state-of-the-art performance by significant margins.

2 META-LEARNING PRELIMINARIES

Meta-learning targets generalization across related tasks, where a learner must use prior experience to make predictions or decisions with limited data. SNAIL combines temporal convolutions and causal attention to provide broad access to past information while selecting relevant experience, using one architecture across supervised and reinforcement learning.

  • Meta-learning problem: Meta-learning optimizes a learner across a distribution of episodic tasks so it can generalize to unseen but related tasks.Each task specifies inputs, outputs, loss, transitions, and episode length; testing uses a similar but different task distribution.
  • Design principle: SNAIL is designed as a simple, versatile meta-learner whose strategy is learned rather than application-specific behavior being built into the architecture.The same design goal covers both supervised and reinforcement learning.
  • Core mechanism: Temporal convolutions provide high-bandwidth access to a finite past context, while causal attention pinpoints information in an arbitrarily large context.Their complementary strengths motivate combining them rather than relying on either mechanism alone.
  • Core mechanism: SNAIL interleaves temporal-convolution blocks with causal-attention layers so it can use past experience extensively while learning which information to retrieve.Attention is applied at multiple stages in an end-to-end trained model.
  • Use across settings: In supervised learning, SNAIL predicts an unlabeled example from preceding example-label pairs; in reinforcement learning, it uses current and previous observations, actions, and rewards.Its internal state is preserved across reinforcement-learning episode boundaries, allowing memory across episodes.
  • Architecture: SNAIL architectures are assembled from causal convolutional and attention blocks that concatenate newly computed features with their inputs.Temporal-convolution blocks use exponentially increasing dilation rates, while attention blocks perform causal key-value lookups.

4 RELATED WORK

Prior meta-learning methods span generic recurrent learners, specialized architectures, and meta-learners that encode gradient-based adaptation. SNAIL is positioned as a generic alternative that can learn strategies exploiting task structure without hard-coding gradient descent.

  • Prior approaches: Meta-learning research includes recurrent, memory-augmented, specialized few-shot, and learned-optimizer approaches, reflecting a tradeoff between performance and generality.The surveyed methods differ in whether they rely on recurrent state, external memory, similarity metrics, or parameter updates.
  • Recurrent learners: RNN-based meta-learners are generic but can be limited by temporally linear hidden-state dependencies, while memory-augmented designs add complicated memory addressing.Prior results reported inadequacy of LSTM architectures for some algorithmic and few-shot tasks.
  • Specialized methods: Several few-shot classification methods improve performance with domain-specific similarity architectures, including Siamese networks, embedding functions, and distance-based prediction.These methods explicitly exploit image or class-similarity structure.
  • Gradient-based adaptation: Learned-optimizer methods use a meta-learner to update a traditional learner’s parameters, commonly interpreting the learned strategy as gradient-based optimization.MAML constrains this adaptation to ordinary gradient descent and achieved equivalent performance in the cited setting.
  • SNAIL’s position: SNAIL offers a generic architecture that can learn algorithms exploiting domain-specific task structure instead of following a prescribed test-time gradient-descent strategy.This reframes the performance–generality tradeoff by leaving the strategy to be learned.
  • Reinforcement learning: In reinforcement-learning benchmarks, SNAIL is compared with LSTM-based meta-learners and MAML on task families studied by prior work.The cited benchmark includes continuous-control tasks involving simulated cheetahs and ants.

5 EXPERIMENTS

The experiments test SNAIL across few-shot classification and reinforcement-learning tasks, emphasizing generality, comparison with specialized methods, and scaling to high-dimensional inputs and long temporal dependencies. Across these settings, SNAIL demonstrates strong performance and adapts by exploiting shared task structure.

  • Experimental scope: The evaluation spans few-shot image classification, multi-armed bandits, tabular MDPs, and continuous-control tasks.These domains probe generality across supervised and reinforcement learning, exploration–exploitation, temporal dependencies, and adaptation to new control objectives.
  • Few-shot image classification: SNAIL is evaluated on 5-way and 20-way Omniglot and 5-way mini-ImageNet, with episodes using randomly selected 1–5 shots.The model predicts the label of a new example after receiving NK example-label pairs in random order.
  • Few-shot image classification: SNAIL outperforms extensively hand-designed or domain-specific methods on Omniglot and mini-ImageNet, and significantly exceeds similarly simple, generic methods.The reported image-classification comparisons are summarized in Tables 1 and 2, with confidence intervals where available.
  • Reinforcement learning: In tabular MDPs, performance is normalized by the value-iteration upper bound and is expected to approach 1 as the agent learns more about the current MDP.Experiments use random MDPs with 10 states, 5 actions, and N ∈ {10, 25, 50, 75, 100} episodes; learning curves compare SNAIL and LSTM.
  • Reinforcement learning: SNAIL and LSTM specialize from shared task structure, identifying the task within the initial timesteps and acting optimally thereafter in the described bandit setting.The bandit task requires initial exploration followed by exploitation, while the MDP and control experiments test longer-term adaptation.
  • Reinforcement learning: In maze experiments, SNAIL explores during the first episode and then goes directly to the goal during the second episode after finding it.The study measures average episode length separately for the first and second episodes; LSTM shows similar behavior but has more difficulty remembering the goal location.

6 CONCLUSION AND FUTURE WORK

SNAIL is presented as a simple, generic meta-learning architecture that combines temporal convolutions with causal attention and achieves strong results across supervised and reinforcement-learning benchmarks. Future work considers applying it to other sequence-to-sequence tasks and extending attention across an agent’s lifetime of experience.

  • 6 CONCLUSION AND FUTURE WORK: SNAIL combines temporal convolutions and causal attention to quickly incorporate and refer to past experience without application-specific components or algorithmic priors.The architecture is designed for generality across supervised and reinforcement learning.
  • 6 CONCLUSION AND FUTURE WORK: SNAIL achieves state-of-the-art performance by significant margins on widely benchmarked meta-learning tasks in supervised and reinforcement learning.
  • 6 CONCLUSION AND FUTURE WORK: Future work would test whether SNAIL transfers to sequence-to-sequence tasks such as language modeling or translation.
  • 6 CONCLUSION AND FUTURE WORK: This work attends only to a few recent episodes, motivating future meta-learners that attend over an agent’s entire lifetime of experience.Such lifelong memory would require deciding which experiences are worth remembering to keep computation practical.

A FEW-SHOT CLASSIFICATION ARCHITECTURES

The few-shot classification architecture uses a shared SNAIL design with dataset-specific image embeddings, including a deeper residual embedding for mini-Imagenet. The architecture interleaves attention and temporal-convolution blocks before producing class predictions.

  • A FEW-SHOT CLASSIFICATION ARCHITECTURES: For N-way, K-shot classification, SNAIL processes a sequence of length T = NK + 1 with alternating attention and temporal-convolution blocks.The architecture ends with a 1 × 1 convolution having N filters.
  • A FEW-SHOT CLASSIFICATION ARCHITECTURES: The Omniglot embedding repeats four convolution, batch-normalization, ReLU, and max-pooling blocks before producing a 64-dimensional feature vector.
  • A FEW-SHOT CLASSIFICATION ARCHITECTURES: For mini-Imagenet, the authors use a deeper embedding network because the shallow embedding did not adequately use SNAIL’s expressive capacity.The deeper embedding was intended to prevent underfitting.
  • A FEW-SHOT CLASSIFICATION ARCHITECTURES: Each residual block applies two 3x3 convolutions with batch normalization and leaky ReLU, adding the block input to the transformed output.
  • A FEW-SHOT CLASSIFICATION ARCHITECTURES: The mini-Imagenet embedding uses several residual blocks in a smaller ResNet-style architecture.

B FEW-SHOT CLASSIFICATION: ABLATIONS

Ablations indicate that SNAIL’s strongest few-shot classification performance depends on combining temporal convolutions with attention, rather than on the deeper embedding alone. Additional experiments show robustness to architectural ordering and transferability across datasets.

  • B FEW-SHOT CLASSIFICATION: ABLATIONS: Both temporal-convolution and attention layers are essential for maximal performance; removing either leaves a competitive but weaker model.Temporal convolutions alone match full-model 1-shot performance but perform worse on 5-shot classification.
  • B FEW-SHOT CLASSIFICATION: ABLATIONS: SNAIL’s improvement is not explained entirely by the deeper embedding, because MAML overfits with it and domain-specific RNN methods use its capacity less effectively.
  • B FEW-SHOT CLASSIFICATION: ABLATIONS: Nearest-neighbor classifiers using Omniglot features reach 65.1% and 67.1% with Euclidean distance and 67.7% and 68.3% with cosine distance for 1-shot and 5-shot tasks.The comparison suggests SNAIL learns a strategy more sophisticated than either distance metric.
  • B FEW-SHOT CLASSIFICATION: ABLATIONS: Randomly permuted 13-layer SNAIL architectures achieve 98.62% ± 0.13% and 99.71% ± 0.08% on 1-shot and 5-shot Omniglot, essentially matching the primary architecture.
  • B FEW-SHOT CLASSIFICATION: ABLATIONS: Cross-dataset experiments obtain 50.62% and 62.34% when Omniglot-trained SNAIL weights transfer to mini-Imagenet, and 98.66% and 99.56% in the reverse direction.A model combining an Omniglot embedding with a mini-Imagenet-trained SNAIL achieves 98.5% and 99.5% on Omniglot.
  • B FEW-SHOT CLASSIFICATION: ABLATIONS: The transfer results are reported as very competitive with state-of-the-art performance, suggesting transferability of SNAIL’s algorithm and feature representation.

C.1 MULTI-ARMED BANDIT AND TABULAR MDP ARCHITECTURES

The bandit and tabular-MDP architectures share a fully connected layer and use separate temporal-convolution and attention stacks for policy and value functions. Attention is unnecessary for bandits but required for SNAIL to learn tabular MDPs.

  • C.1 MULTI-ARMED BANDIT AND TABULAR MDP ARCHITECTURES: The shared architecture begins with a fully connected layer with 32 outputs, followed by separate policy and value-function blocks.
  • C.1 MULTI-ARMED BANDIT AND TABULAR MDP ARCHITECTURES: The policy uses two TCBlock(T, 32) layers and AttentionBlock(32, 32), while the value function uses two TCBlock(T, 16) layers and AttentionBlock(16, 16).
  • C.1 MULTI-ARMED BANDIT AND TABULAR MDP ARCHITECTURES: Removing attention makes no difference on bandit problems, whereas SNAIL without attention cannot learn to solve MDPs.

C.2 CONTINUOUS CONTROL ARCHITECTURES

The continuous-control architecture uses a shared policy-and-value network with two attention blocks surrounding two temporal-convolution blocks.

  • C.2 CONTINUOUS CONTROL ARCHITECTURES: The shared policy-and-value architecture uses two 256-unit tanh fully connected layers, followed by AttentionBlock(32, 32), two TCBlock(T, 16) layers, and AttentionBlock(32, 32).Policy and value functions then use separate fully connected layers for their required output dimensions.

C.3 VISUAL NAVIGATION ARCHITECTURES

The visual-navigation setup preprocesses images into 256-dimensional features and uses separate temporal-convolution and attention configurations for policy and value functions.

  • C.3 VISUAL NAVIGATION ARCHITECTURES: Image observations are processed by two 5 × 5 convolutional layers with 16 filters, stride 2, and ReLU, then flattened into 256-dimensional features.This preprocessing follows the convolutional architecture used by Duan et al. (2016).
  • C.3 VISUAL NAVIGATION ARCHITECTURES: For trajectories of T = 500, the policy uses two TCBlock(T, 32) layers and two AttentionBlock(16, 16) layers.The blocks alternate between temporal convolutions and attention.
  • C.3 VISUAL NAVIGATION ARCHITECTURES: The value function uses two TCBlock(T, 16) layers for the same visual-navigation setup.

C.4 ADDITIONAL REINFORCEMENT LEARNING HYPERPARAMETERS

The reinforcement-learning experiments use TRPO with generalized advantage estimation, while several task settings reuse Duan et al.’s hyperparameters for direct comparison.

  • C.4 ADDITIONAL REINFORCEMENT LEARNING HYPERPARAMETERS: All policies were trained with trust-region policy optimization and generalized advantage estimation, abbreviated TRPO with GAE.The hyperparameters are listed in Table 7.
  • C.4 ADDITIONAL REINFORCEMENT LEARNING HYPERPARAMETERS: The multi-armed bandit, tabular MDP, and visual-navigation experiments reused Duan et al. (2016)’s hyperparameters for direct comparability.The authors note that additional tuning could potentially improve SNAIL’s performance.
  • C.4 ADDITIONAL REINFORCEMENT LEARNING HYPERPARAMETERS: Table 7 reports the TRPO + GAE hyperparameters used in the reinforcement-learning experiments.

D REINFORCEMENT LEARNING: ABLATIONS

Ablations show that temporal convolutions and attention contribute differently: temporal convolutions suffice for bandits but struggle on MDPs, while attention alone performs no better than random.

  • D REINFORCEMENT LEARNING: ABLATIONS: The ablation study tests SNAIL variants using only temporal-convolution layers or only attention layers on bandit and MDP tasks.
  • D REINFORCEMENT LEARNING: ABLATIONS: The TC-only model performs as well as complete SNAIL on bandits, where successful performance does not require maintaining a large memory of past experience.The authors relate this to algorithms that update running statistics at each timestep.
  • D REINFORCEMENT LEARNING: ABLATIONS: The TC-only model struggles in MDPs, where a more sophisticated algorithm is required, and its asymptotic suboptimality suggests saturated internalization of past experience.
  • D REINFORCEMENT LEARNING: ABLATIONS: Table 8 presents the SNAIL ablations in the MDP domain.
  • D REINFORCEMENT LEARNING: ABLATIONS: The attention-only model cannot solve either bandit or MDP tasks and performs no better than a random agent despite multiple attention blocks and heads.The variant uses positional encoding and is equivalent to the Transformer architecture described by Vaswani et al. (2017b).
  • D REINFORCEMENT LEARNING: ABLATIONS: The authors hypothesize that temporal convolutions are essential because they locally analyze contiguous sequence segments, unlike pure attentive lookups.They argue this supports better contextual representations for sequential decision-making.
Loading 1707.03141v3…