PaperlayerView exact revision on arXiv

Source-linked AI summary

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber

arXiv:2108.12284v4cs.LGcs.AIcs.NE

TL;DR

Systematic-generalization benchmarks expose failures in standard Transformer baselines, but it is unclear whether those failures reflect architectural limits or configuration choices. The paper revisits basic model and training configurations across five datasets and reports substantial gains, while showing that IID validation can miss them. These findings support using generalization-specific validation and identify scope boundaries in the study’s focus on basic Transformer architectures.

  • Problem

    Standard Transformer baselines often fail dramatically on systematic-generalization datasets, while IID validation does not reveal the models’ generalization differences.

  • Method

    The paper revisits embedding scaling, early stopping, relative positional embeddings, and Universal Transformer variants across five systematic-generalization datasets.

  • Results

    Across five datasets, carefully configured Transformers substantially improve over existing baselines, including PCFG productivity accuracy from 50% to 85%, COGS from 35% to 81%, and SCAN length-split accuracy of 100% with cutoff 26.

  • Takeaways & Limitations

    Proper generalization validation sets are needed to develop systematic-generalization models and avoid false progress from weak baselines.

  • Takeaways & Limitations

    The study focuses on basic Transformer architectures, although the discussed details may also be relevant to other architectures and Transformer variants.

Abstract

from arXiv · show

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

1 Introduction

Systematic generalization remains unsolved, while standard Transformer baselines can fail on proposed benchmarks because their configurations are questionable. This work shows that revisiting model and training choices substantially improves performance across five datasets, although IID validation can obscure these differences.

  • Systematic generalization requires extrapolating compositional rules beyond the training distribution, such as combining known rules differently or applying them to longer problems.
  • Baseline Transformers often fail dramatically on systematic-generalization datasets because standard machine-translation practices are applied without modification and relevant techniques are omitted.Relative positional embeddings are cited as one relevant technique absent from the baselines.
  • The study evaluates Transformer configurations on SCAN, CFQ, PCFG, COGS, and the Mathematics Dataset.
  • 50% to 85% accuracy is reported for the PCFG productivity split, while COGS improves from 35% to 81% over existing baselines.
  • On SCAN, relative positional embeddings largely mitigate the EOS decision problem, achieving 100% accuracy on the length split with a cutoff at 26.
  • Models with large performance gaps on generalization can nevertheless perform equally well on IID validation datasets, motivating proper generalization validation sets.

2 Datasets and Model Architectures for Systematic Generalization

The paper frames systematic generalization datasets as sequence-to-sequence tasks whose generalization splits differ systematically from training, while IID splits remain conventional. It surveys five datasets and compares standard and Universal Transformer architectures with absolute or relative positional embeddings.

  • All considered language datasets formulate systematic generalization as sequence-to-sequence mapping with test distributions that differ systematically through length, combinations, or composition.
  • Datasets: SCAN maps natural-language navigation instructions to command sequences, with length and add-primitive splits testing distinct generalization patterns.For example, “jump twice” maps to “JUMP JUMP.”
  • Datasets: CFQ translates natural-language questions into Freebase SPARQL queries and includes compound-divergence and length-based splits.Compound divergence measures parse-tree differences between splits and correlates with generalization difficulty.
  • Datasets: PCFG tests executable list manipulations, including productivity on longer sequences and systematicity through novel constituent recombinations.
  • Datasets: COGS maps English sentences to logical forms and provides IID validation alongside out-of-distribution generalization testing; Mathematics uses category-based modules with some extrapolation sets.The study focuses on the Mathematics “add_or_sub” and “place_value” modules.
  • Model architectures: The study compares standard and Universal Transformers with absolute or relative positional embeddings; Universal variants share weights across layers without adaptive computation time or timestep embeddings.Positional embeddings are added only to the first layer.
  • Model architectures: Universal Transformers are relevant to algorithmic reasoning because shared weights can support recombination of operations across positions rather than tying operations to separate successive layers.

3 Improving Transformers on Systematic Generalization

Careful choices in positional embeddings, model selection, embedding scaling, and validation procedures substantially improve Transformer systematic generalization, while IID performance can conceal large generalization differences.

  • 3.1 Addressing the EOS Decision Problem with Relative Positional Embedding: Relative positional embeddings largely mitigate Transformers’ EOS-overfitting problem on longer SCAN sequences.The authors identify absolute positional embeddings as the main cause and apply relative embeddings in the encoder and decoder.
  • 3.1 Addressing the EOS Decision Problem with Relative Positional Embedding: Relative-embedding models excelled across length cutoffs, outperforming models evaluated with oracle output lengths, whereas absolute-embedding models had near-zero accuracy.The comparison used models trained to predict EOS and evaluated without oracle length.
  • 3.2 Model Selection Should Be Done Carefully: 81% COGS test accuracy was achieved without early stopping, compared with 35% for the original Kim and Linzen baseline.In the original codebase setting, disabling early stopping alone improved accuracy to 65%.
  • 3.2 Model Selection Should Be Done Carefully: IID validation is inadequate for selecting models for systematic generalization because considered models reach nearly 100% IID accuracy despite divergent generalization performance.The authors recommend validation sets for the generalization split while warning against using the test set for development.
  • 3.2 Model Selection Should Be Done Carefully: Generalization accuracy and loss can diverge, so accuracy rather than loss should guide early stopping and hyperparameter tuning.On CFQ and COGS, loss and generalization accuracy can both increase during training; PCFG also exhibits epoch-wise double descent in loss.
  • 3.3 Large Impacts of Embedding Scaling: Position Embedding Downscaling performed consistently best on PCFG and COGS, showing that embedding scaling materially affects test-set performance.The authors report a large gap between the best and worst scaling configurations on test sets.

4 Results Across Different Datasets

Across the evaluated datasets, relative positional embeddings and Universal Transformer variants generally improve systematic generalization, while training and model-selection choices also materially affect results.

  • Relative positional Transformer variants outperform absolute variants on almost all tested datasets.Universal variants outperform standard ones except on COGS and CFQ MCD 1.
  • CFQ: 81% accuracy on CFQ output length is achieved with relative positional embeddings, compared with 77% using absolute embeddings and approximately 66% previously reported.The authors attribute the comparison to Universal Transformer variants but could not identify the exact reason for the absolute-embedding improvement.
  • CFQ: On CFQ MCD splits, the results slightly outperform the reported baseline, with Relative Universal Transformers generally performing best except on MCD 1.The standard Transformer wins MCD 1 by a slight margin.
  • PCFG: 85% productivity and 96% systematicity accuracy are achieved on PCFG with Universal Transformers using relative positional embeddings.These results improve over reported baselines of 50% and 72%, respectively.
  • COGS: 81% generalization accuracy on COGS exceeds the previously reported 35% baseline.Removing early stopping raises performance to 65%, while further configuration changes reach 81%.
  • Mathematics Dataset: Relative positional embeddings are advantageous for generalization on both evaluated Mathematics Dataset subsets.The experiments use the place_value and add_or_sub subsets, which are not directly comparable with results from joint training on the full dataset.

5 Related Work

Related work has pursued systematic generalization through specialized architectures, data augmentation, metalearning, and hybrid neuro-symbolic methods. This paper instead emphasizes basic Transformer configuration choices and reports results across standard benchmarks that compare favorably with prior work.

  • Prior SCAN research includes specialized architectures, data augmentation, and metalearning, while CFQ has become an alternative benchmark.These approaches reflect broader efforts to improve compositional generalization.
  • Length generalization remains especially difficult, with prior solutions relying on hybrid task-specific neuro-symbolic approaches.This work isolates the EOS decision problem and reports that relative positional embeddings can mitigate it.
  • The paper analyzes training-stage effects because its experiments reveal large generalization-performance gaps not discussed in prior work.This connects systematic generalization to broader research on neural-network generalization across training stages.
  • Unlike sophisticated initialization proposals, this work revisits basic Transformer scaling methods and finds them particularly important for systematic generalization.The comparison concerns the simplicity of the configuration changes rather than a claim that they replace initialization methods.
  • Across PCFG, COGS, and CFQ, the paper reports better results than Ontañón et al. without intermediate representations.The related-work comparison notes that Ontañón et al. also studied relative positional encodings and Universal Transformers.
  • The study is limited to basic Transformer architectures, although the discussed details may also matter for other Transformer and algorithmic-reasoning architectures.The authors explicitly frame this as a scope boundary and possible relevance beyond the tested models.

6 Conclusion

Transformer performance on systematic-generalization datasets improves substantially when basic model and training configurations are revisited. Relative positional embeddings, early stopping, and embedding scaling also expose why IID validation can mislead model selection.

  • Relative positional embedding often outperforms absolute positional embedding and mitigates the EOS decision problem in length generalization.This leaves composition as the remaining problem for length generalization.
  • Reconsidering early stopping and embedding scaling greatly improves baseline Transformers, particularly on COGS and PCFG.
  • IID validation can conceal generalization differences, making a validation set for systematic generalization necessary for strong baselines.

A Evaluation Metrics

The evaluation uses sequence-level accuracy and average token-wise cross-entropy loss.

  • Sequence-level accuracy counts an output as correct only when every token in the sequence is correct.
  • Losses are reported as average token-wise cross-entropy loss.

B Hyperparameters

The experiments use standardized optimization and architecture settings while varying learning rate and selected configuration choices across tasks and Transformer variants.

  • All models use Adam with PyTorch default hyperparameters except for the learning rate, plus dropout probability 0.1 after attention heads and linear transformations.
  • Universal Transformer experiments use both No scaling and Positional Embedding Downscaling methods.
  • Standard Transformers with absolute positional embeddings test different scaling variants across datasets, selecting the best available methods for reported results.
  • Encoder and decoder use the same number of layers, while decoder embeddings and final softmax weights are tied.
  • Model parameter counts and representative execution times are reported for different models.

C Relative Positional Embedding

The relative-positional and embedding-scaling configurations combine distance-aware attention with alternative treatments of word and positional embeddings. The implementation keeps relative attention separate from absolute positional encoding and evaluates scaling choices across datasets.

  • Relative positional attention: Relative positional self-attention uses a decomposed attention matrix with content and relative-position components.The formulation also includes global content and position biases.
  • Relative positional attention: Hidden states, distance embeddings, query mappings, key mappings, and learned vectors form the inputs to the decomposed attention terms.
  • Relative positional attention: Sinusoidal positional embeddings represent relative positions that may be positive or negative.
  • Relative positional attention: Relative attention logits are scaled by 1/√dmodel before softmax.
  • Relative positional attention: Relative positional variants omit absolute positional encodings from word embeddings and use relative attention in every layer except the encoder-decoder interface.
  • Embedding scaling: Token Embedding Upscaling combines word and positional embeddings after multiplying word embeddings by √dmodel.
  • Embedding scaling: No scaling combines word and positional embeddings directly as Hi = Ewi + Pi.
  • Embedding scaling: Position Embedding Downscaling uses Kaiming-initialized word embeddings and scales positional embeddings by 1/√dmodel.

E Analyzing the Positively Correlated Loss and Accuracy

On some generalization splits, accuracy and loss increase together during training, contrary to the usual loss–accuracy relationship. Figure 5 provides a further analysis of this behavior.

  • Some datasets show accuracy and loss increasing together on their generalization splits during training.
  • This behavior is identified as a positively correlated relationship between generalization accuracy and loss.
  • Figure 5 further analyzes the jointly increasing loss and accuracy behavior.

F Accuracies on the IID Split

IID validation accuracy is generally uninformative about systematic-generalization performance: models can achieve perfect IID accuracy while their generalization accuracy varies widely. On COGS, generalization behavior also contrasts with the expected IID loss–accuracy relationship.

  • IID accuracy does not provide a useful signal for assessing final-model quality.
  • On COGS, high accuracy on the generalization split corresponds to higher loss, unlike the expected IID relationship.
  • 100% IID validation accuracy is achieved by nearly all reported settings, while generalization accuracy varies widely.Exceptions are standard Transformer on PCFG and the “place_value” module of the Mathematics dataset.

G Additional Results

Additional analyses examine training dynamics, positional embeddings, and Transformer variants across the evaluated datasets. Relative positional embeddings usually improve convergence speed and reduce accuracy loss from smaller batch sizes, while some variants fail to converge.

  • On COGS, both test loss and accuracy grow during training, contrasting with the expected IID behavior.
  • On CFQ MCD 1, increasing total loss decomposes into decreasing loss on good samples and increasing loss on bad samples.
  • Relative positional embeddings usually converge faster than absolute positional embeddings.Convergence speed is measured by the mean steps required to reach 80% of final model performance.
  • Universal Transformers benefit more from relative positional embeddings than non-universal Transformers in convergence speed.
  • Non-universal relative variants do not converge on PCFG and Mathematics “place_value”.
  • Relative variants experience less accuracy drop than big variants when CFQ batch size decreases.The CFQ comparison uses a big batch size of 4096 versus a small batch size of 512.
Loading 2108.12284v4…