PaperlayerView exact revision on arXiv

Source-linked AI summary

Applying the Transformer to Character-level Transduction

Shijie Wu, Ryan Cotterell, Mans Hulden

arXiv:2005.10213v2cs.CL

TL;DR

Character-level transduction has seen few transformer models outperform recurrent models, despite transformers’ success on word-level tasks. This paper studies batch size and feature-guided input encoding, and reports strong transformer results across four character-level tasks. It achieves state-of-the-art performance on morphological inflection and historical text normalization while outperforming recurrent baselines on grapheme-to-phoneme conversion and transliteration.

  • Problem

    Although transformers had outperformed recurrent models on several word-level tasks, recurrent models remained dominant in character-level transduction, often trained with substantially less data than machine translation.

    Source: §1 · p. 1

  • Method

    The paper evaluates transformers on four character-level tasks, examines batch size, and uses feature-invariant input encoding for feature-guided transduction.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    The transformer achieves state-of-the-art results on morphological inflection and historical text normalization and outperforms recurrent baselines on grapheme-to-phoneme conversion and transliteration.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Takeaways & Limitations

    When properly tuned, transformers can outperform recurrent transducers on common character-level tasks.

    Source: §3.2 · p. 5

    The paper leaves open what linguistic errors transformers make compared with recurrent models on these tasks.

    Source: §5 · p. 5

Abstract

from arXiv · show

The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. Yet for character-level transduction tasks, e.g. morphological inflection generation and historical text normalization, there are few works that outperform recurrent models using the transformer. In an empirical study, we uncover that, in contrast to recurrent sequence-to-sequence models, the batch size plays a crucial role in the performance of the transformer on character-level tasks, and we show that with a large enough batch size, the transformer does indeed outperform recurrent models. We also introduce a simple technique to handle feature-guided character-level transduction that further improves performance. With these insights, we achieve state-of-the-art performance on morphological inflection and historical text normalization. We also show that the transformer outperforms a strong baseline on two other character-level transduction tasks: grapheme-to-phoneme conversion and transliteration.

1 Introduction

Transformers perform well on word-level transduction, but recurrent models remained dominant in character-level tasks, where prior experiments often found transformers underperformed. The paper studies this gap and reports strong results across four character-level tasks.

1 Introduction

  • Although transformers had achieved strong results on word-level tasks, recurrent attention-based sequence-to-sequence models remained dominant in character-level transduction, where transformers had often underperformed.The introduction also notes that no system in the 2019 SIGMORPHON cross-lingual transfer shared task used a transformer.Source: §1 · p. 1
  • Character-level tasks often have far fewer training examples than machine translation, and the introduction questions whether transformers offer advantages for largely monotonic, letter-based morphology.The authors note that the 2018 SIGMORPHON high-resource setting provided approximately 10k training examples per language.Source: §1 · pp. 1–2
  • The transformer achieves state-of-the-art results on morphological inflection and historical text normalization and outperforms recurrent baselines on grapheme-to-phoneme conversion and transliteration.The authors also report that it outperforms the recurrent sequence-to-sequence baselines on all four tasks despite having fewer parameters.Source: §1 · p. 2
  • The paper identifies batch size as a key factor in transformer performance and introduces a technique for feature-guided character-level transduction.Figure 1 highlights batch-size dependence; Figure 2 depicts special position and type embeddings for features and characters.Source: Figure 1 · p. 1, §1 · p. 2, Figure 2 · p. 2

2 The Transformer for Characters

The paper adapts the transformer for character-level tasks with a smaller model and a feature-invariant encoding. The encoding removes feature-position effects while distinguishing features from word characters.

2 The Transformer for Characters

  • The transformer is a self-attention encoder-decoder with multi-head attention and feed-forward layers, skip connections, layer normalization, and sinusoidal positional embeddings.The decoder also attends to the encoder output between its self-attention and feed-forward layers.Source: §2 · p. 2
  • For character-level tasks, the authors use a smaller transformer with four encoder-decoder layers, four attention heads, 256-dimensional embeddings, and a 1024-dimensional feed-forward hidden layer.The model has around 7.37M parameters, excluding character embeddings and the linear mapping before softmax.Source: §2 · p. 2
  • To handle feature-guided transduction, the model assigns features zero positional encoding, counts positions only for characters, and uses a special token to distinguish features from word characters.This makes the relative distance between characters and features consistent when feature order varies.Source: §2 · pp. 2–3

3 Empirical Findings

The feature-invariant transformer achieves state-of-the-art results on morphological inflection and historical text normalization, and outperforms recurrent models on grapheme-to-phoneme conversion and transliteration. Its character-level performance depends strongly on batch size, while other hyperparameters also benefit the tasks.

3 Empirical Findings

  • The evaluation covers four character-level tasks, using 52 languages for morphological inflection and multiple datasets for historical text normalization.Transliteration results use development performance, and the authors lack access to its test set.Source: §3 · p. 3

3.2 New State-of-the-Art Results

  • The feature-invariant transformer improves morphological inflection results over ensemble-based systems by 0.27 points.Errors show no consistent advantage across linguistic phenomena, while the transformer advantage shrinks as output forms grow longer.Source: Figure 3 · p. 3, Table 2 · p. 4, §3.2 · p. 4

3.1 A Controlled Hyperparameter Study

  • The transformer only outperforms the recurrent baseline at batch sizes of at least 128, and performance increases steadily with batch size.The main experiments use batch size 400; applying the same learning-rate scheduler to the recurrent baseline does not produce similar improvements.Source: §3.1 · pp. 3–4
  • Using Adam, both β2 choices—the default 0.999 and 0.98 used by Vaswani et al. (2017)—benefit character-level transduction tasks as well (see Tab. 1).Source: §3.1 · p. 4

3.2 New State-of-the-Art Results

  • The transformer improves historical text normalization over the previous state of the art, while dropout 0.3 yields slightly better CERi than dropout 0.1.The table reports average test performance and marks subset results with s.Source: Table 3 · p. 4, §3.2 · p. 4
  • The transformer outperforms previously published strong recurrent models on grapheme-to-phoneme conversion and transliteration despite having fewer parameters.Dropout 0.3 is stronger for transliteration, while dropout 0.1 is stronger for grapheme-to-phoneme conversion.Source: Table 4 · p. 4, §3.2 · pp. 4–5

4 Related Work

Character-level transduction is dominated by attention-based LSTM sequence-to-sequence models, and prior work has focused on exploiting monotonic alignments. Approaches include using external alignments, explicitly modeling alignments, and enforcing monotonicity.

4 Related Work

  • Attention-based LSTM sequence-to-sequence models dominate character-level transduction.Character-level inputs and outputs often share large substrings, with frequently monotonic alignments.Source: §4 · p. 5
  • External monotonic alignments were used in approaches to morphological inflection, including the CoNLL-SIGMORPHON 2017 shared-task winner.Aharoni and Goldberg (2017) used alignments from the SIGMORPHON 2016 baseline; Makarov et al. (2017) used that approach.Source: §4 · p. 5
  • Explicitly modeling source-target character alignments with hard attention outperforms soft attention.Wu et al. (2018) report this comparison.Source: §4 · p. 5
  • Enforcing monotonicity in a hard-attention model improves performance.Wu and Cotterell (2019) report this finding.Source: §4 · p. 5

5 Conclusion

Large batch sizes and feature-invariant input allow the transformer to achieve strong performance on character-level tasks. Its error profile relative to recurrent models remains unclear, and how to enforce monotonicity in multi-headed self-attention is not evident.

5 Conclusion

  • Large batch sizes and feature-invariant input allow the transformer to achieve strong performance on character-level tasks.The conclusion identifies these as the conditions associated with strong performance.Source: §5 · p. 5
  • The transformer’s linguistic errors relative to recurrent models remain unclear, motivating detailed error analysis as future work.The passage points to Gorman et al. (2019) as an example of error analysis for recurrent models.Source: §5 · p. 5
  • How to enforce monotonicity in multi-headed self-attention is not evident; future work could enforce it strictly or pretrain the model to copy.The cited options are Wu and Cotterell (2019) and Anastasopoulos and Neubig (2019), respectively.Source: §5 · p. 5

Related papers

Loading 2005.10213v2…