Source-linked AI summary
TranSFormer: Slow-Fast Transformer for Machine Translation
Bei Li, Yi Jing, Xu Tan, Zhen Xing, Tong Xiao, Jingbo Zhu
TL;DR
Prior multiscale Transformers have largely treated subwords as basic units, leaving character-level features unexplored. TranSFormer combines a subword branch with a lightweight character branch and cross-granularity attention. It reports consistent BLEU improvements across three machine-translation benchmarks, with negligible added parameter and computational cost.
Problem
Prior multiscale Transformer work treats subwords as basic units and leaves finer-grained character features unexplored, despite the need to model interactions across linguistic granularities.
Source: §1 · p. 1Method
TranSFormer pairs a wide subword branch with a lightweight character branch and uses cross-granularity attention to exchange information between them.
Source: §1 · p. 2, §3.2 · p. 3Results
TranSFormer reports consistent BLEU improvements on three widely used machine-translation benchmarks.
Source: §5 · p. 9Takeaways & Limitations
The model reports consistent gains with negligible increases in parameters and computational cost, and robustness to suboptimal tokenization or subword segmentation.
Source: §1 · p. 2The current evaluation is limited to machine-translation tasks, and effective batching of variable word-boundary information remains an ongoing challenge for computational efficiency.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Learning multiscale Transformer models has been evidenced as a viable approach to augmenting machine translation systems. Prior research has primarily focused on treating subwords as basic units in developing such systems. However, the incorporation of fine-grained character-level features into multiscale Transformer has not yet been explored. In this work, we present a \textbf{S}low-\textbf{F}ast two-stream learning model, referred to as Tran\textbf{SF}ormer, which utilizes a ``slow'' branch to deal with subword sequences and a ``fast'' branch to deal with longer character sequences. This model is efficient since the fast branch is very lightweight by reducing the model width, and yet provides useful fine-grained features for the slow branch. Our TranSFormer shows consistent BLEU improvements (larger than 1 BLEU point) on several machine translation benchmarks.
1 Introduction
The introduction identifies a gap in multiscale Transformers: prior work uses subwords as basic units, leaving character-level features underexplored. TranSFormer combines a wide subword branch with a lightweight character branch and cross-granularity mechanisms; it reports consistent gains across three translation benchmarks with negligible added cost.
1 Introduction
- Prior multiscale Transformer work treats subwords as basic sequence units, leaving finer-grained character features unmodeled.Character sequences can be about 5× longer than subword sequences, and the model must account for interactions among linguistic units at different granularities.Source: §1 · p. 1
- TranSFormer pairs a wide subword branch that captures correlations with a thin character branch that learns fine-grained features, linked by cross-granularity attention.The attention layer enables the slow branch to use fine-grained features while passing optimized high-level input representations to the fast branch.Source: §1 · p. 2
- The model also uses character-to-word boundary information and boundary-wise positional encoding to represent within-word character positions.Experiments cover WMT’14 English-German, WMT’17 Chinese-English, and WMT’16 English-Romanian.Source: §1 · p. 2
- On Romanian tasks, TranSFormer yields consistent performance gains with a negligible increase in both the number of parameters and computational cost.It is also robust to errors caused by suboptimal tokenization or subword segmentation.Source: §1 · p. 2
2 Related Work
Related work explores multiscale Transformer representations through local patterns, multiple segmentation schemes, and multi-branch designs. The paper positions TranSFormer as extending this work by incorporating character-level features through a lightweight branch.
Multiscale Transformer
- Multiscale Transformer studies model local patterns or correlations across scales, including fixed windows, phrase structures, and subword, word, and phrase representations.UMST models correlations among subword, word, and phrase scales, but the passage says character-level features had not been explored in prior work.Source: §2 · p. 2
Character-level NMT
- Fully character-level translation can be flexible and, under full optimization, outperform subword systems, but long sequences make it time-consuming.Prior character-based studies include multilingual, low-resource, and typologically diverse translation settings, while effective multiscale learning on character features remains rare.Source: §2 · p. 2
Multi-Branch Transformer
- Prior Transformer branches include multiple self-attention branches, branch dropout, switching noise, and dual cross-attention over two subword representations.TranSFormer advances this line by investigating whether a lightweight branch can encode useful character features.Source: §2 · p. 2
3 Method
TranSFormer pairs a high-capacity subword branch with a thin character branch, exchanging cross-granularity information so the slow branch can use fine-grained features.
3 Method
- The encoder uses a large-capacity slow branch for subword sequences and a thin fast branch for character sequences, with the fast branch providing less precise complementary features.The slow branch models correlations among input units; the fast branch is designed to capture character correlations efficiently.Source: §3 · pp. 2–3
3.1 The Slow Branch for Subwords
- The slow branch applies a standard Transformer to mixed subword and word inputs, using a graph convolutional network to model within-word correlations before self-attention.A cross-granularity sublayer is placed between self-attention and the feedforward network.Source: §3.1 · p. 3
3.2 The Fast Branch for Characters
- The fast branch uses a tiny hidden size for long character sequences, character-level graph convolution for local features, and within-word relative positions for stable training.Boundary-wise positional encoding restricts the relative window to characters within each word.Source: §3.2 · pp. 3–4
3.3 Cross-Granularity Fusion
- Cross-granularity attention sits between self-attention and the feedforward network, enabling bidirectional information exchange between the independently operating branches.The fast branch can receive slow-branch features through attention, with its hidden size reduced by the key projection.Source: §3.3 · pp. 4–5
3.4 Interactions Between Encoder and Decoder
- Only the slow-branch output is exposed to the decoder by default; the paper also compares fused outputs and separate decoder attention to both branches.The fast branch is treated as an auxiliary source of fine-grained features.Source: §3.4 · p. 5
4 Experiments
TranSFormer is evaluated on three WMT translation tasks and shows BLEU gains over subword-only systems, while its ablations examine branch width, fusion, interaction placement, and input granularity.
4.1 Experimental Setups
- The experiments use WMT’14 En-De, WMT’16 En-Ro, and WMT’17 Zh-En, with task-specific data, validation sets, and test sets.En-De contains about 4.5 million sentence pairs, En-Ro 610K, and filtered Zh-En 18 million.Source: §4.1 · p. 5
4.2 Results
- Up to 1.16/1.05 BLEU gains over Slow-only on En-De base/big, about 1 BLEU over subword systems on Zh-En, and consistent En-Ro gains including over ODE Transformer with less compute.En-De adds 15% training cost with negligible inference latency; Zh-En outperforms prior local-pattern work while remaining computationally efficient.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6
4.3 Analysis
- Fast-branch widths of 32 for En-De base and 64 for Zh-En big perform best; increasing width further adds compute without further gains.A width of 16 still supplies useful fine-grained features and yields nearly 1 BLEU-point gains with modest additional parameters.Source: §4.3 · p. 7
- Downsampling character representations before concatenation or summation beats the Slow-only baseline but does not match CGA; linear-projection cross-attention performs similarly to CGA with more parameters.The authors suggest downsampling may impede optimization because text has a lower compression ratio than images.Source: §4.3 · p. 7
- Using the Slow branch as encoder output gives the highest performance; a decoder with separate Slow- and Fast-attending streams does not improve it further.The authors attribute the latter result to negative Fast-branch interactions that increase optimization difficulty.Source: §4.3 · p. 8
- Replacing the Fast branch’s character input with the Slow branch’s subwords degrades BLEU by up to 1 point, while different segmentations can still benefit the two-branch design.A character-level Fast branch with a SentencePiece Slow branch scores 28.60 BLEU, slightly above the subword-level variant.Source: §4.3 · p. 8
Ablations on interactions in different levels
- The default interaction frequency performs best on En-De; top-three-block interactions outperform bottom-three-block interactions, while last-block-only fusion is insufficient.TranSFormer also gains an average 1.1 BLEU over Slow-only across BPE settings, and improves consistently with increased encoder depth.Source: Figure 5 · p. 8, Ablations on interactions in different… · p. 9
- On En-De training subsets of 50K, 500K, and 1000K pairs, TranSFormer scores 11.87, 22.75, and 25.30 BLEU; it beats subword-only systems on Negation, Past, P-Gender, and P-Number.The model adds 0.3G/1.1G FLOPs in base/big configurations, with much of the added cost from cross-granularity upsampling and downsampling.Source: Table 7 · p. 8, Ablations on interactions in different… · p. 9, Figure 6 · p. 9
5 Conclusions
TranSFormer combines a subword-level slow branch with a character-level fast branch to use complementary features while preserving computational efficiency. It delivers consistent BLEU improvements on three machine-translation benchmarks.
5 Conclusions
- The two-branch encoder uses the slow branch for subword features and the fast branch for fine-grained character correlations, with the fast branch supplying complementary features.The paper presents the architecture as a potential universal multiscale learning framework.Source: §5 · p. 9
Limitations
TranSFormer’s current evaluation is limited to machine translation, while efficient batching of its cross-granularity design remains unresolved. Its use on other sequence tasks is proposed for future study.
Limitations
- Variability in word boundaries across sentences makes batching the cross-granularity design challenging for computational efficiency.The authors identify this as ongoing research and future work.Source: Limitations · p. 10
- The evaluation covers machine translation only; effectiveness on natural language understanding and other sequence-generation tasks remains unverified.The authors name abstractive summarization as an example for future evaluation.Source: Limitations · p. 10
- Longer sequences make the proposed natural language understanding and sequence-generation applications more challenging to encode.The authors describe TranSFormer as a potential versatile backbone for these tasks, with verification left to future work.Source: Limitations · p. 10