Source-linked AI summary
A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation
Junyoung Chung, Kyunghyun Cho, Yoshua Bengio
TL;DR
Existing machine translation systems generally rely on word-level modelling with explicit segmentation, motivating the question of whether neural translation can generate characters directly. The paper evaluates attention-based encoder-decoder models with subword sources and character or subword targets across four WMT’15 language pairs, and finds character-level decoders consistently stronger. Character-level ensembles surpass previous best neural and non-neural systems on three pairs and are comparable on En-Ru.
Problem
Existing machine translation systems rely almost exclusively on word-level modelling with explicit segmentation, leaving open whether neural translation can generate character sequences without segmentation.
Method
The paper evaluates attention-based encoder-decoder models with byte-pair-encoded source subwords and either subword-level or character-level target decoders across four WMT’15 language pairs.
Results
Character-level decoders outperform subword-level decoders on all four language pairs; ensembles outperform previous best neural and non-neural systems on En-Cs, En-De, and En-Fi and are comparable on En-Ru.
Takeaways & Limitations
The results strongly suggest that neural machine translation can translate at the character level and benefits from doing so.
Takeaways & Limitations
The source side is represented with subword symbols; character-level source representation remains for future investigation.
Abstract
from arXiv · showhide
The existing machine translation systems, whether phrase-based or neural, have relied almost exclusively on word-level modelling with explicit segmentation. In this paper, we ask a fundamental question: can neural machine translation generate a character sequence without any explicit segmentation? To answer this question, we evaluate an attention-based encoder-decoder with a subword-level encoder and a character-level decoder on four language pairs--En-Cs, En-De, En-Ru and En-Fi-- using the parallel corpora from WMT'15. Our experiments show that the models with a character-level decoder outperform the ones with a subword-level decoder on all of the four language pairs. Furthermore, the ensembles of neural models with a character-level decoder outperform the state-of-the-art non-neural machine translation systems on En-Cs, En-De and En-Fi and perform comparably on En-Ru.
1 Introduction
The paper asks whether neural machine translation can generate target character sequences without explicit word segmentation. It evaluates subword-encoded sources with character- or subword-level decoders across four WMT’15 language pairs, finding consistent advantages for character-level decoding.
- Motivation: Neural machine translation is examined as a way to translate directly between sequences without explicit target word segmentation.The question arises because existing systems predominantly use word-level modelling, while neural networks have issues specific to large word-level target vocabularies.
- Approach: The experiments use byte-pair-encoded source subwords and compare subword-level, character-level base, and character-level bi-scale decoders.The evaluation covers En-Cs, En-De, En-Ru, and En-Fi from WMT’15.
- Findings: Character-level decoders outperform subword-level decoders on En-Cs, En-De, En-Ru, and En-Fi.The paper tests both a naive stacked recurrent decoder and a bi-scale recurrent decoder on the target side.
- Findings: Character-level ensembles outperform previous best neural and non-neural systems on En-Cs, En-De, and En-Fi, while matching them comparably on En-Ru.Each ensemble uses eight independent models.
2 Neural Machine Translation
The neural machine translation model combines a bidirectional recurrent encoder, an attention-based decoder, and end-to-end training. At each target step, the decoder updates its state and uses soft alignment to summarize relevant source representations.
- Model Architecture: Neural machine translation maps a source-symbol sequence to a target-symbol sequence using an encoder network and a decoder network.The encoder produces a continuous representation of the input sentence, while the decoder generates the translation.
- Model Architecture: The bidirectional recurrent encoder reads the source sentence in both directions and concatenates forward and reverse hidden states into a context set.Each source position contributes a contextual representation to the set used by the decoder.
- Decoder: The decoder factorizes translation probability over target positions and updates its hidden state using the previous target symbol and a context vector.The context vector is computed from the encoder context set by the soft-alignment mechanism.
- Attention: Soft alignment weights encoder representations according to their relevance given what has already been translated.The scoring function produces an unnormalized score for each source representation based on the previous decoder state and target symbol.
- Training: The complete encoder, decoder, and soft-alignment model is trained end-to-end by minimizing negative log-likelihood with stochastic gradient descent.The decoder ultimately computes a distribution over possible target symbols.
3 Towards Character-Level Translation
The paper revisits character-level translation by contrasting word-level motivations, data-sparsity and recurrent-dependency challenges, and the difficulty of segmentation. It frames these issues as empirical questions about whether recurrent neural networks can generate long target character sequences.
- 3.1 Motivation: Neural translation represents sentences as sequences of indexed vocabulary tokens, whose one-hot vectors are transformed into learned embeddings.The embedding functions map one-hot token representations through linear transformations.
- 3.1 Motivation: Because neural networks learn token semantics in embeddings rather than assuming them, they can operate on words, subwords, or other symbol units.The paper uses this flexibility to motivate character-level modelling.
- 3.1 Motivation: Recent character-based translation approaches still depend on segmenting character sequences into words, ranging from simple tokenization to separately trained morpheme segmentation.The paper broadly uses segmentation for methods that split character sequences into subword symbols.
- 3.2 Why Word-Level Translation?: Character-level modelling intensifies data sparsity because subsequences become 4–5 times longer and the state space grows exponentially with subsequence length.This challenge is especially relevant to count-based components such as language models and phrase tables.
- 3.2 Why Word-Level Translation?: Word-level translation faces imperfect segmentation, which creates separate vocabulary entries for morphological variants that share a lexeme.The paper illustrates this with “run”, “runs”, “ran”, and “running”, which receive independent word vectors.
- 3.4 Challenges and Questions: Target-side character translation must generate long, coherent sequences, often 300–1000 symbols long, while modelling the required dependencies.The paper therefore evaluates these challenges empirically using recurrent neural networks focused on the target side.
4 Character-Level Translation
The paper tests character-level decoders using a standard GRU base decoder and a novel bi-scale recurrent network designed to capture character- and word-level timescales. The bi-scale architecture uses faster and slower layers whose gated interactions support multi-timescale decoding and alignment.
- Decoder Designs: The experiments compare a GRU base decoder with a novel bi-scale recurrent decoder for character-level translation.The base decoder tests whether an existing recurrent architecture can handle character-level decoding, while the bi-scale decoder explores a specialized alternative.
- Bi-Scale Architecture: The bi-scale recurrent network contains faster and slower hidden layers designed to model separate timescales.The faster layer models rapidly changing information, while the slower layer models more slowly changing information.
- Gated Updates: Gated interactions make the slower layer update only when the faster layer resets, constraining the two layers to operate at different rates.The faster layer influences the slower layer after processing the current chunk, rather than at every character step.
- Alignment: Soft alignment can use decoder-layer representations, with the slower layer intended to align larger target chunks to source subwords.The slower layer is motivated by the need to represent meaningful chunks across character-level decoding.
- Decoder Output: The decoder concatenates the faster and slower layer outputs to compute the probability distribution over target symbols.This combined representation is used as the final decoder output.
5 Experiment Settings
The experiments evaluate BPE-based source representations against subword- and character-level target decoders on four WMT’15 language pairs. They use controlled preprocessing, beam-search decoding, BLEU evaluation, and ensemble comparisons with phrase-based systems.
- Representation: Sources are represented with BPE subword symbols, while targets use either BPE symbols or characters.Target corpora are tokenized to facilitate comparison with word- and subword-level modelling.
- Corpora and Preprocessing: The study uses WMT’15 parallel corpora for En-Cs, En-De, En-Ru, and En-Fi, totaling 12.1M, 4.5M, 2.3M, and 2M sentence pairs.Only sentence pairs within specified source and target length limits are retained, and no monolingual corpus is used.
- Results Reporting: Table 1 reports BLEU scores for single models and ensembles, marking the best result in each category and summarizing available variation.The table uses boldface for the best single-model scores, underlining for the best ensemble scores, and subscripts or superscripts for minimum and maximum values.
- Models and Training: The compared settings are BPE→BPE, BPE→Char with a base decoder, and BPE→Char with a bi-scale decoder.GRUs are used for the encoder and for the first two decoder settings; the bi-scale network is used for the third.
- Decoding and Evaluation: Translations are generated with beam search using widths 5 for subword decoders and 15 for character decoders, then evaluated with BLEU.The beam widths were selected using development-set translation quality.
- Ensembles: Ensembles average output probabilities at each decoding step and are compared with state-of-the-art phrase-based translation systems.The ensemble evaluation covers all four language pairs.
6 Quantitative Analysis
Character-level decoders outperform subword-level decoders across the evaluated language pairs, with ensembles also matching or surpassing strong translation baselines. Alignment analysis further favors using the decoder’s slower layer for character-level models.
- Using only the slower decoder layer for soft alignment improves the character-level decoder on En-De.The authors attribute this to aligning a larger target chunk with a source subword unit.
- Character-level decoders outperform subword-level decoders across En-Cs, En-De, En-Ru, and En-Fi.
- On En-De, the character-level base decoder outperforms both the subword-level and character-level bi-scale decoders.
- The experiments suggest character-level translation can surpass word-level translation, but the evaluation restricts unsegmented characters to the decoder.
- Eight-model character-level ensembles outperform subword-level ensembles in every language pair and often exceed state-of-the-art non-neural systems.They outperform previous best neural and non-neural systems on En-Cs, En-De, and En-Fi, while matching results on En-Ru.
7 Qualitative Analysis
Character-level decoders remain coherent despite producing much longer sequences, improve modeling of rare words, and capture meaningful source-target chunks. This comes with a modest online decoding-speed reduction relative to subword decoding.
- Character-level translations are generally 5–10 times longer, yet show no significant coherence difference from subword-level translations.
- The character-level decoder assigns increasingly lower average negative log-probability to rarer words than the subword-level decoder.This pattern offers one potential explanation for character-level decoding’s experimental success.
- Character-level decoding captures alignments between source subwords and meaningful target-character chunks in examples such as “lights” with “Lichtersets.”
- 31.9 words per second is achieved by the subword-level base decoder, versus 27.5 and 25.6 for character-level base and bi-scale decoders.These online En-De measurements translate one sentence at a time on a single Titan X GPU.
8 Conclusion
The paper finds that neural machine translation can generate translations character by character without explicit target-side word segmentation and may benefit from doing so. Its evidence remains bounded by a subword-represented source side.
- Experiments on En-Cs, En-De, En-Ru, and En-Fi strongly suggest that character-level translation is feasible and beneficial for neural machine translation.
- The principal limitation is that the source side uses subword symbols rather than an unsegmented character sequence.The authors identify character-level source representation as a direction for future investigation.