Source-linked AI summary
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Jason Lee, Kyunghyun Cho, Thomas Hofmann
TL;DR
Word-level translation requires segmentation and struggles with rare or out-of-vocabulary words, while character sequences are longer and harder to model. This paper introduces a fully character-level NMT system with convolutional source compression and tests shared multilingual encoders. Character-level models match or exceed subword systems, with especially strong gains in multilingual translation and quality surpassing bilingual models for three language pairs.
Problem
Word-level NMT requires explicit segmentation and struggles with rare or out-of-vocabulary words, while character representations create longer sequences and harder modeling challenges.
Method
The paper maps source character sequences directly to target character sequences and uses convolutional, pooling, and highway layers to shorten source representations.
Results
Character-level models match or exceed subword systems, with the strongest gains in multilingual many-to-one translation; multilingual quality surpasses bilingual models on three language pairs.
Takeaways & Limitations
Character-level translation can share one encoder across languages efficiently and naturally handle intra-sentence code-switching.
Takeaways & Limitations
The study leaves many-to-many translation and more thorough architecture and hyperparameter investigation for future work.
Abstract
from arXiv · showhide
Most existing machine translation systems operate at the level of words, relying on explicit segmentation to extract tokens. We introduce a neural machine translation (NMT) model that maps a source character sequence to a target character sequence without any segmentation. We employ a character-level convolutional network with max-pooling at the encoder to reduce the length of source representation, allowing the model to be trained at a speed comparable to subword-level models while capturing local regularities. Our character-to-character model outperforms a recently proposed baseline with a subword-level encoder on WMT'15 DE-EN and CS-EN, and gives comparable performance on FI-EN and RU-EN. We then demonstrate that it is possible to share a single character-level encoder across multiple languages by training a model on a many-to-one translation task. In this multilingual setting, the character-level encoder significantly outperforms the subword-level encoder on all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilingual character-level translation even surpasses the models specifically trained on that language pair alone, both in terms of BLEU score and human judgment.
1 Introduction
The paper addresses weaknesses of word-level NMT by removing explicit segmentation and operating directly on character sequences. It introduces a character-level model designed for both bilingual and multilingual translation.
- Motivation: Word-level NMT struggles with rare and out-of-vocabulary words, especially in morphologically rich languages.Increasing the target vocabulary to address this raises training and decoding complexity linearly with vocabulary size.
- Contribution: Character sequences avoid explicit segmentation and allow the model to learn word boundaries and internal structure from translation data.The proposed approach maps source character sequences directly to target character sequences.
- Contribution: The model uses convolutional, pooling, and highway layers to shorten source representations, making character-level training reasonably fast.This addresses the prohibitive training speed of an earlier purely character-level baseline.
- Results: On WMT'15, the character-to-character model outperforms a subword-encoder baseline on DE-EN and CS-EN and is comparable on FI-EN and RU-EN.
- Multilingual translation: A single character-level encoder translates German, Czech, Finnish, and Russian into English without increasing model size.The multilingual model is evaluated as a many-to-one translation system.
- Multilingual translation: The multilingual character-level model outperforms the subword-level baseline across all four language pairs and surpasses bilingual quality on three.The paper reports this superiority in both BLEU score and human evaluation.
2 Background: Attentional Neural Machine Translation
Attentional NMT uses an encoder, decoder, and attention mechanism within one neural network. The encoder represents the source, while attention supplies target-specific source context during decoding.
- Overview: NMT maps a source sentence X to a target sentence Y using one neural network with encoder, decoder, and attention components.
- Encoder: A bidirectional recurrent encoder reads the source in both directions and concatenates forward and backward hidden states into continuous representations.These representations form the source sequence that the decoder can access.
- Attention: At each target timestep, attention computes a context vector as a weighted sum of source hidden states.The weights represent how relevant each source token is to the current target token.
- Attention: The attention weights are normalized scores measuring compatibility between source symbols and target-side decoder information.The scoring network uses the previous target embedding, previous decoder state, and a source hidden state.
- Decoder: The decoder uses the source context vector to compute its hidden state and predict the next target symbol.
- Training: The complete model is trained end-to-end by minimizing negative conditional log-likelihood over paired source and target sentences.The objective is defined over the training sentence pairs and their target symbols.
3 Fully Character-Level Translation
Fully character-level translation removes segmentation but creates longer sequences, harder character-to-meaning mappings, and longer-range dependencies. The paper addresses these challenges while motivating character-level multilingual modeling.
- Motivation: Character-level translation avoids out-of-vocabulary problems, represents rare morphological variants, and removes dependence on explicit segmentation.Segmentation can be difficult across languages and may inject decisions unrelated to the translation objective.
- Motivation: Character-level models are naturally suited to multilingual translation because overlapping alphabets can support cross-lingual parameter sharing.Word-level systems require separate vocabularies for each language.
- Motivation: Fully character-level translation lets the model discover sentence structure and map symbol sequences to continuous meaning representations.
- Challenges: Character-level sentences are on average 6 times longer for DE, CS, and RU and 8 times longer for FI.The increased length creates major computational and modeling challenges.
- Challenges: Attention cost grows quadratically with sentence length because every target token attends to every source token.Reducing the source sequence length is therefore key to reasonable training and decoding speed.
- Challenges: Character encoders must learn a highly nonlinear mapping from long character sequences to meaning representations.The orthographic form of a word has an arbitrary relationship to its meaning.
- Challenges: Character-level encoders also need to model dependencies over longer timespans than word-level encoders.
4 Fully Character-Level NMT
The proposed char2char encoder uses convolution, pooling, and highway layers to shorten character sequences while capturing local patterns before recurrent encoding. Its dynamic, overlapping segment units avoid fixed subword segmentation.
- Encoder architecture: Convolutional, pooling, and highway layers shorten source character sequences before a bidirectional GRU resolves long-term dependencies.The shortened representation is passed through the recurrent layer instead of the full character sequence.
- Convolution: Filters with widths up to 8 characters extract character n-grams of varying lengths.A filter bank stacks outputs from multiple widths into one representation before ReLU activation.
- Max pooling with stride: Max-pooling selects salient features within non-overlapping stride segments, whose n-gram contexts can overlap across segment boundaries.Each segment embedding summarizes meaningful character n-grams, while the rightmost segment may capture a pattern beginning in the previous segment.
- Max pooling with stride: Attention operates over segment embeddings, shortening the source representation s-fold; s = 5 balances performance and training time.Smaller strides improve performance empirically but increase training time.
- Dynamic segmentation: Unlike BPE, the model dynamically discovers salient character patterns during training rather than fixing segmentation beforehand.The extracted character sequences can change over the course of training.
- Decoder: The two-layer character-level decoder uses attention over source segments to predict target characters.A feedforward network scores each source segment for the next target character.
5 Experiment Settings
The experiments compare bilingual and multilingual char2char models with subword-level baselines across four WMT’15 language pairs. Multilingual models translate German, Czech, Finnish, and Russian into English under matched core-model capacity and balanced data scheduling.
- Evaluation setup: The study evaluates char2char against bpe2bpe and bpe2char baselines on WMT’15 DE→EN, CS→EN, FI→EN, and RU→EN.Word-level models are excluded because prior work showed subword models outperform them on closed-vocabulary issues.
- Experimental scenarios: Bilingual experiments train separate models on one language pair, while multilingual experiments train one many-to-one model on all four pairs.The multilingual task translates German, Czech, Finnish, or Russian sentences into English.
- Model architecture: The multilingual char2char encoder uses more convolutional filters while keeping encoder and decoder sizes otherwise unchanged.This fixes the core translation module’s capacity while allowing detection of more character patterns.
- Data scheduling: Multilingual minibatches balance language-pair proportions according to each pair’s share of the full corpus.This scheduling gives roughly comparable update counts per full corpus pass and helps avoid early overfitting to one language.
- Cyrillic treatment: Russian Cyrillic characters are converted to Latin using ISO-9 to facilitate cross-lingual parameter sharing.The paper illustrates this conversion with Czech and Russian words for school and schools.
6 Quantitative Analysis
The experiments compare character- and subword-level translation in bilingual and multilingual settings using BLEU and human evaluation. Character-level models match or outperform subword baselines, while multilingual char2char is strongest across most language pairs and gains mainly in fluency.
- Character- vs. subword-level: Bilingual char2char outperforms both subword baselines on DE-EN and CS-EN, while matching bpe2char on FI-EN and RU-EN.
- Character- vs. subword-level: Multilingual char2char significantly surpasses the subword-level encoder consistently across all four language pairs.The authors associate this with discovering shared constructs between languages and more flexible capacity allocation.
- Multilingual vs. bilingual: Multilingual char2char exceeds bilingual char2char by 2.64 BLEU on FI-EN and 0.78 BLEU on CS-EN, with comparable results on DE-EN and RU-EN.
- Multilingual vs. bilingual: Multilingual translation provides less performance benefit at the subword level, and multilingual bpe2char requires more updates to reach bilingual bpe2char performance.The authors suggest that useful subword segmentation across languages is difficult.
- Multilingual char2char vs. others: Multilingual char2char is best on CS-EN, FI-EN and RU-EN and runner-up on DE-EN, outperforming single-pair models in three language pairs.This supports using one multilingual character-level model instead of N separate models for N language pairs.
- Human evaluation: BLEU does not fully capture translation quality, so the study also evaluates adequacy and fluency using standardized human ratings from approximately 1k turkers.The human evaluation compares bilingual bpe2char, bilingual char2char and multilingual char2char.
- Human evaluation: Human evaluation finds multilingual char2char tied or superior on adequacy, while its clearest advantages over other models appear in fluency.For FI-EN and RU-EN, it significantly outperforms all other evaluated models in fluency.
- Human evaluation: The proposed fluency explanation remains a conjecture because the authors leave confirmation of the English decoder’s language-model effect for future work.
7 Qualitative Analysis
Qualitative tests examine robustness, translation speed, and multilingual behavior on difficult inputs. Character-level models handle spelling errors, long and morphologically novel words, nonce words, and mixed-language sentences, with a modest speed cost.
- Robustness: Character-level translation remains robust to a few spelling mistakes, whereas word-based translation quality would severely drop for non-canonical forms.
- Robustness: For long concatenative words with ambiguous segmentation, char2char translates the German example more accurately than the subword-level model.The subword model’s segmentation of “Siebentausendzweihundertvierundfünfzig” is inaccurate.
- Robustness: Char2char correctly understands the novel morphological inflection “gesperrt,” a past participle of “sperren.”
- Robustness: The character-level model detects salient character patterns in nonce words and produces correct translations in constructed German-English examples.
- Code-switching: Both multilingual models generate reasonable translations for mixed-language sentences without a language identifier, although multilingual bpe2char fails on high language intermixing.The mixed inputs combine German, Czech and Russian; the models translate them into coherent English sentences.
- Training and decoding speed: Char2char models are approximately 35% slower to train than bpe2char baselines at the same batch size.Bilingual character-level models can be trained in roughly two weeks on a single Titan X GPU.
- Training and decoding speed: Bilingual char2char requires 71.71 minutes to translate 3,000 sentences, compared with 66.63 minutes for bilingual bpe2char.The comparison is online rather than batched.
- Overfitting: Multilingual models are less prone to overfitting than bilingual models, particularly for the low-resource FI-EN pair.The FI-EN validation BLEU trajectory shows rapid bilingual overfitting while multilingual training appears to regularize learning.
8 Conclusion
The paper shows that fully character-level translation can match or exceed subword-level models while supporting multilingual translation and code-switching. Its multilingual model is also more parameter efficient, although many-to-many translation and broader architecture studies remain future work.
- Fully character-level NMT learns word concepts and boundaries from translation alone, without explicitly hard-coded segmentation.
- Character-level models perform as well as or better than subword-level translation models, with especially pronounced gains in multilingual many-to-one translation.
- Multilingual character-level translation is more parameter efficient than bilingual models and naturally handles intra-sentence code-switching.
- Extending multilingual many-to-one models to many-to-many translation and investigating architectures and hyperparameters remain future work.