Source-linked AI summary
Character-based Neural Machine Translation
Marta R. Costa-Jussà, José A. R. Fonollosa
TL;DR
Neural MT still struggles with very large vocabularies and morphologically rich languages despite state-of-the-art results. The paper replaces lookup-based source word representations with character-based embeddings using convolutional and highway layers, obtaining improvements of more than 3 BLEU points in English-to-German and about 1.5 points in German-to-English.
Problem
Neural MT remains challenged by very large vocabularies and morphologically rich languages, while lookup-based embeddings ignore word orthography and affixes.
Method
The system computes source word embeddings from characters with convolutional and highway layers within an attention-based bidirectional recurrent neural MT architecture.
Results
About 1.5 BLEU points improve German-to-English and more than 3 points improve English-to-German over the baseline systems.
Takeaways & Limitations
Character-based source embeddings provide unlimited-vocabulary, affix-aware representations and improve translation quality, including when the source language is not morphologically rich.
Takeaways & Limitations
The target vocabulary remains limited by the output softmax, and extending the character-based approach to the target side remains future work.
Abstract
from arXiv · showhide
Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional and highway layers to replace the standard lookup-based word representations. The resulting unlimited-vocabulary and affix-aware source word embeddings are tested in a state-of-the-art neural MT based on an attention-based bidirectional recurrent neural network. The proposed MT scheme provides improved results even when the source language is not morphologically rich. Improvements up to 3 BLEU points are obtained in the German-English WMT task.
1 Introduction
Neural MT achieves state-of-the-art results but remains constrained by vocabulary size. The paper replaces independent word representations with character-derived source embeddings to capture intra-word information and reduce unknown words.
- Neural MT achieves state-of-the-art results but has a strong vocabulary limitation and expensive parameter tuning.
- The proposed system computes each source word embedding from its characters rather than using an independent lookup vector.A CNN and highway network process characters before the attention-based encoder.
- Character-based source embeddings exploit intra-word information, especially for morphologically rich languages.
- The source input contains no unknown words, while the target vocabulary remains limited by the standard target-side setup.
- Up to 3 BLEU points of translation-quality improvement are obtained with the character-based neural MT architecture.
2 Neural Machine Translation
The baseline neural MT system encodes source sentences and decodes translations with an attention-based bidirectional recurrent architecture. Encoder and decoder parameters are trained jointly to maximize conditional translation probability.
- The encoder converts a source sentence into a sequence of hidden states, which the decoder uses to generate the translation.
- The encoder and decoder are jointly trained to maximize the conditional log-probability of the target sentence.
- Attention uses a bidirectional GRU encoder to improve performance on long sentences.
- Each target word depends on the decoder state, the previous target word, and an alignment-derived context vector.
3 Character-based Machine Translation
The character-based system replaces lookup-based source embeddings with representations built from character sequences. Convolution, max pooling, and highway layers produce fixed-length, affix-aware vectors, while target vocabulary remains limited.
- Standard lookup embeddings impose a finite vocabulary and ignore stems, prefixes, suffixes, and other affixes.
- Character-based embeddings address these drawbacks by representing words through their character sequences.
- Convolution filters spanning 1 to 7 characters process character vectors, followed by max pooling to form fixed-length word representations.
- Two highway layers refine the pooled representation, whose second-layer output replaces the standard source word embedding.
- The target vocabulary remains limited by the output softmax, although source affix-aware representations positively influence the network components.
4 Experimental framework
The experiments evaluate character-based neural MT on German-English WMT data using corpus statistics, baseline systems, translation examples, and BLEU results. Character-based neural MT improves translation quality and reduces unknown-word counts relative to the neural baseline.
- 4.1 Data: German-English WMT data included EPPS, NEWS, and Commoncrawl, with tokenization, truecasing, punctuation normalization, and language-based sentence filtering.Corpus statistics report sentences, words, vocabulary, and out-of-vocabulary words by set and language.
- 4.2 Baselines and setup: The evaluation compared phrase-based and neural baselines with character-based neural MT, including variants that post-process unknown words using the corresponding source word.The neural system used settings from prior work, while the character-based system was evaluated as CHAR.
- 4.3 Results: More than 3 BLEU points improved English-to-German translation, while German-to-English improved by almost 1.5 points.These comparisons concern the character-based neural MT results against the baseline systems reported in Table 3.
- 4.3 Results: The number of post-processing unknown words fell from 1491 to 1260 for German-to-English and from 3148 to 2640 for English-to-German.The reductions compare the neural baseline (NN) with character-based neural MT (CHAR).
- 4.3 Results: Translation examples attribute improvements to fewer source unknowns, handling morphological variation, and semantic disambiguation.The reported effects also include better alignment, reordering, and morphological generation.
5 Conclusions
The paper addresses neural MT's vocabulary-size limitation by modifying the encoder-decoder architecture to use character-based source word embeddings. The reported BLEU improvements are about 1.5 points for German-to-English and more than 3 points for English-to-German, while extending the approach to the target side remains future work.
- Neural MT remains challenged by limitations in vocabulary size despite its trainable, end-to-end advantages over earlier approaches.
- The proposed modification uses unlimited-vocabulary character-based source word embeddings in the standard encoder-decoder neural MT architecture.
- About 1.5 BLEU points improved German-to-English, and more than 3 BLEU points improved English-to-German.
- Extending the character-based approach to the target side is identified as further work.