Source-linked AI summary
Learning Joint Multilingual Sentence Representations with Neural Machine Translation
Holger Schwenk, Matthijs Douze
TL;DR
The paper addresses how to learn sentence representations that are independent of language and can be compared across languages. It uses a multilingual neural machine translation architecture and a scalable similarity-search evaluation. Across six languages, the learned representations achieve low similarity error and support semantically related cross-lingual nearest neighbors, while some training strategies and settings impose limitations.
Problem
The paper seeks fixed-size sentence representations independent of language, motivated by their potential to capture shared semantics across multilingual formulations.
Method
The authors use neural machine translation with multiple encoders and decoders, partial training paths, and an automatically scalable multilingual similarity-search evaluation.
Results
1.2% average cross-lingual similarity error across all 21 language pairs between six languages, with evaluation scaling to 1.4 million sentences.
Takeaways & Limitations
Sentences close in the learned embedding space are semantically highly related despite often differing substantially in structure and syntax, including across languages.
Takeaways & Limitations
The common-output-language training strategy cannot learn an embedding for that language because it is never used at the input.
Abstract
from arXiv · showhide
In this paper, we use the framework of neural machine translation to learn joint sentence representations across six very different languages. Our aim is that a representation which is independent of the language, is likely to capture the underlying semantics. We define a new cross-lingual similarity measure, compare up to 1.4M sentence representations and study the characteristics of close sentences. We provide experimental evidence that sentences that are close in embedding space are indeed semantically highly related, but often have quite different structure and syntax. These relations also hold when comparing sentences in different languages.
1 Introduction
The paper motivates fixed-size multilingual sentence representations that are independent of language and likely to capture shared sentence semantics. It uses neural machine translation to learn and evaluate these representations for cross-lingual applications.
- Background: Seq2seq systems encode an input sentence into an internal representation and generate the output sentence from it.Neural machine translation is described as a successful application of this encoder-decoder paradigm.
- Motivation: Multilingual sentence representations are designed to remain comparable across languages by using a fixed-size representation.The paper focuses on comparing representations for the same or multiple languages.
- Motivation: Shared meaning across differently formulated languages motivates using multilingual representations to capture underlying sentence semantics.The authors identify meaning as the common characteristic across multilingual formulations.
- Motivation: Such representations could transfer sentence-processing applications to other languages without language-specific training data.The listed applications include classification, sentiment analysis, and semantic similarity.
- Motivation: Multilingual sentence representations also have potential for multilingual search and a continuous-space interlingua.These are presented as motivations for learning a shared representation.
- Approach: The proposed approach uses neural machine translation with multiple encoders and decoders to train multilingual sentence embeddings.The paper presents the model and then evaluates it experimentally.
2 Architecture
The architecture combines multiple language-specific encoders and decoders through flexible partial training paths to learn shared sentence representations. Different paths trade off language coverage, training-data requirements, and representation availability.
- Architecture: The architecture contains one encoder and one decoder for each language, while each mini-batch can activate only selected models.Training paths may use pairwise or N-way parallel data, and can also use monolingual data.
- Training paths: Partial training paths include 1:1, M:1, 1:N, and M:N configurations that vary the numbers of source and target languages.The strategies range from one source and target to combinations of multiple inputs and outputs.
- Training paths: A shared decoder and limited model capacity provide an incentive for different encoders to produce the same representations.This is the rationale for using a common decoder across input languages.
- Limitations: The common-output-language strategy requires bitexts involving that language but cannot learn its input embedding because it is never used as input.Using the common output language as input would instead create an autoencoder, which may negatively affect training.
- Training paths: The 1:N strategy translates one input language simultaneously into all other languages, enabling embeddings for every language but requiring L-way parallel data.Each language serves as an input and has at least one target language in common with the others.
- Training paths: The most generic N:M strategy combines the preceding training techniques, illustrated with four-language paths in Figure 2.Figure 2 shows 1:1, 2:1, 3:1, and 1:3 configurations for English, French, Spanish, and Russian.
3 Evaluation protocol
The evaluation defines desired properties for multilingual sentence embeddings and introduces similarity search as an automatically scalable alternative to existing cross-lingual evaluations. It measures multilingual and semantic closeness while considering content preservation, scalability, corpus size, and language-pair coverage.
- Evaluation criteria: The protocol seeks multilingual closeness, semantic closeness, content preservation, and scalability to many languages.Semantic closeness allows similar meanings despite differences in syntax and word choice.
- Existing evaluations: Existing evaluations primarily use cross-lingual Reuters document classification and cross-lingual semantic textual similarity.STS scores semantic equivalence from 0, completely dissimilar, to 5, completely equivalent, across several language pairs.
- Similarity search: The proposed framework uses translated collections of S sentences in L languages, requiring no new human-labeled data and scaling to many languages and large corpora.Suitable L-way parallel resources include Europarl2, the UN corpus, and TED.
- Similarity search: Similarity search retrieves the closest sentence among S candidates and counts an error when it is not the reference translation.The procedure requires S^2 distance calculations and assumes the corpus contains no duplicate sentences.
- Similarity metrics: Cosine distance performed best among L2, inner product, and cosine, while normalized vectors make all three metrics equivalent.The evaluation reports how similarity error changes with the number of language pairs and corpus size.
4 Experimental evaluation
Experiments evaluate multilingual sentence embeddings across architectures, training strategies, languages, and domains. BLSTM max-pooling and one-to-many training perform strongly, while simple many-to-one averaging fails to improve the baseline; retrieved sentences are semantically related across languages.
- Experimental setup: The experiments use UN data spanning six languages, with 8.3M six-way parallel training sentences and predefined 4,000-sentence development and test sets.Texts are lowercased, limited to 50 words, and encoded with BPE using a 20k vocabulary.
- 4.1 Different network architectures: BLSTM max-pooling substantially outperforms LSTM last-state representations, especially as the number of languages increases.The gain is not explained solely by dimensionality: a 1024-dimensional LSTM reaches 1.36% similarity error.
- 4.1 Different network architectures: 1.01% similarity error is achieved on three languages by the all-language system, comparable to 1.03% when training only on those three languages.Increasing the number of languages does not appear to make joint embedding learning harder in this comparison.
- 4.2 Many-to-one training strategies: 0.92% similarity error from one-to-many training outperforms the 1:1 result of 1.01% and provides embeddings for all languages.The 1:1 strategy excludes the common output language from the sentence embeddings.
- 4.2 Many-to-one training strategies: Averaging three input representations in the 3:1 strategy does not learn joint representations, while 2:1 reaches 1.85% versus the 1:1 baseline of 1.03%.Adding 2:1 or 3:1 paths to alternating training increases similarity error, so more complex M:N paths were not attempted.
- 4.3 Large scale out-of domain similarity search: 7.7% similarity error is achieved by both evaluated training strategies on 1.46M Europarl sentences, compared with 12.2% for a 3x512 LSTM last-state model.The evaluation requires 1.46M^2 distances per language pair and uses FAISS brute-force L2 search.
- 4.4 Examples of multilingual search: Nearest-neighbor searches retrieve paraphrases and semantically related sentences within and across languages, with cosine distances reflecting decreasing relatedness.Cross-lingual examples show French and Spanish neighbors that are semantically related, while lower-scoring later neighbors are less related.
5 Conclusion
The NMT framework learns joint fixed-size multilingual sentence representations and supports scalable similarity evaluation across six substantially different languages.
- 1.2% average cross-lingual similarity error across all 21 language pairs between six languages.The languages differ significantly in morphology, inflection, and word order.
- The approach learns joint fixed-size sentence representations with interesting linguistic characteristics.
- The proposed multilingual similarity-search protocol scales to many languages and large corpora.
- Similarity error rates were also studied at a scale of 1.4 million sentences from an out-of-domain corpus.