Source-linked AI summary
Multilingual Models for Compositional Distributed Semantics
Karl Moritz Hermann, Phil Blunsom
TL;DR
The paper addresses the challenge of learning compositional semantic representations across languages without requiring extensive linguistic supervision. It uses sentence-aligned parallel data and multilingual compositional objectives to learn shared word, sentence, and document representations. The models outperform prior state of the art on cross-lingual document classification, while their representations remain comparable to a machine-translation baseline despite using one embedding per document.
Problem
Prior compositional models typically require parse trees, word alignments, or annotated semantic signals, limiting their use for low-resource languages.
Method
The approach uses sentence-aligned parallel data with multilingual compositional vector models and a bilingual hinge-loss objective to align semantic representations across languages.
Results
The models outperform prior state of the art on cross-lingual document classification, and their overall performance is comparable to a machine-translation system.
Takeaways & Limitations
Bilingual signals enable representations to abstract away from monolingual surface forms into a deeper multilingual semantic space.
Takeaways & Limitations
The machine-translation baseline slightly outperforms the models on most language pairs, while accessing all document words rather than one document embedding.
Abstract
from arXiv · showhide
We present a novel technique for learning semantic representations, which extends the distributional hypothesis to multilingual data and joint-space embeddings. Our models leverage parallel data and learn to strongly align the embeddings of semantically equivalent sentences, while maintaining sufficient distance between those of dissimilar sentences. The models do not rely on word alignments or any syntactic information and are successfully applied to a number of diverse languages. We extend our approach to learn semantic representations at the document level, too. We evaluate these models on two cross-lingual document classification tasks, outperforming the prior state of the art. Through qualitative analysis and the study of pivoting effects we demonstrate that our representations are semantically plausible and can capture semantic relationships across languages without parallel data.
1 Introduction
The paper extends distributional learning to multilingual joint-space semantic representations using parallel data. It evaluates these representations on cross-lingual document classification, outperforming prior state of the art.
- Distributed word representations support NLP applications by capturing syntactic and semantic information.
- The approach extends the distributional hypothesis to multilingual data and learns meaning across languages in a shared semantic space.
- The model learns from parallel corpora and uses compositional representations to transfer semantic information across languages.
- The experiments outperform prior state of the art on cross-lingual Reuters document classification and show relevance across many languages using TED data.
2 Overview
The overview motivates cross-lingual compositional representations because parallel texts share semantics, while prior approaches largely learn multilingual representations only at the word level. The proposed method uses sentence-aligned data without parse trees, word alignments, or annotated training signals.
- The model uses a compositional vector model to build sentence and document representations from constituent word representations.
- Sentence-aligned parallel data provide the otherwise unsupervised learning signal, avoiding syntactic parse trees, word alignments, and annotated data.
- The paper extends multilingual representation learning from words to larger cross-lingual expressions such as sentences and documents.
3 Approach
The approach learns multilingual semantic representations from sentence-aligned parallel data without parse trees, word alignments, or annotated training signals, using contrastive objectives and compositional vector models. It extends composition recursively from words to sentences and documents, allowing document-level signals to train representations.
- Training assumptions: The method requires only sentence-aligned parallel data, avoiding syntactic parse trees, word alignments, and annotated training signals.Using multiple compositional vector models transfers information between languages and broadens the semantic context.
- Multilingual objective: The model learns multilingual semantic representations by forcing parallel sentences to share a representation of their common semantics across languages.The approach uses multilingual data and joint-space embeddings to abstract from language-specific surface realizations.
- Multilingual objective: A bilingual contrastive objective minimizes distances for equivalent sentence pairs while imposing a margin between aligned and sampled non-equivalent pairs.Noise samples provide additional sentence pairs that are unlikely to be semantically equivalent, and the hinge loss uses margin m.
- Two composition models: ADD composes a sentence by summing word vectors, whereas BI applies a non-linearity to bigram pairs and can learn word interactions that ADD cannot.ADD ignores sentence ordering, while BI is designed to capture bigram information.
- Document-level semantics: Document representations are learned recursively by composing sentence vectors with a second CVM, with document-level error signals propagated to individual words.Parallel documents can provide this signal even when their sentences are not aligned; aligned documents can combine document- and sentence-level signals.
- Experimental models: The experiments compare ADD and BI models with and without document-level composition and error signals, producing ADD, BI, DOC/ADD, and DOC/BI variants.This design directly evaluates the contribution of document-level learning.
4 Corpora
The experiments use Europarl for development, testing, and bilingual representation learning, and a massively multilingual TED-derived corpus for document-level evaluation. The TED training data contains 12,078 parallel documents across 12 language pairs and 1,678,219 non-English sentences.
- Europarl: Europarl v71 supplies development data, initial testing, and representations for cross-lingual document classification using English-German and English-French pairs.The final 100,000 sentences from each pair were reserved for development.
- TED corpus: The TED-derived corpus contains 12,078 parallel documents distributed across 12 language pairs.Approximately 10 percent of each language’s training data was removed to create a test corpus.
- TED corpus: The TED corpus includes 1,678,219 non-English sentences, while the number of unique English sentences is smaller because many documents have multiple translations.
- Pre-processing: Both corpora were tokenized and lowercased with cdec tools, and empty sentences together with their translations were removed.
5 Experiments
The experiments evaluate multilingual sentence and document representations on cross-lingual, multilingual, and monolingual classification tasks, plus qualitative word-embedding analyses. Across these settings, the models outperform prior or strong baselines and exhibit semantically meaningful cross-language structure.
- RCV1/RCV2 Document Classification: The experiments replicate cross-lingual document classification on Reuters RCV1/RCV2 using Europarl-trained embeddings and classifier transfer between English and German.Documents are represented by averaging their sentence vectors, and classifiers are trained with an averaged perceptron.
- TED Corpus Experiments: The TED experiments evaluate document classification across a wider range of languages using document-level models and multilingual training data.The TED corpus supports a multi-label classification task and enables evaluation of the DOC model at larger multilingual scale.
- RCV1/RCV2 Document Classification: The models outperform the prior state of the art on RCV1/RCV2, with BI models performing slightly better than ADD models.Adding a second language improves performance in both English-to-German and German-to-English directions, despite no additional German data for the multilingual extensions.
- TED Corpus Experiments: The models achieve performance comparable to the MT system on most language pairs, despite using one embedding per document rather than all document words during classifier training.The comparison indicates that the learned representations capture nearly as much information as the Naïve Bayes classifier’s raw-data input.
- TED Corpus Experiments: Joint DOC/ADD embeddings support classification across thirteen languages, including language pairs without parallel training data, indicating a shared multilingual semantic space.The overall performance compared with the paired-language results implies that the learned vectors are semantically meaningful.
- TED Corpus Experiments: In monolingual TED classification, the models outperform SENNA and Polyglot embeddings on all languages, even surpassing the Naïve Bayes system on several.The authors note that in-domain training may partly explain this result.
- Linguistic Analysis: BI+ t-SNE projections cluster English, French, and German words by semantic and gender-related relationships without parallel French-German training data.The neutral English expression “the president” appears near the midpoint between gender-specific translations, supporting semantically plausible shared embeddings.
6 Related Work
Prior work learns distributed representations mainly within single languages, while multilingual approaches use parallel or comparable data to transfer information across languages. Composition functions extend representations from words to larger structures such as phrases, sentences, and documents.
- Compositional vector models represent larger structures as functions of their constituent representations, addressing the insufficiency of individual word semantics.
- Simple composition functions support bigram semantic similarity, while matrix-vector, convolutional, and tensor models have been applied to sentiment, relational similarity, and dialogue tasks.
- Most distributed-representation research has focused on single languages, especially English because of its annotated resources.
- Multilingual representation methods include joint embeddings, comparable-document models, cross-lingual autoencoders, and approaches that transfer information through aligned or transformed representations.
7 Conclusion
The paper introduces multilingual word embeddings trained with parallel data and a multilingual objective for compositional vector models. These representations outperform the state of the art in cross-lingual document classification, while bilingual signals support abstraction beyond monolingual surface forms.
- The method learns multilingual word embeddings from parallel data using a multilingual objective function for compositional vector models.
- The approach extends the distributional hypothesis to multilingual joint-space representations.
- The learned vectors, combined with simple composition functions, outperform the state of the art on cross-lingual document classification.
- Bilingual signals help models abstract away from monolingual surface realisations into a deeper semantic space.