Source-linked AI summary
Multilingual Distributed Representations without Word Alignment
Karl Moritz Hermann, Phil Blunsom
TL;DR
The paper addresses the gap between monolingual compositional representations and cross-lingual word-level embeddings. It learns multilingual sentence representations from parallel data without word alignments, achieving state-of-the-art cross-lingual document-classification results and semantic relationships across languages lacking direct parallel data.
Problem
Existing work focuses on compositional semantic representations in monolingual settings and multilingual representations mainly at the word level.
Method
The model learns shared multilingual representations by training sentence-level compositional embeddings so parallel sentences receive similar representations, without requiring word alignments.
Results
The approach achieves state-of-the-art results on cross-lingual document classification, while multilingual training further improves performance and enables semantic relationships across languages without direct parallel data.
Takeaways & Limitations
Sentence-level multilingual training can transfer semantic information beyond directly observed language pairs, including through additional languages and multilingual data.
Takeaways & Limitations
The evaluated additive composition function is simplified and does not account for word order or other effects that more complex compositional models might capture.
Abstract
from arXiv · showhide
Distributed representations of meaning are a natural way to encode covariance relationships between words and phrases in NLP. By overcoming data sparsity problems, as well as providing information about semantic relatedness which is not available in discrete representations, distributed representations have proven useful in many NLP tasks. Recent work has shown how compositional semantic representations can successfully be applied to a number of monolingual applications such as sentiment analysis. At the same time, there has been some initial success in work on learning shared word-level representations across languages. We combine these two approaches by proposing a method for learning distributed representations in a multilingual setup. Our model learns to assign similar embeddings to aligned sentences and dissimilar ones to sentence which are not aligned while not requiring word alignments. We show that our representations are semantically informative and apply them to a cross-lingual document classification task where we outperform the previous state of the art. Further, by employing parallel corpora of multiple language pairs we find that our model learns representations that capture semantic relationships across languages for which no parallel data was used.
1 Introduction
The paper combines compositional semantic representations with cross-lingual embeddings by learning sentence-level representations from parallel data without requiring word alignments.
- Existing work largely separates monolingual compositional representations from cross-lingual word-level representations.
- Sentence-aligned data offers a more plausible semantic correspondence than token-level alignment for some cross-lingual expressions.The German compound “Schwerlastverkehr” aligns naturally with the English phrase “heavy goods vehicle traffic.”
- The proposed model learns cross-lingual embeddings at the sentence level by combining cross-lingual embedding research with compositional semantic representations.
- The model does not require word-aligned data and can technically use sentence-level, document-level, or comparable data.The paper evaluates it with sentence-aligned data but states that word alignment is not theoretically required.
- The authors report that a simple compositional vector model outperforms the state of the art on cross-lingual semantic-transfer evaluation.
2 Models of Compositional Distributed Semantics
Prior research developed distributed semantic representations for words, larger structures, and cross-lingual transfer largely in separate settings. This work identifies multilingual representations beyond the word level as the next step.
- Word-level distributed representations use contextual information to encode semantic relationships and support tasks such as word similarity and synonym detection.
- Representations of phrases and sentences are needed for tasks where individual word semantics are insufficient.Compositional approaches represent larger structures as functions of their constituent representations.
- Earlier multilingual methods transferred word representations using bilingual data, including word-aligned parallel data, multitask learning, and autoencoders.
- Compositional semantic models have mainly focused on monolingual data, while cross-lingual representation learning has mainly focused on word-level embeddings.
- The paper presents inducing multilingual representations for larger expressions as a natural next step beyond predominantly word-level multilingual work.
3 Model Description
The model learns multilingual sentence-level representations by bringing encodings of parallel sentences together, using compositional vector models trained jointly across languages. Contrastive training prevents degenerate solutions by separating parallel from non-parallel sentences.
- Bilingual training signal: The model learns sentence-level cross-lingual embeddings from parallel data by encouraging semantically equivalent sentences to share representations.The approach uses sentence-level semantic similarity rather than requiring word alignments.
- Compositional vector models: A compositional vector model maps word vectors into representations of larger syntactic units through recursive composition.The general binary composition function combines child vectors with an encoding matrix, bias, and activation function.
- Compositional vector models: For evaluation, the model simplifies composition by setting weight matrices to identity and biases to zero, yielding additive sentence representations.This simplification produces a bag-of-words-style CVM that omits word order and related structural effects.
- Bilingual training signal: The bilingual objective minimizes distance between sentence encodings of parallel inputs and transfers semantic knowledge between the two language models.The sentence-root vectors represent the encoded sentences, and joint training avoids requiring one model to be fully trained beforehand.
- Contrastive objective: Contrastive estimation penalizes small distances between parallel sentences and sampled non-parallel sentences, preventing all embeddings from collapsing to zero.The final objective uses a hinge-loss formulation over negative sentence samples.
- Optimization: Because the joint objective is differentiable, the models can be trained with standard gradient-based optimization and backpropagation through structure.The paper mentions stochastic gradient descent, L-BFGS, and AdaGrad as possible optimization methods.
4 Experiments
The experiments train bilingual and multilingual models on parallel Europarl data and evaluate them on cross-lingual document classification and multilingual word representations. Adding English-French data improves classification and enables semantic similarity between French and German without direct parallel data.
- Experimental setup: Training uses AdaGrad for 50 iterations, with 50 noise elements, margin size 50, and embedding dimensionality d=40.The same iteration count is used for BICVM and BICVM+ for comparability, although BICVM converges faster.
- Experimental setup: BICVM uses 500k English-German sentence pairs, while BICVM+ adds 500k English-French pairs for 1 million English sentences overall.Each language uses distinct vocabulary encodings.
- Cross-Lingual Document Classification: The CLDC evaluation trains on one language and tests on the other using supervised data in only the training language.Experiments vary classifier training sizes from 100 to 10,000 documents and include both English-to-German and German-to-English directions.
- Cross-Lingual Document Classification: Both BICVM models outperform all prior work on cross-lingual document classification.BICVM+ additionally outperforms BICVM, indicating the usefulness of training data from a separate language pair.
- Visualization: BICVM+ learns semantic word-word similarity between French and German despite using no parallel French-German training data.The model evaluates English as a pivot language, with t-SNE visualizations covering English, French, and German words.
5 Conclusions
The paper introduces cross-lingual compositional representations trained from sentence-aligned data. Results are state of the art on CLDC, improve with multilingual training, and show cross-language semantic relationships without direct parallel data.
- 5 Conclusions: The proposed method induces cross-lingual distributed representations for compositional semantics.The approach uses sentence-aligned data and a simple composition method.
- 5 Conclusions: The method achieves state-of-the-art results on CLDC, a task designed to evaluate semantic transfer across languages.Adding multilingual training data in BICVM+ further improves the model.
- 5 Conclusions: The approach learns semantically related embeddings across languages without direct training data between those languages.The conclusion is supported by qualitative experiments and visualizations.
- 5 Conclusions: The approach requires little to no annotation and can use flexible training data, including sentence-aligned data.The authors propose exploring document-aligned or comparable corpora for low-resource languages.
- 5 Conclusions: Future work will test more complex compositional models and additional tasks such as machine translation and multilingual information extraction.The current results were obtained with a relatively simple CVM.