Source-linked AI summary
An Autoencoder Approach to Learning Bilingual Word Representations
Sarath Chandar A P, Stanislas Lauly, Hugo Larochelle, Mitesh M. Khapra, Balaraman Ravindran, Vikas Raykar, Amrita Saha
TL;DR
The paper seeks bilingual word representations that support cross-language learning without word-level sentence alignments. It trains autoencoders to reconstruct aligned sentence bag-of-words, adds correlation regularization, and evaluates the representations in cross-language classification. The approaches are competitive with the state of the art, with improvements of up to 10-14 percentage points over prior results.
Problem
Bilingual representation methods commonly require word-level alignments, while many languages lack annotated resources needed for strong NLP systems.
Method
A bilingual autoencoder reconstructs paired aligned-sentence bag-of-words and uses an explicit correlation regularizer to align the learned encoder representations.
Results
The approaches achieve state-of-the-art cross-language classification performance, improving over the best previously reported results by up to 10-14 percentage points.
Takeaways & Limitations
Meaningful bilingual word representations can be learned from fairly coarse sentence-level alignments without word-level alignments.
Takeaways & Limitations
The reported future work leaves extensions to bags-of-ngrams and multilingual representations for further investigation.
Abstract
from arXiv · showhide
Cross-language learning allows us to use training data from one language to build models for a different language. Many approaches to bilingual learning require that we have word-level alignment of sentences from parallel corpora. In this work we explore the use of autoencoder-based methods for cross-language learning of vectorial word representations that are aligned between two languages, while not relying on word-level alignments. We show that by simply learning to reconstruct the bag-of-words representations of aligned sentences, within and between languages, we can in fact learn high-quality representations and do without word alignments. Since training autoencoders on word observations presents certain computational issues, we propose and compare different variations adapted to this setting. We also propose an explicit correlation maximizing regularizer that leads to significant improvement in the performance. We empirically investigate the success of our approach on the problem of cross-language test classification, where a classifier trained on a given language (e.g., English) must learn to generalize to a different language (e.g., German). These experiments demonstrate that our approaches are competitive with the state-of-the-art, achieving up to 10-14 percentage point improvements over the best reported results on this task.
1. Introduction
The paper addresses unequal NLP resources across languages by learning bilingual word representations without word-level alignments. Its bilingual autoencoder uses aligned sentences and achieves state-of-the-art cross-language classification performance.
- Motivation: Resource-rich languages have stronger NLP tools because many other languages lack comparable annotated resources.English has high-quality POS taggers, parsers, and sentiment analyzers, unlike many languages including Hindi, Marathi, Bodo, Farsi, and Urdu.
- Motivation: Improving annotations in under-resourced languages is costly, time-consuming, and effort-intensive.
- Motivation: Aligned bilingual representations could transfer resources and NLP capabilities from resource-rich to resource-poor languages.Prior approaches seek common word or document representations across languages for this purpose.
- Prior work: Prior resource-transfer methods often relied on bilingual resources such as machine translation systems or word-level alignments.Recent common-representation methods aimed to eliminate the need for an MT system, but typically still used alignment information.
- Contribution: The proposed bilingual autoencoder reconstructs paired bag-of-words sentences while making their hidden representations predictive of each other.It requires aligned sentences rather than word-level alignments, and word embeddings are extracted from the encoder for supervised NLP tasks.
- Contribution: Up to 10-14 percentage point improvements over the best previously reported results were achieved on cross-language document classification.
2. Autoencoder for Bags-of-Words
The paper adapts autoencoders to sparse, high-dimensional bag-of-words observations using binary and tree-based reconstruction strategies. These designs address computational costs while learning word representations from summed or frequency-weighted word embeddings.
- Input representation: A bag-of-words sentence is encoded from word-presence information, with the goal of learning D-dimensional word representations.The representation discards word order and uses a fixed vocabulary of V words.
- Encoder-decoder: The encoder sums embeddings for words in a bag-of-words and applies a nonlinearity before a decoder reconstructs the original observation.The embedding matrix W has one D-dimensional column per vocabulary word.
- Binary reconstruction: Binary reconstruction converts each bag-of-words into a sparse V-dimensional vector indicating whether each vocabulary word is present.Training minimizes reconstruction cross-entropy, but reconstructing the full vector is slow for large vocabularies and millions of sentences.
- Binary reconstruction: Merged mini-batches reduce the number of parameter updates per epoch while retaining the efficiency of stochastic-gradient updates.The bags-of-words in each mini-batch are merged into one bag before optimization.
- Tree-based reconstruction: The second architecture models bags directly, using word-frequency-weighted sums or optionally averages of embeddings.A multinomial negative log-likelihood serves as the reconstruction loss, with averaging selected by cross-validation.
- Tree-based reconstruction: A probabilistic binary tree decomposes word probabilities, reducing decoder computation from vocabulary-linear normalization to logarithmic path evaluation.For a bag containing |x| words, at most O(|x| log V) decoder outputs are required in the worst case, with shared paths allowing reuse.
3. Bilingual autoencoders
The bilingual autoencoder learns aligned word representations from paired sentence bag-of-words by reconstructing sentences within and across languages. It also adds correlation-based regularization to make paired sentence embeddings more correlated, then uses the learned word vectors for document classification.
- Paired sentence bag-of-words provide the basis for learning aligned representations in two languages, without requiring word-level alignment.
- The model uses language-specific word matrices with the same embedding dimension and decodes representations into either language.The encoders share a bias before the nonlinearity to encourage representations on the same scale, while language-specific decoders retain their own parameters.
- Training jointly optimizes four losses: reconstructing each sentence from the other language and reconstructing each sentence from itself.
- Correlation regularization encourages encoder representations of paired sentences to be highly correlated and balances its contribution with the reconstruction terms.The correlation term can be used with either binary bag-of-words or tree-based reconstruction autoencoders.
- The learned word matrices represent documents as tf-idf weighted sums of word vectors for cross-lingual document classification.
4. Related Work
Earlier bilingual representation methods commonly rely on word-level alignments or bilingual resources, whereas related neural approaches also explore mappings between monolingual representations.
- Klementiev et al. and Zou et al. learn bilingual embeddings using neural language models and regularization based on aligned or frequently aligned words.
- Gao et al. learn a useful linear mapping between separately trained monolingual skip-gram models, but still require specified cross-language word pairs.
- The tree-based bilingual autoencoder illustration depicts shared parameters across input-to-hidden connections and, similarly, hidden-to-output connections.
5. Experiments
The experiments train bilingual embeddings on parallel English–German data and evaluate them by transferring a classifier trained in one language to documents in the other. They compare three autoencoder variants, including one with correlation regularization, under the established cross-language classification setup.
- The bilingual embeddings are learned from roughly 2 million English–German Europarl parallel sentences without word alignments.The corpus is tokenized, lowercased, and stripped of punctuation without removing stopwords.
- The cross-language classification setup trains on 10,000 single-topic documents and tests on 5,000 documents in each language.
- The procedure trains bilingual word representations, fits a classifier on Reuters documents in language X, and applies it to Reuters test documents in language Y.Monolingual Reuters documents may optionally reinforce the embeddings, with that choice crossvalidated.
- Three models are compared: BAE-tr with tree-based decoding, BAE-cr with reconstruction-error decoding, and BAE-cr/corr with correlation regularization.
- BAE-cr is identified as the worst-performing model, motivating evaluation of whether correlation regularization improves representation quality.
- All models use 40-dimensional word embeddings, train for up to 20 epochs, and accelerate reconstruction-error training by merging five adjacent sentence pairs.
6. Results and Discussions
The learned bilingual embeddings capture cross-language and semantic similarity, while correlation regularization improves cross-language classification, especially with limited supervision. Performance remains strong under some coarse alignment settings but can deteriorate with larger merged mini-batches.
- Qualitative embedding analysis: BAE-cr/corr places each selected English word near its German translation and semantically similar English and German words.Similarity is measured using Euclidean distance between the learned embeddings.
- Model comparison: Correlation regularization makes BAE-cr the best-performing method, exceeding other methods by more than 10% accuracy on EN-DE classification.BAE-tr is comparable to Klementiev et al.’s word-alignment-based embeddings.
- Effect of training size: BAE-cr/corr clearly outperforms the other evaluated embedding models at almost all supervised training sizes.The evaluation varies classifier training sizes from 100 to 10,000 examples in both EN-DE and DE-EN directions.
- Effect of training size: BAE-cr/corr performs remarkably well with only 100 supervised training examples, indicating strong generalization at low data sizes.The comparison includes BAE-tr, BAE-cr/corr, and Klementiev et al.’s embeddings.
- Effect of coarser alignments: Larger merged mini-batches can deteriorate performance, as observed for BAE-cr/corr on the DE-EN task.The coarser-alignment experiments compare merged mini-batch sizes of 5, 25, and 50.
7. Conclusion and Future Work
The model learns meaningful bilingual representations without word-level alignments, outperforming alignment-based and machine-translation baselines. Future work extends the bag-of-words model to phrases and multiple languages.
- The approach learns meaningful bilingual word representations without word-level alignments, even with fairly coarse sentence-level alignments.
- The model outperforms a state-of-the-art word-representation method using word-level alignments and a strong machine-translation baseline.
- Correlation-based regularization produces highly correlated bilingual embeddings that perform better on cross-language classification.
- Future work would extend the bag-of-words autoencoder to bags-of-ngrams so it can learn representations for short phrases.The proposed extension is intended to support machine-translation systems.
- Future work would convert the bilingual model into a multilingual model using different amounts of parallel data between languages.