Source-linked AI summary
Not All Neural Embeddings are Born Equal
Felix Hill, KyungHyun Cho, Sebastien Jean, Coline Devin, Yoshua Bengio
TL;DR
The paper examines whether translation-based and monolingual embeddings capture the same kinds of conceptual information. It compares the embeddings on similarity and analogy evaluations, finding translation-based embeddings stronger on conceptual similarity and syntactic role, while monolingual embeddings better capture relatedness.
Problem
Method
The authors train translation models and monolingual baselines, then evaluate their English embeddings on similarity, synonym, and analogy tasks.
Source: §1.2 · p. 2, §2 · pp. 2–3, §2.2 · p. 4Results
Translation-based embeddings perform better on conceptual similarity and syntactic-role tasks, while monolingual embeddings better capture general relatedness and semantic analogies.
Source: §2 · pp. 2–3, §2.2 · p. 4, §3 · p. 4Takeaways & Limitations
The authors conclude that translation objectives seem particularly effective at inducing embeddings that encode ontological or functional information about concepts.
Source: §3 · p. 4Scarce bilingual corpora prevented testing translation models on increasingly large training datasets.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Neural language models learn word representations that capture rich linguistic and conceptual information. Here we investigate the embeddings learned by neural machine translation models. We show that translation-based embeddings outperform those learned by cutting-edge monolingual models at single-language tasks requiring knowledge of conceptual similarity and/or syntactic role. The findings suggest that, while monolingual models learn information about how concepts are related, neural-translation models better capture their true ontological status.
1 Learning embeddings from language data
Both monolingual language models and translation models learn word embeddings by updating them to serve a training objective. Their different objectives shape how embeddings are used: monolingual models predict words, while translation models encode source sentences and decode target sentences.
1 Learning embeddings from language data
- Training updates embeddings and model weights to help each model satisfy its objective.Models score training examples, compare the score with the objective, and backpropagate the error.Source: §1 · p. 1
1.1 Monolingual models
- Monolingual models predict either a sentence’s next word or nearby words, updating embeddings to improve those predictions.Skipgram-style models use cue and context embeddings and learn from nearby word pairs.Source: §1.1 · pp. 1–2
1.2 Translation-based embeddings
- Translation models encode source-language word sequences and use the resulting representation to decode a target-language sentence.Their source embeddings must support ordered-sequence representations, while target embeddings must support decoding correct sentences.Source: §1.2 · p. 2
2 Comparing Mono-lingual and Translation-based Embeddings
The comparisons show translation-based embeddings perform better on conceptual similarity and syntactic analogies, while monolingual embeddings better capture general relatedness and semantic analogies. Increasing monolingual training data does not close the reported similarity-task gap.
2 Comparing Mono-lingual and Translation-based Embeddings
- The comparison used English source embeddings from translation models trained on 300 million English–French words, alongside three monolingual baselines.The monolingual models were trained on the English half of the bilingual corpus or, for the full-sentence model, a larger 1-billion-word corpus.Source: §2 · p. 2
- Translation embeddings perform best on SimLex-999, while monolingual embeddings perform better on MEN and WordSim-353, which measure relatedness.SimLex-999 measures conceptual similarity; Table 1 also reports translation embeddings outperforming alternatives on similarity-focused evaluations.Source: §2 · p. 2, Table 1 · p. 3
- Translation-based embeddings outperform all monolingual embeddings on the TOEFL synonym test and a supervised synonym–antonym classification task.The TOEFL test uses nearest cosine neighbors; the classifier uses concatenated embeddings and 8-fold cross-validation.Source: §2 · p. 3
2.1 Quantity of training data
- Monolingual embeddings’ similarity-task performance converges below translation embeddings even when trained on increasingly large Wikipedia subsets.Figure 1 presents the training-corpus-size comparison; WordSim-353 results were similar to MEN.Source: §2.1 · p. 3, Figure 1 · p. 3
2.2 Analogy questions
- Scarce bilingual corpora prevented training the translation models on the larger datasets used to test monolingual embeddings.Source: §2.2 · p. 3
- Among 11,166 lexical analogies, translation embeddings are effective on syntactic analogies but perform poorly on semantic analogies.They outperform monolingual embeddings on syntactic analogies, even those trained on much more data.Source: §2.2 · p. 4, Figure 2 · p. 4
3 Conclusions
The conclusions distinguish the information captured by translation-based and monolingual embeddings: translation models better capture conceptual similarity and syntactic role, while monolingual models capture relatedness. The authors suggest translation objectives may help encode ontological information, while noting that the mechanism remains uncertain.
3 Conclusions
- Translation models better encode conceptual similarity and syntactic role, whereas monolingual models capture general inter-concept relatedness.The authors note that even the full-sentence monolingual model struggles with similarity, possibly because dissimilar words such as antonyms often occur in similar contexts.Source: §3 · p. 4
- Strong similarity-task performance is not simply attributable to richer syntax, since those evaluations use word groups or pairs with identical syntactic roles.The authors argue that success on SimLex-999 requires information approximating what concepts are, even when co-occurrence points otherwise [10].Source: §3 · p. 4
- The authors hypothesize that cross-language differences in meaning partitions may bring some source-language concepts closer, while antonyms remain distinguishable.They caution that the proposed account is uncertain and note that a clear difference such as gagner mapping to both win and earn may hinder similarity modeling.Source: §3 · p. 4