Source-linked AI summary
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
Nikola Mrkšić, Ivan Vulić, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, Steve Young
TL;DR
Existing distributional vectors need richer semantic information and better support for cross-lingual and lower-resource settings. The paper introduces ATTRACT-REPEL, which injects mono- and cross-lingual lexical constraints, and reports state-of-the-art similarity results alongside gains in multilingual dialogue state tracking.
Problem
Distributional vectors require semantic constraints and cross-lingual resources to improve representations across languages and support language-understanding tasks.
Method
ATTRACT-REPEL jointly injects synonymy and antonymy constraints from monolingual and cross-lingual lexical resources into pretrained vector spaces.
Results
The method achieves state-of-the-art semantic-similarity results, improves dialogue state tracking across languages, and enables multilingual models that outperform monolingual models.
Takeaways & Limitations
Cross-lingual specialised spaces support semantic transfer and can bootstrap dialogue-state-tracking models for lower-resource languages.
Takeaways & Limitations
The approach is not suited to languages with no lexical resources, and German dialogue-state-tracking performance remains substantially weaker than English and Italian.
Abstract
from arXiv · showhide
We present Attract-Repel, an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. Attract-Repel facilitates the use of constraints from mono- and cross-lingual resources, yielding semantically specialised cross-lingual vector spaces. Our evaluation shows that the method can make use of existing cross-lingual lexicons to construct high-quality vector spaces for a plethora of different languages, facilitating semantic transfer from high- to lower-resource ones. The effectiveness of our approach is demonstrated with state-of-the-art results on semantic similarity datasets in six languages. We next show that Attract-Repel-specialised vectors boost performance in the downstream task of dialogue state tracking (DST) across multiple languages. Finally, we show that cross-lingual vector spaces produced by our algorithm facilitate the training of multilingual DST models, which brings further performance improvements.
1 Introduction
The paper introduces ATTRACT-REPEL, which injects synonymy, antonymy, and cross-lingual constraints into word vectors to improve semantic quality across languages and downstream dialogue systems.
- 1 Introduction: ATTRACT-REPEL tunes word vectors with synonymy and antonymy constraints that conventional distributional training may not capture.The method advances semantic specialisation by incorporating linguistic information from lexical resources.
- 1 Introduction: Cross-lingual ATTRACT-REPEL uses BabelNet relations to embed multiple languages in one vector space and transfer information from high-resource to lower-resource languages.The approach injects constraints between words from different languages.
- 1 Introduction: The nearest-neighbour examples show that most neighbours across three language-group spaces are meaningful synonyms or translations.The table covers Slavic, Germanic, and Romance groups, with English included in each collection.
- 1 Introduction: The work reports state-of-the-art results on multilingual SimLex-999 and significant intrinsic improvements for Hebrew and Croatian.Hebrew and Croatian are presented as comparatively low-resource languages.
- 1 Introduction: Specialised vectors improve dialogue state tracking in English, Italian, and German, while a single multilingual model outperforms monolingual models in each language.The paper also releases Italian and German DST datasets and reports potential for bootstrapping language-understanding models.
2 Related Work
Prior semantic-specialisation methods either train representations jointly with lexical information or post-process pretrained vectors; ATTRACT-REPEL extends the latter approach to cross-lingual spaces.
- 2 Related Work: Semantic-specialisation methods combine distributional information with lexical resources, either during training or as post-processing of pretrained vectors.WordNet, FrameNet, and PPDB are examples of commonly used resources.
- 2 Related Work: Joint-training approaches modify training priors, regularisation, or objectives to incorporate linguistic constraints, including constraints targeting antonymy.These approaches use distributional and lexical information together.
- 2 Related Work: Although joint models could theoretically match or exceed fine-tuning methods, their performance had not surpassed fine-tuning approaches.The comparison is stated for the reviewed semantic-specialisation literature.
- 2 Related Work: Fine-tuning methods refine pretrained vectors using synonymy, paraphrase, antonymy, semantic intensity, or rich concept-dictionary constraints.Retrofitting and related post-processing methods are prominent examples.
- 2 Related Work: ATTRACT-REPEL is a portable, lightweight post-processor that incorporates external knowledge into arbitrary vector spaces and ties languages into unified spaces.The paper reports state-of-the-art SimLex-999 performance and positive semantic transfer between languages.
- 2 Related Work: Cross-lingual embedding research commonly uses parallel, sentence-aligned, or other cross-lingual distributional signals to connect languages.The paper contrasts these approaches with its use of pre-existing linguistic resources.
- 2 Related Work: Shared cross-lingual spaces can improve monolingual similarity tasks and support bilingual lexicon induction, information retrieval, and transfer learning.These capabilities motivate constructing unified multilingual representations.
- 2 Related Work: ATTRACT-REPEL uses cross-lingual constraints from repositories such as BabelNet, addressing prior work's limited exploitation of such resources.The paper reports that its resulting spaces outperform five representative cross-lingual models.
3 The ATTRACT-REPEL Model
ATTRACT-REPEL specialises existing distributional vectors by injecting synonymy and antonymy constraints through mini-batch updates, while regularisation preserves compatible semantic information. It differs from counter-fitting through context-sensitive updates and standard L2 regularisation.
- Model overview: ATTRACT-REPEL injects synonymy and antonymy constraints into existing distributional vector spaces to improve semantic similarity.The procedure builds on Paragram and counter-fitting while providing a portable approach for arbitrary vector spaces.
- Mini-batch construction: Mini-batches contain synonymy and antonymy pairs, with negative examples selected from the remaining in-batch vectors.For synonymy pairs, negatives are closest to each endpoint; for antonymy pairs, negatives are furthest away.
- Constraint objectives: The cost function pulls synonymous words together and pushes antonymous words apart relative to their negative examples.The synonymy and antonymy terms enforce stronger relations than those expressed by the corresponding negative examples.
- Regularisation: A regularisation term pulls each updated vector toward its original distributional representation while preserving compatible semantic content.The regularisation constant is λreg, and bxi denotes the original distributional vector for xi.
- Comparison to prior work: ATTRACT-REPEL updates example pairs and their negative examples in context, unlike counter-fitting's context-insensitive attract and repel terms.It also uses standard L2 regularisation rather than preserving pairwise distances within an initial neighbourhood.
- Evaluation and optimisation: ATTRACT-REPEL significantly outperforms counter-fitting in both monolingual and cross-lingual intrinsic evaluations.The optimisation uses AdaGrad for five epochs without early stopping and tunes six hyperparameters.
4 Experimental Setup
The experiments cover sixteen languages and combine distributional vectors with monolingual PPDB and cross-lingual BabelNet constraints. This setup evaluates semantic specialisation across high- and lower-resource languages using shared cross-lingual spaces.
- Languages and vectors: The experimental setup covers sixteen languages, including English, German, Italian, Russian, Dutch, Swedish, French, Spanish, Portuguese, Polish, Bulgarian, Croatian, Irish, Persian, and Vietnamese.The first four are the languages of the Multilingual SimLex-999 dataset.
- Languages and vectors: The study uses established high-quality vectors for the four SimLex languages and trains skip-gram word2vec vectors for all sixteen languages.The supplied passage identifies the vector collections for English, German, Italian, and Russian.
- Constraint resources: PPDB supplies monolingual paraphrase constraints, while BabelNet supplies cross-lingual synonymy and monolingual and cross-lingual antonymy constraints.BabelNet links words across languages through shared Babel synsets; it is not used for monolingual synonymy.
- Cross-lingual setup: Cross-lingual constraints bring language-specific vector spaces together into a shared space intended to improve semantic content, especially for languages with limited monolingual resources.The evaluation includes six lower-resource languages without monolingual synonymy constraints.
- Resource availability: PPDB covers 22 languages and BabelNet provides cross-lingual mappings for 271 languages, enabling their joint use across varied resource settings.PPDB depends on large, high-quality parallel corpora, whereas BabelNet uses Wikipedia interlanguage links and statistical machine translation.
5 Intrinsic Evaluation
The intrinsic evaluation tests semantic specialisation with mono- and cross-lingual synonymy and antonymy constraints across multiple starting spaces and languages. ATTRACT-REPEL consistently improves semantic similarity performance, including for lower-resource languages and over prior methods.
- Evaluation Setup: Spearman’s rank correlation on SimLex-999 measures semantic similarity while distinguishing similarity from mere conceptual association.SimLex-999 assigns low ratings to related but nonsimilar pairs such as book and read.
- Evaluation Setup: ATTRACT-REPEL applies monolingual and cross-lingual synonymy and antonymy constraints to distributional vectors across English, German, Italian, and Russian.The experiments compare monolingual constraints with shared multilingual spaces and also compare ATTRACT-REPEL with counter-fitting.
- Cross-Lingual Results: Cross-lingual specialisation further improves all four SimLex languages, with Italian approaching the top-performing English vectors.Bilingual specialisation also improves substantially over monolingual specialisation across language pairs, indicating broadly language-independent gains.
- Comparison to Baselines: ATTRACT-REPEL achieves stronger performance than counter-fitting in all reported intrinsic experiments.The authors attribute this advantage to fine-grained updates and L2 regularisation.
- State-of-the-Art Results: 0.751 on original SimLex-999 and 0.674 on SimVerb-3500 establish new English state-of-the-art scores, exceeding prior scores of 0.706 and 0.628, respectively.The corresponding Multilingual SimLex-999 score is 0.71.
- Lower-Resource Languages: Tying lower-resource Hebrew and Croatian vectors to high-resource languages produces substantial improvements, with English transfers consistently strongest.The authors suggest this may support high-quality spaces for languages covered by BabelNet but absent from PPDB.
6 Downstream Task Evaluation
The downstream evaluation tests whether semantically specialised vectors improve dialogue state tracking and whether cross-lingual spaces support multilingual and low-resource transfer. Specialisation improves DST broadly, while intrinsic gains do not always predict downstream gains.
- 6.1 Dialogue State Tracking: DST maintains a belief distribution over possible dialogue states as users express goals through slot-value pairs.The evaluation uses the NBT model on task-oriented restaurant dialogues, including English, German, and Italian WOZ 2.0 data.
- 6.1 Dialogue State Tracking: Delexicalisation-based DST models use generic tags and semantic lexicons to handle unseen ontology values and their rephrasings.As ontologies grow, manually or automatically constructed dictionaries become a deployment bottleneck, motivating dictionary-free semantic specialisation.
- 6.1 Dialogue State Tracking: The NBT uses fixed initial word vectors, so semantically related unseen words can be recognised from their positions in the vector space.This makes semantic specialisation important because distributional proximity between antonyms can harm DST decisions.
- 6.3 Results and Discussion: Both mono- and cross-lingual specialisation improve DST performance over distributional vectors and five baseline bilingual spaces.Table 8 evaluates NBT-CNN accuracy across English, German, and Italian using language-specific vector-space subspaces.
- 6.3 Results and Discussion: Large SimLex improvements produce only minor DST gains, showing that intrinsic evaluation does not fully capture downstream vector usability.A bilingual baseline can outperform the proposed vectors for English DST despite weaker SimLex performance, though not for Italian or German.
- 6.3 Results and Discussion: Ontology grounding yields substantial multilingual-model gains, especially in the low-data setting, enabling transfer across the three languages.The multilingual model combines training data from all three languages and uses a shared English-grounded ontology with cross-lingual vectors.
- 6.3 Results and Discussion: German DST remains substantially weaker than English and Italian, reflecting challenges associated with cases and compounding.The authors release German and Italian datasets to support downstream evaluation for morphologically richer languages.
7 Conclusion
The paper presents ATTRACT-REPEL for injecting monolingual and cross-lingual linguistic constraints into word vectors. It reports strong intrinsic and DST results, including cross-lingual transfer and multilingual bootstrapping, while highlighting evaluation discrepancies and challenges for morphologically rich languages.
- 7 Conclusion: ATTRACT-REPEL jointly injects synonymy and antonymy constraints to create semantically specialised unified cross-lingual vector spaces.The method is designed as a portable approach for incorporating linguistic constraints into vector representations.
- 7 Conclusion: The resulting spaces achieve state-of-the-art performance on SimLex-999 and multilingual variants while improving representations for lower-resource languages.Lower-resource vectors are induced by tying them into bilingual spaces with high-resource languages.
- 7 Conclusion: ATTRACT-REPEL-specialised vectors improve the downstream task of dialogue state tracking, accompanied by the release of German and Italian datasets.The conclusion links intrinsic improvements with gains in DST and introduces two novel non-English evaluation resources.
- 7 Conclusion: Semantically rich cross-lingual vectors facilitate language transfer and bootstrapping belief-tracking models for new languages.The paper presents this as an effective method for transferring DST capability across languages.
- 7.1 Further Work: Intrinsic and task-based experiments reveal discrepancies between conclusions drawn from vector-space evaluation and downstream evaluation.The authors identify improving vector-space models for morphologically rich languages as a major direction for future work.