Source-linked AI summary
Counter-fitting Word Vectors to Linguistic Constraints
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, Steve Young
TL;DR
Distributional word vectors can conflate semantic similarity with conceptual association, causing antonyms to appear similar and limiting domain-specific dialogue understanding. The paper counter-fits pre-trained vectors with synonymy and antonymy constraints from lexical resources or ontologies. It reports new state-of-the-art SimLex-999 performance and robust dialogue state tracking improvements across two domains, while limiting “improvement” to the specific task or purpose.
Problem
Distributional word vectors often conflate semantic similarity with conceptual association, causing antonyms to receive similar representations and creating problems for semantic judgments and dialogue state tracking.
Method
Counter-fitting is a lightweight post-processing method that fine-tunes pre-trained word vectors with synonymy and antonymy constraints from lexical resources or application-specific ontologies.
Results
The method achieves new state-of-the-art performance on SimLex-999 and improves dialogue state tracking across two dialogue domains.
Takeaways & Limitations
Counter-fitting can tailor word-vector spaces for semantic similarity judgments and downstream dialogue systems by separating antonymous words and injecting domain ontologies.
Takeaways & Limitations
The paper’s use of “improve” is purpose-specific: vectors tuned for semantic similarity are not expected to perform better on semantic relatedness.
Abstract
from arXiv · showhide
In this work, we present a novel counter-fitting method which injects antonymy and synonymy constraints into vector space representations in order to improve the vectors' capability for judging semantic similarity. Applying this method to publicly available pre-trained word vectors leads to a new state of the art performance on the SimLex-999 dataset. We also show how the method can be used to tailor the word vector space for the downstream task of dialogue state tracking, resulting in robust improvements across different dialogue domains.
1 Introduction
Distributional word vectors learn useful representations but often conflate semantic similarity with conceptual association, making antonyms appear similar. Counter-fitting injects synonymy and antonymy constraints to improve semantic similarity judgments and dialogue state tracking.
- Motivation: Distributional learning can make words with opposing meanings receive similar vectors because antonyms often occur in near-identical contexts.Examples include east and west, and expensive and inexpensive.
- Motivation: False synonyms can seriously mislead dialogue systems that must distinguish semantically different but conceptually related words.Restaurant dialogue state tracking may need to distinguish cheap from expensive preferences while generalising across variants such as cheaper and pricey.
- Approach: Counter-fitting fine-tunes pre-trained word vectors using synonymy and antonymy relations from lexical resources or application-specific ontologies.It is a lightweight post-processing procedure that does not require retraining the original vectors.
- Results: Counter-fitting produces more intuitive nearest neighbours than the original GloVe vectors and improves vector quality regardless of the input vectors.Table 1 compares nearest neighbours before and after counter-fitting.
- Results: The method achieves new state-of-the-art performance on SimLex-999 and improves dialogue state tracking across two dialogue domains.It injects domain ontology knowledge into vectors used to construct semantic dictionaries.
2 Related Work
Prior work mainly used lexical resources to bring related words closer or extracted antonymy rather than directly exploiting it. The paper positions counter-fitting as a lightweight post-processing alternative focused on separating similarity from antonymy.
- Similarity-focused methods: Earlier methods primarily brought semantically related words closer through modified training objectives, regularization, or lexical resources.Paraphrase Database-trained vectors had state-of-the-art SimLex-999 performance before this work.
- Scope: Semantic similarity and semantic relatedness are distinct objectives, so improving one does not imply improving the other.Antonymous concepts may be related while not being similar.
- Post-processing: Lightweight post-processing methods refine off-the-shelf vectors with lexical knowledge without requiring large corpora for retraining.Retrofitting uses similarity constraints from WordNet and other resources.
- Antonymy: Most prior antonymy research extracted antonym pairs from text or used them as features for contradiction and entailment detection.This contrasts with directly injecting antonymy constraints into word-vector spaces.
- Antonymy: Other approaches encoded antonymy through negative co-occurrences, Siamese networks, or embeddings specialised for antonymy.These methods used distributional data together with patterns or thesauri.
3 Counter-fitting Word Vectors to Linguistic Constraints
Counter-fitting transforms pretrained word vectors by injecting synonymy and antonymy constraints while preserving distributional structure. Its objective combines attraction, repulsion, and vector-space preservation, and the resulting representations support ontology-based semantic dictionaries for dialogue state tracking.
- Method: Counter-fitting starts with pretrained vectors V and produces transformed vectors V′ using antonymy constraints A and synonymy constraints S.The constraints contain indexed word pairs, and the objective contains three terms.
- Method: Antonym Repel pushes antonymous word vectors apart, while Synonym Attract brings known synonymous pairs closer together in V′.The antonym term uses a margin; the experiments set δ = 1.0, while the synonym term uses γ = 0.
- Method: Vector Space Preservation bends V′ toward the original topology to retain semantic information learned from large textual corpora.For efficiency, preservation distances are computed over precomputed neighborhoods rather than every vocabulary pair.
- Method: The training objective is a weighted sum of antonym repulsion, synonym attraction, and vector-space preservation terms.The three weights are nonnegative and were set equal in the experiments; stochastic gradient descent ran for 20 epochs.
- Dialogue state tracking: Domain ontology constraints and semantic lexicons are injected to create semantic dictionaries for dialogue state tracking, addressing exact-string matching limitations.Ontology antonymy constraints can distinguish values such as Chinese and Indian or expensive and cheap.
- Evaluation: Table 2 summarizes SimLex-999 performance for unaltered, retrofitted, and counter-fitted vectors, with retrofitting evaluated using the authors’ code and PPDB data.The table compares competitive scores across these vector configurations.
4 Experiments
Experiments evaluate counter-fitting with pre-trained vectors and lexical resources on semantic similarity and dialogue state tracking. The method improves SimLex-999 performance, corrects erroneous similarity predictions, and further improves tracking with ontology-informed dictionaries.
- 4.1 Word Vectors and Semantic Lexicons: 0.685 Spearman correlation on SimLex-999 was reported for counter-fitted Paragram-SL999 vectors, above the 0.67 average inter-annotator agreement.The authors also report that all model and vector results remain below the 0.78 average annotator agreement with the gold-standard ranking.
- 4.1 Word Vectors and Semantic Lexicons: Counter-fitting substantially improves both GloVe and Paragram-SL999 vectors, whereas retrofitting improves GloVe but not already specialised Paragram-SL999 vectors.The experiments attribute the broader gains to injecting antonymy relations.
- 4.2 Improving Lexical Similarity Predictions: Eight of seventeen highest-error Paragram word pairs are corrected by counter-fitting, including five pairs absent from the injected constraint sets.The corrected pairs have predicted-to-gold rank differences of 100 or less, indicating effects from indirect interactions in the cost function.
- 4.3 Improving Dialogue State Tracking: Ontology-informed semantic dictionaries improve dialogue-state-tracking performance over no-dictionary baselines, with counter-fitted vectors improving it further.Dictionaries treat vocabulary words within a tuned radius of slot values as rephrasings, using domain-specific antonymy constraints among slot values.
5 Conclusion
The paper presents counter-fitting as a lightweight way to inject linguistic constraints into word vectors. It improves semantic similarity judgments and supports domain-tailored semantic dictionaries for dialogue systems.
- 5 Conclusion: Counter-fitting postprocesses word vectors by injecting linguistic constraints, especially by separating representations of antonymous word pairs.The method is intended for tasks involving semantic similarity judgments.
- 5 Conclusion: Domain ontologies can tailor counter-fitted word vectors for constructing semantic dictionaries used in dialogue systems.The paper demonstrates this application for downstream dialogue-state-tracking tasks.