Source-linked AI summary
SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, Anna Korhonen
TL;DR
Verb semantics is important for language understanding, but existing evaluation resources provide limited evidence for analyzing verb representations. The paper introduces SimVerb-3500, a large, broadly covered human-rated verb-similarity benchmark, and finds strong annotation agreement alongside weaknesses in raw-text models for low-frequency and highly polysemous verbs.
Problem
Existing verb-similarity resources are too limited for statistically robust analyses, while verb semantics remains important for understanding sentence meaning.
Method
The paper constructs SimVerb-3500 with human similarity ratings for 3,500 verb pairs, broad USF and VerbNet coverage, and standardized development and test sets.
Results
Distributional models trained on raw text perform very poorly on low-frequency and highly polysemous verbs, with degradation partially mitigated by more principled contexts.
Takeaways & Limitations
SimVerb-3500 supports statistically robust comparisons of representation models across specific verb classes and analyses of diverse verb semantics.
Takeaways & Limitations
Spearman’s ρ can be problematic for categories such as antonyms because their rankings are arbitrary.
Abstract
from arXiv · showhide
Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.
1 Introduction
Verbs are central to sentence meaning and many language tasks, yet representation learning usually applies one approach to all word types. SimVerb-3500 addresses the resulting evaluation gap with broad, scalable verb coverage.
- Verbs encode event semantics, participant relations, and diverse syntactic and semantic behavior relevant to parsing, semantic role labeling, information extraction, and translation.
- Existing distributed representation methods typically use a single learning algorithm and representational form for every vocabulary word.
- Limited verb-focused evaluation resources make statistically robust analyses and comparisons of representation architectures practically impossible.
- The paper presents the dataset, annotation process, model evaluations, benchmark comparisons, and linguistic analyses enabled by its scale and coverage.
2 Related Work
Existing human-rated word-pair resources are too small, narrowly sampled, or dominated by nonverbs for representative verb-similarity evaluation. SimVerb-3500 is introduced as a more comprehensive remedy.
- Human-rated word-pair similarity datasets are insufficient for verb evaluation because of their small size or narrow verb coverage.
- RG-65 and WordSim-353 contain only 65 and 353 word pairs, respectively, while larger resources are dominated by noun pairs or low-frequency phenomena.
- Baker et al. and SimLex-999 include only 134 and 222 verb pairs, leaving the variety of verb semantic phenomena unrepresented.
- The paper presents SimVerb-3500 as a more comprehensive and representative resource for evaluating verb pairs.
3 The SimVerb-3500 Data Set
SimVerb-3500 was designed to satisfy representativeness, clear similarity annotation, and reliable scoring while covering diverse verb classes and phenomena. Its construction combines USF associations with VerbNet-guided sampling.
- The resource follows criteria requiring representative coverage, a clearly defined relation, and consistent, reliable judgments from untrained native speakers.
- SimLex guidelines define similarity ratings from 0 to 10, instructing annotators to rate antonyms low and distinguish similarity from relatedness.
- VerbNet contributes broad coverage of syntactico-semantic classes, including subcategorization frames, selectional preferences, and lexical-semantic verb classes.
- USF is a large English free-association database whose association strengths support sampling both related-but-nonsimilar and similar verb pairs.
- The sampling procedure filtered USF pairs, supplemented under-represented VerbNet classes, and added 347 unassociated pairs to reach 3,500 total.
- The dataset contains 827 distinct verbs, with 29 top-level VerbNet classes represented by 3 member verbs and 40 verbs outside any VerbNet class.
4 Word Pair Scoring
Verb-pair ratings were collected through a structured crowdsourcing survey with repeated pairs and checkpoint questions for quality control. Accepted ratings were averaged and rescaled to the dataset’s final range.
- Each of the 3,500 pairs was rated by at least 10 annotators across 70 tranches, with shared and duplicated pairs supporting consistency checks.
- Participants rated pairs on a 0–6 slider, while repeated pairs were used to detect inconsistent annotations.
- Three checkpoint questions tested whether annotators understood the guidelines and remained attentive throughout the survey.
- The study included 843 raters and produced over 65,000 ratings from screened native English-speaking participants.
- Ratings were excluded for failed checkpoints, duplicate-pair inconsistency, or suspicious patterns; accepted ratings were averaged and scaled from 0–6 to 0–10.
5 Analysis
SimVerb-3500 is evaluated with human-agreement measures, comparisons to existing benchmarks, and a broad set of representation models. Its larger, diverse verb coverage supports standardized development/test evaluation and reveals a substantial human–system gap.
- Human agreement: ρ = 0.84 for IAA-1 and ρ = 0.86 for IAA-2, indicating strong agreement among SimVerb-3500 annotators.IAA-1 averages pairwise rater correlations, while IAA-2 compares each rater with the average of the others.
- Model evaluation: The evaluation compares unsupervised text-based models with models using hand-crafted linguistic resources or curated knowledge bases.The model set includes SGNS variants, symmetric-pattern vectors, PMI-weighted vectors, sparse resource-based vectors, and paraphrase-based vectors.
- Benchmark comparison: 170 shared pairs produce ρ = 0.91 between SimVerb-3500 and SL-222, showing consistent ratings across the two datasets.
- Model evaluation: Paragram+CF achieves the highest SimVerb-3500 score, ρ = 0.628, below the IAA-1 value of 0.84.The same model performs substantially better on SL-222, where its score is on par with IAA-1 of 0.72.
- Evaluation design: SimVerb-3500 provides a 500-pair development set and a 3,000-pair test set, addressing the lack of standard splits in earlier scored word-pair datasets.The development set spans similarity ranges and top-level VerbNet classes; the test set contains the remaining pairs.
6 Evaluating Subsets
SimVerb-3500 supports subset analyses showing that model performance varies with verb frequency, polysemy, VerbNet class, and lexical relation, while human rating disagreement is largely stable across frequency and synset groups.
- Frequency: 390-490 verb pairs form each of three BNC-frequency groups, with both verbs in each pair assigned to the same frequency interval.
- Frequency: All models improve as verb-pair frequency increases, with steeper gains for distributional models and only slight effects for Faruqui and Dyer’s model.
- WordNet Synsets: All models decline as WordNet synset counts increase, but models using information beyond corpus co-occurrence remain more robust.Performance drops substantially beyond 10 synsets for these approaches, compared with beyond 5 synsets for other models.
- VerbNet Classes: Among 1,378 same-top-level-VerbNet-class pairs, 603 receive similarity scores below 5, indicating that within-class membership does not guarantee high human similarity.
- VerbNet Classes: VerbNet classes show clear performance differences, motivating finer-grained specialized representations for verb classes.
- Human Agreement: Average rating standard deviations are approximately 1.3 across frequency and WordNet-synset subsets, but relation-based disagreement differs: 0.86 for antonyms, 0.92 for unrelated pairs, and 1.34 for synonyms.Other relation subsets have average standard deviations of approximately 1.4.
7 Conclusions
SimVerb-3500 supports analysis and evaluation of how humans and machines represent verb meaning, including syntactico-semantic phenomena. Initial analyses reveal weaknesses in raw-text distributional models and motivate methods better suited to diverse verb semantics.
- SimVerb-3500 enables statistically robust comparisons of representation models across specific syntactico-semantic verb classes.Its size and coverage support analyses of model strengths and weaknesses.
- Existing representation-learning models were evaluated to demonstrate SimVerb-3500’s utility.
- Raw-text distributional models such as SGNS perform poorly on low-frequency and highly polysemous verbs.
- Focusing on principled contexts such as symmetric patterns can partially mitigate this performance degradation.
- The findings suggest that diverse verb semantics may require algorithms that learn quickly from few examples and flexibly represent sense distinctions.
- Improved understanding of verb diversity may support better encoding and interpretation of facts, propositions, relations, and events in language.
Vector Space Models
The evaluation compares distributional, pattern-based, count-based, knowledge-based, and retrofitted vector-space models. Models are trained or obtained using varied corpora, contexts, linguistic resources, and dimensionalities.
- The study evaluates models trained on two corpora because the models primarily learn from co-occurrence statistics in large text collections.The 8B corpus contains 8 billion tokens from various sources, while PW is English Polyglot Wikipedia.
- SGNS-BOW uses bag-of-words contexts with window size 2 and dimensionality 500 on 8B or 300 on PW.
- SGNS-UDEP and SGNS-DEP use universal-dependency or labeled Stanford-dependency contexts, respectively.SGNS-UDEP is trained on PW, whereas SGNS-DEP uses the 8B corpus.
- SymPat represents words using co-occurrence in symmetric patterns with an antonym-detection mechanism.
- Count-SVD uses PMI-weighted traditional count vectors with SVD dimensionality reduction, using window size 2 and dimensionality 500.
- Non-distributional vectors are sparse binary representations built from resources including WordNet, FrameNet, and lexical sentiment resources.
- Paragram retrofits word2vec vectors using PPDB paraphrase constraints, while Paragram+CF additionally injects antonymy constraints.