Source-linked AI summary
SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, Stan Szpakowicz
TL;DR
Semantic-relation classification lacked a standard multi-way benchmark for comparing approaches. The paper defines SemEval-2010 Task 8, constructs its dataset and annotation process, and reports results from 28 systems, with performance generally well above random guessing despite the task’s difficulty.
Problem
The paper addresses the need for a standard testbed to compare approaches to automatic classification of semantic relations between nominal pairs.
Method
The authors define a nine-relation-plus-OTHER multi-way task, construct and annotate its dataset, and evaluate participating systems under controlled training-data sizes.
Results
Participating systems generally achieved performance well over an order of magnitude above random guessing on the non-trivial 19-way classification challenge.
Takeaways & Limitations
The task provides a benchmark for comparing semantic-relation classification systems and for studying distinctions among difficult, potentially overlapping classes.
Takeaways & Limitations
Creating trustworthy training data required more than 1000 person-hours, and the gain from TD3 to TD4 suggests that additional data would be beneficial but difficult to obtain.
Abstract
from arXiv · showhide
In response to the continuing research interest in computational semantic analysis, we have proposed a new task for SemEval-2010: multi-way classification of mutually exclusive semantic relations between pairs of nominals. The task is designed to compare different approaches to the problem and to provide a standard testbed for future research. In this paper, we define the task, describe the creation of the datasets, and discuss the results of the participating 28 systems submitted by 10 teams.
1 Introduction
SemEval-2010 Task 8 establishes a multi-way classification task for semantic relations between nominal pairs, motivated by broad applications of automatic relation recognition. It addresses task design challenges while expanding on an earlier SemEval benchmark.
- Semantic-relation recognition supports information extraction, summarization, machine translation, thesaurus construction, and several auxiliary language tasks.
- The task targets automatic classification of semantic relations between pairs of nominals.
- SemEval-2010 Task 8 uses multi-way classification over ten relations, replacing seven separate binary-labeled datasets from SemEval-1 Task 4.
- The new dataset contains 10,717 annotated examples, compared with 1,529 examples in SemEval-1 Task 4.
2 Dataset Creation
The dataset was built around a broad but mutually exclusive relation inventory, explicit truth-conditional and syntactic annotation rules, and multi-stage collection and adjudication. The resulting 10,717-instance dataset shows substantial variation in positivity and agreement across relations.
- 2.1 The Inventory of Semantic Relations: The relation inventory balances broad coverage against mutual exclusivity because no inventory fully satisfies both goals.
- 2.1 The Inventory of Semantic Relations: The inventory contains nine relations plus OTHER, including paired distinctions such as Entity-Origin versus Entity-Destination and Content-Container versus Component-Whole.
- 2.2 Annotation Guidelines: Annotations target relations true under the most plausible truth-conditional interpretation, excluding merely hypothetical or counterfactual cases.
- 2.2 Annotation Guidelines: Candidates use common-noun heads, require syntactic locality, and exclude discourse-dependent pronoun arguments.
- 2.3 The Annotation Process: Around 1,200 sentences per relation were collected through pattern-based Web searches, initially emphasizing positive and near-miss candidates rather than relation labels.
- 2.3 The Annotation Process: Two independent annotators labeled candidates using all nine relations plus OTHER, with systematic ambiguity illustrated by static versus dynamic containment examples.
- 2.3 The Annotation Process: Disagreements were resolved or examples removed, producing 10,717 instances split into 8,000 training examples and the remainder for testing.
- 2.3 The Annotation Process: Positive-instance rates averaged closer to 75% than 90%, while inter-annotator agreement ranged from 60% to 95% and varied by relation.
3 The Task
The task asks systems to predict a directional semantic relation for two tagged nominals in a sentence. Evaluation uses official scoring and controlled training-data sizes to compare system performance.
- Systems predict the relation and its direction for two tagged nominals given a sentence.
- The scorer outputs a confusion matrix, accuracy and coverage, per-relation precision, recall, and F1, plus micro- and macro-averaged metrics.
- The released training data included 891 re-annotated SemEval-1 examples, assigned to training so the test set remained unseen.
- The official metric is direction-sensitive macro-averaged F1 for nine relations plus OTHER, while additional scores include accuracy, coverage, and per-relation and aggregate P, R, and F1.
- Teams submitted predictions using 1,000, 2,000, 4,000, and 8,000 training instances, designated TD1 through TD4.
4 Participants and Results
The task compared 28 systems from 10 teams on semantic-relation classification, showing strong effects from training-data size, relation difficulty, and resource integration.
- Overall Ranking and Training Data: UTD submitted the best system, exceeding 82% F-score and outperforming the second-best system by more than 4%.FBK-IRST ranked second with 77.62%, narrowly ahead of ISI at 77.57%.
- Overall Ranking and Training Data: Increasing training data substantially improved all systems, while reducing the spread between the top three systems from around 10% at TD1 to below 5% at TD4.The transition from TD3 to TD4 produced diminishing returns for many, but not all, systems.
- The Influence of System Architecture: Neither Max-Ent nor SVM showed a clear advantage, and sequence models did not demonstrate a benefit.Two-step classification was used by some high- and lower-ranked systems, so the results did not establish a general advantage for that design.
- The Influence of System Architecture: Richer feature sets generally improved performance, but integrating semantic resources remained difficult because many resources also appeared in less successful systems.WordNet, Google n-grams, and deeper semantic resources were among the resources discussed.
- Relation-specific Analysis: Relation difficulty varied sharply: CE was easiest, whereas IA and PP were hardest, with recall problems and frequent misclassification as OTHER.PP’s difficulty was not attributed to questionable annotation because it had very high inter-annotator agreement.
- Difficult Instances: 152 examples were misclassified by every system, including cases requiring more lexical knowledge and complex reasoning than shallow models provided.Some cases also involved annotation errors or borderline examples.
5 Discussion and Conclusion
The task demonstrates that 19-way semantic-relation classification is challenging, while participating systems achieved performance well above random guessing. Results also suggest that larger training sets could improve performance, although creating trustworthy data required substantial human effort.
- 19-way classification is non-trivial because lexical-semantic classes are idiosyncratic and not necessarily disjoint.
- Participating systems performed well over an order of magnitude above random guessing.
- The TD3 →TD4 gain suggests that more training data could further improve performance.
- Constructing trustworthy training data, refining guidelines, and running the task took organizers more than 1000 person-hours.