Source-linked AI summary
FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun
TL;DR
Relation classification requires costly annotation and remains insufficiently studied in the few-shot setting, especially for long-tail relations. The paper introduces FewRel, constructs it through distant supervision followed by crowdworker filtering, adapts state-of-the-art few-shot methods, and evaluates them. These models outperform vanilla RC baselines but remain substantially below humans, while the dataset requires diverse reasoning skills and leaves few-shot relation classification open for further research.
Problem
Relation classification depends heavily on time-consuming annotated data, while long-tail relations suffer when training instances are scarce and few-shot NLP research lacks systematic study.
Method
The paper constructs FewRel by aligning Wikipedia sentences with Wikidata relations, filtering labels with crowdworkers, and adapting recent few-shot learning models for relation classification.
Results
Few-shot learning methods significantly outperform CNN/PCNN baselines using finetuning or kNN, but their performance remains far below humans on FewRel.
Takeaways & Limitations
FewRel is a challenging testbed showing that few-shot relation classification remains an open problem requiring further research and diverse reasoning skills.
Takeaways & Limitations
FewRel’s diverse ways of expressing the same relation make classification challenging and motivate incorporating commonsense knowledge or improved causal modules.
Abstract
from arXiv · showhide
We present a Few-Shot Relation Classification Dataset (FewRel), consisting of 70, 000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers. The relation of each sentence is first recognized by distant supervision methods, and then filtered by crowdworkers. We adapt the most recent state-of-the-art few-shot learning methods for relation classification and conduct a thorough evaluation of these methods. Empirical results show that even the most competitive few-shot learning models struggle on this task, especially as compared with humans. We also show that a range of different reasoning skills are needed to solve our task. These results indicate that few-shot relation classification remains an open problem and still requires further research. Our detailed analysis points multiple directions for future research. All details and resources about the dataset and baselines are released on http://zhuhao.me/fewrel.
1 Introduction
Relation classification depends on costly annotated data, while existing distant-supervision benchmarks underrepresent long-tail relations and contain labeling noise. FewRel frames RC as few-shot learning, introduces a supervised dataset, adapts recent methods, and evaluates them comprehensively.
- 58% of relations in NYT-10 are long-tail with fewer than 100 instances, and performance drops dramatically as training instances decrease.
- FewRel formulates relation classification as a few-shot learning task requiring models to classify relations from a handful of training instances.
- The paper proposes a new large supervised few-shot relation-classification dataset.
- The authors systematically adapt recent state-of-the-art few-shot learning methods for relation classification, with potential benefit to other NLP tasks.
- They conduct a comprehensive evaluation of few-shot learning methods on FewRel to identify promising research directions for relation classification.
2 FewRel Dataset
FewRel is constructed by aligning Wikipedia sentences with Wikidata relations through distant supervision, then using human annotation to remove incorrect labels. The resulting dataset contains 100 relations with 700 instances each and separate training, validation, and test relation sets.
- The construction pipeline first creates a distant-supervision candidate set, then has human annotators filter incorrectly labeled sentences into a clean relation-classification dataset.
- Wikipedia supplies the corpus and Wikidata supplies the knowledge base used to align sentences with relation facts.
- Entity linking identifies additional unanchored entities by applying named entity recognition, matching mentions to knowledge-base names, and linking successful matches.
- The candidate set keeps one instance per unique entity pair, removes relations with fewer than 1000 instances, and retains 1000 instances for each remaining relation.
- Each instance receives at least two agreeing annotations, with disagreements sent to a third annotator before finalization.
- The final FewRel dataset has 100 relations with 700 instances each, using 64, 16, and 20 relations for training, validation, and testing.
3 Experiments
The experiments formulate relation classification as few-shot prediction from support examples and evaluate multiple training strategies, encoders, and recent few-shot methods. They compare these approaches across four N-way K-shot settings, including human performance in the one-shot cases.
- 3.1 Task Formulation: Few-shot relation classification maps a relation set, support set, and query instance to a predicted relation.
- 3.1 Task Formulation: The experiments use four settings: 5 way 1 shot, 5 way 5 shot, 10 way 1 shot, and 10 way 5 shot.
- 3.2 Experiment Settings: FewRel evaluations cover vanilla CNN and PCNN encoders with finetune or kNN strategies, plus four adapted few-shot learning methods.The evaluated methods are Meta Network, GNN, SNAIL, and Prototypical Network.
- 3.2 Experiment Settings: CNN forms sentence embeddings through convolution, max-pooling, and nonlinear activation, whereas PCNN uses piecewise max-pooling.
- 3.2 Experiment Settings: Human labelers are evaluated only in 5-way 1-shot and 10-way 1-shot settings because the corresponding 5-shot tasks are considered easier.Labelers receive support instances and one test instance without relation names or extra information.
- 3.3 Baselines of Few-shot Learning Models: Meta Network uses fast weights generated by a meta learner alongside slow weights updated through classification loss.Fast weights are intended to help generalize to new tasks with very few training instances.
- 3.3 Baselines of Few-shot Learning Models: GNN propagates information among support and query instances, while SNAIL sequences support-label pairs before the query and applies temporal convolution and attention.
- 3.3 Baselines of Few-shot Learning Models: Prototypical Networks average support embeddings into class prototypes and classify queries by their distances to those prototypes.
4 Result Analysis and Future Work
FewRel results show that few-shot learning methods substantially outperform conventional CNN/PCNN baselines, yet remain far below human performance. The dataset’s diverse expressions and varied reasoning modes make relation classification challenging and motivate future work.
- Results: Few-shot learning methods significantly outperform CNN/PCNN models trained with finetuning or kNN.The experiments report this comparison as evidence that adapting few-shot methods to relation classification is promising.
- Results: Even the best evaluated few-shot model performs much worse than humans on FewRel.This performance gap identifies FewRel as a challenging testbed for relation classification and few-shot learning.
- Future Work: FewRel contains examples of the relation “educated at” that require different reasoning modes for classification.Table 5 illustrates the diversity of expressions for the same relation through differently marked head and tail entities.
- Future Work: The diversity of expressions for the same relation is the dataset’s most challenging characteristic.The authors suggest commonsense knowledge and improved causal modules as possible directions for future research.