Source-linked AI summary
FewRel 2.0: Towards More Challenging Few-Shot Relation Classification
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou
TL;DR
FewRel 2.0 asks whether few-shot relation classifiers can adapt across domains and detect queries outside the sampled relations. It introduces a biomedical test set and NOTA setting, then finds that existing models and common techniques still struggle with both challenges.
Problem
FewRel omitted few-shot domain adaptation and none-of-the-above detection, leaving two real-world challenges insufficiently addressed in few-shot relation classification.
Method
The paper constructs FewRel 2.0 with a biomedical-domain test set and an explicit NOTA setting, then evaluates state-of-the-art models and proposed solutions.
Results
State-of-the-art few-shot models struggle on both new tasks, while commonly used domain-adaptation and NOTA techniques remain unsatisfactory.
Takeaways & Limitations
Few-shot domain adaptation and NOTA detection require further exploration as real-world challenges in relation classification.
Takeaways & Limitations
BERT-PAIR retains gaps of 8 points for 5-way 1-shot and 7 points for 5-way 5-shot between conventional and 50% NOTA settings.
Abstract
from arXiv · showhide
We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel.
1 Introduction
FewRel 2.0 extends few-shot relation classification with few-shot domain adaptation and none-of-the-above detection, addressing issues ignored by FewRel. Experiments indicate that state-of-the-art models still struggle on these new tasks.
- Motivation: FewRel 2.0 adds few-shot domain adaptation and few-shot none-of-the-above detection to the FewRel relation-classification benchmark.The task targets adaptation to a different domain and queries that do not match the sampled relations.
- Findings: Even state-of-the-art models struggle on the new few-shot domain-adaptation and NOTA tasks.Figure 1 compares the best current-model results on FewRel, few-shot DA, and few-shot NOTA.
- Motivation: Few-shot domain adaptation tests whether models can transfer across domains when the test domain differs substantially from the training data.The new test set is described as having great disparities from the original FewRel dataset.
- Motivation: Few-shot NOTA adds a classification option for queries that do not belong to the sampled relation classes.This extends the conventional N-way K-shot assumption that every query belongs to one of the sampled classes.
- Contributions: Few-shot NOTA remains underexplored, so the paper proposes solutions based on state-of-the-art few-shot models and evaluates them in the new setting.The authors present the task and experimental directions for addressing this challenge.
2 FewRel 2.0
FewRel 2.0 retains the N-way K-shot formulation while introducing a biomedical domain-shift test set and an explicit NOTA label. The NOTA setting allows queries outside the sampled relations, making classification more variable than conventional N-way K-shot prediction.
- Original FewRel: In the original N-way K-shot setting, each batch provides N relations, K support instances per relation, and a query whose label belongs to the sampled relation set.Models predict the query relation from the given support set S and relation set R.
- Few-Shot Domain Adaptation: FewRel’s training and test data share the Wikipedia and Wikidata domain, unlike real-world settings where training and test domains may differ.The paper motivates domain adaptation by contrasting Wikipedia training with other application domains.
- Few-Shot Domain Adaptation: The new domain-adaptation test set aligns PubMed biomedical literature with UMLS and contains 25 relations with 100 validated instances per relation.Instances were checked by at least two annotators, with a third used when annotations disagreed.
- Few-Shot Domain Adaptation: Few-shot domain adaptation uses the original FewRel training set, the newly annotated biomedical data for testing, and SemEval-2010 Task 8 for validation.The validation set differs in corpus and schema from both the original training data and the new test set.
- Few-Shot NOTA: Few-shot NOTA adds a label for queries expressing none of the given relations, whose variable semantic space makes detection difficult.Unlike conventional N-way K-shot classification, the correct label can be outside the sampled relation set.
- Few-Shot NOTA: The NOTA rate α is the proportion of test queries sampled outside the given N relations, with the remaining 1 − α queries drawn from those relations.The formalization expands the possible labels to {r1, r2, ..., rN, NOTA}.
3 Approaches for Few-Shot DA
The paper adopts adversarial training for few-shot domain adaptation, using an encoder and discriminator trained on corpora from the source and test domains. The discriminator identifies domains while the encoder seeks domain-invariant features.
- Related Approaches: Domain-adaptation methods include subspace mapping, domain-invariant spaces, feature augmentation, minimax estimators, and adversarial training.The paper highlights adversarial training as effective for finding domain-invariant features.
- Adversarial Training: The adversarial approach frames adaptation as a game in which the encoder generates domain-invariant features and the discriminator predicts their domain.This min-max interaction is intended to reduce domain-specific information in the encoded representations.
- Adversarial Training: Following Wang et al. (2018), the method uses a two-layer perceptron discriminator alongside the sentence encoder.Both components receive corpora from the training and test domains during few-shot learning-task training.
- Adversarial Training: The encoder and discriminator are optimized through a min-max objective using the training corpus C0 and test corpus C1.The notation defines [·]i as a vector element and identifies C0 and C1 as the training and test corpora.
4 Approaches for Few-Shot NOTA
The paper compares simple NOTA-as-an-extra-class handling with BERT-PAIR, which scores query-supporting-instance pairs to address few-shot NOTA.
- Treating NOTA as an extra class samples supporting instances outside the N relations and performs (N + 1)-way K-shot learning.
- BERT-PAIR pairs each query with every supporting instance and uses BERT sequence classification to score whether each pair expresses the same relation.
- For each candidate relation, BERT-PAIR calculates the relation score by averaging the pair scores across its supporting instances.
- The model includes NOTA in the relation set and optimizes predictions with the commonly used cross-entropy loss.
5 Experiments
The experiments evaluate few-shot relation classification under domain shift and NOTA conditions, comparing established baselines, adversarial training, and BERT-PAIR. Results show substantial domain-transfer degradation, partial gains from adversarial training, stronger overall performance from BERT-PAIR, and persistent difficulty as NOTA rates rise.
- Few-shot domain adaptation: All few-shot models suffer dramatic performance falls when tested on a different domain.
- Few-shot domain adaptation: Adversarial training improves results on the new test domain, yet still leaves large room for growth.
- Few-shot domain adaptation: BERT-PAIR outperforms all other few-shot models on both the 1.0 and 2.0 test sets.
- Few-shot domain adaptation: Using 1,000 additional training instances yields 72.30% for 5-way 1-shot and 80.50% for 5-way 5-shot.These results are 16 and 13 points higher than the current best results, respectively.
- Few-shot NOTA: Treating NOTA as the N + 1 relation is beneficial, but results still fall quickly as the NOTA rate increases.
- Few-shot NOTA: BERT-PAIR performs better under NOTA because of its binary-classification style and remains stable as the NOTA rate rises.
- Few-shot NOTA: At a 50% NOTA rate, gaps from the conventional setting reach 8 points for 5-way 1-shot and 7 points for 5-way 5-shot.The remaining gaps motivate further research on few-shot NOTA.
6 Conclusion
FewRel 2.0 extends few-shot relation classification with cross-domain adaptation and none-of-the-above detection, revealing that existing methods remain inadequate for both challenges.
- FewRel 2.0 adds a biomedical test set and a none-of-the-above setting to evaluate previously ignored challenges.
- Existing state-of-the-art few-shot models struggle on the new task.
- Commonly used techniques produce promising improvements but remain unsatisfactory for few-shot domain adaptation and few-shot NOTA.
- The paper identifies few-shot domain adaptation and few-shot NOTA as real-world challenges requiring further exploration.