Source-linked AI summary
Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, Eneko Agirre
TL;DR
Relation extraction needs costly labeled data, motivating methods that work with few or no examples. This paper reformulates it as textual entailment using quick hand-made relation verbalizations and pretrained NLI models, achieving strong TACRED results and identifying no-relation detection as the main low-data gap.
Problem
Relation extraction is difficult in real-world settings because annotation is costly and labeled examples are often scarce across languages and domains.
Method
The paper verbalizes relations with templates, uses an entailment engine for zero-shot prediction, and fine-tunes it on labeled examples recast as entailment pairs.
Results
The method reports excellent zero- and few-shot TACRED performance, scales with larger language models, and achieves the best TACRED results to date when fully trained.
Takeaways & Limitations
Few-shot systems are especially effective at discriminating between relations, while their low-data performance difference mainly comes from identifying no-relation cases.
Takeaways & Limitations
The approach scores low on PER:OTHER_FAMILY, whose many specific relations are only sparsely covered by the templates.
Abstract
from arXiv · showhide
Relation extraction systems require large amounts of labeled examples which are costly to annotate. In this work we reformulate relation extraction as an entailment task, with simple, hand-made, verbalizations of relations produced in less than 15 min per relation. The system relies on a pretrained textual entailment engine which is run as-is (no training examples, zero-shot) or further fine-tuned on labeled examples (few-shot or fully trained). In our experiments on TACRED we attain 63% F1 zero-shot, 69% with 16 examples per relation (17% points better than the best supervised system on the same conditions), and only 4 points short to the state-of-the-art (which uses 20 times more training data). We also show that the performance can be improved significantly with larger entailment models, up to 12 points in zero-shot, allowing to report the best results to date on TACRED when fully trained. The analysis shows that our few-shot systems are specially effective when discriminating between relations, and that the performance difference in low data regimes comes mainly from identifying no-relation cases.
1 Introduction
Relation extraction is difficult in low-data settings because annotation is costly and real-world applications often provide few labeled examples. The paper reformulates relation extraction as entailment using manually constructed verbalizations and an off-the-shelf entailment engine.
- Relation extraction systems perform poorly when differing languages, domains, and annotation costs limit labeled examples.
- The paper reformulates relation extraction as entailment because elaborate relation descriptions are difficult for prompt-based language-model prediction.
- Manual relation verbalizations use entity-type constraints and require at most 15 minutes of labor per relation.The verbalizations support zero-shot inference and can also produce entailment pairs for few-shot fine-tuning.
- The entailment approach achieves results well above comparable state-of-the-art systems in zero- and few-shot TACRED settings and scales to large pretrained language models.The authors report the best TACRED results to date when using large amounts of training data.
2 Related Work
Prior work uses pretrained language models, prompts, and pivot tasks for relation extraction, but fully unseen labels and competing positive relations remain challenging. This paper targets the more demanding setting in which relation labels are fully unseen.
- Textual entailment reformulates an input task around whether a premise entails, contradicts, or is neutral to a hypothesis.
- TACRED is the largest and most widely used English relation-extraction dataset and is derived from the TAC-KBP relation set with crowdsourced labels.
- Prompt-based methods and pivot-task approaches use task descriptions, questions, or label verbalizations to support few- or zero-shot prediction.
- Earlier entailment-based relation-extraction work evaluated binary entailment rather than full relation extraction, avoiding competing positive labels and no-relation detection.
- Existing zero- and few-shot relation-extraction models often observe some labels during training, whereas this work addresses fully unseen labels.
3 Entailment for RE
The system converts relation labels into hypotheses, applies an NLI model to estimate entailment for each hypothesis, and selects a relation using probabilities and entity-type constraints. It supports zero-shot inference and fine-tuning on transformed labeled examples, with explicit mechanisms for no-relation cases.
- Zero-shot relation extraction: The system uses an input containing two entity mentions as premise and relation verbalizations as hypotheses, then returns the highest-probability compatible relation.Its workflow comprises verbalization, NLI scoring, and relation selection using probabilities and entity types.
- Verbalizing relations as hypothesis: Templates generate relation hypotheses by substituting the two entity mentions into verbalized relation descriptions.Multiple templates can represent one relation, while entity types disambiguate templates shared by different relations.
- Zero-shot relation extraction: Entity coherence filters relations whose allowed argument types do not match the two entities, assigning incompatible relations zero probability.
- Verbalizing relations as hypothesis: Specific verbalizations tested within the 15-minute limit had very low recall and were not selected.
- Detection of no-relation: No-relation detection either treats a dedicated no-relation template as another relation or applies a threshold to the highest relation probability.Without development data, the threshold is set to 0.5; with development data, it is estimated experimentally.
- NLI for inferring relations: Labeled relation examples become entailment pairs for fine-tuning, with positive templates labeled entailment and other sampled hypotheses labeled neutral or contradiction.No-relation examples receive entailment labels for the no-relation template, while positive examples receive contradiction labels for that template.
4 Experimental Setup
The experiments evaluate entailment-based relation extraction across TACRED low-resource settings, verbalization construction, pretrained NLI models, and comparisons with existing systems.
- Dataset and scenarios: TACRED evaluation covers zero-shot, few-shot, full-training, and data-augmentation scenarios with stratified splits preserving the original label distribution.The dataset contains 42 relation labels, including NO-RELATION; few-shot settings use approximately 4, 16, or 32 examples per relation.
- Dataset and scenarios: Zero-shot evaluation uses either no development data or approximately two examples per relation, with development examples adjusting hyperparameters rather than model parameters.The 0% setting uses no development data, while the 1% setting provides around two examples per relation.
- Dataset and scenarios: Few-shot evaluation tests approximately 4, 16, and 32 examples per relation, while full training uses all available training and development data.The development set is reduced using the same ratio as the training data in few-shot settings.
- Pre-Trained NLI models: The study tests ALBERT, RoBERTa, BART, and DeBERTa v2 NLI models, using all models for zero-shot and full training but RoBERTa and DeBERTa for few-shot experiments.Data-augmentation experiments use only RoBERTa; models differ in architecture, size, and pretraining objectives and are mainly fine-tuned on MNLI.
- Baselines and implementation: The experiments run three random-seed trials, tune learning rates on the development set, and compare against SpanBERT, K-Adapter, LUKE, and a vanilla RoBERTa baseline.The selected learning rate is 4e−6, and the compared systems have a comparable number of parameters.
5 Results
On TACRED, the entailment-based approach performs strongly in zero-shot and few-shot settings, scales with larger models and more training data, and remains competitive under full training.
- Zero-Shot: The two DeBERTa v2 models achieved the best F1 scores, and small MNLI differences expanded into larger relation-extraction performance gaps.The reported examples include a 1.5-point MNLI difference becoming a 7-point gap in one RE setting and a 1-point gap in another.
- Zero-Shot: 57.8 F1 was the maximum in the most challenging default-threshold setting, while roughly 2 examples per relation significantly improved results.The experiments varied the threshold T and used the 1% development setting for about 2 examples per relation.
- Zero-Shot: 63.4% F1 with all development data was only 0.6 points better than using 1%, showing that a small number of examples suffice to set the threshold.The largest improvement occurred between 0% and 1% development data, while performance was nearly flat from 1% to 100%.
- Few-Shot: 48, 22, and 13 points of improvement over the vanilla RoBERTa baseline were obtained by NLIRoBERTa across the three few-shot settings.The competing K-Adapter and LUKE systems improved over the baseline but remained below NLIRoBERTa.
- Full Training: 71.0 F1 was achieved by NLIRoBERTa with full training, while NLIDeBERTa achieved the best TACRED results to date.NLIRoBERTa improved from 67.9 to 71.0 with additional training data; NLIDeBERTa benefited from a larger pre-trained NLI model.
- Data Augmentation: NLI-based zero-shot and few-shot models produced silver data that significantly improved a traditional RE baseline across all evaluated gold-data settings.Using silver annotations also offers easier integration into existing pipelines through a trained Transformer model.
6 Analysis
The zero-shot system distinguishes most positive relations well, but its main weakness is recognizing no-relation cases. Errors also arise from overlapping relations and incomplete coverage of broad relation categories.
- Auxiliary metrics: The zero-shot system is only 7 points below the fully trained system on positive-relation discrimination but 18 points behind on positive-versus-no-relation detection.Tuning the T threshold narrows the positive-versus-negative gap to 10 points, while the positive-relation gap remains the stronger result.
- Confusion analysis: 85.6% accuracy accompanies a strong confusion-matrix diagonal, indicating proper discrimination among most relations except the no-relation column.The matrix is row-normalized, so the diagonal represents recall for each true label.
- Confusion analysis: Most relation-to-relation confusions involve overlapping labels, including organization membership relations and concurrent origin, birth-country, and residence relations.These relations can describe similar or simultaneously valid situations, making their distinctions difficult.
- Confusion analysis: PER:OTHER_FAMILY performs poorly because it groups many specific relations while templates cover only a handful of them.The limitation reflects incomplete verbalization coverage rather than ordinary class-frequency imbalance.
7 Conclusions
The paper reformulates relation extraction as entailment and finds that simple hand-made verbalizations remain effective with limited manual effort. Larger language models and available labeled examples yield strong zero-shot, few-shot, and fully trained TACRED results.
- 7 Conclusions: 15 minutes per relation suffices for hand-made verbalizations that produce excellent zero-shot and few-shot results.The templates are designed to keep manual construction practical while supporting entailment-based relation extraction.
- 7 Conclusions: The entailment reformulation makes effective use of labeled examples and, with larger language models, achieves the best TACRED results reported to date.The conclusion links the method's data efficiency with its scalability to larger pretrained models.
- 7 Conclusions: The main performance gap against supervised models comes from discriminating no-relation examples, while performance among positive examples is stronger.This conclusion is consistent with the auxiliary-metric analysis.
A Pre-Trained models
The experiments use pretrained natural-language-inference models from the Transformers library as entailment engines. The tested models range from ALBERT, RoBERTa, and BART to two larger DeBERTa variants.
- A Pre-Trained models: The pretrained entailment models tested are ALBERT, RoBERTa, BART, DeBERTa v2 xLarge, and DeBERTa v2 xxLarge.All are listed as pretrained NLI systems from the Transformers library.
B Experimental details
Experiments were run primarily on Titan V GPUs, with DeBERTa fine-tuning using a four-GPU cluster. The setup used mixed precision, limited fine-tuning schedules, and three random seeds.
- B Experimental details: Most experiments used a single 16GB Titan V, while DeBERTa fine-tuning used four 32GB Titan V100 GPUs.The reported average inference time was 1–1.5 hours, with fine-tuning taking up to 5 hours for DeBERTa.
- B Experimental details: Mixed precision was used to speed runtime, with zero- and few-shot inference averaging 1–1.5 hours.RoBERTa fine-tuning took at most 2.5 hours and DeBERTa fine-tuning at most 5 hours.
- B Experimental details: Fine-tuning hyperparameters were explicitly listed, including random seeds {0, 24, 42}.The systems were fine-tuned from already trained NLI models, motivating low epoch and learning-rate settings.
C TACRED templates
The TACRED experiments use manually created relation templates organized for PERSON and ORGANIZATION relations. The section also specifies valid argument types accepted for each relation.
- The experiments use manually created templates for TACRED PERSON and ORGANIZATION relations.The templates are presented in separate tables for the two relation groups.
- The templates were based on TAC KBP Slot Descriptions annotation guidelines.
- Each relation is accompanied by the valid argument types accepted for that relation.