Source-linked AI summary
RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction
Yew Ken Chia, Lidong Bing, Soujanya Poria, Luo Si
TL;DR
Relation extraction has limited support for generalizing to unseen relation labels, especially when complete entity-relation triplets must be extracted without annotated examples. The paper introduces ZeroRTE and uses RelationPrompt to generate structured synthetic data, supplemented by Triplet Search Decoding for multiple triplets. On FewRel and Wiki-ZSL, the method surpasses prior ZeroRC methods and strong baselines on ZeroRTE.
Problem
Existing relation extraction methods often require costly annotations and fixed relation sets, while related zero-shot tasks do not extract complete triplets for unseen labels.
Method
RelationPrompt prompts language models to generate structured synthetic relation examples from unseen relation labels, while Triplet Search Decoding handles multiple candidate triplets.
Results
The method surpasses prior ZeroRC methods and strong baselines on ZeroRTE across evaluations on FewRel and Wiki-ZSL.
Takeaways & Limitations
ZeroRTE establishes a zero-shot benchmark for full relation triplet extraction, and RelationPrompt provides effective synthetic training data for the task.
Takeaways & Limitations
Generated entity pairs may not match the intended relation meaning, as illustrated by a Political Partner sample that expresses Political Party instead.
Abstract
from arXiv · showhide
Despite the importance of relation extraction in building and representing knowledge, less research is focused on generalizing to unseen relations types. We introduce the task setting of Zero-Shot Relation Triplet Extraction (ZeroRTE) to encourage further research in low-resource relation extraction methods. Given an input sentence, each extracted triplet consists of the head entity, relation label, and tail entity where the relation label is not seen at the training stage. To solve ZeroRTE, we propose to synthesize relation examples by prompting language models to generate structured texts. Concretely, we unify language model prompts and structured text approaches to design a structured prompt template for generating synthetic relation samples when conditioning on relation label prompts (RelationPrompt). To overcome the limitation for extracting multiple relation triplets in a sentence, we design a novel Triplet Search Decoding method. Experiments on FewRel and Wiki-ZSL datasets show the efficacy of RelationPrompt for the ZeroRTE task and zero-shot relation classification. Our code and data are available at github.com/declare-lab/RelationPrompt.
1 Introduction
The paper introduces ZeroRTE, which extracts full relation triplets for unseen relation labels, and proposes synthetic-data generation with RelationPrompt to make the task tractable.
- Motivation: Existing relation extraction methods commonly depend on costly annotated datasets and fixed relation sets, while related zero-shot tasks do not extract complete triplets.Zero-shot relation classification assumes entities are given, whereas slot filling predicts tails and relies on entity detection.
- Task setting: ZeroRTE extracts full relation triplets from sentences even when test relation labels have no annotated training examples.The task extends relation triplet extraction to disjoint seen and unseen relation sets.
- Approach: RelationPrompt reframes zero-shot relation extraction as generating synthetic training samples conditioned on the semantics of unseen relation labels.Language models generate structured texts that can supervise a downstream relation extractor.
- Approach: Triplet Search Decoding enumerates candidate head, tail, and relation sequences so one sentence can yield multiple relation triplets.A likelihood threshold filters the generated candidate sequences.
- Results: RelationPrompt surpasses prior ZeroRC methods and strong baselines on ZeroRTE, while generated samples are reported as reasonable and diverse synthetic training data.The reported evaluation covers FewRel and Wiki-ZSL, and the contribution claim describes the method as setting a bar for future work.
2 RelationPrompt: Methodology
RelationPrompt reframes ZeroRTE as synthetic-data generation: a relation generator uses unseen relation labels to create structured samples, and a relation extractor predicts triplets from sentences. Triplet Search Decoding extends extraction to multiple triplets per sentence while also supporting single-triplet ZeroRTE and entity-conditioned ZeroRC.
- 2.1 Task Formulation: The framework defines ZeroRTE with disjoint seen and unseen relation-label sets, training on the seen dataset and testing on the unseen dataset.Each output triplet contains a head entity, tail entity, and relation label.
- 2.2 Relation Generator: RelationPrompt fine-tunes a relation generator on seen data, prompts it with unseen relation labels, and trains an extractor on the resulting synthetic samples.The generator produces structured context and entity-pair outputs; the extractor is then fine-tuned on both seen and synthetic data.
- 2.2 Relation Generator: The method represents generated samples as “Context,” “Head Entity,” and “Tail Entity” fields, which are decoded into relation triplets by string processing.Samples with entities absent from their generated contexts are discarded until a fixed number of valid samples is obtained.
- 2.3 Relation Extractor: The relation extractor uses sequence-to-sequence learning to generate entity pairs and relation labels from input sentences, with greedy decoding for single-triplet prediction.Providing the entity pair as decoder input changes the same trained model into a ZeroRC classifier.
- 2.4 Extracting Multiple triplets using Triplet Search Decoding: Triplet Search Decoding enumerates candidate head entities, tail entities, and relation labels, then retains multiple triplets whose probabilities exceed a tuned threshold.Unlike greedy decoding, it produces multiple output sequences corresponding to different candidate relation triplets.
3 Experiments
Experiments evaluate RelationPrompt on ZeroRTE and ZeroRC using FewRel and Wiki-ZSL, comparing it with baselines and prior zero-shot classification methods. RelationPrompt performs best overall, particularly for multi-triplet extraction and larger unseen label sets.
- Experimental Setup: Evaluation separates single-triplet Accuracy, multi-triplet Micro F1 with precision and recall, and ZeroRC Macro F1.Datasets use disjoint seen, validation, and unseen relation-label sets.
- Triplet Extraction: RelationPrompt consistently outperforms baseline methods on single-triplet and multi-triplet ZeroRTE across Accuracy and F1 metrics.The comparison uses FewRel and Wiki-ZSL; the reported advantage is strongest for overall multi-triplet F1.
- Triplet Extraction: Synthetic samples are critical: RelationPrompt improves F1 by more than two times over NoGen in some settings.NoGen constrains labels to target relation names but does not train on generated samples.
- Triplet Extraction: TableSequence performs significantly worse than RelationPrompt, especially for multi-triplet extraction.RelationPrompt’s extractor and decoding method enumerate and rank multiple triplets at inference time.
- Relation Classification: RelationPrompt consistently improves ZeroRC over ZS-BERT and maintains relatively high F1 as the unseen label set grows.The paper attributes this pattern to stronger use of relation-label semantics through token-level conditional generation and extraction.
4 Analysis
Analysis examines decoding, fine-tuning, synthetic-data quantity, and generation quality. Triplet Search Decoding is important for multi-triplet ZeroRTE, while excessive synthetic data and semantic mismatches limit performance.
- 4.1 Ablation Study: Triplet Search Decoding creates a large performance gap in multi-triplet ZeroRTE, showing that candidate enumeration and ranking are critical.The ablation is conducted on Wiki-ZSL validation data with 10 unseen labels.
- 4.1 Ablation Study: Fine-tuning first on seen-relation sentences and then on synthetic unseen-relation samples improves adaptation to the zero-shot extraction task.The analysis connects initial task learning with later relation-specific adaptation.
- 4.2 Effect of Generated Data Size: Increasing synthetic samples from 125 to 250 per label improves F1, but increasing them to 2000 provides no further gain.The authors suggest excessive noisy data can cause over-fitting.
- 4.3 Qualitative Analysis: Generated samples usually match the semantics and contexts of Investor, Defeated By, and Currency Of, but Political Partner can produce an incompatible entity pairing.The erroneous output is closer to Political Party despite remaining in a political context.
5 Related Work
Related work spans zero-shot extraction, data augmentation, prompting, knowledge retrieval, and structured prediction. RelationPrompt combines synthetic generation with structured triplet extraction to address gaps in prior approaches.
- Zero-Shot Relation Extraction: Prior zero-shot extraction methods use slot filling or entailment, but they require manual templates or support sentence classification rather than full triplet extraction.These limitations motivate a task formulation that extracts head, relation, and tail jointly.
- Knowledge Retrieval: RelationPrompt leverages language-model knowledge to compose relation samples in realistic contexts, while factual accuracy depends on strong knowledge retrieval.The approach is connected to broader language-model knowledge and retrieval research.
- Language Model Prompts: Prompting methods can generate synthetic data, but earlier work had not demonstrated effectiveness for complex triplet extraction.RelationPrompt extends prompting-based synthesis to a structured prediction setting.
- Structured Prediction: The structured-data generation approach may also apply to named entity recognition, event extraction, and aspect sentiment triplet extraction.These tasks are presented as related structured prediction applications.
6 Conclusions and Future Work
The paper introduces ZeroRTE and RelationPrompt, using prompted synthetic data and Triplet Search Decoding to address unseen relations and multiple triplets. Results surpass prior ZeroRC methods and strong ZeroRTE baselines, while future work targets better semantic compatibility of generated entities.
- Conclusions and Future Work: ZeroRTE evaluates extraction of head, relation, and tail triplets when test relation labels have no annotated training samples.The task is introduced to address limitations of earlier zero-shot relation settings.
- Conclusions and Future Work: RelationPrompt generates structured synthetic training data from relation-label prompts, and Triplet Search Decoding supports multiple-triplet extraction.The decoding method is described as effective and interpretable.
- Conclusions and Future Work: RelationPrompt surpasses prior ZeroRC methods and strong ZeroRTE baselines, establishing a benchmark for future work.The conclusion summarizes the reported experimental outcome without giving task-specific scores.
- Conclusions and Future Work: Future improvement should make generated entity spans more compatible with the semantics of their prompted relations.The paper identifies semantic mismatch in generated entities as a remaining issue.
A.1 Additional Data Samples
Figures 7 and 8 provide additional FewRel and Wiki-ZSL test samples, including examples generated for ten unseen relation labels.
- Figure 7 presents test samples from the FewRel and Wiki-ZSL test sets with 10 unseen relation labels.
- Figure 8 presents synthetic samples generated from FewRel and Wiki-ZSL test-set labels with 10 unseen relation labels.
A.2 Implementation Details
The implementation generates structured synthetic relation samples, decodes them into triplets, and reports additional experimental settings, examples, hyperparameters, and diversity comparisons.
- Structured outputs are converted into relation triplets by separating template content from natural text through string processing.Generation continues after small conversion errors until the required number of sentences per label is reached.
- The appendix includes example generated sentences for relations such as League, Military Branch, Position Played, and an estuary-related relation.
- Figure 8 shows generated outputs with head and tail entities displayed in blue and orange, respectively.
- Table 6 reports additional hyperparameters, including searches over 125, 250, 500, 1000, and 2000 generated samples per label.The Triplet Search Decoding threshold is tuned over fifty evenly spaced values spanning candidate-triplet output scores.
- Experiments run on single NVIDIA V100 GPUs with 32 GB memory and mixed-precision settings.
- Table 7 compares the diversity of real and synthetic samples using unique words and entities.
A.3 Further Analysis
Further analysis examines synthetic-data diversity and relation-specific ZeroRTE performance, showing stronger results for some highly specific relations than others.
- The synthetic-data analysis compares real and generated samples using the number of unique words and entities.
- RelationPrompt performs well on “Drafted By” and “Sports Discipline Competed In” but more poorly on “Official Language” and “Employer.”The evaluation uses single-triplet ZeroRTE on the Wiki-ZSL test set with 10 unseen labels.
- The analysis suggests that RelationPrompt performs best for highly specific relations because they constrain the output context more effectively.