Source-linked AI summary
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen
TL;DR
Prompt-tuning for relation extraction is hindered by labor-intensive template design, difficult label-word selection, and underused relation-label knowledge. KnowPrompt addresses this with learnable virtual type and answer words optimized under structural constraints, and it improves over baselines across standard and low-resource settings. The paper notes that its current formulation uses heterogeneous language-model and knowledge-graph embeddings rather than pretrained knowledge-graph embeddings.
Problem
Prompt-tuning for RE requires domain expertise for templates, costly label-word search, and better use of semantic and prior knowledge among relation labels.
Method
KnowPrompt injects relation-label knowledge through learnable virtual type and answer words, then synergistically optimizes them with implicit structural constraints and context-aware calibration.
Results
KnowPrompt improves over all baselines and is especially beneficial in low-resource settings, with average absolute gains over fine-tuning of up to 22.4% and 13.2%.
Takeaways & Limitations
Injecting semantic knowledge maintained in relation labels is particularly valuable for relation extraction when labeled data are scarce.
Takeaways & Limitations
The approach avoids pretrained knowledge-graph embeddings because they are heterogeneous with pretrained language-model embeddings.
Abstract
from arXiv · showhide
Recently, prompt-tuning has achieved promising results for specific few-shot classification tasks. The core idea of prompt-tuning is to insert text pieces (i.e., templates) into the input and transform a classification task into a masked language modeling problem. However, for relation extraction, determining an appropriate prompt template requires domain expertise, and it is cumbersome and time-consuming to obtain a suitable label word. Furthermore, there exists abundant semantic and prior knowledge among the relation labels that cannot be ignored. To this end, we focus on incorporating knowledge among relation labels into prompt-tuning for relation extraction and propose a Knowledge-aware Prompt-tuning approach with synergistic optimization (KnowPrompt). Specifically, we inject latent knowledge contained in relation labels into prompt construction with learnable virtual type words and answer words. Then, we synergistically optimize their representation with structured constraints. Extensive experimental results on five datasets with standard and low-resource settings demonstrate the effectiveness of our approach. Our code and datasets are available in https://github.com/zjunlp/KnowPrompt for reproducibility.
1 INTRODUCTION
Relation extraction benefits information extraction and knowledge-base construction, but prompt-tuning for RE remains difficult because templates and label words require costly, expert-guided design. KnowPrompt addresses these challenges by injecting relation-label knowledge into learnable prompts and jointly optimizing virtual words under structural constraints.
- Relation extraction extracts structured knowledge from unstructured text for information extraction, knowledge-base construction, information retrieval, web mining, and question answering.
- Fine-tuning PLMs requires extra classifiers and annotated data, while prompt-tuning instead uses cloze prediction to connect pre-training with downstream classification.
- Prompt-tuning for RE requires domain expertise to design templates and substantial computation to generate and verify entity-aware prompts.
- Searching suitable label words becomes computationally complex when relation-label lengths vary, making verbalizer construction non-trivial.
- KnowPrompt injects relation-label knowledge through learnable virtual answer words and context-adaptive virtual type words surrounding entities.
- Synergistic optimization uses context-aware calibration and implicit structural constraints to associate prompt embeddings and optimize virtual type and answer words.
2 RELATED WORK
Prior RE research spans neural architectures, knowledge-enhanced PLMs, few-shot learning, and prompt-tuning. KnowPrompt differs from existing RE prompt methods by using flexible learnable virtual words and synergistic knowledge-constrained optimization.
- PLM-based RE has become standard, while knowledge-enhanced models incorporate relational information through pre-training objectives, input features, or model architectures.
- Few-shot RE methods use meta-learning or metric learning to train models with few labeled sentences, including practical settings such as 16 examples per class.
- Prompt-tuning research includes manual, automatically searched, knowledge-calibrated, and continuous prompt designs, but these methods do not directly adapt to RE.
- PTR constructs RE prompts with logic rules and several sub-prompts, whereas KnowPrompt represents relations with virtual answer words and uses learnable virtual type words.
- KnowPrompt is model-agnostic, avoids predefined rules, and jointly optimizes virtual type and answer words with knowledge constraints.
3 BACKGROUND
RE predicts the relation between subject and object entities in context. Fine-tuning adds a classifier over PLM representations, whereas prompt-tuning maps labels to vocabulary words predicted at masked positions.
- An RE dataset consists of examples X and relation labels Y, with the task of predicting y ∈ Y between subject and object entities.
- 3.1 Fine-tuning PLMs for RE: Fine-tuning encodes an input sequence with a PLM and applies a softmax classifier to the [CLS] representation, optimizing the PLM and classifier with cross-entropy.
- 3.2 Prompt-Tuning of PLMs: Prompt-tuning constructs a template T and an injective mapping M from relation labels Y to vocabulary label words V.
- 3.2 Prompt-Tuning of PLMs: The prompt retains the original input, inserts one or more [MASK] tokens, and asks the PLM to predict mapped label words at masked positions.
- 3.2 Prompt-Tuning of PLMs: For sentiment classification, predicting “great” or “terrible” at the masked position identifies the corresponding positive or negative label.
4 METHODOLOGY
KnowPrompt injects relation and entity-type knowledge into learnable prompts through virtual answer and type words, then jointly calibrates them with context and structural constraints. A two-stage procedure first optimizes prompt embeddings and then tunes the prompted language model.
- Prompt Construction with Knowledge Injection: KnowPrompt constructs prompts with learnable virtual type words around entities and virtual answer words for relation labels.The approach uses knowledge injection rather than ordinary fixed label words or additional entity-type annotations.
- Prompt Construction with Knowledge Injection: Prior entity-type distributions for each relation initialize virtual type words, which can adapt to context during optimization.The initialized words aggregate embeddings over candidate subject and object types and serve a role similar to Type Marker methods.
- Prompt Construction with Knowledge Injection: Relation-label semantics initialize virtual answer-word embeddings through weighted averages of decomposed relation-label words.These learnable embeddings provide an anchor that can later be optimized using context.
- Synergistic Optimization with Knowledge Constraints: Context-aware prompt calibration optimizes virtual type and answer words using masked-token prediction over the virtual answer-word set.The objective is cross-entropy between the target relation and the probability distribution at the masked position.
- Synergistic Optimization with Knowledge Constraints: Implicit structured constraints represent relational facts as subject-type, relation, and object-type triplets and use language-model output embeddings in the scoring calculation.Negative samples corrupt a subject or object with an entity having an impossible type for the current relation.
- Training Details: KnowPrompt uses two-stage optimization: a large learning rate optimizes virtual words, followed by a small-learning-rate update of the prompted PLM using masked-token loss.The combined objective weights the structured-constraint and masked-prediction losses with λ.
5 EXPERIMENTS
KnowPrompt is evaluated on five relation-extraction datasets under standard and low-resource settings, using micro F1 as the primary metric. It consistently outperforms comparison methods, with especially large gains when labeled data are scarce.
- Experimental Settings: Experiments cover SemEval, DialogRE, TACRED, TACRED-Revisit, and Re-TACRED under standard and low-resource settings.Low-resource experiments use 8-, 16-, and 32-shot configurations with five random samples.
- Experimental Settings: Micro F1 scores are the primary evaluation metric, with RoBERTa_large used for fair comparisons except DialogRE.DialogRE uses RoBERTa_base when comparing with prior methods.
- Main Results: KnowPrompt improves over all standard-setting baselines, including knowledge-enhanced models using data augmentation or architecture enhancement.The result supports injecting task-specific knowledge through prompt-tuning rather than relying only on fine-tuning.
- Main Results: Up to 22.4% and 13.2% absolute average gains over fine-tuning occur in low-resource experiments, especially at 8 and 16 shots.The advantage decreases as K increases from 8 to 32 because more labeled instances reduce the marginal benefit of injected semantic knowledge.
- Prompt-Tuning Comparisons: KnowPrompt exceeds PTR by approximately 5.4 points on DialogRE in the standard supervised setting.The authors associate this advantage with PTR's multiple-mask rule potentially confusing multi-label prediction.
- Ablation Study: Removing knowledge injection for virtual answer words reduces relation F1 from 74.3% to 52.5% in the 8-shot setting.Removing virtual type words lowers performance to 72.8, while random initialization lowers it to 68.1; the authors conclude all modules contribute.
6 ANALYSIS AND DISCUSSION
KnowPrompt extends beyond RoBERTa to GPT-2 and analyzes how its learned virtual words represent relations and entity types. The method remains model-agnostic while its optimized virtual answer and type words acquire interpretable semantic structure.
- Can KnowPrompt Applied to Other LMs?: KnowPrompt achieves impressive improvement with low variance on RoBERTa and GPT-2 in low-resource settings, compared with fine-tuning.GPT-2 is supported by appending the prompt template with [MASK] and extending the PLM word-embedding layer with a relation embedding head.
- Can KnowPrompt Applied to Other LMs?: Figure 3 compares RoBERT-large and GPT-2 across different K values on TACRED-Revisit.K denotes the number of instances per class.
- Interpreting Virtual Answer Words: Optimized virtual answer words are semantically intuitive and resemble prototypical representations for relation labels.The analysis uses t-SNE and normalization to inspect relations in vocabulary space; nearby vocabulary words provide semantic interpretations.
- Interpreting Virtual Type Words: Virtual type words are initialized from candidate head and tail entity types and synergistically optimized with virtual answer words using context and knowledge constraints.Their representations are analyzed by applying the MLM head and finding nearest vocabulary words with L2 embedding distance.
7 CONCLUSION AND FUTURE WORK
The paper presents KnowPrompt as knowledge-aware prompt construction combined with synergistic optimization under knowledge constraints. Future work targets semisupervised learning with unlabelled data and lifelong learning with adaptive task optimization.
- KnowPrompt combines knowledge-aware prompt construction with synergistic optimization using knowledge constraints for relation extraction.
- Future work will extend KnowPrompt to semisupervised learning and lifelong learning.The proposed directions are leveraging unlabelled data and optimizing prompts for adaptive tasks.
A DETAILED STATISTICS OF DATASET
The experiments use five relation extraction datasets spanning sentence-level, relation classification, dialogue-level, corrected, and refactored TACRED variants. Their sizes, relation inventories, entity types, and labeling structures differ substantially.
- Experiments cover TACRED, TACRED-Revisit, Re-TACRED, SemEval 2010 Task 8, and DialogRE.
- Dataset Statistics: TACRED contains more than 106K sentences and 42 relations, with person and organization subjects and 16 fine-grained object types.
- Dataset Statistics: TACRED-Revisit corrects errors in TACRED’s development and test sets while leaving its training set unchanged.
- Dataset Statistics: Re-TACRED refactors all dataset splits, modifies several relation types, and contains 40 relation types.
- Dataset Statistics: SemEval contains 10,717 annotated examples covering nine bidirectional relations and one no_relation category.
- Dataset Statistics: DialogRE contains 1,788 human-annotated dialogues and supports multilabel classification for entity pairs.
B IMPLEMENTATION DETAILS FOR KNOWPROMPT
KnowPrompt training uses AdamW with warmup and decay, gradient clipping, and fixed regularization settings. Hyperparameter searches vary optimization rates, epochs, batch sizes, sequence lengths, and gradient accumulation by dataset.
- Training uses AdamW, linear warmup over the first 10% of updates, linear decay thereafter, and gradient clipping above norm 1.0.
- The margin γ, λ, and non-bias weight decay are set to 1, 0.001, and 0.01, respectively.
- Hyperparameter Search: The search includes lr1 values of [5e-5, 1e-4, 2e-4] and lr2 values of [1e-5, 2e-5, 3e-5, 5e-5].lr1 is used for synergistic optimization of virtual template and anchor words; lr2 is used for overall parameters.
- Hyperparameter Search: The search uses five epochs, except 20 for DialogRE, with batch size 16 and reduced batch size 8 for TACREV, Re-TACRED, and DialogRE.
- Hyperparameter Search: Maximum sequence length is 256, increased to 512 for TACREV, Re-TACRED, and DialogRE; DialogRE uses four gradient accumulation steps.
C IMPLEMENTATION DETAILS FOR FINE-TUNING
The fine-tuning method follows the procedure shown in Figure 2 and uses the same entity marker and hyperparameters as KnowPrompt for comparison.
- The fine-tuning procedure is conducted according to Figure 2.
- The raw text uses the same entity marker as KnowPrompt for a fair comparison.
- Batch size, epoch, and learning rate are the same as KnowPrompt.
D IMPLEMENTATION DETAILS FOR PTR
PTR experiments are supplemented by rerunning its public code across additional datasets and scenarios, with process adjustments for DialogRE.
- The authors rerun PTR’s public code to supplement experiments on datasets and scenarios not covered originally.
- For SemEval, the experiment process completely follows the original setting in PTR’s code.
- For DialogRE, PTR’s code is modified to better fit the dataset setting.
- The implementation specifies batch size, epoch, and learning rate as hyperparameters.