Source-linked AI summary
PTR: Prompt Tuning with Rules for Text Classification
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, Maosong Sun
TL;DR
Many-class classification remains difficult for prompt tuning because manual prompt design is cumbersome and auto-generated prompts are costly to verify. PTR composes sub-prompts with logic rules to encode prior knowledge, and experiments on relation classification show significant and consistent improvements over state-of-the-art baselines. The paper concludes that PTR is a promising way to combine human prior knowledge with PLMs.
Problem
Prompt tuning has promising results on few-class tasks, but many-class classification remains difficult because manual prompts are cumbersome and auto-generated prompts are expensive to verify.
Method
PTR manually designs essential sub-prompts and composes them into task-specific prompts using logic rules, encoding prior knowledge about tasks and classes.
Results
PTR significantly and consistently outperforms existing state-of-the-art baselines on relation classification across four popular benchmarks.
Takeaways & Limitations
PTR provides a prompt-tuning approach for many-class classification that uses human prior knowledge without additional model layers, manual annotations, or augmented data.
Takeaways & Limitations
The paper identifies the need for more effective prompt construction and theoretical analysis of the mechanisms underlying prompting strategies.
Abstract
from arXiv · showhide
Fine-tuned pre-trained language models (PLMs) have achieved awesome performance on almost all NLP tasks. By using additional prompts to fine-tune PLMs, we can further stimulate the rich knowledge distributed in PLMs to better serve downstream tasks. Prompt tuning has achieved promising results on some few-class classification tasks such as sentiment classification and natural language inference. However, manually designing lots of language prompts is cumbersome and fallible. For those auto-generated prompts, it is also expensive and time-consuming to verify their effectiveness in non-few-shot scenarios. Hence, it is still challenging for prompt tuning to address many-class classification tasks. To this end, we propose prompt tuning with rules (PTR) for many-class text classification and apply logic rules to construct prompts with several sub-prompts. In this way, PTR is able to encode prior knowledge of each class into prompt tuning. We conduct experiments on relation classification, a typical and complicated many-class classification task, and the results show that PTR can significantly and consistently outperform existing state-of-the-art baselines. This indicates that PTR is a promising approach to take advantage of both human prior knowledge and PLMs for those complicated classification tasks.
1 Introduction
Prompt tuning addresses the objective mismatch between PLM pre-training and downstream classification, but many-class tasks make prompt design difficult. PTR uses logic rules to compose sub-prompts that encode task and class knowledge, and outperforms strong baselines on relation classification.
- PLM fine-tuning faces a gap between pre-training objectives and downstream objectives, hindering knowledge transfer and adaptation.
- Prompt tuning bridges this objective gap by converting classification into cloze-style prediction with templates and label words.
- Many-class classification makes manually selecting distinguishing templates and label words difficult, especially for closely related relations.
- Auto-generated prompts require expensive computation and are therefore better suited to few-shot settings than tasks with many instances and classes.
- PTR composes manually designed sub-prompts with logic rules to encode prior knowledge about task and class structure.For relation classification, sub-prompts can separately identify entity types and parent-child semantics.
- PTR significantly and consistently outperforms existing state-of-the-art baselines on four relation-classification benchmarks.The evaluated benchmarks are TACRED, TACREV, ReTACRED, and SemEval 2010 Task 8.
2 Preliminaries
The preliminaries contrast vanilla fine-tuning with prompt tuning. Vanilla fine-tuning uses a task-specific classification head, whereas prompt tuning maps inputs to masked-token predictions and then maps label words back to classes.
- Vanilla Fine-tuning for PLMs: Vanilla fine-tuning encodes the input with a PLM and uses a randomly initialized task-specific head to predict class probabilities.The model, bias, and classification matrix are tuned during training.
- Prompt Tuning for PLMs: Prompt tuning applies a template to place the input and at least one [MASK], which the PLM fills with a candidate label word.The template controls input-token placement and added tokens.
- Prompt Tuning for PLMs: The PLM scores candidate label words at the masked position using the hidden representation of [MASK].
- Prompt Tuning for PLMs: A verbalizer maps each task class to a label word, converting masked-token probabilities into probabilities over classes.For sentiment classification, positive and negative classes can map to “great” and “terrible.”
- Prompt Tuning for PLMs: Prompt tuning trains the template, label-word set, and verbalizer to maximize the probability of the correct mapped label word.
3 Prompting Tuning with Rules (PTR)
PTR addresses many-class classification by decomposing class decisions into logic-based conditional functions and composing their sub-prompts into task-specific prompts. The framework encodes prior knowledge about entity types and relations while aggregating multiple masked predictions.
- 3.1 Overall Framework of PTR: For relation classification, the relation “person:parent” requires checking both entity types and whether the sentence expresses parental semantics.The analogous “organization:parent” relation differs through the entity-type condition.
- 3.1 Overall Framework of PTR: PTR defines conditional functions as predicates that test whether inputs satisfy task-specific conditions.Examples classify an input as a person or determine whether one entity is another’s parent.
- 3.2 Sub-Prompts for Conditional Functions: Each conditional function receives a manually designed template and label-word set, forming a sub-prompt that can be combined according to class semantics.Unary functions cover properties such as entity types, while binary functions model relations between two inputs.
- 3.2 Sub-Prompts for Conditional Functions: Binary conditional functions distinguish relations between two sentences or entities, including entailment, neutral, contradiction, and complex entity connections.For relation classification, label words can include “’s parent was” and “was born in”.
- 3.2 Sub-Prompts for Conditional Functions: Unary and binary functions usually suffice, while multi-variable functions extend the design for classification tasks with more complex semantics.The extension provides more powerful sub-prompts.
- 3.3 Composing Sub-Prompts for Tasks: PTR composes rule-related sub-prompts with conjunctive normal-form logic and direct concatenation into a complete task-specific prompt.The illustrated template aggregates three masked positions and corresponding label-word sets.
- 3.3 Composing Sub-Prompts for Tasks: The aggregated prompt may include learnable randomly initialized tokens, and classification considers all masked positions jointly.For each masked position, φj maps a class to its permitted label words; n denotes the number of masked positions.
4 Experiments
PTR is evaluated on four relation-classification benchmarks using relation-specific prompts, including variants that reverse selected relations. Across comparisons, PTR consistently outperforms conventional, knowledge-enhanced, and prompt-tuning baselines, with additional benefits in few-shot settings and convergence.
- Datasets and Experimental Settings: PTR is evaluated on TACRED, TACREV, ReTACRED, and SemEval 2010 Task 8, using F1 as the main metric.
- Datasets and Experimental Settings: PTR uses manually designed sub-prompts, logic-rule composition, and two settings that either preserve or reverse selected relations.Reversed variants are named “...(Reversed)”.
- Comparison between PTR and Fine-Tuning Methods: ROBERTA_LARGE outperforms conventional models trained from scratch, while knowledge-enhanced PLMs outperform vanilla ROBERTA_LARGE.The comparisons include recurrent, graph-neural, and knowledge-enhanced baselines.
- Comparison between PTR and Fine-Tuning Methods: PTR significantly improves over all baselines in both normal and relation-reversed settings, including knowledge-enhanced models on TACRED.
- Comparison between PTR and Prompt Tuning Methods: PTR performs comparably to or better than TYP MARKER in few-shot settings, especially on ReTACRED, despite using no extra human annotations or neural layers.With the full training dataset, PTR performs worse than TYP MARKER (PUNCT).
- Effect of Reversing Relations: Reversing relation labels yields significant improvements for PTR on TACRED, TACREV, and ReTACRED, while smaller gains occur for ENTITY MARKER and TYPED MARKER.The reported absolute improvements are 3.5% for PTR, 0.2% for ENTITY MARKER, and 1.4% for TYPED MARKER on TACRED.
5 Related Work
Related prompt-tuning work automates templates and label-word selection to reduce manual effort, but PTR emphasizes rule-composed prompts as a balance among efficiency, effectiveness, generalization, and workload.
- Fine-tuned PLMs have demonstrated effectiveness across major NLP tasks, motivating continued work on prompt-based adaptation.
- Automatic prompt search reduces labor-intensive prompt design by generating templates, label words, or both.Methods include automatic label-word identification, gradient-guided generation, and sequence-to-sequence prompt candidates.
- PTR composes human-picked sub-prompts with predefined logic rules to form complete task-specific prompts.This design is presented as a balance among model efficiency, effectiveness, generalization, and human workload.
6 Conclusion
PTR applies logic-rule composition of sub-prompts to many-class text classification, encoding prior human knowledge while simplifying prompt design. On relation classification, it significantly outperforms state-of-the-art baselines without extra layers, annotations, or augmented data.
- PTR composes sub-prompts according to logic rules, encoding prior human knowledge into prompts for many-class text classification.
- PTR significantly outperforms existing state-of-the-art baselines on relation classification.
- PTR requires no additional model layers, manual annotations, or augmented data.