Source-linked AI summary
OpenPrompt: An Open-source Framework for Prompt-learning
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun
TL;DR
Prompt-learning lacks a standard, comprehensive implementation framework despite its growing use for adapting PLMs through prompted objectives. OpenPrompt provides a modular, extensible toolkit that combines PLMs, task formats, and prompting modules, supporting deployment and evaluation across NLP tasks.
Problem
Prompt-learning requires detailed choices and lacks a comprehensive standard framework, while existing codebases provide limited, poorly regulated implementations for specific scenarios.
Method
OpenPrompt modularizes the prompt-learning lifecycle into independent but interacting components, including templates, verbalizers, prompt models, tokenization, and PLM integration.
Results
OpenPrompt implements baselines and processors across a broad range of NLP tasks, including generation, understanding, relation extraction, entity typing, classification, and knowledge probing.
Takeaways & Limitations
OpenPrompt supports efficient deployment, targeted improvements, and generalization evaluation by enabling flexible combinations of PLMs, task formats, and prompting modules.
Abstract
from arXiv · showhide
Prompt-learning has become a new paradigm in modern natural language processing, which directly adapts pre-trained language models (PLMs) to $cloze$-style prediction, autoregressive modeling, or sequence to sequence generation, resulting in promising performances on various tasks. However, no standard implementation framework of prompt-learning is proposed yet, and most existing prompt-learning codebases, often unregulated, only provide limited implementations for specific scenarios. Since there are many details such as templating strategy, initializing strategy, and verbalizing strategy, etc. need to be considered in prompt-learning, practitioners face impediments to quickly adapting the desired prompt learning methods to their applications. In this paper, we present {OpenPrompt}, a unified easy-to-use toolkit to conduct prompt-learning over PLMs. OpenPrompt is a research-friendly framework that is equipped with efficiency, modularity, and extendibility, and its combinability allows the freedom to combine different PLMs, task formats, and prompting modules in a unified paradigm. Users could expediently deploy prompt-learning frameworks and evaluate the generalization of them on different NLP tasks without constraints. OpenPrompt is publicly released at {\url{ https://github.com/thunlp/OpenPrompt}}.
1 Introduction
Prompt-learning adapts PLMs through prompts but lacks a standard, readable, and comprehensive implementation framework. OpenPrompt addresses this gap with a modular toolkit that supports flexible combinations of PLMs, task formats, and prompting modules.
- Prompt-learning adapts PLMs by projecting downstream tasks into pre-training objectives using textual or soft-encoding prompts.
- Existing implementations lack a standard paradigm, offer limited scenario-specific support, and can suffer from poor readability and unstable reproducibility.
- Template and verbalizer choices can substantially affect prompt-learning performance, increasing implementation and comparison barriers.
- OpenPrompt modularizes prompt-learning into clearly defined components while preserving their independence and interactions.Templates wrap inputs, Verbalizers map labels to vocabulary words, and PromptModels handle training and inference.
- OpenPrompt supports flexible combinations of diverse PLMs, task formats, and prompting modules for evaluating generalization across tasks.The framework can, for example, adapt prefix-tuning to text classification.
- The toolkit aims to help beginners understand prompt-learning, researchers deploy research pipelines, and engineers apply methods to practical NLP systems.
2 Background
Prompt-learning is motivated by limitations of conventional fine-tuning, including its growing computational demands for large PLMs and its separation from pre-training. It mimics pre-training and can be effective in low-data settings, while OpenPrompt provides a unified implementation framework for developing and deploying such methods.
- Conventional fine-tuning introduces task-specific heads and objectives, creating a gap between model tuning and pre-training.
- Increasing model size makes conventional fine-tuning increasingly difficult to operate because of massive computational volume.
- Prompt-learning mimics pre-training to bridge the gap between pre-training and model tuning.
- Prompt-learning is reported to be effective in low-data regimes, with zero-shot prompting sometimes outperforming 32-shot fine-tuning when templates are appropriate.
- OpenPrompt provides a unified framework for implementing existing algorithms and exploring new templates, verbalizers, and prompt-learning attributes.
3 Design and Implementation
OpenPrompt organizes prompt-learning into modular, combinable components spanning PLMs, tasks, templates, verbalizers, tokenization, and training. Its interfaces and template language support flexible implementation and evaluation across prompting strategies.
- Framework architecture: OpenPrompt provides a full prompt-learning lifecycle whose modules remain independent while supporting targeted interactions and improvements.The architecture includes task processing, prompt components, model execution, and training control.
- Combinability: OpenPrompt combines classification and generation tasks, MLM, LM, and Seq2Seq PLMs, and diverse prompt modules within one flexible framework.The framework treats downstream prediction as word prediction conditioned on context, enabling these combinations.
- Pre-trained language models: The toolkit supports MLM, LM, and Seq2Seq PLMs, loading them directly through Hugging Face Transformers and selecting the appropriate tokenizer by model type.Different pre-training objectives yield different PLM groups and adaptation capabilities.
- Tokenization: OpenPrompt simplifies prompt-learning tokenization by jointly processing inputs and templates while handling masked-token indices, truncation, and PLM-specific tokenization.These details are important because small tokenization errors can have serious consequences.
- Templates: Its template language enables token-level customization of hard and soft tokens, initialization, shared embeddings, and post-processing under a unified syntax.Template nodes carry text and attribute descriptions, avoiding separate implementations for each prompt format.
- Verbalizers and PromptModel: Verbalizers map labels to vocabulary words, while PromptModel unifies masked-position prediction, training, and inference with an optional verbalizer.OpenPrompt also implements automatic verbalizers and calibration operations.
4 Evaluation
OpenPrompt is evaluated by implementing baselines across a broad validation space of NLP tasks. The evaluation spans generation, language understanding, relation extraction, entity typing, classification, and knowledge probing.
- Evaluation scope: OpenPrompt implements and evaluates prompt-learning baselines across a broad set of NLP tasks.Figure 4 presents the framework’s validation space and corresponding implementation flows.
- Evaluation scope: The evaluation covers WebNLG, GLUE, SuperGLUE, SemEval, Few-NERD, MNLI, AG’s News, DBPedia, IMDB, and LAMA.These datasets represent conditional generation, NLU, relation extraction, fine-grained entity typing, classification, and knowledge probing.
5 Conclusion and Future Work
The paper presents OpenPrompt as a unified, easy-to-use, and extensible toolkit with clearly defined blocks and flexible interactions. It is intended to facilitate efficient deployment of prompt-learning pipelines, with continued integration of techniques and features planned.
- Conclusion: OpenPrompt provides a unified, easy-to-use, and extensible toolkit with clearly defined blocks and flexible interactions.The framework is designed to support research and application deployment under the prompt-learning paradigm.
- Future work: OpenPrompt is intended to help researchers and developers deploy prompt-learning pipelines effectively and efficiently.The authors plan to continue integrating new techniques and features to facilitate prompt-learning research.