Source-linked AI summary
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-Jian Jiang, Alexander M. Rush
TL;DR
Prompt engineering needs tools for creating, refining, and sharing prompts collaboratively, because prompt choices affect language-model behavior and prompts differ from ordinary labels. PromptSource provides a templating language, interfaces for development and management, and community quality standards, supporting a growing shared collection used in research. It was originally designed for zero-shot learning, with few-shot extensions possible when needed.
Problem
Prompt engineering requires collaborative tools because prompt choices affect predictions and prompts are dataset-level functions rather than ordinary example-level labels.
Method
PromptSource combines a Jinja2-based templating language, interfaces for browsing and iterating on prompts, and community-driven documentation and review standards.
Results
PromptSource provides a shared system supporting prompt creation, management, review, and use across datasets and research projects.
Takeaways & Limitations
As of January 2022, PromptSource included 2,000 public prompts for roughly 170 datasets and had supported multiple recent research projects.
Takeaways & Limitations
PromptSource was originally designed for zero-shot learning, so it emphasizes explicit task instructions and no priming examples.
Abstract
from arXiv · showhide
PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common pool. Over 2,000 prompts for roughly 170 datasets are already available in PromptSource. PromptSource is available at https://github.com/bigscience-workshop/promptsource.
1 Introduction
Prompt engineering is emerging in NLP, especially for zero- and few-shot learning, creating a need for tools that support collaborative prompt creation, refinement, and sharing. PromptSource addresses this need with a templating language, prompt-development interfaces, and community quality standards.
- Prompting represents tasks as natural-language utterances used to query language models for responses.
- Prompting can facilitate adaptation to ad-hoc tasks and improve sample efficiency in low-data settings.
- Prompt choices can significantly affect downstream predictions, particularly in zero-shot settings.
- PromptSource provides a Web-based GUI for writing prompts in a templating language and immediately viewing outputs on examples.
- Its design combines flexible templating, prompt-management tools, and community-driven quality standards.
2 Background and Related Work
PromptSource focuses on human-written prompts for zero- and few-shot learning, distinguishing prompts from conventional labels and other annotations because they are semi-structured functions. Its design emphasizes prompts that remain understandable, modifiable, and usable across models.
- Prompting has emerged as an NLP focus because it can improve few-shot and zero-shot performance across downstream tasks.
- PromptSource facilitates research with human-written prompts that describe tasks in natural language and are not tied to a specific model.
- PromptSource was originally designed for zero-shot learning, emphasizing explicit task instructions without priming examples.
- Traditional annotation systems can use metrics such as inter-annotator agreement and model confidence for label collection.
- Unlike labels collected for individual examples, prompts are semi-structured functions requiring tools for developers.
3 System Design and Workflow
Prompt creation in PromptSource treats prompts as dataset-level functions rather than example-level labels, requiring evaluation across examples and support for controlled variation. The workflow moves from dataset exploration through writing, documentation, iteration, and global review.
- Functions, not Labels: A prompt maps dataset examples to natural-language input/target pairs, making prompt creation more like programming than traditional annotation.
- Dataset-Level Choices: Authors must evaluate prompt choices across all dataset examples rather than treating them as local annotations.
- Variation in Prompt Construction: Prompt variation is desirable because different construction choices may produce different results, but it complicates quality judgment.
- S1: Dataset Exploration: The workflow begins by reading dataset documentation and browsing examples, then testing and adjusting prompt wording in the Sourcing view.
- S3: Prompt Documentation: Authors document prompts with metadata, iterate across wording and targeted tasks, and submit drafts for code review, automatic tests, and prompted-instance validation.
4 Prompting Language
PromptSource adopts a templating language as a middle ground between expressive code and restrictive rule-based generation. Jinja2 templates link natural-language prompts to dataset fields while supporting simple control logic and prompt-instance generation.
- Pure Python offers expressivity but makes prompts difficult to interpret, analyze, and manipulate downstream.
- Rule-based generation provides structure but cannot feasibly enumerate rules for the desired range of tasks and data formats.
- PromptSource therefore uses a templating language as a middle ground between expressivity and explicit structure.
- Jinja2 templates combine dataset-field placeholders, hard-coded text, and built-in functions to produce prompted examples.
- Conditional statements allow inapplicable examples to be skipped, while a choice function supports multiple training instances from one example.
5 The PromptSource UI
PromptSource’s UI supports the prompt-engineering cycle from inspecting datasets to writing, testing, and organizing prompts. Its three views connect example browsing, iterative template development, and dataset-level prompt management.
- UI workflow: The system is designed to let prompt creators view data, write prompts in a standard format, and verify that templates work correctly.A lightweight Streamlit interface supports local use and uploading results to a central repository.
- V1: Browse: Browse presents original dataset examples alongside the prompted examples generated by selected prompts.Substituted text is highlighted, allowing users to inspect many examples and return to sourcing when changes are needed.
- V2: Sourcing: Sourcing lets users select a dataset, inspect examples, enter a template, and see the saved template’s output beside the editor.It also collects metadata such as the template name and bibliographic reference or rationale.
- V3: Helicopter: Helicopter shows which datasets are available for template writing and how many prompts each has.The view helps prioritize attention, move between datasets, and support prompt review.
6 Community Guidelines and Process
PromptSource uses community guidelines and varied prompt creation to address the difficulty of judging prompt quality and coordinating a shared prompt corpus. The process combines standardized metadata with explicit requirements for natural-language, task-clear templates.
- Community guidelines: Because prompt variation makes quality judgment difficult, PromptSource developed community guidelines for shared vocabulary, valid prompts, and common errors.The guidelines were developed iteratively rather than from a simple example-level agreement metric.
- Validity requirements: A valid prompt combines a template function with metadata and must express both input and target in natural language.Pure code is forbidden, and the task should be understandable to a non-specialist adult.
- Prompt quality: Early experiments found that prompts listing possible valid completions tended to outperform otherwise corresponding prompts without those completions.The guidelines also recommend keeping target templates limited to the task answer and moving extra wording into the input template.
- Variation and diversity: Prompt authors are encouraged to create 5–10 or more prompts per dataset while varying prompt formulation and sometimes task formulation.This supports research on whether training with diverse formulations improves robustness to prompt formulation at test time.
- Metadata: Prompt metadata records names, references or rationales, task alignment, valid outputs, whether outputs are stated in the input, and possible evaluation metrics.These fields support the community process around shared prompts.
7 Case Studies
PromptSource served as a resource for multiple research projects by providing a shared collection of prompts and a flexible templating system. The case studies illustrate its use in large-scale prompted training and related prompting methods.
- Case studies: PromptSource is presented as a key resource for research projects studying prompting and zero-shot behavior.The paper highlights three recent projects using the system.
- Templating: Its Jinja-based templating language supports expressive prompt editing, including string manipulation and conditional statements.The QA-ZRE example demonstrates these capabilities within the Sourcing view.
- Massively multitask prompted training: Sanh et al. used PromptSource to create diverse prompts for a large collection of NLP datasets in massively multitask prompted training.Their training and evaluation prompts were a subset of P3.
- Reuse: Users were able to reuse PromptSource’s template collection and templating language for priming methods, although the system was not originally designed specifically for that form of prompting.This demonstrates use beyond its original design context.
8 Conclusion
PromptSource is an open-source system for creating, sharing, and using natural-language prompts through centralized tools for prompt development and collaboration. By January 2022, it included 2,000 public prompts for roughly 170 datasets and supported multiple research projects.
- Conclusion: PromptSource combines a flexible template language, prompt-management tools, and community-driven quality standards.These components address collaborative and centralized needs in prompting research.
- Conclusion: By January 2022, PromptSource contained 2,000 public prompts for roughly 170 datasets.The collection was growing and had already supported multiple recent research projects.
A Data and Statistics
P3 is a large, actively growing collection containing thousands of English prompts across hundreds of datasets and subsets. Its development involved a broad international contributor community.
- A Data and Statistics: 2’052 English prompts cover 170 English datasets and 269 subsets as of January 2022.The collection averages 7.6 prompts per data subset and 5.6 original-task prompts per data subset.
- A Data and Statistics: P3 is described as the largest public collection of English prompts and is actively growing.
- A Data and Statistics: P3 was developed through a BigScience open hackathon focused on collecting prompts for English NLP datasets.Almost 50 contributors from more than 25 institutions in 10 countries participated.
B Complete Views
PromptSource provides complete interface views for browsing datasets and prompts, sourcing prompts, and inspecting prompt behavior. The paper presents these views through dedicated examples.
- B Complete Views: The Browse view is illustrated through a complete example interface.
- B Complete Views: The Sourcing view is illustrated through a complete example interface.
- B Complete Views: The Helicopter view is illustrated through a complete example interface.