Source-linked AI summary
PromptChainer: Chaining Large Language Model Prompts through Visual Programming
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, Carrie J Cai
TL;DR
Complex applications often exceed what a single LLM run can handle, motivating chains of coordinated prompts whose authoring needs remain underexplored. This paper studies those needs and presents PromptChainer, a visual programming interface with scaffolds for transformations and multilevel debugging. Case studies with four designers and developers show support for varied application prototypes, while leaving open how to scale chains and prototype alternatives cheaply.
Problem
Complex applications are difficult to prototype with a single LLM prompt, while how users should author their own chains remains an open question.
Method
The paper studies LLM-chain authoring and designs PromptChainer, a visual interface with scaffolds for data transformation, chain construction, and multilevel debugging.
Results
Case studies with four designers and developers show that PromptChainer supports chains for diverse tasks, including branching and iterative content-generation applications.
Takeaways & Limitations
Users chain prompts both to address LLM limitations and to make prototypes more extensible or allow intervention over intermediate results.
Takeaways & Limitations
The study may have favored tasks that were easy to decompose, leaving task decomposition for larger and more complex tasks as future work.
Abstract
from arXiv · showhide
While LLMs can effectively help prototype single ML functionalities, many real-world applications involve complex tasks that cannot be easily handled via a single run of an LLM. Recent work has found that chaining multiple LLM runs together (with the output of one step being the input to the next) can help users accomplish these more complex tasks, and in a way that is perceived to be more transparent and controllable. However, it remains unknown what users need when authoring their own LLM chains -- a key step for lowering the barriers for non-AI-experts to prototype AI-infused applications. In this work, we explore the LLM chain authoring process. We conclude from pilot studies find that chaining requires careful scaffolding for transforming intermediate node outputs, as well as debugging the chain at multiple granularities; to help with these needs, we designed PromptChainer, an interactive interface for visually programming chains. Through case studies with four people, we show that PromptChainer supports building prototypes for a range of applications, and conclude with open questions on scaling chains to complex tasks, and supporting low-fi chain prototyping.
1 INTRODUCTION
LLMs support rapid prototyping of individual functionalities, but complex applications often require multiple coordinated steps. PromptChainer investigates how users author such chains and provides scaffolding for transformation, prompting, and debugging, while identifying open challenges for more complex and low-fi chains.
- Motivation: Complex applications such as music chatbots may require multiple LLM steps rather than a single prompt.The chatbot example first determines query type and then generates a response based on that type.
- Motivation: Chaining decomposes an overarching task into targeted LLM steps, passing each step’s output to the next.Prior work found chaining could support more complex tasks in a more transparent and controllable way.
- Authoring challenge: Authoring chains requires users to manage task decomposition in addition to prompting individual LLM steps.The paper frames this need as targeted tooling akin to end-user programming.
- PromptChainer: PromptChainer addresses chain-authoring challenges with scaffolds for understanding LLM capabilities, transforming arbitrary data, defining step interfaces, and debugging cascading errors.The interface was informed by formative studies and supports the authoring process across these needs.
- Open challenges: The paper identifies future challenges in scaling chains to highly interdependent or logically complex tasks while preserving global context and coherence.This is presented as an open question rather than a resolved capability.
- Open challenges: It also raises the need for low-fi prototyping that lets users test multiple alternative chains without overinvesting in individual prompts.The paper describes this as finding a prompting sweet spot for rapidly exploring alternatives.
2 BACKGROUND: LARGE LANGUAGE MODELS, PROMPTING AND CHAINING
LLMs can be adapted through prompts for user-defined tasks, but single prompts remain limited for complex applications. Prompt chaining addresses this by decomposing tasks across multiple prompts and passing outputs between steps.
- Large language models: LLMs can support user-defined tasks through natural-language prompts without retraining.Zero-shot prompts describe the task directly, whereas few-shot prompts provide examples of desired inputs and outputs.
- Prompt chaining: Single LLMs are less capable for complex applications because of inherent limitations such as weak multi-step reasoning.The paper motivates chaining as a response to these limitations.
- Prompt chaining: Prompt chaining decomposes an overarching task into multiple LLM prompts whose outputs become inputs to subsequent steps.The approach is presented as a way to prototype applications beyond the capabilities of a single prompt.
3 PROMPTCHAINER: INTERFACE REQUIREMENT ANALYSIS & DESIGN
PromptChainer is a visual programming interface designed around the challenges of authoring LLM chains: arbitrary intermediate formats, unstable step interfaces, and cascading errors. Its views, node types, examples, synchronized handles, and multilevel debugging support chain construction and iteration.
- 3.1 Requirement Analysis: Formative studies with software engineers and designers examined unsupported chain authoring and iterative prototype design to identify authoring challenges.The studies occurred over three months and included participants proposing and building chains of interest.
- 3.1 Requirement Analysis: Arbitrary LLM output formats make it difficult to transform upstream results into inputs compatible with downstream steps.This challenge arises because users must develop a mental model of highly versatile LLM capabilities while connecting steps.
- 3.1 Requirement Analysis: Unstable LLM function signatures mean prompt edits can change semantic output types and introduce downstream input errors.For example, an edit can change short phrases into numbered lists and break the chain.
- 3.1 Requirement Analysis: Black-box or sub-optimal output from one step can produce cascading errors across the chain.PromptChainer therefore includes debugging support at multiple granularities.
- 3.2 Interface Design: The Chain View represents nodes and their connections, while the Node View supports implementing, improving, and testing individual steps.Edges represent how one node’s output is used as another node’s input.
- 3.2 Interface Design: PromptChainer combines generic LLM and classifier nodes with helper nodes for transformation, evaluation, custom JavaScript, and external communication.These node types target diverse prototyping and data-exchange needs.
- 3.2 Interface Design: Example chains provide reusable composition patterns and encourage outputs compatible with predefined processing nodes.An Ideation example producing numbered lists is paired with a node that splits text by number.
- 3.2 Interface Design: Prompt templates synchronize input handles with placeholders, keeping the global chain consistent after local edits.The interface also offers node previews, status indicators, unit tests, end-to-end logs, breakpoints, and editable intermediate outputs.
4 USER FEEDBACK SESSIONS
In a preliminary study with four participants, PromptChainer supported diverse chain structures, construction strategies, and multi-level debugging, while exposing challenges in coherence, traceability, and low-fi exploration.
- Chaining patterns: Participants created parallel branching chains for conditional processing and iterative chains that divided content generation before recombining it.P1 and P2 used branching logic, while P3 and P4 incrementally processed content using divide-and-conquer strategies.
- Chaining rationales: Participants used chaining to make prototypes more generalizable and to retain control over intermediate outputs, not only to address single-prompt limitations.Examples included swapping classifiers for different variables and selecting one candidate story spine for later expansion.
- Chain construction: PromptChainer supported both top-down and node-by-node construction, although participants commonly drafted one node at a time before refining prompts.P1 connected placeholder nodes first, while the other participants either tested or rough-drafted nodes sequentially.
- Node utilities: Pre-defined helpers covered most observed chaining needs: participants used 13 helpers versus 4 customized JavaScript nodes across eight study and pilot-user chains.Among 27 LLM nodes, 7 categorized inputs, 13 sourced information, and 7 reorganized inputs.
- Chain debugging: Participants debugged chains end-to-end and locally, using output previews, breakpoints, and independent node tests to identify failures without repeatedly executing upstream steps.PromptChainer also helped users discover interaction errors, though resolving them through alternative solutions remained an open challenge.
- Remaining challenges: Remaining challenges included maintaining coherence across interdependent parallel tasks, tracing complex decompositions, and exploring multiple low-fi chain alternatives.The study also noted that pre-created prompts and time constraints may have narrowed participants’ exploration of alternative structures and more complex tasks.
5 CONCLUSION
PromptChainer addresses three challenges in authoring LLM chains and supports transforming intermediate outputs and debugging interacting steps. The conclusion also identifies scaling complex chains and enabling low-fi construction as future directions.
- The authors identify three challenges arising from LLMs’ versatile and open-ended capabilities.These challenges motivate the design of PromptChainer.
- PromptChainer helps users transform intermediate LLM outputs and debug chains with interacting steps.
- Future work includes supporting more complex chains and half-baked chain construction before users invest heavily in prompts.The latter would let users sketch chain structure without extensive upfront prompting.
A SAMPLE CHAINS FROM PILOT USERS
The paper includes figures showing full details of user-study chains and chains built by pilot users. A separate example concerns generating descriptions for imaginary summer vacation photos.
- One listed example is an image-query generator for descriptions of imaginary summer vacation photos.
- Figure 6 presents the full details of the user-study chains.
- Figure 7 presents chains built by pilot users.