PromptChainer: Chaining Large Language Model Prompts through Visual Programming

Background

• Prompting enables non-ML experts to prototype custom ML functionality with less effort and time by avoiding upfront data collection and model training.LLMs can be customized at run time through natural-language prompts, supporting tasks such as code generation, question answering, and creative writing [3] [13].Source: §1 · p. 1

Motivation

• Many real-world applications require multi-step tasks that are nontrivial for one LLM run, making realistic applications difficult to prototype with a single prompt.For example, a music chatbot may need to classify a query before generating a response.Source: §1 · p. 1
• Chaining decomposes an overarching task into targeted LLM steps, passing each step’s output to the next; prior work found it can make complex tasks more transparent and controllable [17].Authoring chains also requires users to design the overall task decomposition, motivating targeted tooling akin to end-user programming.Source: §1 · p. 2

Case studies

• Four designers and developers built chains for realistic application ideas, revealing that users chain prompts for extensibility and need both local and global debugging.Some built one step at a time, while others first sketched abstract placeholders for all steps.Source: §1 · p. 2

Challenges and interface

• Formative studies identified capability-utilization overhead, prompting errors, and cascading errors from black-box, unstable generations as challenges in authoring LLM chains.PromptChainer scaffolds users’ mental models of LLM capabilities, arbitrary data formats, function signatures for each step, and debugging cascading errors.Source: §1 · p. 2

Open challenges

• The paper leaves open how to scale chains to highly interdependent or logically complex tasks while preserving global context and coherence, and how to help users quickly prototype alternative chains without overinvesting in individual prompts.The latter challenge is finding a “sweet spot” for low-fidelity prompting.Source: §1 · p. 2

2 BACKGROUND: LARGE LANGUAGE MODELS, PROMPTING AND CHAINING

• Pre-trained on billions of internet samples, recent LLMs can be adapted on the fly for user-defined tasks, which users invoke with task-appropriate prompts [3] [1] [11].The passage gives code generation, question answering, and creative writing as examples of user-defined tasks.Source: §2 · p. 2
• Zero-shot prompts directly describe the task, whereas few-shot prompts provide examples of desired inputs and outputs for the LLM to follow.The passage illustrates both approaches with a music-classification task.Source: §2 · p. 2
• Because a single LLM is less capable of prototyping complex applications given limitations such as weak multi-step reasoning, prior work proposed chaining prompts [17].This paper builds on that work to explore how users can author effective chains for prototyping.Source: §2 · p. 2

3.1 Requirement Analysis

• Over three months, formative studies combined reflection on unaided LLM-chain authoring with iterative sessions in which participants built chains in early prototypes and reported pain points.The studies involved two software engineers and three designers.Source: §3.1 · pp. 2–3
• LLMs’ versatile capabilities require users to develop a mental model, while their arbitrary string outputs often need transformation to fit downstream inputs.Source: §3.1 · pp. 3–4
• Because LLM outputs lack stable semantic types, prompt edits can change a node’s output format and break downstream steps.For example, a step may begin returning numbered lists instead of short phrases.Source: §3.1 · p. 4
• A suboptimal or unsafe output from a black-box LLM step can cascade into errors across the chain, as also observed in traditional machine-learning pipelines.Source: §3.1 · p. 4

3.2 Interface Design

• The Chain View represents each chain step as a node and its connections as edges showing how one node’s output feeds the next.Nodes can expose named inputs and outputs, status icons, and inline or detailed data previews.Source: §3.2 · p. 4
• PromptChainer offers generic LLM and classifier nodes, transformation and evaluation helpers, custom JavaScript nodes, and communication nodes for external data exchange.An example gallery supplies frequently composed subchains and prompting patterns intended to help users anticipate useful capabilities and compatible processing nodes.Source: Figure 3 · p. 3, §3.2 · p. 4
• The Node View lets users inspect, implement, and test individual nodes, while automatically synchronizing input handles with names in prompts or JavaScript function signatures.Changing a prompt placeholder updates the corresponding handle and global chain to match.Source: §3.2 · p. 4
• Debugging supports independent node tests, end-to-end chain runs with per-node output logs, breakpoints, and direct edits to intermediate outputs.Editing an intermediate output lets users test downstream nodes independently of earlier errors.Source: §3.2 · pp. 4–5

4.1 Study design

• The study involved four participants—three designers and one developer—who prepared prompts before a one-hour session authoring and revising chains.Participants had some prior experience with non-chained prompts and spent about 90 minutes in total, including preparation.Source: §4.1 · p. 5

4.2 Study Results

• Participants chained prompts to address single-prompt limits and to make prototypes extensible, with one also adding a step to select and refine candidate outputs.P3 separated a general ideator from a replaceable classifier; P4 generated three story-spine candidates before selecting one for refinement.Source: §4.2 · p. 5
• All four participants built desired chains averaging 5.5 ± 0.9 nodes, spanning parallel branching tasks and iterative content-processing tasks.Their applications ranged from music chatbots and ad generators to writing assistants.Source: §4.2 · p. 5, Figure 5 · p. 6
• Participants used varied construction strategies, while all ran chains end-to-end and then debugged locally; observing node interactions also prompted refinements that helped uncover errors.P1 connected placeholder nodes top-down, while others generally worked node by node; breakpoints and independent testing supported local debugging.Source: §4.2 · pp. 5–6
• Participants used three times as many predefined helpers (13) as customized JavaScript nodes (4), and their 27 LLM nodes covered categorizing, sourcing, and reorganizing information.A participant also created a classifier by adapting a simple example despite limited prompting experience.Source: §4.2 · pp. 5–6
• Independent parallel subtasks could undermine coherence, while complex decompositions made it difficult to trace how intermediate entities mapped back to the original input.The authors suggest investigating ways to manage interdependencies and improve tracing, including grouping selected nodes and visualizing input-to-output mappings.Source: §4.2 · p. 7

4.3 Discussion and Limitations

• Future work could encourage low-fi prototyping of multiple chains and explore decomposition strategies for larger, more complex tasks.Prewritten prompts may have discouraged participants from considering alternatives; the authors suggest simpler initial prompts and splitting nodes further when prompting proves unsuccessful.Source: §4.3 · p. 7

• The authors identify three challenges for LLM chain authoring arising from LLMs’ versatile, open-ended capabilities.Source: §5 · p. 7
• PromptChainer helped users transform intermediate LLM outputs and debug chains when steps had interacting effects.Source: §5 · p. 7
• Future directions include supporting more complex chains and letting users sketch chain structures before investing heavily in prompts.Source: §5 · p. 7

A SAMPLE CHAINS FROM PILOT USERS

• Figure 6 gives the full details of the user-study chains.Source: Figure 6 · p. 9
• Figure 7 shows the chains built by pilot users.Source: Figure 7 · p. 10