AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts

Tongshuang Wu; Michael Terry; Carrie J. Cai

Source-linked AI summary

AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts

Tongshuang Wu, Michael Terry, Carrie J. Cai

arXiv:2110.01691v3Oct 4, 2021cs.HCcs.CL

TL;DR

Complex tasks expose LLM limitations in reasoning, transparency, and controllability, making human assistance difficult. The paper introduces Chains of scoped LLM steps with an interactive interface for editing prompts, intermediate data, and connections, and evaluates them in a 20-person study. Chaining improved task outcomes and human-AI experience while supporting calibration, strategy comparison, and debugging.

Problem
LLMs can be less effective, opaque, and difficult to control when assisting with complex tasks requiring multiple sub-parts or objectives.
Method
Chaining decomposes complex tasks into independently prompted LLM steps whose outputs feed later steps, supported by primitive operations and an interactive editing interface.
Results
In a 20-person within-subject study, Chaining significantly improved transparency, controllability, collaboration, mental support, and outcome quality, with higher-quality results about 82% of the time.
Takeaways & Limitations
Users used Chains to calibrate model expectations, compare prompting strategies through downstream effects, and debug outputs by isolating Chain components.
Takeaways & Limitations
Chaining can provide scaffolding while limiting free exploration, motivating more customizable prompts, parsers, and step implementations.

Abstract

from arXiv · show

Although large language models (LLMs) have demonstrated impressive potential on simple tasks, their breadth of scope, lack of transparency, and insufficient controllability can make them less effective when assisting humans on more complex tasks. In response, we introduce the concept of Chaining LLM steps together, where the output of one step becomes the input for the next, thus aggregating the gains per step. We first define a set of LLM primitive operations useful for Chain construction, then present an interactive system where users can modify these Chains, along with their intermediate results, in a modular way. In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration. Additionally, we saw that users developed new ways of interacting with LLMs through Chains: they leveraged sub-tasks to calibrate model expectations, compared and contrasted alternative strategies by observing parallel downstream effects, and debugged unexpected model outputs by "unit-testing" sub-components of a Chain. In two case studies, we further explore how LLM Chains may be used in future applications

1 INTRODUCTION

LLMs support many prompted tasks but become less effective and harder to collaborate with on complex, multi-objective work. Chaining decomposes such work into editable sub-tasks, improving outcomes while increasing transparency and control.

LLMs can perform diverse tasks through natural-language prompts, including translation, question answering, and story writing.
Complex real-world tasks can challenge single model runs because they require multi-step reasoning or simultaneous objectives.
Opaque prompts make unexpected outputs difficult to debug and users may struggle to form accurate models of LLM capabilities.
Chaining breaks a task into prompted sub-tasks whose outputs feed later steps, allowing the same LLM to address multiple components.
The interactive system exposes Chain structure and intermediate results so users can edit prompts, outputs, connections, and suggestions.
In a 20-person within-subject study, Chaining significantly improved transparency, controllability, collaboration, mental support, and task outcomes, with higher-quality results about 82% of the time.The same underlying LLM powered both interfaces.

2 BACKGROUND AND RELATED WORK

Prior work establishes LLM prompting and human-AI collaboration as promising but identifies persistent interaction and workflow challenges. Chaining draws on decomposition and pipelining ideas while targeting limitations specific to LLM reasoning and prompt variability.

LLMs are versatile in-context learners, but their outputs depend on careful prompt design and mined prompts can be less human-readable.
Human-AI interaction research spans domains including classification, drawing, translation, creative writing, and design ideation, while transparency, controllability, and agency remain concerns.
Prior collaboration systems position AI as exploration-oriented creators or assistants that compensate for human weaknesses.
Chaining is inspired by crowdsourcing pipelines that decompose complex tasks into independently performed pieces and combine them.
Unlike crowdsourcing workflows, Chaining uses more scoped and granular sub-tasks because LLM limitations include weak reasoning, variable prompt effectiveness, and exposure bias.

3 CHAINING LLMS

The paper frames Chaining as a response to LLM weaknesses on broad tasks: independently prompted, connected steps narrow inputs and outputs to more targeted operations. It defines the approach and derives primitive building blocks from existing LLM use cases.

LLMs may struggle when inputs are poorly formatted, contain extraneous data, or require multiple sub-parts, while targeted operations may perform better.
Chaining breaks complex tasks into smaller steps, with each step completed by an independent LLM run and outputs passed to later steps.
The paper surveys LLM literature and 73 online demonstrations to identify challenges and promising primitive operations for Chain construction.
The approach scopes inputs and outputs to make sub-tasks more amenable to LLM capabilities.

C.1 LLMs lack multi-step reasoning capabilities. Because

The paper links LLM failures to broad, sequential, and poorly structured inputs, then proposes primitive operations and connected data layers to scope and organize model work. Prompt templates operationalize these steps across custom inputs and outputs.

LLM challenges: LLMs struggle with multi-step reasoning, branching logic, long text generation, and diverse content because of language-form learning and exposure bias.
LLM challenges: Prompt effectiveness varies with wording and information structure, with unnatural formats and some word types receiving less effective treatment.
Primitive operations: The paper reviews 73 LLM demonstrations to identify primitive operations that narrow inputs and outputs around recurring model challenges.
Primitive operations: Classification can validate or triage inputs, while rewriting, information extraction, and splitting can make complex content more manageable.
Chain structure: An LLM Chain connects multiple operation-defined steps through shared data layers, such as splitting feedback into problems before ideating suggestions.
Prompt design: Operations use natural-language prompts with task-specific or few-shot patterns and meaningful prefixes that reinforce intended structure.
Prompt design: Prompt templates support custom input and output data layers through placeholders and optional few-shot examples, but the presented template is not exhaustive.

4 INTERACTIVE USER INTERFACE

The interface exposes LLM Chains through linked Chain and Step views, enabling users to inspect, execute, and modify prompts, intermediate data, and overall structure. It supports local edits to prompts or outputs and global rewiring of Chain operations, with changes reflected across affected steps.

Interface views: The interface provides a Chain view for high-level structure and a Step view for refining and executing individual LLM steps.The Chain view shows the workflow, while the Step view exposes inputs, outputs, and prompt structure.
Design rationales: The design evolved from pilot feedback emphasizing visible Chain progress, flexible local fixes, editable structures, and preservation of natural-language interaction.The final interface uses in-line text fields rather than structured tables and distinguishes local from global modifications.
Interface views: The Chain view visualizes LLM operations, data layers, execution progress, and the links connecting each output to its downstream input.Operation glyph shapes indicate 1-1, 1-N, and N-1 mappings; updated data rows help users track execution.
Interface views: The Step view separates shared instructions from parallel running blocks, allowing users to inspect how each complete prompt is assembled.Each running block represents a distinct LLM invocation, combining shared instruction text with path-specific content.
Execution and iteration: Users can run individual parallel blocks or all blocks together, preview the resulting prompt, and edit parsed outputs for further iteration.Comparing parallel results supports inspection of alternative executions within a step.
Execution and iteration: The interface offers local prompt edits, intermediate-output edits, and global Chain restructuring, with broader changes triggering a full re-render.Prompt edits affect one step, output edits affect steps sharing the data layer, and structural edits refresh defaults such as temperature and instructions.

5 USER STUDY

In a within-subject study, Chaining helped participants use LLMs across multiple sub-tasks, make finer-grained interventions, and produce higher-quality task results. Participants also customized Chain structures and used intermediate outputs to evaluate and improve individual components.

Study Design: 20 participants completed Flashcard or Review tasks with both Chaining and Sandbox interfaces.The study counterbalanced interface order and analyzed ratings, clickstreams, and final results.
Quantitative Results: Chaining reduced consecutive model reruns without prompt changes from 51% in Sandbox to 36%, a significant difference (t=3.5, p=.001).Participants instead made more manual interventions in Chaining.
Quantitative Results: 77% of Chaining edits curated existing text, compared with 41% in Sandbox; undo and create accounted for 18% and 5% in Chaining.The shift toward curation was significant (t=−6.75, p<.001).
Quantitative Results: 85% and 80% of independent-rater comparisons preferred Chaining results, while participants rated their own Chaining results 6.0 ± 0.9 versus 5.0 ± 1.1 for Sandbox.The two independent raters consistently preferred Chaining results, and participants’ ratings showed the same direction.
Chain Customization: 15 participants suggested at least one Chain change, and 11 successfully implemented and executed one of their own solutions.Five participants preferred the provided default Chains.
User Strategies: Chaining supported multiple model roles, including generating diverse flashcard content while preserving translation correctness.Participants used Ideation for diversity and Rewriting for translation, whereas Sandbox users often treated the model only as a translator.
User Strategies: Participants tuned temperature by sub-task, increasing it for Ideation to broaden diversity and lowering it for Rewriting to improve correctness.Four Flashcard participants made this adjustment in Chaining, but none did so in Sandbox.
Intermediate Outputs: Participants could freeze preferred intermediate data points, allowing them to improve later components without losing earlier progress.This supported evaluation and improvement of individual Chain components independently of previous failures.

6 CASE STUDIES

Two case studies illustrate how LLM Chains can support complex applications in visualization debugging and assisted text entry. They use intermediate processing and branching logic to address tasks that are difficult for a single prompt.

6 CASE STUDIES: Two case studies apply Chaining to visualization code debugging and assisted text entry.Both applications use the same model as the user study.
6.1 Case 1: Visualization code debugging: A single LLM run can struggle with visualization debugging because visualization specifications are difficult to parse in JSON format.The difficulty is especially relevant when the goal is to reason about violated design constraints.
6.1 Case 1: Visualization code debugging: Visualization debugging uses a Chain that rewrites JSON specifications, validates natural-language descriptions, and rewrites the specification.This structure helps expose violated design constraints when multiple valid fixes exist.
6.2 Case 2: Assisted Text Entry: Assisted text entry uses gradual expansion and if-else logic to disambiguate shorthand inputs and autocomplete already-expanded phrases.Users can select among expansions or add further shorthand when the desired option is absent.

7 DISCUSSION & FUTURE DIRECTIONS

Chaining improves transparency, controllability, and debuggability by treating multiple LLM runs as modular, steerable units for complex-task support and rapid prototyping. The discussion also identifies boundaries around task decomposition, user authoring, exploration, information loss, and computational cost.

Implications: Chaining reshapes interaction rather than retraining the model, boosting transparency, controllability, and debuggability.The authors describe this as a first step toward realizing latent LLM potential through interaction design.
Chaining as a new paradigm of control on multiple model units: Multiple LLM runs can serve as controllable units whose upstream changes cascade through downstream steps and whose parallel outputs support comparison.This extends control from properties within one model unit to interactions among multiple model runs.
Future directions: Future Chain designs may include model units for task decomposition, correctness checkpoints, and interleaving LLM steps with human computation.These units could help users create sub-tasks and verify data along the Chain.
Chaining for rapid prototyping of integrated applications: Chaining supports rapid prototyping by letting developers envision alternative application flows and A/B test Chains before building bespoke models.Assisted text entry illustrates testing an added context layer for shorthand expansion.
Limitations and challenges: Each additional LLM run introduces computational overhead, which must be weighed against the effort saved by avoiding bespoke single-purpose AI systems.The authors believe the saved effort outweighs the added cost.
Limitations and challenges: Chaining may limit free exploration, especially when structural constraints prevent users from experimenting and forming mental models of AI.The authors suggest loosening constraints within steps as one possible balance.
Limitations and challenges: End-user Chain authoring remains open, and a pre-built decomposition may reduce transparency when it mismatches how users naturally divide a task.The paper leaves effective end-user authoring to future work.
Limitations and challenges: Decomposition can lose coherence or quality for interdependent tasks, while omitted data layers can distort or compress the task.The discussion recommends further assistance for crafting Chain steps.

B.1 Questions in the Exit Survey

The exit survey collected participants’ comparative ratings and explanations after they completed tasks with Sandbox and Chaining. Questions addressed satisfaction, thought support, transparency, controllability, differences between interfaces, and preferred usage situations.

Survey ratings: Participants rated Sandbox and Chaining on seven-point Likert scales after completing the task in both conditions.They described their reasoning alongside the ratings.
Survey ratings: Survey dimensions included satisfaction with the task goal, support for thinking through outputs and procedures, transparency, and controllability.The questions asked whether users could track progress and steer creation.
Open-ended questions: Free-form questions asked participants to compare the interfaces and identify work situations or examples where they would prefer Sandbox or Chaining.The survey solicited one to three concrete examples of preferred use.

B.2 Clickstream Categorization

The clickstream categorization reconstructs how participants changed text around model runs and distinguishes consecutive runs from runs followed by human edits.

Action categories: Logs captured text before and after each model run, then categorized actions as RUN, UNDO, FORMAT, CREATE-CONTENT, CURATE-CONTENT, or CHANGE-TEMPERATURE.CURATE-CONTENT includes edits to instructions, prefixes, and data entries in Chaining.
Run classification: Consecutive runs were those without substantive changes after the prior run, while other cases counted as human edits.Formatting-only changes such as line breaks or stopwords were included with consecutive runs.

B.3 Case 0: Metaphor Creation (Used in tutorial)

The tutorial illustrates prompting variations for metaphor creation and related language tasks, including question-answering, commands, list enumeration, and multi-step rewriting.

Metaphor creation is demonstrated through examples that map a concept to a simile and a shared aspect.Examples describe crowdsourcing as a team sport or beehive, emphasizing collective work toward a goal.
Question-answering and command-style prompts provide alternative baseline formats for generating metaphors.The examples ask for a good metaphor directly or instruct the model to write one, producing different metaphor formulations.
The tutorial also contrasts prompt formats for personalized French flashcard creation, including question answering and list enumeration.The task targets diverse travel-oriented sentence pairs, with examples covering restaurants, bus stations, and language ability.
Peer-review rewriting is framed as producing a friendly paragraph with concrete suggestions for each presentation problem.A two-step version first elicits improvement suggestions and then integrates them with the original problems.

C FULL LLM CHAINS FOR CASE STUDIES

The case-study Chains decompose visualization debugging and assisted text entry into staged operations. Each Chain routes intermediate results into later classification, rewriting, or generation steps.

The visualization bug-fixing Chain rewrites a VegaLite JSON specification into natural language, extracts related rules, classifies validity, and generates a fixed specification.The stages progressively remove data noise, identify relevant visualization rules, verify concrete errors and fixes, and produce the repair.
The assisted text-entry Chain classifies shorthand, expands it when present with human selection among alternatives, or otherwise autocompletes the sentence.For “LTSG,” rewriting may produce “Let’s go” or “Let’s get”; without shorthand, the Chain uses generation.

D THE FULL IMPLEMENTATION OF PRIMITIVE OPERATIONS

The paper presents primitive operations as reusable building blocks for constructing LLM Chains, supported by default prompting templates and temperatures.

Primitive operations include methods for gathering additional clues when desired outputs are too long or diverse.These operations target cases where a single generation may not adequately cover the requested content.
The primitive building blocks are grouped by intended objectives and paired with default prompting templates and temperatures.The listed examples are drawn from OpenAI’s GPT-3 example collection.
The operations are intended to support modular Chain construction rather than constitute an exhaustive inventory.The paper’s operation set is presented as a practical design basis for composing LLM steps.