PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Code Generation: LLM-supported Exploration of the Program Design Space

J. D. Zamfirescu-Pereira, Eunice Jun, Michael Terry, Qian Yang, Björn Hartmann

arXiv:2503.06911v1cs.HC

TL;DR

Program design requires iterative exploration of alternative problems and solutions, but code-generation LLMs typically present a single point solution. The authors develop Pail, an IDE that supports higher-level design exploration and decision tracking, and find broader exploration alongside challenges in attention and update awareness.

  • Problem

    Program-design support is limited because code-generation LLMs typically deliver a particular point solution rather than exposing alternative problem formulations and solutions.

  • Method

    The authors develop Pail, an IDE combining chat and LLM-based agents to elicit alternatives, explore problem and solution formulations, track requirements, and surface implicit decisions.

  • Results

    100% of participants found at least one unconsidered alternative that influenced their design work, while participants also struggled to keep up with LLM-originated changes and generated information.

  • Takeaways & Limitations

    Supporting program design with LLM agents broadens exploration but creates challenges in managing attention, maintaining awareness of updates, and balancing information relevance with user agency.

  • Takeaways & Limitations

    The authors identify Pail’s generalizability and transitions across levels of abstraction as key limitations requiring future work.

Abstract

from arXiv · show

In this work, we explore explicit Large Language Model (LLM)-powered support for the iterative design of computer programs. Program design, like other design activity, is characterized by navigating a space of alternative problem formulations and associated solutions in an iterative fashion. LLMs are potentially powerful tools in helping this exploration; however, by default, code-generation LLMs deliver code that represents a particular point solution. This obscures the larger space of possible alternatives, many of which might be preferable to the LLM's default interpretation and its generated code. We contribute an IDE that supports program design through generating and showing new ways to frame problems alongside alternative solutions, tracking design decisions, and identifying implicit decisions made by either the programmer or the LLM. In a user study, we find that with our IDE, users combine and parallelize design phases to explore a broader design space -- but also struggle to keep up with LLM-originated changes to code and other information overload. These findings suggest a core challenge for future IDEs that support program design through higher-level instructions given to LLM-based agents: carefully managing attention and deciding what information agents should surface to program designers and when.

1 Introduction

Program design iterates over both problem formulations and implementation details, whereas conventional code-generation visions often begin from a fixed goal and produce a point solution. Pail extends LLM programming support to explore alternatives, track design decisions, and surface implicit choices within an IDE.

  • 1 Introduction: Program design iterates on both implementation details and design goals, so exploring alternative problem formulations precedes and informs solution development.The paper contrasts this process with visions that assume one known goal and restart after major changes.
  • 1 Introduction: A formative study found that chat-based design caused lost requirements and difficult retrieval of prior questions, decisions, and discussions.These findings motivated the development of Pail.
  • 1 Introduction: LLM code-generation systems commonly provide a single point solution, which can obscure alternatives and create anchoring bias.The formative study found that individual point solutions without consideration of alternatives led to significant anchoring bias.
  • 1 Introduction: Pail embeds LLM assistance in an IDE that elicits design alternatives, explores alternative problem and solution formulations, and tracks requirements and implicit decisions.Its agents support higher-level design reasoning while still enabling direct code manipulation.
  • 1 Introduction: In a lab study with 11 participants, 100% found at least one unconsidered alternative that influenced their design work.Participants also appreciated direct summaries and manipulating design decisions in situ.
  • 1 Introduction: The study identifies attention management as a challenge because agents and interfaces can generate more information than developers can effectively track.Future tools must balance information surfacing with users’ attentional capacities and agency.

2 Related Work

Prior work frames design as iterative exploration of problem and solution spaces, including rapid sketching, exploratory programming, and explicit comparison of alternatives. This paper builds on those traditions while examining how multiple LLM-supported design affordances can be integrated into one programming environment.

  • 2 Related Work: Design processes iteratively reformulate problems while exploring solution spaces, with sketches supporting rapid, disposable, and sometimes parallel alternative exploration.Sketches are distinguished from prototypes by their suggestive, plentiful, and cognitive-offloading role.
  • 2 Related Work: Exploratory programming uses code to prototype open-ended ideas, while opportunistic programming prioritizes speed and ease over robustness or maintainability.LLM code generation provides a new source of code for such prototypes.
  • 2 Related Work: HCI systems have supported explicit comparison of alternatives across text-to-image prompting, computational notebooks, code, and parameterized variants.These systems establish a precedent for making alternative design paths visible and comparable.
  • 2 Related Work: Research on LLM assistance spans programming support, creative design, prompting, agency, perception, and structured workflows for composing and evaluating model calls.The paper draws together these strands to study more complex cognitive support tools.
  • 2 Related Work: The paper targets the challenges that arise when multiple design, organizational, evaluative, and grounding affordances are integrated into one system.This focus complements prior work on individual interventions and multi-agent coordination.

3 Designing Pail

The authors use Research through Design to develop Pail, a web-based IDE for probing how LLMs might support program design and prototyping. The tool and study focus on experienced programmers creating small interactive prototypes because model fidelity and the study’s design-support focus constrain the target setting.

  • 3 Designing Pail: Pail is a web-based IDE for p5.js sketches developed through a Research through Design approach to probe LLM-supported program design.The system was developed iteratively using prior work, a formative study, and co-author feedback.
  • 3 Designing Pail: The tool and study focus primarily on experienced programmers creating small interactive prototypes for personal or non-deployment professional use.This focus was established by practical constraints on the tool and study design spaces.
  • 3 Designing Pail: Current LLM code-synthesis fidelity limits the complexity of programs that can be effectively constructed or iterated with limited human intervention.The authors therefore target smaller JavaScript programs using p5.js that fit within model context windows.
  • 3 Designing Pail: Because the study examines design support rather than programming education, it primarily targets participants with strong programming knowledge.The authors distinguish this aim from work supporting end-users learning to program.

3.2 Formative Study

A formative study examined how participants used ChatGPT or Claude to create interactive p5.js prototypes through chat. Participants valued rapid code generation but struggled to preserve design discussions, locate experiments, review generated code, and articulate user needs for others.

  • 3.2 Formative Study: Five HCI and design students with varied interactive-prototyping experience participated in a formative study of LLM-supported program design.Participants were in their final two undergraduate years or doctoral programs.
  • 3.2 Formative Study: Over six weeks, participants used ChatGPT or Claude to generate p5.js prototypes for interactive programs, copying generated code into a standard p5.js environment.The tasks included games, simulations, art showcases, real-estate aggregators, and sensemaking tools.
  • 3.2 Formative Study: Chat discussions rapidly disappeared into history, making requirements and design decisions difficult to find or recall and less likely to influence later iterations.Participants identified tracking discussion outcomes as a major challenge.
  • 3.2 Formative Study: Participants appreciated quickly generating code to test ideas but struggled to identify experiments, revert to prior code, and inspect generated code for errors.A common loop involved running generated code and asking the LLM to fix observed errors rather than reviewing the code directly.
  • 3.2 Formative Study: Participants who mediated generation for others spent substantial time identifying user needs and synthesizing solutions, indicating a need for more concrete epistemic goals in prototyping.These activities arose when creating interactive artifacts for someone else or observing a novice user.

3.3 Design Goals

Pail’s design goals support explicit exploration of alternatives, visible tracking of requirements and decisions, abstraction-level communication, and attention to end users.

  • Alternative designs: Pail should generate, track, and compare alternative designs, including prospective comparisons before alternatives are selected.The design process uses a lightweight pull model and suggests alternatives whenever possible.
  • Requirements and decisions: Pail should track requirements and explicit decisions outside the conversation so they remain salient to both the designer and the LLM.
  • Abstraction-level interaction: Pail should help users communicate at a level above code by surfacing natural-language decision points, grounding jargon, and summarizing code changes.Grounded terminology gives users and Pail shared names for concepts, while change summaries reduce context switching between design and code.
  • User-centered design: Pail should help users consider the needs and goals of the programs’ ultimate users.
  • Requirements and decisions: Pail should surface implicit LLM decisions, such as data representation and constraint operationalization, to enable consideration of alternatives.The formative study found that unsurfaced decisions were discussed only when they visibly affected the program.

3.4 System Design

Pail integrates chat with a design panel whose agents and views help users manage goals, requirements, implicit decisions, abstractions, and alternative code changes.

  • Architecture: Pail combines a ConversationAgent, DesignAgent, and ReflectionsAgent with views aggregated in a right-side design panel.
  • Design panel: The design panel tracks design questions and goals, confirmed requirements, implicit LLM decisions, and useful abstractions.Useful abstractions provide terminology and descriptions for concepts shared by the user and AI.
  • Design panel: Each design-panel item includes a rationale and two or three alternatives, with some alternatives hidden unless identified as important.
  • Chat and code: The ConversationAgent reads, patches, or replaces project code while guiding users through identifying users, assessing needs, considering goals, and generating prototypes.
  • Chat and code: Chat updates the design panel through the ReflectionsAgent, while trying a design alternative triggers speculative, uncommitted code changes through the DesignAgent and ConversationAgent.
  • Scope: Pail focuses on design-process support and excludes debugging, nonfunctional-code handling, automated QA, and simulated user testing.The authors leave these areas for future work.

3.5 Pail Implementation

Pail’s implementation supports rapid, incremental updates to both code and design-panel content while using speculative assessments to preview alternative impacts.

  • Application architecture: Pail is a React single-page application that proxies GPT-4o calls through a Node.js backend and uses Monaco to display inline code differences.
  • Application architecture: The browser runs p5.js code in an iframe with loop protection and proxies console messages to the IDE.
  • Speculative assessment: Pail stores code and design-panel JSON as separate versioned artifacts and speculatively assesses selected alternatives to indicate likely impacts and improve latency.
  • Speculative assessment: Speculative assessments examine effects on design goals, newly revealed requirements, and newly emerging design questions.
  • Incremental updates: Pail supports incremental code edits by sending line-numbered active code to the LLM and extracting tool-call replacements using line numbers and fuzzy matching.This approach avoids unreliable full-file diff formatting, whose edits frequently misaligned or inserted code incorrectly.
  • Incremental updates: Design-panel updates progressively match streaming incomplete JSON against existing items and display new content only when confidence is high that it is genuinely new.

4 Pail Usage: An Example

In the usage scenario, Pail helps Sam iteratively frame a reading game, inspect and try alternatives, and test prototypes with her child while retaining design knowledge.

  • Framing the design: Sam begins by asking Pail for a reading game, then reviews design questions about skills, activities, themes, users, and goals.
  • Framing the design: Sam selects “Focus on simple words,” after which Pail follows up with further design exploration.
  • Trying alternatives: When the agent proposes drag-and-drop, Sam finds it unsuitable and tries Pail’s alternative of tap-to-select, which is assessed as simpler and more intuitive for very young children.
  • Trying alternatives: Sam then tries a grid layout instead of the initial single-line display, and Pail records the resulting requirement and code changes.
  • Rapid prototyping: Pail lets Sam sketch with code by reducing the activation energy of choosing an environment, data model, rendering pipeline, and other boilerplate decisions.
  • Rapid prototyping: After 10 minutes, Sam learns that Alex likes emoji iconography but finds the targets too small, then continues iterating with Alex’s feedback.

5 User Study Procedure

The study used Pail as a probe to examine how its design-oriented affordances support program design and what challenges arise during AI-assisted exploration. Participants completed ambitious interactive-software tasks, while researchers observed their attention, decisions, and use of Pail.

  • The study aimed to identify how Pail’s affordances helped or failed to help users and to identify challenges inherent to AI-supported design.
  • Participants were encouraged to use Pail as a design tool, requesting broad goals while considering target users and high-level communication goals.
  • Tasks were chosen to combine ambitious, ambiguous goals with robustness to common LLM failure modes and observable diversity in design approaches.
  • Participants completed three tasks: an article-related interactive feature, a child-focused learning game, and a simulation of medical overtesting.
  • The sample included 11 participants with varied programming, design, and LLM experience, including five academics, six professionals, and three parents.
  • The participant pool was not representative of the broader population and was skewed toward design- and STEM-related professionals and graduate students.
  • Researchers analyzed think-aloud videos and interviews by tracking attention, design decisions, influencing factors, and interviewer interventions, then comparing participants through affinity diagramming.

6 Findings

Pail supported rapid, high-level exploration across problem formulations and solutions, but participants struggled with attention, initial orientation, and keeping track of changes. Interactive prototypes helped expose mismatches and enabled broad design iteration, while the design panel supported decision tracking and alternative exploration.

  • Rapid iteration: Participants frequently iterated rapidly at higher abstraction levels, treating prototypes as disposable sketches across discovery, definition, development, and delivery.A single action could span multiple design phases.
  • Starting design exploration: 9/11 participants initially stumbled over choosing the first action, and 7/11 were unsure which abstraction level to use for their first request.ConversationAgent questions helped many participants explicitly consider goals, users, needs, and outcomes.
  • Reconsidering design: Some users preferred starting from a point solution or example, suggesting that future tools should support comparable alternatives to reduce anchoring.Pail’s design-question workflow encouraged explicit goal reflection for many participants, but not all.
  • First contact and local iteration: First prototypes exposed mismatches between participants’ expectations and the code or agents’ interpretations, prompting low-level implementation fixes and further exploration.These mismatches often led to rapid local iteration lasting from under a minute to more than 20 minutes.
  • Attention during iteration: Participants shifted attention among code, output, chat, and the design panel; programming expertise affected code inspection, while 7/11 participants eventually ignored the code.More experienced participants often inspected code for confirmation rather than editing it directly.
  • Design panel: Nearly all participants found the design panel useful for scanning decisions, tracking updates, and reviewing rationales and alternatives.Eight participants valued summarization and tracking, while seven appreciated rationales and alternatives.
  • High-level change: A participant found Pail’s high-level design changes more effective than ChatGPT’s output for making changes at that abstraction level.The participant reported that ChatGPT took longer and lacked the appropriate mechanism for high-level changes.

7 Discussion

The discussion identifies attention management as a central challenge for LLM-supported program design, especially as tools accelerate nonlinear exploration and generate information across multiple design activities. Effective systems must align their support with users’ starting points while balancing information relevance, agency, and control.

  • Open challenges: LLM-supported program design must help users define goals, evaluate progress, and move between problem and solution spaces.The authors identify goal definition, progress evaluation, and transitions between abstraction levels as open challenges.
  • Nonlinear design: Pail’s rapid code generation enabled participants to shift nonlinearly between exploring solutions and reconsidering whether their problem formulations were addressable.Running code functioned as an exploratory sketch, supporting movement between extrapolating designs and concretizing them.
  • Attention management: Design controls broadened exploration but sometimes overwhelmed users with incoming information whose low signal-to-noise ratio risked being ignored.The overload arose partly because automation compressed activities that normally unfold at a human timescale and occurred concurrently across multiple facets.
  • Information management: Future systems need a new interaction layer for managing information that changes substantially over time when its importance to users is difficult to assess.The authors connect this challenge to deciding what feedback is useful, when it is useful, and how it should be delivered.
  • Agency and control: LLM-powered tools raise questions about who directs user attention because black-box model behavior can make design affordances unpredictable and demanding to learn.The authors caution that showing many affordances at once may make future systems less empowering rather than more empowering.

8 Limitations and Future Work

The authors position Pail as one exploratory point in the design space rather than a system shown superior to conversational baselines. They identify generalizability, design-process literacy, and transitions across abstraction levels as priorities for future work.

  • Scope of evaluation: Pail was designed to expose challenges in integrating explicit design support, not to demonstrate superiority over ChatGPT or Claude.The evaluation therefore did not explicitly compare Pail with those baselines.
  • Limitations: The work identifies Pail’s generalizability and transitions across abstraction levels as two key limitations requiring future study.These limitations concern both where Pail can be used and how designers move between high-level concerns and code-level details.
  • Generalizability: Pail is not intended for every programming task or programmer, and effective use requires some design-process literacy.The authors note that this literacy could be taught or designed into tools.
  • Abstraction transitions: Designers must know when to move from high-level goals into implementation details and when to return from the weeds to broader design concerns.Pail currently supports code, established requirements and decisions, and higher-level design concerns as abstraction levels.
  • Future work: Future tools must address information overload and determine how much agency or initiative automated systems should receive in directing design work.The authors connect these questions to managing attention when users provide higher-level instructions amid uncertain signal-to-noise ratios.

9 Conclusion

The paper presents Pail as a design probe for helping developers explore program formulations and solutions while tracking requirements and decisions. The study found evidence of broader design-space consideration, alongside challenges in managing attention and maintaining awareness of program updates.

  • Contribution: Pail encourages user-centered program design by tracking requirements discovered and decisions made through prototyping.It focuses on problem formulation and assessing solution suitability rather than code generation alone.
  • Findings: The user study found evidence that Pail can help participants consider the program design space broadly.The conclusion presents this as evidence of helpfulness, not as a comparative claim against other tools.
  • Challenges: The study also revealed challenges in managing attention and maintaining awareness of program updates.These challenges connect to broader trade-offs involving information generation, relevance, sharing, and agency.
Loading 2503.06911v1…