Source-linked AI summary
Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support
Kevin Pu, Daniel Lazaro, Ian Arawjo, Haijun Xia, Ziang Xiao, Tovi Grossman, Yan Chen
TL;DR
Proactive AI programming support promises to reduce the effort of user-initiated assistance, but its effects on programming workflows remain underexplored. The paper designs and evaluates Codellaborator, a proactive LLM-agent probe, in a three-condition within-subject study with 18 programmers. Proactivity improved efficiency while disrupting workflows, and visible presence plus flexible context reduced disruptions and improved awareness, alongside concerns about control, ownership, and code understanding.
Problem
The effects and user-experience trade-offs of proactive programming assistance, compared with prompt-initiated tools, remain insufficiently assessed.
Method
Codellaborator analyzes editor activity and work state to initiate contextual assistance, using heuristics, visual presence, and localized context management in a three-condition study with 18 programmers.
Results
Proactive assistance improved efficiency but disrupted workflows, while visual presence and interaction context reduced disruptions and improved awareness of the AI’s actions.
Takeaways & Limitations
Proactive support can shift programming toward collaboration, but its design must account for user control, ownership, code understanding, and programming-process context.
Takeaways & Limitations
Inconsistent LLM responses produced variable assistance quality and affected participants’ perceptions, trust, expectations, task length, and interaction counts.
Abstract
from arXiv · showhide
AI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N=18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and interaction context support alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.
1 Introduction
Existing AI programming tools commonly require users to initiate help, while the effects of proactive assistance on programming workflows remain underexplored. This study designs and evaluates Codellaborator across three interaction conditions to examine proactive support, its benefits, and its disruptions.
- Motivation: User-initiated AI programming tools require prompts or partial code snippets, imposing effort to formulate requests and provide context.These tools return generated code and natural-language explanations for coding tasks.
- Research gap: The effects of system-driven programming assistance on human workflows, including potential harms from excessive automation and reduced control, remain insufficiently explored.The study frames these concerns through questions about when proactivity helps or harms programming processes.
- Approach: Codellaborator explores proactive assistance through the timing of help, AI representation, and interaction-context scope, guided by interruption management, social transparency, and help-seeking theories.The agent initiates messages in response to activities in the coding environment.
- Evaluation: The within-subject study compared PromptOnly, CodeGhost, and Codellaborator with 18 participants.PromptOnly responded to user prompts and inline comments; CodeGhost added proactive interaction without visual representation; Codellaborator used all three design dimensions.
- Findings: Proactive assistance reduced response-comprehension time but disrupted workflows, whereas agent presence and flexible context reduced disruptions and improved awareness of AI actions.Participants described Codellaborator as more like collaborating with a partner than using a tool.
- Contributions: The work contributes a design exploration, the Codellaborator technology probe, and an empirical study of proactive support in a code editor.It also proposes five design implications for proactive assistance in human-AI programming.
2 Related Work
Prior work spans user-initiated programming assistance, proactive agents, interruption management, information seeking, and pair programming. These strands motivate studying how proactive AI support affects workflow, awareness, and collaboration rather than assuming autonomy is beneficial.
- AI programming tools: AI-generated assistance can mismatch user expectations, complicate code interpretation, and require expanded context or explanations to support understanding.Prior systems address these issues through visible explanations, broader editor context, interaction histories, and provenance information.
- AI programming tools: Existing AI programming tools predominantly use command-response interactions, while newer agents proactively support users and autonomously complete tasks.The shift raises questions about how system-initiated assistance changes programming workflows.
- AI programming tools: Autonomous coding agents remain limited in real-world task completion, with SWE-agent and OpenDevin reporting 12.5% and 26.0% completion on SWE-bench.These results motivate more balanced human-AI programming designs.
- Proactive assistance and interruption: Proactive collaboration must manage interruptions because interruptions can harm memory, well-being, and ongoing task execution.The paper applies interruption-management principles to choose programming contexts for proactive service.
- Design dimensions: The design framework organizes Codellaborator around timing of assistance, AI representation, and interaction scope, each informed by distinct collaboration and programming-support literatures.The figure positions these dimensions as intersecting design considerations for evaluating programming experiences.
- Help-seeking and pair programming: Pair programming links awareness of a collaborator’s focus with better design, more concise code, and fewer errors, while also incurring cost, scheduling, and interpersonal challenges.Codellaborator adapts this paradigm by making the AI’s presence and actions visible.
3 Design Goals
The study uses a technology probe to explore proactive programming support across timing, agent representation, and interaction scope. These goals connect anticipated programmer needs with visible AI behavior and flexible contextual interaction.
- Design probe: The technology probe is used to investigate the unknown design space of timing, representation, and interaction in proactive AI programming support.The probe is intended to produce useful data about user workflow effects.
- DG1: Timing of assistance: DG1 establishes heuristics for timely proactive assistance by anticipating programmer needs from editor activities and offering task-relevant support.The goal is to determine when suggestions, insights, or corrections should be initiated.
- DG2: Agent representation: DG2 represents the AI agent through visible cues that communicate its actions, intentions, and decision-making processes.The design aims to enhance users’ awareness of the agent’s assistance.
- DG3: Interaction scope: DG3 provides flexible interaction scopes spanning global editor context and localized interactions.This design goal connects assistance to different levels of programming context.
4 The Codellaborator Probe
Codellaborator is a design probe for proactive programming assistance, combining context-sensitive timing, visible agent activity, and multiple interaction scopes. Its six proactivity features use editor activity and code context to decide when and how the agent should intervene.
- Probe design: Codellaborator combines timely proactive support, visual agent representation, and multi-level interaction scopes as modular design components.The probe was designed to support comparisons of these mechanisms.
- Timely proactive assistance: The agent delays intervention until extended inactivity, treating idle editing and cursor movement as a rough signal of low workload or being stuck.The authors note that idleness may also reflect thought and suggest more advanced models for future cognitive-state estimation.
- Timely proactive assistance: Task beginnings and subtask boundaries are used as intervention opportunities because interruptions are perceived as less disruptive at those points.The system detects programming boundaries such as completing a block of code.
- Timely proactive assistance: Six proactivity features evaluate changed code, caret location, and local file content before selecting an editor action.The agent uses prompts to assess working context and can employ tools to modify the editor.
- Adaptive behavior: The agent adapts its proactivity by responding to ignored interventions and canceling pending actions when users initiate conversations.It preserves actions during parallel editor work, allowing the agent to modify other code while the user continues coding.
- Agent representation and context: Visual carets, cursors, thought bubbles, progress signals, provenance highlights, and topic-organized chats make the agent’s actions and working state visible.Breakout chats support localized conversations, while the system tracks caret position, file contents, activity, console output, and prior dialogue across context scopes.
5 Evaluation
The evaluation compared three Codellaborator variants in a counterbalanced within-subject study of 18 participants completing three Python programming tasks. It combined surveys, interviews, task statistics, and interaction-level analysis to examine proactive assistance against a user-initiated baseline.
- Study design: The within-subject study compared PromptOnly, CodeGhost, and Codellaborator conditions with 18 participants.Condition orders were counterbalanced, and the same underlying LLM configuration was used across conditions.
- Study design: PromptOnly responded only to explicit prompts, CodeGhost added proactive messages and code changes without visual agent representation, and Codellaborator added visual features and scoped context.These conditions isolate increasingly salient forms of proactive support.
- Tasks and participants: Participants completed three Python tasks covering an event scheduler, word guessing game, and budget tracker.The tasks were derived from LeetCode problems and selected to reflect typical programming activities.
- Procedure: Each task lasted up to 30 minutes and was followed by Likert-scale ratings of disruption, awareness, control, and related experience measures.Sessions also included think-aloud protocols, recordings, and a concluding semi-structured interview.
- Analysis: The analysis combined thematic coding of interviews with task-level and interaction-level quantitative measures.Recorded measures included task duration, completed test cases, and human-AI interaction episodes.
6 Results
Proactive assistance improved perceived efficiency and reduced AI-response interpretation effort, but also introduced disruptions that varied by interface design and programming process. Presence, context management, and process-sensitive timing helped users remain aware and better coordinate with the agent.
- Efficiency: Participants found CodeGhost and Codellaborator more efficient than PromptOnly because proactive suggestions reduced interaction effort.Prompt-based tools required more effort, while proactive systems supplied suggestions preemptively and made interaction feel more natural.
- Efficiency: The proactive conditions significantly reduced the time needed to interpret AI actions compared with PromptOnly.Across 857 interaction episodes, interpretation time differed significantly across conditions (F(2,856)=41.1, p < 0.001).
- Disruption: Disruption was highest in CodeGhost, lower in Codellaborator, and lowest in PromptOnly.Mean disruption ratings were CodeGhost 4.61, Codellaborator 3.78, and PromptOnly 1.56; CodeGhost and Codellaborator both exceeded PromptOnly significantly.
- Timing: 398 proactive interventions produced 212 effective engagements, 48 disruptions, and 138 ignored interactions.These corresponded to 53.3% effective engagement, 12.1% disruptions, and 34.7% ignored proactivity.
- Timing: Task-boundary interventions were generally effective, whereas code-completion responses and implicit signals such as comments or selections often misinterpreted user intent.Code-block completion generated excessive affirmatory responses, while comments and cursor selections produced false positives and workflow disruptions.
- Collaboration: Participants adapted toward observer and code-reviewer roles, with half relying on the AI’s generative capability while shifting attention toward higher-level design.Nine participants exhibited this reliance and role change, including reduced attention to exactly what the AI wrote.
- Collaboration: Participants retained overall control in some collaborations by adjusting assistance levels and negotiating the division of labor with the AI.Four participants explicitly described maintaining control while steering whether the AI implemented code or provided suggestions.
- Programming processes: Participants preferred proactive assistance at process boundaries, during repetitive work, or when debugging, but preferred user-initiated control during high-focus implementation.Implementation accounted for 32.7% of disruptions, compared with 7.27% during debugging and 1.82% during refactoring.
7 Discussion
The discussion frames proactive programming assistance as a productivity-supporting but disruptive form of collaboration whose benefits depend on matching AI behavior to programming processes and user control needs. It also identifies limits to the study’s generalizability and to consistent AI behavior.
- Code understanding: Highly efficient generation may reduce time for users to understand code, supporting gradual semantic presentation, scaffolding, and explanations without overloading users.Participants appreciated hints and partial code, such as code skeletons and comments illustrating logical steps.
- Shared design context: Proactive agents can make task direction harder to steer because low-level assistance may obscure high-level design communication and earlier suggestions.The discussion recommends maintaining shared context around the overall task design rather than focusing only on local code changes.
- Salience and awareness: AI presence should match action significance: visible presence helps users understand consequential code changes, while less significant fixes can use lower salience.Participants expected clear signals of the AI’s working process when it changed editor code.
- Adaptation to programming processes: Users preferred proactive support differently across programming stages, with task design, refactoring, and debugging generally favored but individual preferences shaping disruption.Interventions in less-preferred processes often disrupted workflows, motivating adaptation to users’ programming processes.
- Trade-offs of proactive assistance: Proactive assistance can improve productivity, but participants also reported losses of control, ownership, and code understanding.Participants appreciated reduced prompting effort and enhanced productivity while expressing concerns about over-reliance on AI support.
- Scope and limitations: The study provides an exploratory basis for proactive programming-system design, not a comprehensive account of real-world engineering contexts.Codellaborator supported only single-file Python tasks without persistent interaction or code provenance, limiting external validity.
- Scope and limitations: Inconsistent LLM responses affected participants’ perceptions, trust, expectations, task duration, and recorded interactions.Future research could impose greater control over AI behavior and collect more data.
8 Conclusion
Codellaborator demonstrates that proactive AI assistance can reduce users’ effort to convey intent, while also disrupting workflows. Visual presence and localized context can alleviate these disruptions and make collaboration feel more like working with a partner than a tool.
- Proactive assistance lowered users’ expression effort but incurred more workflow disruptions in a three-condition study with 18 programmers.Codellaborator alleviated disruptions and increased users’ awareness of the AI.
- Visual presence, threaded breakout messages, and provenance signals supported more contextualized collaboration with the AI agent.These design elements helped users understand the interaction process and manage localized context.
- The study produced design implications and identified opportunities and risks for future proactive AI systems in programming workflows.
B.1 Task 1: Scheduling API
The scheduling task required implementing event management with conflict checking and time-based display. Its subtasks specify maintaining event information, adding events, detecting conflicts, and sorting results.
- The task requires a scheduling system that maintains events and creates new events while checking location and participant conflicts.
- The subtasks require storing event details, implementing event creation, checking conflicts, and displaying events sorted by time.
B.2 Task 2: Word Guessing Game
The word-guessing task involves implementing a Wordle-style game around a five-letter word and a Dictionary API.
- The game manager initializes with a verified five-letter word and provides methods for unguessed letters and guesses that return feedback.
B.2.1 Subtasks.
The subtasks define initialization, state tracking, guessing, feedback, and dictionary validation for the word-guessing game.
- The subtasks require validating five-letter input, storing game state, returning unguessed letters, and implementing five-character guesses with feedback.
- The game must use the provided Dictionary API to verify guessed words and return an error for invalid words.
B.3 Task 3: Budget Tracker
The task is to implement a budget tracker class that manages income, spending, balances, category limits, and spending reports.
- The budget tracker is initialized with a starting amount and records income and expenses by category and amount.
- Its methods calculate the existing balance, set spending limits for expense categories, and generate a spending report.
B.3.1 Subtasks.
The subtasks specify methods for recording income and expenses, calculating balances, enforcing category limits with warnings, and producing ordered spending reports.
- Users can add income sources and track expenses with descriptions and amounts.
- The current balance is calculated from the recorded income and expenses.
- Budget limits can be set for expense categories, with warnings when those limits are exceeded.
- The spending report breaks expenses down by category, listing limited categories first by ascending distance from their limits.
- Categories without limits follow, sorted by total expense amount in descending order.