Developers as Managers of AI Agents: A Workplace Shadowing Study

1 Introduction

• Prior research offers limited insight into how professional developers incorporate agents into everyday work, including directing and checking them while coordinating tasks and collaborators.Repository outcomes, controlled tasks, and public streams capture different conditions from workplace development.Source: §1 · p. 2
• 33.4% of observed session time went to prompt formulation or AI-response checking, while task-switching rates ranged from 0 to 101.6 switches per hour.Participants worked on 1–17 distinct tasks.Source: §1 · p. 2
• Supervisor, Delegator, and Learner patterns describe direct artifact inspection, dialogue-based coordination and delegated work, and explanation seeking to build understanding.The patterns overlap within sessions rather than defining fixed developer types.Source: §1 · p. 3
• Developers used agents for technical information and turned to colleagues for organizational context, human authority, and collective alignment.Source: §1 · p. 3
• The study contributes an empirical account of attention allocation, agent management, and output checking, alongside three empirically derived management patterns.Source: §1 · p. 3

2.1 Understanding Developers Working with AI

• Earlier AI-development studies used repositories, surveys, controlled experiments, and livestream observations, but provided limited insight into everyday workplace practices and concurrent work.Workplace studies of AI coding agents remain rare; existing settings differ in task scope and audience conditions.Source: §2.1 · p. 3

2.2 Observing Developers at Work

• Research on everyday software work documented information needs and fragmented attention, motivating attention to task allocation when agent work can continue in parallel.Source: §2.2 · p. 4
• The study combines direct observation, field notes, recordings, and follow-up interviews; participants reviewed visualizations to check and refine interpretations.The researchers avoided concurrent think-aloud to reduce interference with ongoing work.Source: §2.2 · p. 4

3 Method

• The two-phase mixed-methods study combined coded activity traces and prompt analysis with follow-up interviews in which participants reflected on session visualizations.Source: §3 · p. 4

3.1 Recruitment and Participants

• Participants came from two distinct organizations, and 11 of 13 recorded sessions were included in the analysis.One session was excluded because a meeting left insufficient observation time; another because the participant could not use the agent after reaching its token limit.Source: §3.1 · pp. 4–5

3.2.1 Observational Study.

• Developers worked on their own activities during 70–90-minute observed sessions, without a think-aloud protocol; field notes contextualized off-screen activity.The dataset included around 15 hours, 4,481 activity segments, and 272 submitted prompts.Source: §3.2.1 · pp. 5–6

3.2.2 Phase 1 Data Analysis.

• Researchers coded observable actions, linked prompts to preceding visible context and subsequent checks, and counted transitions between assigned tasks.Prompt-related checks were traced across intervening tasks until the next prompt in the same task or the recording’s end; brief task visits counted as transitions.Source: §3.2.2 · pp. 6–7

3.3 Phase 2: Follow-up Interviews

• Follow-up interviews used participant-specific questions and session visualizations, with deductive topic coding and inductive subcodes to contextualize observed patterns.Participants were asked whether activity and task-switching visualizations reflected typical work.Source: §3.3 · pp. 7–8

3.4 Threats to Validity

• Recordings capture visible actions, not attention or intentions, and the 11 sessions cannot represent the full range of developers’ agent use or estimate its prevalence.Participants came from two workplaces, sessions excluded meetings, and off-screen messages could not be linked to later prompts.Source: §3.4 · pp. 8–9

4.1 RQ1-1: Time Allocations Among Activities

• 33.4% of session time went to prompting or checking AI responses, more than direct code inspection for 10 of 11 developers, with substantial variation across participants.Window switching averaged 4.6% of session time. Participants linked time allocations to roles, organizational context, and team norms, including agent-first practices; some said their practices had since changed.Source: §4.1 · pp. 9–10, Figure 2 · p. 10

4.2 RQ1-2: Switches Among Tasks

• Task switching ranged from none to 101.6 switches per hour, and task count did not predict switching frequency.Participants worked on 1–17 tasks; nine had median task runs under three minutes, though brief runs could coexist with sustained work. The table summarizes switching statistics, and the figure illustrates contrasting interleaving patterns.Source: §4.2 · p. 11, Table 3 · p. 11, Figure 3 · p. 11
• Participants attributed switching to mental load, work responsibilities, agent orchestration, and waiting for CI/CD blockers, while reporting both within-task focus benefits and cross-task attention difficulties.Some dispatched long-running agent tasks and worked elsewhere while waiting; participants described fatigue and shallow attention across concurrent tasks, alongside strategies such as timers, deliberate switching, and notes to preserve context.Source: §4.2 · p. 12

4.3 RQ2-1: Directing and Informing AI Agents

• Prompts advanced development and helped developers understand and manage agents; reviewing an AI response was the sole observable preceding activity for 41% of prompts on average.Prompt uses included implementation, documentation, code and tool explanations, agent status or capability inquiries, configuration, and coordination; developers also supplied clarifications in dialogue and sometimes copied information into prompts.Source: §4.3 · pp. 13–14

4.4 RQ2-2: Information from AI to Developers

• 40% of retained prompts were followed solely by checking AI responses on average, ranging from 0% for P1 and P5 to 72% for P7.Developers also opened code, executed programs or checked data, and checked documentation; they sometimes prompted agents to reassess their work. Reported checking strategies relied on advance harnesses, personal expertise, or model trust, which could change over time.Source: §4.4 · pp. 14–15

4.5 RQ2-3: Management Patterns: Supervisors, Delegators, and Learners

• Supervisors mainly requested artifacts and directly inspected them, whereas Delegators relied more on dialogue, agent coordination, and delegated implementation and checking.Supervisors’ prompts were 78% requests and 5% were followed solely by AI-response checking; Delegators’ prompts included 39% inquiries, 35% requests, 30% replies, and 23% about agent information or configuration, while 69% were followed solely by response checking. Delegators averaged 11.7 tasks and 79.7 switches per session, compared with Supervisors’ 5.3 tasks and 27.0 switches.Source: §4.5 · pp. 16–17
• Learners primarily used agents to build understanding, while some sessions combined or shifted among patterns depending on task and context.Learners’ prompts were 76% inquiries, and they averaged 3.3 tasks and 18.3 switches per session; mixed Supervisor–Learner participants averaged 2.0 tasks and 11.0 switches, with 45% inquiries and 36% requests. The patterns describe observed sessions, not fixed or exhaustive developer types.Source: §4.5 · pp. 16–17

4.6 RQ2-4: Information from AI vs. Colleagues

• Developers turned to colleagues for organizational context, human authority, and collective alignment, and used agents for technical questions answerable from code, documentation, or public knowledge.Colleagues supplied feature intent, historical reasoning, permissions, domain meanings, and coordination on scope, design, or ownership. Communication content was not analyzed; interviews addressed which information developers sought from agents versus colleagues.Source: §4.6 · pp. 17–18
• Documentation served two purposes: helping people understand systems and giving agents context for implementation.Participants also questioned whether documentation remained worthwhile when agents could explain code from the codebase alone.Source: §4.6 · p. 18
• Developers used documentation to preserve shared understanding and decisions that might otherwise remain implicit or be lost.Some created pre-work documents as code production accelerated; another configured an agent to log design and architectural decisions.Source: §4.6 · p. 18
• Agents generated wiki pages, notes, and drafts, reducing documentation effort, but developers still had to clean up their outputs.One participant also moved team documentation into a repository so an agent could access it and check consistency with the code.Source: §4.6 · p. 18

5.1 Developers as Managers of AI Agents

• Agent management combines bounding autonomy, exercising technical judgment, and allocating attention across concurrent delegated work.Control occurs both upstream, through task scope and tests, and downstream, through review and follow-up; developers also use reminders, notes, and deliberate switching to manage parallel work.Source: §5.1 · pp. 18–19
• Developers use expertise to decide when closer inspection is warranted, while agents can help build the understanding needed for that judgment.Source: §5.1 · p. 19
• Agents shift rather than eliminate information seeking: developers use them for technical information and colleagues for organizational history, intent, authority, and alignment.Source: §5.1 · p. 19

5.2 Designing for Supervising, Coordinating, and Learning with AI Agents

• Tools should make verification evidence visible, including generated changes, relevant artifacts, fragile areas, and executed test results.For Supervisors, tools could also connect changes to requirements and business rules, surface uncertainties, and help direct review effort.Source: §5.2 · p. 19
• Coordination tools could show which concurrent agent tasks are progressing, blocked, awaiting approval, or need human intervention.The discussion also proposes exposing dependencies and information flow across agent sessions, and studying effects of task switching on cognitive load, resumption, and perceived control.Source: §5.2 · p. 19
• Learning-oriented tools could connect agent explanations to artifacts and preserve evidence, reasoning, and tacit knowledge behind software.The paper also suggests personalized pedagogical support; because patterns overlap and shift within sessions, support should adapt to task and context rather than fixed developer types.Source: §5.2 · pp. 19–20

• 33.4% of observed time went to formulating prompts or checking AI responses, while task-switching rates ranged from 0 to 101.6 switches per hour.Source: §6 · p. 20
• Workplace shadowing and follow-up interviews with 11 professional developers examined attention allocation, agent direction, and output checking.Source: §6 · p. 20
• Supervisor, Delegator, and Learner patterns involved artifact inspection, dialogue-based coordination, and seeking explanations to build understanding.Source: §6 · p. 20
• Participants used agents for technical information and colleagues for organizational context, authority, and collective alignment.The conclusion calls for further study of attention across concurrent agent tasks and tools supporting verification, coordination, and learning.Source: §6 · p. 20

• The replication package includes codebooks and coding guidelines for observational activity coding and prompt-centered analysis.Source: §7 · p. 20
• Anonymized sequences of coded activities for all participants support analyses of time allocation, task switching, and activities surrounding prompts.Source: §7 · p. 20
• The package provides participant-level visualizations for reported figures, including complete-session task and management-pattern visualizations for all 11 participants.Source: §7 · p. 20