Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents

1 Introduction

• Holistic empirical research on how people oversee agents in messy real-world settings remains limited.Existing work is largely conceptual or narrowly focused, leaving on-the-ground practices and challenges poorly understood.Source: §1 · p. 2
• The study explores software engineers’ oversight practices and asks how developers oversee agents and address the challenges they encounter.It focuses on power users, whose practices and challenges may offer early signals for other user groups.Source: §1 · p. 2
• The authors present an early empirical account intended to bridge theory and practice, alongside future research directions and design and practice implications.They explicitly do not claim fully generalizable findings or an exhaustive taxonomy.Source: §1 · pp. 2–3
• Developers perform four forms of oversight: a priori control, co-planning, real-time monitoring, and post hoc review.These practices serve varied goals.Source: §1 · p. 2
• Oversight extends beyond reactive monitoring and retrospective evaluation to preventative configuration and proactive co-planning.The paper contrasts these practices with how prior research has portrayed oversight.Source: §1 · p. 2
• Developers face challenges such as reviewing agent-generated code and use heuristics such as treating test results as guarantees of correctness.The authors characterize this as efficient rather than perfect oversight, exposing a disconnect between research aspirations and practical use.Source: §1 · pp. 2–3

2.1 Human oversight of AI

• Reviewing AI outputs, especially spotting and correcting mistakes, is widely treated as a key part of oversight.Source: §2.1 · p. 3
• Empirical research on human oversight is only beginning to examine real-world practice; much prior work is conceptual or narrowly focused on controlled verification tasks.The literature describes an overall disconnect between research and practice.Source: §2.1 · p. 3

2.2 Agents in software development: Applications and challenges

• Software agents have been applied across software-engineering tasks, but studies also report code flaws and note that controlled evaluations miss real-world lifecycle complexity.Source: §2.2 · pp. 3–4
• Early studies describe developers monitoring and debugging agent actions, iteratively steering outputs, and facing usability challenges when using agents.Source: §2.2 · p. 4

2.3 The emerging focus on human oversight of agentic systems in FAccT scholarship

• FAccT research has only recently turned to generative AI and agents, and highlights trade-offs between system understanding, user workload, and human cognition in oversight aids.Source: §2.3 · pp. 4–5
• This study contributes an exploratory account of developers’ oversight work, the challenges and trade-offs they face, and the heuristics they use to navigate them.Source: §2.3 · p. 5

3.1 Participants: Participation criteria, recruitment process, and organizational context

• Researchers criterion-sampled 17 participants based on professional software-agent experience; all used agents at least weekly, 16 several times a week, and 12 daily.Source: §3.1 · p. 5
• Twelve of 17 participants worked at the authors’ organization, an unintended consequence of recruitment challenges; the sample focused on experienced users in well-resourced organizations.Source: §3.1 · p. 6

3.2 Data collection and analysis

• Semi-structured interviews asked developers about agent use, evaluation, and difficulties, using temporal prompts to elicit work before, during, and after agent execution.Source: §3.2 · p. 6
• Inductive analysis progressed through open, axial, and selective coding to form categories of oversight work and heuristics; the temporal structure serves as an organizing device, not an exact representation.The researchers distinguished configuring pre-established controls from actively steering agents, and interpreted heuristics as imperfect-yet-necessary shortcuts for good-enough review.Source: §3.2 · p. 6

4.1 Four forms of human oversight work

• Developers performed four forms of oversight: a priori control, co-planning, real-time monitoring, and post hoc review.The supplied findings elaborate on a priori control and co-planning.Source: §4.1 · p. 7
• A priori control uses agent settings and custom instructions to set boundaries, configure autonomy, restrict libraries, and provide project context before delegation.Its purpose is to set agents up properly and minimize failure; participants varied in how much they configured agents.Source: §4.1 · p. 7
• Despite controls, developers reported little control and limited information about agents’ operation, reasoning, and data policies.Uncertainty about agent behavior led some to experiment in secure environments such as sandboxes.Source: §4.1 · pp. 7–8
• Co-planning establishes common ground through iterative prompting, joint plans, and partial solutions that help agents identify requirements and reduce misalignment.Developers used plans to build shared understanding of a solution’s structure and components.Source: §4.1.2 · p. 8
• Developers decomposed work into small, independently testable tasks to contain side effects and make code changes easier to review and modify.They relied on coherent mental models to judge when minimal detail sufficed and when agents needed more context.Source: §4.1.2 · p. 8
• Co-planning was iterative: developers refined plans, negotiated decisions with agents, and used specifications to keep work grounded.Participants described this as an emerging form of spec-driven development.Source: §4.1.2 · p. 9
• Developers struggled to choose instruction specificity: under-specified plans led agents to misinterpret goals or overlook constraints, while the needed level of guidance was hard to anticipate.Participants also noted that hand-holding can eventually outweigh the benefits of using agents.Source: §4.1.2 · p. 9
• Natural-language prompting made it difficult to articulate goals, widening gaps between developer intent and agent work, especially when constructing coherent instructions for complex tasks.Developers described overexplaining even simple requests and struggling to assemble a coherent account of what code should do.Source: §4.1.2 · p. 9
• Real-time monitoring was rare; task decomposition made quick glances or skipped monitoring seem sufficient, though cursory scans could miss subtle errors.Monitoring involved observing logs or reasoning traces and pausing or stopping agents, but participants rarely examined agent thought processes proactively.Source: §4.1.3 · pp. 9–10
• Monitoring was difficult because agent explanations could diverge from actions, indirect cues such as long runtimes helped spot drift, and mid-task corrections were unreliable.Participants sometimes had to stop an agent, update the prompt, and restart from the beginning.Source: §4.1.3 · p. 10
• Post hoc review was the most discussed oversight work: developers verified, corrected, and re-prompted outputs because agents made mistakes and hallucinated.They sought to ensure outputs were correct and produced with appropriate approaches.Source: §4.1.4 · p. 10
• Review intensity rose when agents integrated features into legacy codebases, while developers accepted reduced scrutiny for prototypes.For serious software engineering, participants said they reviewed every line; for prototypes, some granted agents unrestricted autonomy.Source: §4.1.4 · p. 11
• Reviewing agent-generated code was cognitively demanding because developers felt distant from unfamiliar, high-volume code, increasing verification effort and making troubleshooting harder.Developers described their understanding of agent-written code as surface-level compared with code they had written themselves.Source: §4.1.4 · p. 11
• Each requested code fix could trigger a full re-review because agents might change multiple places, including unnecessarily.Participants said they had to review again after each regeneration or fix request.Source: §4.1.4 · p. 11

4.2 Four heuristics developers use for post hoc review

• Developers treated an agent’s plan as a faithful proxy for its execution, using plan verification as an easier substitute for extensive code review.This heuristic assumes agents follow their plans and that their actual work is congruent with the plan.Source: §4.2 · p. 12
• Developers focused on plans and outcomes rather than reading every agent step, and adopted spec-driven workflows to structure work before coding.One participant described solidifying product and technical specifications, then creating an implementation plan and tasks for the agent to follow.Source: §4.2 · p. 12
• Participants treated passing tests as evidence that agent-generated code was correct and used failures to locate problems, but this heuristic depends on test quality.Participants emphasized that humans remain responsible for reviewing test cases and ensuring the specification captures intended requirements.Source: §4.2 · pp. 12–13
• To manage information overload, developers spot-checked outputs, rationales, and change summaries, and sometimes asked agents to generate diagrams to aid review.Examples included checking method signatures and using data-flow diagrams to assess whether components in large pull requests were connected correctly.Source: §4.2 · p. 13
• In unfamiliar domains, some developers deferred to agents when they lacked expertise, shifting from explicit verification toward implicit reliance.The paper also reports a contrasting case: a developer unfamiliar with Go reviewed every line and said it looked legitimate.Source: §4.2 · p. 13
• In uncharted areas, participants also gained confidence when two agents independently gave similar responses to the same prompt.One participant used the same prompt with both agents and viewed their agreement as evidence of a grounded, comprehensive report.Source: §4.2 · p. 13

5.1 Oversight entails not only reviewing and monitoring, but also controlling and steering

• Oversight extends beyond reviewing and monitoring: developers configure agents before prompting and revise plans before execution to steer agents away from pitfalls.The authors note that although some anticipatory work resembles good software engineering, they do not claim that software engineering practices should be redefined as oversight.Source: §5.1 · p. 14

5.2 Oversight functions not through exhaustive awareness, but through practical heuristics

• Under bounded rationality, developers use good-enough heuristics: plans proxy for agent work, test results for code correctness, and skimming for information processing.These are described as adaptations to constraints, not simply lazy shortcuts or attempts at exhaustive awareness.Source: §5.2 · pp. 14–15

5.3 Implications for the human-centered design of software agents

• For a priori control and co-planning, agent builders can offer configuration guidance, surface constraints and defaults, and provide contextual prompting and task-decomposition help.The cited design suggestions address insufficient information and difficulties translating goals into prompts or decomposing tasks.Source: §5.3 · p. 15
• For monitoring and post hoc review, suggested supports include contextual signals, inline controls, and code diffs augmented with intent, reasoning, and proposed tests.The discussion also proposes ways to help developers reverse-engineer agent logic, such as interweaving rationale and reasoning traces and visualizing change impacts.Source: §5.3 · p. 15

5.4 Implications for the software engineering practice: Individuals and organizations

• The discussion describes a shift from hands-on coding toward oversight, with developers’ competencies moving toward architectural design, critique, and review.It highlights tensions between efficiency-oriented heuristics and organizational compliance obligations, and the need for situational awareness and epistemic access, especially in high-risk AI work.Source: §5.4 · pp. 15–16

Limitations and Future Work

• Most participants worked in well-resourced technology firms, where monetary and computational costs mattered little; resource-constrained oversight settings need further study.The authors cite generous token limits, rich tool integrations, and early access to new features as characteristics of these settings.Source: §6 · p. 16
• Twelve of 17 participants worked at one technology company, which may have biased findings despite similar experiences among participants from other firms.Research across diverse organizational contexts could complement this study with a more holistic view of oversight work.Source: §6 · p. 16
• Participants delegated relatively small tasks, making real-time monitoring optional and post hoc review easier; future work should examine complex, longer-running tasks.The authors also call for study of how agent builders conceptualize and facilitate more demanding oversight work.Source: §6 · p. 16

Conclusion

• Developers perform diverse oversight work to pursue different goals, while encountering pain points and adopting heuristics to navigate uncertainty when working with agents.The study presents an exploratory analysis of software developers overseeing coding agents in practice.Source: §7 · p. 16
• Oversight begins as early as configuration and planning and continues through ongoing verification, rather than being a single act.The authors characterize it as a coordinated set of ad hoc, evolving practices.Source: §7 · p. 16
• The findings provide early empirical anchors for researching and improving agent oversight, with implications for tooling, individuals, organizations, and future research.The paper connects these implications to supporting effective collaboration and complementarity between humans and agents.Source: §7 · p. 16

Generative AI Usage

• The authors used generative AI for interview transcription, proofreading, and sometimes improving sentence fluency, but not for data analysis or generating technical content.Source: Generative AI Usage · p. 16

• Table 1 covers participant demographics, programming and software-agent experience, agent-usage frequency, and agents used.Source: A Appendix: Participant demographics · p. 25, Table 1 · p. 25

• Participants used agents across development phases for tasks including requirements work, feature implementation, debugging, log analysis, and code refactoring.Source: B Appendix: Agent usage scenarios · p. 26
• Participants also used agents to support codebase learning and collaboration, including catching up on codebase changes after leave.Source: B Appendix: Agent usage scenarios · p. 26
• For production-grade enterprise software, participants usually delegated small, well-bounded tasks that agents could complete in a few dozen seconds to a minute.Source: B Appendix: Agent usage scenarios · p. 26
• Agent run times were longer for proof-of-concept or prototype work and for advanced power users undertaking extended tasks.Examples included 20–25 minutes for prototype work and 15–20 minutes for an advanced power user.Source: B Appendix: Agent usage scenarios · p. 26

• The guide asked participants to describe agent-use scenarios, how they chose agents, and trade-offs in those choices.Source: §C.0.2 · p. 27, §C.0.3 · p. 27
• Questions covered tailoring agents to projects and considering risks and evaluations during the initial phase.Source: §C.0.3 · p. 27
• Deployment prompts addressed risks, governance, interventions, output correctness, goodness, inspected artifacts, and verification tools.Source: §C.0.4 · p. 27, §C.0.5 · p. 27
• The guide also asked about workflow changes, activities added or dropped, and ways participants got unstuck.Source: §C.0.6 · p. 27

• Table 2 captures four forms of emergent oversight work and describes them.Source: D Appendix: Four forms of oversight work · p. 28, Table 2 · p. 28