PaperlayerView exact revision on arXiv

Source-linked AI summary

"To Boldly Go Where No One Has Gone Before" What Has AI Let Developers Unlock?

Rudrajit Choudhuri, Jenna L. Butler, Christian Bird, Carmen Badea, Brian Houck, Anita Sarma

arXiv:2610.23708v1cs.HC

TL;DR

Measuring AI only by how quickly developers complete familiar tasks can miss what new work it makes possible. This study surveys 331 Microsoft developers and analyzes their open-ended responses using reflexive thematic analysis with a human-in-the-loop, multi-model council. Developers report faster familiar work, newly reachable work, and new work around AI agents, while the authors frame developers and AI as joint learners meeting at a skill–judgment gap.

  • Problem

    Measures of AI’s impact focused on speed and familiar workflows may not capture what new work AI lets developers do.

    Source: §I · p. 1

  • Method

    The authors analyze open-ended responses from 331 Microsoft developers using reflexive thematic analysis and a human-in-the-loop, multi-model AI council.

    Source: §I · p. 1, A. Survey · p. 3

  • Results

    45% of responses described familiar work done faster, 36.9% work brought within reach, and 12.4% new work around directing agents and building for them.

    Source: §IV · p. 4

  • Takeaways & Limitations

    The authors frame developers and AI as joint learners who meet at a skill–judgment nexus, where developers must bridge the gap between AI’s skills and its reliable judgment.

    Source: §I · pp. 1–2, §VI · p. 10

    The cross-sectional study captures stated experience at one point in time, and self-selection may mean respondents hold stronger views about AI support than the broader population.

    Source: C. Threats to validity · p. 4

Abstract

from arXiv · show

Measuring AI by how much faster developers work is like judging electricity by how much faster it spun a steam-era factory. A fuller measure asks what new work becomes possible. We surveyed 331 software developers at Microsoft about what AI lets them do that they could not do before and analyzed their answers using reflexive thematic analysis with a human-in-the-loop, multi-model AI council. Developers described doing familiar work faster (Faster, 45%), undertaking work previously beyond their reach (Further, 36.9%), and taking on new kinds of work centered on building for and directing AI agents (Afresh, 12.4%). Across these accounts, learning and scaffolding ran in both directions: developers learned new skills with AI while externalizing their tacit knowledge as the context and standards that guide AI's work. Drawing on Vygotsky's zone of proximal development, we frame developers and AI as joint learners whose zones meet at the skill-judgment nexus: AI can supply skill faster than it can reliably judge its own work, and developers must continuously bridge that gap. Whether they do sets them on one of two paths. On one, developers' judgment grows alongside AI's skill, and they codify what they learn so the loop keeps turning. On the other, they become hollow orchestrators, approving work they can no longer judge and losing the identity and meaning they once drew from it. Based on our findings, we offer a two-way scaffolding playbook to help developers, teams, and tool builders take the first path.

I. INTRODUCTION

Measures of AI’s impact should ask not only whether familiar work gets faster, but what work becomes possible. The paper surveys Microsoft developers, identifies three reported zones of AI-enabled work, and frames developers and AI as joint learners who must bridge a skill–judgment gap.

I. INTRODUCTION

  • Time and output measures alone may miss what AI makes possible and how it changes developers’ work.The paper contrasts productivity measures of familiar tasks with asking what developers can now do that was previously out of reach.Source: §I · p. 1
  • The study analyzes 331 developers’ open-ended responses using reflexive thematic analysis and a human-in-the-loop, multi-model AI council.It asks what developers report they can now do at work because of AI that they could not do before.Source: §I · p. 1
  • 45% of responses described familiar work done faster, 36.9% work newly within reach, and 12.4% new work around directing agents and building for them.The zones are Faster, Further, and Afresh, respectively.Source: §I · p. 1
  • Developers and AI are framed as joint learners whose zones meet at the skill–judgment nexus, where developers continually bridge AI’s skill–judgment gap.Developers learn with AI and codify their discernment as specifications, memory, and success criteria to guide AI.Source: §I · p. 1
  • A two-way scaffolding playbook is offered to help developers and AI grow together, while failure to bridge the gap risks hollow orchestration and loss of skills and professional identity.The authors argue these practices could help developers and their agents grow together.Source: §I · p. 2

II. RELATED WORK

Prior research examines productivity, support needs, work meaning, and changing developer responsibilities, often using existing tasks as its frame. This study instead examines newly reachable work and situates its findings in developers’ and AI’s mutual scaffolding.

II. RELATED WORK

  • Research finds that AI’s productivity effects vary across tasks and settings, with reported speedups alongside a study finding developers took 19% longer while believing AI had sped them up.Other studies reported completing tasks 55% faster, spending about 21% less time, or completing 26% more tasks.Source: §II · p. 2
  • Studies of AI support describe a tension between reducing repetitive toil and preserving identity-defining work, alongside review and verification demands.Work meaning predicts where developers draw limits on AI autonomy.Source: §II · p. 2
  • Existing-work studies ask which current tasks AI can and should support, whereas this study asks what AI lets developers do that they could not before.The paper distinguishes Faster, Further, and Afresh as familiar work accelerated, previously unreachable work made possible, and new work created around AI.Source: §II · p. 2
  • As AI becomes more autonomous, developers’ work shifts toward directing and checking output, with metacognitive demands to specify goals and evaluate results.Participants using an autonomous agent reported understanding its work less well than participants using non-agentic AI.Source: §II · p. 2

III. METHOD

  • The study uses Vygotsky’s zone of proximal development to frame developers and AI as joint learners and presents a two-way scaffolding playbook.It draws on a survey of Microsoft developers across diverse domains, teams, stakeholder contexts, and geographies.Source: §II · p. 2, §III · p. 2

A. Survey

The study uses a broad open-ended survey question to capture developers’ own accounts of newly accessible work, then analyzes responses through a human-reviewed, multi-model process. Authors retain interpretive responsibility while models support discovery and systematic coding.

A. Survey

  • The optional open-ended question asked whether AI enabled work participants could not do before, and responses were the primary unit of analysis.The broad wording intentionally let developers identify what counted as a newly accessible capability.Source: A. Survey · p. 2
  • The survey received 355 responses (8.9% response rate), of which 345 contained substantive answers; after excluding 14 with insufficiently supported code assignments, 331 participant responses were retained for analysis.Source: A. Survey · p. 3

B. Data analysis

  • The analysis combined independent discovery by three model families, author review and approval of a codebook, and systematic coding by three model instances.Models’ candidate patterns were grounded in participant IDs and verbatim excerpts; authors assessed fidelity, grounding, and distinctness before coding began.Source: B. Data analysis · p. 3
  • Model-coder agreement had median Krippendorff’s α = 0.92 across codes, with final assignments requiring a 2-of-3 majority.An author’s blind recoding of 50 responses yielded human–council agreement of κ = 0.87.Source: B. Data analysis · pp. 3–4
  • The authors mapped responses to Faster, Further, and Afresh according to whether AI accelerated familiar work, made previously unreachable work reachable, or created work around AI-enabled systems.The study excluded 14 responses whose zone placement lacked sufficient support; models proposed patterns and applied an approved codebook, while authors decided meanings and synthesis.Source: B. Data analysis · p. 4

C. Threats to validity

The findings concern developers’ stated experiences rather than observed performance, and the study’s cross-sectional Microsoft setting constrains interpretation and generalization. The authors identify possible self-selection and AI-assisted interpretation as additional validity concerns.

C. Threats to validity

  • The study reports stated experience rather than observed performance, and developers may describe gains they would not realize in practice.Self-reported speed can be wrong even in direction; the authors present the zones as testable categories for future measurement.Source: C. Threats to validity · p. 4
  • AI-assisted coding could surface plausible themes that do not fully reflect participants’ meanings, despite independent model discovery, researcher review, rationales, and verification checks.Verbatim quotes let readers judge whether the themes are grounded in participant responses.Source: C. Threats to validity · p. 4
  • The cross-sectional study captures one point in time, while self-selection may mean respondents hold stronger views about AI support than the broader population.The authors note that the zones could help track changes as AI tools and developer experiences co-evolve.Source: C. Threats to validity · p. 4
  • The Microsoft setting may provide AI capabilities and organizational support less available elsewhere, so the study is an in-depth account rather than a claim to represent all developers.Replication in other contexts remains necessary future work.Source: C. Threats to validity · p. 4

IV. RESULTS

Developers described AI enabling familiar work faster, previously unreachable work, and new work around AI. Faster included expanded exploration and learning as well as increased task throughput.

IV. RESULTS

  • 45% of responses were Faster, 36.9% Further, and 12.4% Afresh; the remaining 5.7% named no specific task.Responses could carry several themes, so theme percentages overlap and do not sum to each zone total.Source: §IV · p. 4, Table II · p. 5

A. Faster: the same tasks, faster

  • Participants reported speeding up familiar design, implementation, testing, and debugging work.Examples included completing two comprehensive PRs in a day and doing strong testing in minutes.Source: A. Faster: the same tasks, faster · p. 4
  • Faster also included expanded rigor and exploration, such as generating tests, building prototypes, and comparing approaches before committing.The cost of investigating ideas fell enough that participants could evaluate multiple working approaches.Source: A. Faster: the same tasks, faster · p. 5
  • AI let developers run low-priority tasks in the background or parallelize work, so more work could land during the workday.Participants noted that tasks might not finish faster in isolation even as more work was completed in the same timeframe.Source: A. Faster: the same tasks, faster · pp. 5–6
  • Participants used AI to understand unfamiliar codebases and domains faster, then spent saved time on engineering judgment and higher-value problems.They also described using AI to structure learning steps and challenges that had not been feasible before.Source: A. Faster: the same tasks, faster · p. 6

B. Further: tasks reachable only with AI

Further work became reachable when AI supplied skills, knowledge, or capacity developers lacked, or made previously deferred work practical. Participants described expanding both their expertise and the range of work they could take on.

B. Further: tasks reachable only with AI

  • 36.9% of responses described Further work; 24.2% involved skills, knowledge, or capacity participants lacked, while 12.7% involved work they had deferred.Participants gave examples ranging from front-end contributions to features that had previously been months of work.Source: B. Further: tasks reachable only with AI · p. 6
  • Participants used AI to work outside their expertise, including unfamiliar frameworks, deployment pipelines, data analysis, and broader portions of a system.Some also used it to improve interpersonal work, such as writing feedback and thinking through how to work with direct reports.Source: B. Further: tasks reachable only with AI · p. 6
  • Accessible knowledge let participants troubleshoot without asking colleagues, while AI capacity helped them pursue ideas they could not staff alone.One participant described AI as a virtual team for exploring ideas and magnifying impact.Source: B. Further: tasks reachable only with AI · p. 6
  • Lower effort made discretionary work practical, including tedious automation, technical-debt work, and personal tools.One participant automated portal updates in an hour, saving 80 hours of manual work; without AI, the automation was estimated to take 30 hours.Source: B. Further: tasks reachable only with AI · p. 6
  • Participants also took on broader and more rigorous work, using proofs, tests, and prototypes to test hypotheses and gather stakeholder feedback.Cheaper exploration let one participant check three or four approaches before committing, rather than choosing the first plausible one.Source: B. Further: tasks reachable only with AI · p. 6

C. Afresh: new work because of AI

AI also created new work centered on directing agents, setting standards for their output, and building the context and infrastructure they need. Participants described oversight and trust as part of this shift.

C. Afresh: new work because of AI

  • 12.4% of responses described new work around AI, including directing agents, deciding when work was done, and building what agents need.Participants also described setting standards for AI’s work and creating infrastructure around the model.Source: C. Afresh: new work because of AI · pp. 6–7
  • Developers described shifting from executing steps to orchestrating work streams, managing agent teams, and iterating with agents to produce artifacts.In one account, more than 90% of the code was AI-written, with developers acting as a protective layer and agent managers.Source: C. Afresh: new work because of AI · pp. 6–7
  • Agent oversight included reviewing outputs before merging and checking that specifications and intent carried across workflow stages.Some participants envisioned agents escalating only consequential decisions or evidence-backed blockers.Source: C. Afresh: new work because of AI · p. 7
  • Because AI does not necessarily provide a reliable completion criterion, participants built harnesses that checked outputs against artifacts defining the finished result.Some expected defining success and exit criteria to become a larger part of engineering work.Source: C. Afresh: new work because of AI · p. 7
  • Participants built agent infrastructure including pipelines, shared backlogs, durable memory, and tools for reviewing large AI-generated diffs.Trust also became a design requirement, including ways for agents to report back and for managers to verify their trust.Source: C. Afresh: new work because of AI · p. 7

V. DISCUSSION

Developers in AI-native work act both as learners who use AI to extend their reach and as teachers who supply agents with context and standards. The authors frame this relationship as changing over time as models, developer knowledge, and tasks change.

V. DISCUSSION

  • Developers can be learners using AI for work they could not do alone and teachers who pass experience to AI through specifications, success criteria, and memory.The authors describe these roles as potentially occurring within the same workday.Source: §V · p. 7
  • The skills developers need to judge AI’s work shift as models improve, developers externalize knowledge, and new tasks appear.The paper connects this evolving people-tool system to Engelbart’s account of augmentation and Lehman’s observation that real-world software must keep changing to remain useful.Source: §V · p. 7

A. Developers and AI as joint learners, meeting at the skill– judgment nexus

Developers and AI scaffold one another as joint learners, meeting where AI’s skill exceeds its reliable judgment. Developers can turn borrowed reach into lasting learning by judging AI’s work and codifying what good work looks like.

Joint learners

  • Scaffolding makes the middle distance reachable, and developers and AI can scaffold each other by learning from and teaching one another.Participants play both learner and teacher; scaffolding should adapt as learners grow and eventually hand work over.Source: A. Developers and AI as joint… · p. 8

The developer as learner

  • Developers can turn AI-borrowed reach into their own skill by staying engaged, making them better able to judge later work.Without that engagement, they may need the same help again and remain unable to tell whether it was right.Source: A. Developers and AI as joint… · p. 8

AI as learner

  • Developers teach AI by making tacit experience-based discernment explicit in specifications, success criteria, tests, and standing instructions.Examples include recognizing designs that will age badly, incorrect requirements, or fixes that treat symptoms rather than causes.Source: A. Developers and AI as joint… · p. 8

The skill– judgment nexus

  • The skill–judgment nexus is where AI can supply skills beyond what it can reliably judge, so developers must continually bridge the gap.The loop can turn when developers grow their judgment and codify it for AI; it stalls when skill outpaces judgment.Source: A. Developers and AI as joint… · p. 8, Figure 1 · p. 8

The loop in practice

  • When developers understand an AI solution and codify what makes it good, both their own and AI’s zones can expand, leaving developers free for harder problems.Participants also kept work they felt accountable for; such work can build the judgment the loop needs.Source: A. Developers and AI as joint… · pp. 8–9

B. When the loop stalls: the hollow orchestrator

When AI’s output grows faster than developers’ judgment, developers can become hollow orchestrators: they approve work they can no longer follow, while dependence, verification burden, and loss of expertise or meaning accumulate.

The hollow orchestrator

  • Rapid AI adoption can leave developers approving more code and reviews while writing less, with judgment unchanged—the hollow orchestrator’s starting point.The hollow orchestrator retains the role but can no longer follow the work they approve.Source: B. When the loop stalls: the hollow… · p. 9

Costs of borrowed judgment

  • Hollow orchestrators become dependent, face growing verification burdens, and risk gradual expertise erosion that can surface as skill atrophy and diminished professional identity.A participant worried that stale domain knowledge would make missed AI fabrications more likely; prior clinical research described “intuition rust.”Source: B. When the loop stalls: the hollow… · p. 9

Meaning and the stalled loop

  • Handing over meaningful work can weaken engagement and the judgment that develops through exercising it, stalling both developer learning and teaching.When teaching stops, stale context and vague requests can also leave AI producing work that looks right but is wrong.Source: B. When the loop stalls: the hollow… · p. 9

C. Keeping the loop turning: a two-way scaffolding playbook

The two-way scaffolding playbook adapts teaching functions for developers guiding AI and AI guiding developers. It pairs concrete practices with the need to keep teaching knowledge forward, while recognizing that practice details will change across tools and settings.

Why keep teaching

  • Externalizing tacit knowledge can help teams learn and preserve guidance for agents and future developers, but ownership of those records remains open and organizations need to make time for the work.Codifying knowledge may sharpen developers’ own judgment; the authors also warn that tacit knowledge may otherwise hollow out as routine work shifts to AI.Source: C. Keeping the loop turning: a two-way… · p. 9

Playbook structure

Demonstration

  • Contextualized examples can convey developer standards to AI, while AI demonstrations can show newcomers expert approaches, near misses, and checks.A pair of examples—a convention-following case and a near miss—can convey more than an underspecified instruction such as “handle errors better.”Source: C. Keeping the loop turning: a two-way… · p. 9

Articulation

Exploration

  • Exploration becomes durable learning when developers fold what worked and failed into inherited context and success criteria, while prototypes can build developers’ judgment.Where failure is cheap, developers can give AI a goal rather than steps and let it find a route.Source: C. Keeping the loop turning: a two-way… · p. 9

Scope and adaptation

  • The specific practices may change with models, experience, tasks, teams, and learners, while the underlying scaffolding functions are expected to outlast particular tools.The playbook prompts teams and tool builders to consider who scaffolds whom, how much support is needed, and how it should change as learners grow.Source: C. Keeping the loop turning: a two-way… · p. 9

D. Implications for future research

The paper identifies research questions about how the skill–judgment nexus and stalled learning loops change, and how developers will build expertise as AI takes on work through which it has traditionally developed. It concludes that the future depends on developers growing judgment with AI and externalizing knowledge for AI and future developers to learn from.

D. Implications for future research

  • Future research could test whether model skill gains open new judgment gaps, and whether practices in the playbook can slow or reverse hollow-orchestrator drift.The authors also ask whether a hollow orchestrator can rebuild judgment and what early signs might reveal that drift.Source: D. Implications for future research · p. 10
  • Research and education must address how the next generation builds expertise when AI performs much of the work through which expertise has developed.The authors call for two-way AI literacy, including judging AI output, teaching agents, and deciding what work to keep.Source: D. Implications for future research · p. 10

VI. CONCLUSION

  • The study describes AI enabling familiar work faster, work previously beyond reach, and new work around directing, supervising, and building for agents.It frames developers and AI as joint learners who meet at a skill–judgment nexus that developers must continually bridge.Source: §VI · p. 10
  • The future of AI-native software development depends on developers growing judgment as they learn with AI and externalizing knowledge for AI and future developers to learn from.Source: §VI · p. 10

Related papers

Loading 2610.23708v1…