PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic AI Software Engineers: Programming with Trust

Abhik Roychoudhury, Corina Pasareanu, Michael Pradel, Baishakhi Ray

arXiv:2502.13767v4cs.SEcs.AI

TL;DR

The paper asks how AI software engineers can earn developer trust as LLMs generate and edit code. It presents LLM agents that use software tools and proposes trust-building practices, concluding that programming with AI should mean effective human-AI delegation rather than full automation.

  • Problem

    Although LLM code generation and editing are prevalent, fully automated AI software engineers are not widely deployed in industry, and developer trust is identified as a key adoption barrier.

    Source: §1 · p. 1

  • Method

    The paper argues for software engineering agents that combine LLMs with autonomous use of software tools, alongside quality assurance and collaboration practices.

    Source: §3 · pp. 2–3, §4 · pp. 3–4

  • Results

    The paper concludes that trusted AI software engineering calls for transparent, feedback-responsive agents with quality and security safeguards, and effective delegation rather than full automation.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The authors argue that future programming will emphasize trust alongside scale, with human and AI working hand in hand.

    Source: §1 · p. 2, §5 · p. 4

    The paper distinguishes technical and human trust but does not relate them to interpersonal trust models from sociology and psychology.

    Source: §2 · p. 2

Abstract

from arXiv · show

Large Language Models (LLMs) have shown surprising proficiency in generating code snippets, promising to automate large parts of software engineering via artificial intelligence (AI). We argue that successfully deploying AI software engineers requires a level of trust equal to or even greater than the trust established by human-driven software engineering practices. The recent trend toward LLM agents offers a path toward integrating the power of LLMs to create new code with the power of analysis tools to increase trust in the code. This opinion piece comments on whether LLM agents could dominate software engineering workflows in the future and whether the focus of programming will shift from programming at scale to programming with trust.

1 A Key Barrier for AI Software Engineers

The paper identifies developer trust as a key barrier to deploying AI software engineers and asks how AI-generated and manually written software can coexist in future workflows. It argues that AI’s growing role shifts programming’s emphasis toward trust as well as scale.

1 A Key Barrier for AI Software Engineers

  • Developers’ trust in AI companions, rather than organizational acceptance, is identified as a key barrier to deploying AI software engineers.The paper notes that AI code generation and editing are prevalent in IDEs, while fully automated AI software engineers are not yet widely deployed in industry.Source: §1 · p. 1
  • The paper asks what place AI software engineers will have in future development workflows.It frames coexistence between automatically generated and manually written software as a possible path toward greater deployment.Source: §1 · p. 1
  • As AI code generation grows, the authors argue programming will emphasize trust alongside programming at scale.They contrast this shift with the past fifty years of steady interest in programming in the large.Source: §1 · p. 2

2 Technical and Human Trust

The paper distinguishes technical trust in code from human trust in AI-generated solutions. It does not attempt to connect these categories to interpersonal trust models from sociology and psychology.

2 Technical and Human Trust

  • Technical trust concerns measurable code attributes, including correctness, security, efficiency, maintainability, standards compliance, testing, and dependency quality.Human trust instead concerns psychological and social factors such as explainability, ethical alignment, familiar practices, expertise, collaboration, and adaptation to feedback.Source: §2 · p. 2
  • Human trust is shaped by how developers perceive and accept AI-generated solutions, including whether systems explain choices and adapt to developer feedback.The passage also identifies ethical alignment, familiar team practices, and matching developers’ expertise as factors.Source: §2 · p. 2
  • The authors do not relate their distinction to interpersonal trust models from sociology and psychology.They suggest agent-developer interactions may constitute a new mode of interaction and that new trust models may develop, building on prior human-AI trust work.Source: §2 · p. 2

3 Software Engineering Agents

The paper presents software engineering agents as a promising route to more trustworthy AI software engineers. It characterizes them by LLM back ends, software-tool interaction, and autonomous tool use.

3 Software Engineering Agents

  • An agent is described as a program that uses one or more LLMs as back-end computation and decision engines.This is one of the three aspects the paper highlights for software engineering agents.Source: §3 · p. 2
  • Software engineering agents use tools such as file navigation, code editing, test execution, and program analysis, with appropriate tool use intended to enhance developer trust.The paper characterizes agents as invoking tools autonomously through nondeterministic work plans rather than deterministic algorithms.Source: §3 · p. 3
  • The discussed agents combine LLMs and developer tools in different ways, with RepairAgent guided by a finite-state machine and restricted to test-based bug fixes.Devin and SWE-agent use tools autonomously in a human-like workflow; AutoCodeRover uses program analysis and intent extraction, while SWE-agent does not use program analysis.Source: §3 · p. 3

4 Establishing Trust

The paper proposes combining established quality assurance techniques with effective collaboration to build trust in AI software engineers. Its proposals span testing, formal verification, security guardrails, inferred specifications, and workflow practices.

4 Establishing Trust

  • Trust in an AI software engineer depends on integrating established quality assurance techniques and collaborating effectively with human developers.The paper frames this as necessary for an agent to earn a reputation similar to a senior human colleague.Source: §4 · p. 3
  • Testing and lightweight static analysis can increase technical trust, but generated tests require appropriate oracles to compare actual and expected outputs.Expected outputs might be inferred from natural-language specifications.Source: §4 · p. 3
  • Formal proofs can provide greater trust than testing alone by verifying code generated with formal specifications such as preconditions, postconditions, and loop invariants.The paper names F*, Dafny, and Verus as examples of verifiable languages.Source: §4 · pp. 3–4
  • Guardrails can sanitize inputs before they reach the LLM and validate generated code before returning it to users.The paper identifies prompt injections, malicious code, and vulnerable code as threats guardrails can defend against.Source: §4 · p. 4
  • Specification inference could recover function- or method-level intent from an initial, possibly buggy program and provide justifications for modifications.The proposed agent would navigate the code base using code search and infer intended behavior [6].Source: §4 · p. 4
  • Effective AI-human collaboration may address hesitation caused by the volume of quickly generated code, with confidence scores proposed as one response.The paper also proposes review parity, specialized review agents, and transparent scaffolding that tags AI-generated code with provenance and confidence metrics.Source: §4 · p. 4

5 Outlook

The outlook frames trusted AI software engineering as human–AI delegation, calling for agents that combine core development tasks with transparency, feedback, and safeguards.

5 Outlook

  • Effective delegation, rather than full automation, is the proposed way for humans and AI to work together in software engineering.Source: §5 · p. 4
  • A unified agent could combine coding, testing, and debugging into one coherent, explainable workflow instead of relying on many specialized agents.Source: §5 · p. 4
  • To earn developer trust, such an agent should offer transparency, adapt to feedback, and include safeguards for quality and security.Source: §5 · p. 4

Related papers

Loading 2502.13767v4…