PaperlayerView exact revision on arXiv

Source-linked AI summary

A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents

Wuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang

arXiv:2609.17698v1cs.SE

TL;DR

This study asks how QA is engineered in open-source LLM-agent repositories, where model-influenced actions can affect external systems. It analyzes repository artifacts across four QA layers and finds that coverage remains uneven, with inconsistent safeguards and rare failure-oriented tests. The authors conclude that agent QA should connect safeguards and tests to end-to-end workflows.

  • Problem

    Agents can execute commands, modify files, browse, and call APIs, so QA must consider whether model-influenced actions remain bounded across execution surfaces.

    Source: §I · p. 1

  • Method

    The study codes repository-visible QA practices across execution surfaces, safeguards, testing artifacts, and risk scenarios in 157 open-source agent projects.

    Source: §I · p. 1, §VI · p. 10

  • Results

    QA practices are common but unevenly cover agent workflows, with safeguards inconsistently applied across equivalent routes and boundary, adversarial, or multi-step tool-use tests rare.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    Agent QA should connect explicit tool authority, safeguards, and failure-oriented checks to workflows reaching sensitive actions.

    Source: §VI · p. 11

    The corpus is a fixed May 2026 snapshot of popular open-source GitHub projects and overrepresents coding and browser/computer-use agents.

    Source: §V · p. 10

Abstract

from arXiv · show

Large language model (LLM)-based agents are increasingly used across software engineering, web automation, research, and productivity applications. Their integration of planning, memory, tool use, code execution, and external interactions enables greater autonomy but also introduces new reliability, safety, and security risks. We present a large-scale empirical study of quality assurance (QA) practices in 157 open-source LLM-based agent projects with at least 100 GitHub stars. We analyze documentation, source code, configurations, and tests to characterize QA practices across execution surfaces, safeguards, testing artifacts, risk scenarios, and recurring gaps. We find that current QA primarily focuses on basic functionality and high-risk actions, while coverage remains fragmented. Safeguards are inconsistently applied across equivalent execution routes, tests rarely examine boundary, adversarial, or multi-step tool-use failures, and identified risks are seldom translated into end-to-end QA checks. These findings highlight the need to move beyond feature-level testing toward systematic end-to-end validation that ensures agent workflows remain within intended boundaries when interacting with untrusted inputs, tools, persistent state, and external APIs.

I. INTRODUCTION

Agent QA must address whether model-influenced actions remain bounded across execution surfaces, not only whether outputs are correct. This study examines repository-visible QA practices and finds fragmented coverage, while situating its contribution alongside work on agent risks and evaluations.

I. INTRODUCTION

  • Agent QA must assess whether model-influenced actions remain bounded across execution surfaces, because agents can affect files, systems, and external services.Potential consequences include privacy leakage, unsafe commands, financial loss, and compromised credentials.Source: §I · p. 1
  • Across 157 open-source agent projects, the study characterizes repository-visible QA practices and finds safeguards inconsistently applied across routes and agent-specific failure tests rare.The study covers execution surfaces, safeguards, testing artifacts, and risk scenarios; reported scenarios are candidates for boundary review, not confirmed vulnerabilities.Source: §I · pp. 1–2
  • The study analyzes how actions enter architectures, pass through execution surfaces, meet safeguards, and are validated through testing and risk-scenario review.Its coding describes practices visible in repositories, not whether mechanisms are effective after deployment.Source: §I · pp. 1–2

II. BACKGROUND AND RELATED WORK

  • Prior work identifies prompt-injection and agent-security risks, but does not directly measure how often open-source repositories expose corresponding tools, policies, tests, and configuration paths.The study complements capability benchmarks and individual-attack evaluations with repository-level analysis of QA engineering artifacts.Source: §II · p. 2
  • The study broadens repository-level QA evidence beyond tests to include code, configuration, documentation, safeguards, execution surfaces, and structured risk observations.Repositories may document safeguards without testing them, test permissions without applying them to every interface, or expose risky surfaces without a confirmed vulnerability.Source: §II · p. 2

III. EMPIRICAL STUDY DESIGN

The study selects popular open-source agent repositories, reviews their artifacts using a four-layer rubric, and synthesizes coded records into corpus-level QA findings.

III. EMPIRICAL STUDY DESIGN

  • The study proceeds through repository selection, artifact review and annotation, and cross-repository synthesis.Synthesis aggregates coded records into corpus-level statistics, QA practice families, and undercovered problems.Source: §III · p. 2, Figure 1 · p. 3

A. Terminology

  • The analysis distinguishes repository artifacts, coded records derived from them, and four layers used to organize those records.The layers are execution surfaces, safeguards, testing artifacts, and risk scenarios.Source: A. Terminology · p. 2, Table II · p. 3

B. Repository Selection

  • The corpus targets public GitHub agent projects with at least 100 stars as of May 2026 and enough artifacts to examine QA-relevant behavior.Repositories also had to integrate an LLM or compatible API, provide an executable entry point or package manifest, and expose a model-mediated action surface.Source: B. Repository Selection · pp. 2–3

C. Repository Review and Annotation

1) Coding Rubric:

A four-layer rubric organizes repository evidence about agent action channels, controls, tests, and risk situations; annotations record visible mechanisms rather than proven effectiveness.

1) Coding Rubric:

  • The coding rubric separates execution surfaces, safeguards, testing artifacts, and risk scenarios into four analysis layers.These records respectively capture action channels, controls, tests or evaluations, and situations warranting boundary review; risk scenarios are not confirmed vulnerabilities.Source: 1) Coding Rubric · p. 3

2) Annotation Process:

  • Reviewers familiarize themselves with each repository, inspect execution-related artifacts, then code them using the four-layer rubric.Inspection includes documentation, source code, configuration, manifests, tests, examples, deployment artifacts, and CI workflows.Source: 2) Annotation Process · p. 3
  • Annotations require a repository-visible artifact and distinguish capabilities from QA mechanisms associated with those capabilities.For example, shell access is a capability, while approval prompts, sandboxing, redaction, monitoring, or tests are QA mechanisms.Source: 2) Annotation Process · p. 3
  • QA labels record visible mechanisms or testing intent, not validated runtime effectiveness or proof that failures are prevented after deployment.A sandbox setting or test path demonstrates repository evidence of a mechanism, not its effectiveness in deployed use.Source: 2) Annotation Process · p. 4
  • The label set emerged through open coding and annotation cards that were grouped and refined into final labels.A plugin can be coded as a capability, safeguard, or risk scenario depending on the artifact's role.Source: 2) Annotation Process · p. 4

D. Research Questions

  • RQ1 asks what visible QA practices appear across execution surfaces, safeguards, testing artifacts, and risk scenarios.Its subquestions cover action channels, controls, tests and evaluations, and situations that may reach sensitive actions, assets, or state.Source: D. Research Questions · p. 4
  • RQ2 identifies missing, fragmented, or weak coverage, including gaps in end-to-end validation, safeguard consistency, robustness testing, and bounded-action documentation.It also considers environment-aware validation and documentation of trusted assumptions.Source: D. Research Questions · p. 4

IV. RESULTS

The execution-surface review identifies risks from overlapping action routes, external effects, credentials, file and process authority, extensions, and browser state.

1) QA Challenges:

  • Projects with multiple routes to similar authority face a cross-path consistency challenge because a control on one route may not cover another.The cited example exposes CLI, shell, filesystem, browser, web, and MCP paths.Source: 1) QA Challenges · p. 4
  • Outbound provider, MCP, search, browser, and local API channels can send data or trigger remote actions beyond the project boundary.Browser automation may also operate through persistent signed-in state, interacting with pages, cookies, uploads, downloads, or domains.Source: 1) QA Challenges · pp. 4–5
  • Provider keys, OAuth stores, browser profiles, and MCP environment variables can become reachable by tools or subprocesses.The example project has configured MCP environments and provider settings that must be kept away from logs and tool outputs.Source: 1) QA Challenges · p. 4
  • File tools, downloads, skills, and hooks can read or write beyond an intended workspace, while model-selected commands can alter host processes, environment variables, or workspace contents.The cited shell example's authority depends on approval and sandbox mode.Source: 1) QA Challenges · p. 4
  • MCP servers, plugins, and skills can expand model-callable tools, while configured or runtime-loaded code can execute outside a fixed application path.The cited examples include MCP extensions and extension configuration that changes callable runtime capabilities.Source: 1) QA Challenges · p. 4

2) QA Practice: •

Execution-surface QA focuses on keeping model-influenced actions within authority boundaries across multiple routes. Projects use approval, isolation, policy, secret-handling, extension, and browser-scoping controls, but equivalent routes may be controlled unevenly.

2) QA Practice: •

  • Approval and authorization controls require an explicit grant before privileged execution or access to sensitive state.Examples include bearer authentication for sensitive local API routes and confirmation or trust modes for interactive actions.Source: 2) QA Practice: • · p. 5
  • Sandboxing and isolation bound actions to a workspace, container, or restricted mode, separating file writes, command effects, and runtime-loaded code from the host.OpenHands separates workspace state from sandbox execution, while Codex routes shell commands through sandbox policy.Source: 2) QA Practice: • · p. 5
  • Permission and policy controls specify which actions, paths, tools, routes, or modes are allowed before execution.MaxKB uses token permissions and tool filtering around custom Python tools and generated MCP servers.Source: 2) QA Practice: • · p. 5
  • Secret management and redaction limit where credentials can be stored, inherited, logged, or returned through model/tool paths.Gemini CLI uses logging redaction and environment sanitization around tool and provider flows.Source: 2) QA Practice: • · p. 5
  • MCP/plugin registration makes extension points explicit, helping reviewers assess added authority and overlap with built-in action paths.Goose pairs MCP extension support with permission, confirmation, and server-auth mechanisms.Source: 2) QA Practice: • · p. 5
  • Browser and domain-scoping controls restrict domains, pages, scripts, uploads, downloads, or session data available to browser agents.Browser-use includes domain settings, sensitive-data filtering, IP blocking, storage-state handling, and upload-path gating.Source: 2) QA Practice: • · p. 5

B. RQ1.2: QA Challenges and Practices at the Safeguard Layer

Safeguard QA centers on whether high-authority actions are bounded consistently across routes. The reviewed challenges span commands, workspace access, credentials, extensions, and browser sessions.

B. RQ1.2: QA Challenges and Practices at the Safeguard Layer

  • A project may protect one route to shell, file, browser, plugin, or secret access while leaving another route to similar authority less clearly bounded.The key safeguard issue is incomplete control coverage across routes.Source: B. RQ1.2: QA Challenges and Practices… · p. 5

1) QA Challenges: •

  • Privileged commands need approval and containment, workspace access needs boundary control, and credentials need scoped handling across tools, subprocesses, logs, and provider calls.The examples include Codex shell execution, OpenHands workspace separation, and Gemini CLI configuration and redaction paths.Source: 1) QA Challenges: • · p. 6
  • Plugins and MCP servers can add callable tools and inherited authority, while browser sessions expose persistent state, page interactions, and cross-domain access.Goose treats MCP/plugin support as a primary capability; Browser-use exposes navigation and page interaction over browser state that may persist across tasks or domains.Source: 1) QA Challenges: • · p. 6

2) QA Practices: •

Safeguard practices include approval, isolation, policy, secret handling, extension registration, and browser scoping, but coverage is context-sensitive. Testing remains weighted toward conventional regression, with agent-specific boundary, adversarial, and multi-step checks less systematic.

2) QA Practices: •

  • Approval, sandboxing, and permission policies respectively gate privileged actions, bound execution away from the host, and make allowed actions reviewable.Examples include Codex approval and sandbox policy, OpenHands workspace separation, and MaxKB token permissions and tool filtering.Source: 2) QA Practices: • · p. 6
  • Secret redaction, explicit MCP/plugin registration, and browser/domain scoping address credential exposure, extension authority, and session or cross-domain risks.Examples include Gemini CLI environment sanitization, Goose extension controls, and Browser-use domain and sensitive-data settings.Source: 2) QA Practices: • · p. 6
  • Safeguards are widely present, but coverage must account for how, where, and under what authority a capability is reached.The same capability may be acceptable in one setting and risky in another.Source: 2) QA Practices: • · p. 6

C. RQ1.3: QA Challenges and Practices at the Testing Layer

  • 137 (87.3%) projects contain conventional tests or specs, while 8 (5.1%) contain prompt-injection, adversarial, jailbreak, or red-team testing paths; agent-specific boundary and multi-step tests are less systematic.The counts cover 157 projects.Source: C. RQ1.3: QA Challenges and Practices… · pp. 6–7
  • Conventional tests cover implementation regressions in tool wrappers, APIs, storage, settings, provider clients, parsers, and file operations.Examples include sandbox cleanup and secret-store tests in OpenHands, and session, filesystem, and navigation tests in Browser-use.Source: C. RQ1.3: QA Challenges and Practices… · pp. 6–7
  • Security/safety tests check whether high-authority actions cross boundaries they should not cross, including domain, IP, secret, approval, and shell-safety boundaries.Examples include blocked prohibited domains, filtered sensitive values, and tests against destructive shell commands.Source: C. RQ1.3: QA Challenges and Practices… · p. 7
  • Evaluation and red-team practices test expected tool sequences and workflows, while prompt-injection tests check how untrusted context affects later actions.Examples include Gemini CLI workflow evals, Cordum red-team scenarios, and tests of context ordering and invalid paths.Source: C. RQ1.3: QA Challenges and Practices… · p. 7

D. RQ1.4: QA Challenges and Practices in Risk Scenarios Layer

The risk-scenario layer uses end-to-end risk-chain templates rather than treating surfaces or controls in isolation. It examines how ordinary agent behaviors can connect through model decisions or untrusted context to sensitive actions or assets.

D. RQ1.4: QA Challenges and Practices in Risk Scenarios Layer

  • Risk-chain templates review end-to-end paths in which ordinary behaviors become risky when connected through model decisions or untrusted context before a sensitive action or asset.The unit of analysis is the path, not an individual surface or control.Source: D. RQ1.4: QA Challenges and Practices… · p. 7

1) QA Challenges:

  • Secret-to-tool and workspace-to-persistent-state chains ask whether credentials remain scoped and workspace boundaries hold as state is reused.The examples connect secrets to logs, subprocesses, provider calls, or tool outputs, and workspace operations to settings, hooks, plugins, or secret storage.Source: 1) QA Challenges · pp. 7–8
  • Extension-to-authority, mode-switch-to-privilege, and runtime-code chains ask whether added tools, alternate execution modes, and loaded code remain within review boundaries.These chains cover inherited local or network authority, alternate paths to privileged actions, and executable logic introduced after fixed application code review.Source: 1) QA Challenges · p. 8
  • Model-to-process, session-to-action, and untrusted-context-to-action chains examine whether commands, persistent browser state, or untrusted inputs shape later actions.The examples include host-process or workspace changes, cross-task browser state, and context that moves into a later command, file operation, or disclosure.Source: 1) QA Challenges · p. 8

2) QA Practices:

Risk-scenario QA uses both chain interruption at critical transitions and design-time reduction of risky paths. Common mechanisms include authorization, isolation, redaction, permissions, and scoping.

2) QA Practices:

  • Approval and authorization interrupt model-mediated paths before sensitive actions, with the practice appearing across the reported chain types.Examples include approval around command execution and confirmation or server authentication around extension paths.Source: 2) QA Practices · p. 8
  • Sandboxing and isolation limit inherited host authority for tools, commands, runtime modules, and workspace operations.The practice appears in 104 of 149 secret-to-tool chains and 73 of 99 runtime-code chains.Source: 2) QA Practices · p. 8
  • Secret management and redaction interrupt credential propagation into logs, tool outputs, or provider-facing content.Redaction appears in 96 of 149 secret-to-tool chains and 25 of 45 untrusted-context-to-action chains.Source: 2) QA Practices · p. 9
  • Design-time controls scope allowed actions, make extension authority reviewable, and constrain browser sessions and domains.Permission and policy controls make allowed actions explicit; registration/configuration and browser scoping address extension and session paths.Source: 2) QA Practices · p. 9
  • Risk-scenario practices combine design-time risk reduction with chain interruption at critical points.The reported mechanisms include authorization, isolation, and redaction.Source: 2) QA Practices · p. 9

E. RQ2: Undercovered QA Problems Across the Four RQ1 Layers

Undercovered problems arise when exposed agent risks lack directly relevant QA practices or controls remain local to particular routes, modes, or artifact types. The gaps span action surfaces, safeguards, tests, and risk-chain operationalization.

E. RQ2: Undercovered QA Problems Across the Four RQ1 Layers

  • Common credential, network/API, file, shell, and dynamic-code routes may lack their direct boundary controls.The practical implication is to attach a boundary such as redaction, permission policy, or sandboxing to each action route.Source: E. RQ2: Undercovered QA Problems… · p. 9
  • Broad safeguards may not protect the specific sensitive action, leaving secret handling, workspace containment, extensions, or browser sessions without matching practices.The gap is to bind the appropriate safeguard to the sensitive action, rather than merely adding a general control.Source: E. RQ2: Undercovered QA Problems… · p. 9
  • Agent-specific testing gaps include adversarial tests for untrusted context, replay or regression tests for multi-step behavior, and security tests for high-authority actions.Tests should fail when model-mediated actions cross approval, sandbox, credential, plugin, file, or browser boundaries.Source: E. RQ2: Undercovered QA Problems… · p. 9
  • Risk chains involving credentials, host commands, workspace boundaries, extensions, and runtime code need explicit triggers, protected assets, controls, and regression checks.The study identifies operationalization of these chains as a remaining gap.Source: E. RQ2: Undercovered QA Problems… · p. 10
  • The recommendations are to define controls by protected action or asset across interfaces and test high-authority routes for boundary failures.Suggested failure-oriented checks cover approval, sandboxing, policy, redaction, domain scope, and extension checks.Source: E. RQ2: Undercovered QA Problems… · p. 10
  • Many undercovered problems are agent-specific, including model-mediated tool use, session reuse, extension authority, and untrusted-context propagation.These often lack QA practices beyond ordinary tests, access checks, or configuration review.Source: E. RQ2: Undercovered QA Problems… · p. 10

V. THREATS

The study’s repository-level analysis finds uneven QA coverage across agent workflows, while its counts describe visible practices rather than runtime effectiveness. Its scope is a fixed snapshot of popular open-source projects, and the conclusion calls for workflow-wide boundary checks.

V. THREATS

  • Repository artifacts may omit hosted services, provider-side controls, deployment-specific settings, and dependency behavior outside the inspected snapshot.Automated search and LLM-assisted summarization may also introduce false positives, false negatives, or inconsistent terminology.Source: §V · p. 10
  • The four analysis layers are an operational coding rubric, not a universal taxonomy, and counts do not measure runtime effectiveness.Projects may reasonably classify artifacts differently when one artifact serves multiple roles.Source: §V · p. 10
  • The corpus is a fixed May 2026 snapshot of popular open-source GitHub projects, overrepresenting coding and browser/computer-use agents.It is not a census; closed-source, enterprise, and small prototype agents may differ.Source: §V · p. 10

VI. CONCLUSION

  • In 157 open-source LLM-based agent projects, QA practices were common but coverage across agent workflows remained uneven.The central gap was whether QA artifacts covered the full workflow, not whether projects had artifacts at all.Source: §VI · pp. 10–11
  • Future agent QA should make tool authority explicit and connect safeguards and failure-oriented checks to sensitive actions and workflow boundaries.The conclusion identifies boundary, adversarial, and multi-step tool-use cases as targets for additional checks.Source: §VI · p. 11

Related papers

Loading 2609.17698v1…