PaperlayerView exact revision on arXiv

Source-linked AI summary

Build or Adopt? Custom-Built vs. General-Purpose LLM Agent Harnesses for Automotive GUI Testing

Arsham Gholamzadeh Khoee, Ruyu Fu, Jiaqi Wei, Ruikun Deng, Robert Feldt, Dhasarathy Parthasarathy, Yinan Yu

arXiv:2610.33636v1cs.SEcs.AI

TL;DR

The study asks whether general-purpose harnesses can replace custom-built harnesses in industrial GUI testing and what adoption changes in maintenance, operational cost, and stakeholder experience. At Volvo Group, the study varies the harness while holding tools and models constant, benchmarks IVI and public-app tasks, and interviews ten practitioners. General-purpose harnesses improve open-ended-task success and lower nightly IVI regression total monthly costs by 3–66%, but custom remains competitive and more consistent on stepwise IVI tests (inconsistent outcomes: Setup A 2–3%; Setup B 5–9%).

  • Problem

    The study asks whether general-purpose harnesses can replace custom-built harnesses in industrial GUI testing and what adoption changes in maintenance, operational cost, and stakeholder experience.

    Source: §I · pp. 1–2

  • Method

    At Volvo Group, the study varies the harness while holding tools and models constant, benchmarks IVI and public-app tasks, and interviews ten practitioners.

    Source: §I · pp. 1–2

  • Results

    General-purpose harnesses improve open-ended-task success and lower nightly IVI regression total monthly costs by 3–66%, but custom remains competitive and more consistent on stepwise IVI tests (inconsistent outcomes: Setup A 2–3%; Setup B 5–9%).

    Source: §VI · p. 10, §VIII · p. 10

  • Takeaways & Limitations

    Adoption shifts maintenance from the agent loop toward tools and instructions, while teams have less control over what the model receives and testers still need action visibility and scenario adherence.

    Source: §VI · pp. 9–10

    All interviewees work with Setup A, and three developers co-authored both setups and ran the experiments, potentially biasing their accounts.

    Source: §VII · p. 10

Abstract

from arXiv · show

As general-purpose LLM agent harnesses mature, organizations must decide whether to continue developing in-house systems or adopt existing alternatives. We study this decision at Volvo Group for GUI testing of Android-based in-vehicle infotainment (IVI) systems. We compare a production, custom-built harness with two general-purpose harnesses, Claude Code and OpenCode, keeping the tools, accessed through the Model Context Protocol (MCP), and the models the same. We evaluate the harnesses with up to four models on Volvo's IVI test cases and on open-ended tasks in public apps (Google Maps, Booking.com), and interview ten practitioners: developers, the product owner, and testers. On Volvo's stepwise IVI tests, general-purpose harnesses match the custom-built harness in success but vary more across runs. Their inference cost per task is 2.6-10.7 times higher, yet the engineering effort they save outweighs this: for a nightly IVI regression suite, total monthly cost, including engineering effort and inference, is 3-66% lower. On open-ended public-app tasks, they succeed more often, mainly because they recover better from incorrect actions. Interviews suggest that external advances can turn an in-house harness into technical debt even without deterioration. Adopting a general-purpose harness shifts maintenance from the agent loop to tools and instructions, but gives teams less control over what the model receives. Regardless of the harness, testers want to inspect the agent's actions and ensure that it follows the prescribed test scenario.

I. INTRODUCTION

This paper examines whether an industrial GUI-testing team should maintain a custom-built harness or adopt a general-purpose one as such harnesses mature. At Volvo Group, it compares the alternatives while holding tools and models constant and combines benchmarking, practitioner interviews, and a cost estimate.

I. INTRODUCTION

  • Prior mobile-GUI agent research evaluates purpose-built harness designs, but the authors know of no comparison with a general-purpose harness or study of industrial maintenance.The cited examples include GPTDroid, DroidAgent, AutoDroid, AppAgent, and Mobile-Agent.Source: §I · p. 1
  • The central decision is whether a custom harness's initial investment remains worthwhile as models, requirements, and general-purpose alternatives change.The paper frames this as a build-versus-adopt question about operational cost and the effects on maintainers and testers.Source: §I · pp. 1–2
  • The study pairs quantitative benchmarking with interviews of ten practitioners: four developers, the product owner, and five testers.The evaluation spans four models, Volvo IVI cases, and open-ended public-app tasks.Source: §I · pp. 1–2
  • The study compares one production custom-built harness with two general-purpose harnesses, holding tools and models constant across 66 IVI and 56 public-app tasks.It also examines flakiness, robustness to model and application change, latency, inference cost, and practitioner perspectives.Source: §I · pp. 1–2

II. PROBLEM FORMULATION

The paper models Android GUI testing as an agent choosing UI actions from an intent, current state, and interaction history. The visual screen and structural hierarchy can disagree, while asynchronous transitions make outcomes stochastic.

II. PROBLEM FORMULATION

  • GUI testing is modeled as sequential decisions in which an agent executes a natural-language intent through UI actions.The state combines a screenshot-based visual representation with an XML view hierarchy containing semantic element properties.Source: §II · p. 2
  • Neither the screenshot nor hierarchy alone is sufficient: visible elements may lack semantic text, hierarchy nodes may be hidden, and the representations may disagree about state.Such mismatches can lead the agent to target the wrong element or misjudge the app state.Source: §II · p. 2
  • An action combines a type such as TAP or SCROLL with parameters, which can include coordinates derived from an element's bounding box.Source: §II · p. 2
  • The policy selects the next action from the current state, intent, and history of prior states and actions.The history is the sequence of earlier state-action pairs.Source: §II · p. 2
  • Android's asynchronous transitions make the next state stochastic; acting before an animation or screen change finishes can produce an unintended state and run-to-run variation.The paper measures this variation as flakiness.Source: §II · p. 2

III. SYSTEM ARCHITECTURE

  • The harnesses differ in who controls the next-action policy: custom-built code fixes the workflow, while a general-purpose harness lets the model decide within a generic loop.Both use the same tools and device access; the general-purpose harness is adapted through outer components such as instructions and an MCP server.Source: §III · p. 2

A. Setup A: Based on a Custom-Built Harness

Setup A uses a developer-defined workflow around its primary agent, whereas Setup B replaces that orchestration with a vendor-maintained general-purpose loop while retaining shared tools and device access.

A. Setup A: Based on a Custom-Built Harness

  • Setup A observes both a screenshot and a simplified hierarchy, relying on the multimodal model to align visual and structural elements.The agent is instructed to judge state from hierarchy flags and verify disagreements with a tool.Source: A. Setup A: Based on a Custom-Built… · pp. 2–3
  • Setup A's custom harness fixes the primary agent's four-step ReAct loop; Setup B's agent chooses how to proceed within a generic loop.The setups share the tool layer and device access; Fig. 1 colors identify who maintains each component.Source: A. Setup A: Based on a Custom-Built… · p. 2, Figure 1 · p. 3
  • A single LLM call decomposes the intent into atomic steps, updates a checklist, judges completion, and selects the next tool and arguments.The harness checks tool validity and argument schemas before executing a valid call and recording its outcome.Source: A. Setup A: Based on a Custom-Built… · p. 3
  • Setup A rebuilds each prompt from the intent, current observation, fixed instructions, and step-log history, while retaining only screenshots around the last action.Its run records include step-level screenshots, reasoning, checklists, actions, prompts, responses, and links to subagent runs.Source: A. Setup A: Based on a Custom-Built… · p. 3

B. Setup B: Based on a General-Purpose Harness

  • Setup B adapts Claude Code or OpenCode through a shared instruction template and an MCP server that exposes Setup A's tools.The template prescribes observing before acting, using hierarchy bounds, and performing and verifying one action at a time.Source: B. Setup B: Based on a General-Purpose… · pp. 3–4
  • In Setup B, the model chooses tools, decides when to observe and stop, and uses conversation history that the vendor-maintained harness condenses when long.The MCP server reuses the shared tools but leaves tool choice, timing, and stopping to the primary agent.Source: B. Setup B: Based on a General-Purpose… · p. 4

C. Shared Tool Layer and Device Access

Both setups use the same device-tool code, but differ in how the primary agent invokes it and who maintains the surrounding orchestration. Volvo retains the shared tool layer while the general-purpose harness vendor maintains the agent loop and context management.

C. Shared Tool Layer and Device Access

  • Both setups use the same device-tool code, accessed as local Python functions in Setup A and through an MCP server in Setup B.Device tools capture screenshots and hierarchy data or perform single gestures and key events, deriving target coordinates from hierarchy bounds.Source: C. Shared Tool Layer and Device Access · p. 4
  • Subagents handle multi-interaction subtasks such as scrolling, text entry, and waiting for a screen condition, returning only the final result to the primary agent.Their behavior depends on the underlying model because they call an LLM.Source: C. Shared Tool Layer and Device Access · p. 4

D. Maintenance and Ownership

  • Setup A requires Volvo to revise its loop, prompts, context construction, and sometimes tools or routing as models and requirements change.A model replacement can change intent decomposition, completion timing, or repeated actions, prompting recurring harness adaptation.Source: D. Maintenance and Ownership · p. 4
  • In Setup B, vendors maintain the loop, context management, and tool-calling protocol, while Volvo maintains instructions, the MCP server, and tools.New requirements usually prompt instruction updates; a new device capability requires a shared-tool addition, without changing the harness.Source: D. Maintenance and Ownership · p. 4

IV. QUANTITATIVE EVALUATION

The evaluation compares harnesses on Volvo IVI procedures and open-ended tasks in Google Maps and Booking.com, using shared tools, devices, and models. Repeated runs are judged from execution traces, with task outcomes and efficiency measured per run.

IV. QUANTITATIVE EVALUATION

  • The evaluation covers task success, reliability, and efficiency across harness configurations, addressing effectiveness, robustness to change, and operational cost.The section reports effectiveness, robustness to model or application changes, and per-task latency, tokens, and inference cost.Source: §IV · p. 4

1) Configurations:

  • Setup A uses GPT-5.1 in production; both setups also run shared models to separate harness effects from model effects.Claude Code is used only with Anthropic models.Source: 1) Configurations · p. 4

2) Benchmarks:

  • The 66-task Volvo IVI benchmark uses stepwise test intents, while the 56-task AMEX benchmark covers open-ended goals in Google Maps and Booking.com.AMEX includes 40 Google Maps tasks and 16 Booking.com tasks; task milestones describe intermediate UI states.Source: 2) Benchmarks · pp. 4–5

3) Execution and Evaluation:

  • Configurations share tasks, devices, and tools; each task starts from the target app’s home screen and is repeated two or three times.Tool traces, outputs, and screenshots are scored by an LLM judge; the paper also checks judge agreement and manually validates one Booking.com repetition.Source: 3) Execution and Evaluation · p. 5

4) Metrics:

General-purpose harnesses achieve comparable success on prescribed IVI tests and higher success on open-ended public-app tasks, with their clearest advantage in recovery from incorrect actions. The evaluation also measures milestones, action correctness, flakiness, and per-task efficiency.

4) Metrics:

  • The measures cover task success, milestone reach, tool correctness, fault recovery, flakiness across repeated runs, and per-task latency, tool calls, tokens, and inference cost.Flakiness compares outcomes across runs of a configuration and is compared only within each benchmark.Source: 4) Metrics · p. 5

B. Effectiveness (RQ1)

  • Setup B produces inconsistent outcomes on 3–6 of 66 IVI tasks, compared with 1–2 for Setup A; on Google Maps, flakiness is 13–25% versus 35–48%.The higher success rate partly leaves fewer Setup B tasks with mixed outcomes on Maps.Source: B. Effectiveness (RQ1) · p. 5
  • On Booking.com, Setup B’s success lead is not significant over 16 tasks, but its milestone reach is 72–78% versus 56–59%.Milestones represent intermediate UI states on the path to a task goal.Source: B. Effectiveness (RQ1) · p. 5
  • Setup B does not make fewer incorrect calls, but succeeds after incorrect calls in 78–83% of Google Maps runs and 27–38% of Booking.com runs.Setup A succeeds after incorrect calls in 24–53% of Google Maps runs and 10–18% of Booking.com runs; on IVI, incorrect calls are rare and recovery is low.Source: B. Effectiveness (RQ1) · p. 5
  • The contrast tracks task style: prescribed IVI steps leave little room to recover, while open-ended AMEX goals require planning and allow alternate paths to success.The paper attributes Setup B’s advantage to its accumulated context, which can help the agent detect and correct ineffective or wrong actions.Source: B. Effectiveness (RQ1) · pp. 5–6
  • On IVI, all configurations achieve 55–61% success; on Google Maps, Setup B leads Setup A by 28 points with GPT-5.4 and 17–19 with Sonnet 4.6.The IVI differences are not significant; on Maps, Setup B also leads with Opus 4.6, though that comparison is not significant at p = 0.06.Source: B. Effectiveness (RQ1) · p. 5, Figure 2 · p. 6

C. Robustness to Change (RQ2)

Without adaptation, both harness types tolerate most model changes, but the custom-built harness transfers less successfully beyond its original setting. Application comparisons are qualified because application, task style, and device change together.

C. Robustness to Change (RQ2)

  • On Google Maps, success varies by 21 points in Setup A and 7 points in OpenCode; the difference is driven mainly by GPT-5.4, where success is 53% versus 80%.On IVI and Booking.com, success varies by at most 3 points for models shared by both setups; with Sonnet 4.6 and Opus 4.6, Maps results vary by at most 5 points.Source: Table III · p. 6, C. Robustness to Change (RQ2) · p. 7
  • Setup A falls behind Setup B on both AMEX apps with every model, while its production GPT-5.1 configuration has 10% success on Booking.com.The application comparison also changes task style and device, so the effect cannot be attributed to the application alone.Source: Table III · p. 6, C. Robustness to Change (RQ2) · p. 7
  • The results suggest that the custom-built harness’s comparable IVI performance does not extend to AMEX applications and task styles.The paper concludes that general-purpose harnesses tolerate most model changes, while the custom-built harness transfers less successfully beyond its original setting.Source: C. Robustness to Change (RQ2) · p. 7

D. Efficiency (RQ3)

General-purpose harnesses cost more in inference per task but do not increase latency. For a nightly IVI regression suite, estimated engineering savings outweigh inference costs.

D. Efficiency (RQ3)

  • Setup B’s conversation context grows with tool calls and outputs and is resent at each turn, while Setup A rebuilds a concise prompt from prior actions and the current screen.On IVI, Claude Code with Sonnet 4.6 costs $4.97 per task against $0.46 for Setup A.Source: D. Efficiency (RQ3) · p. 7
  • With the same model, Setup B’s mean latency per task is 3 to 57 seconds lower than Setup A’s.On IVI, Setup A uses fewer tool calls but takes longer because every step deterministically captures a screenshot and view hierarchy.Source: D. Efficiency (RQ3) · p. 7
  • 2.6–10.7 times higher inference cost per task on IVI accompanies 3–66% lower monthly total cost for a nightly regression suite.The total includes engineering effort and inference; across all benchmarks, general-purpose harness inference costs 1.9–10.7 times more per task with the same model.Source: D. Efficiency (RQ3) · p. 7

V. QUALITATIVE EVALUATION

This section explains how Volvo interviewed practitioners and analyzed their accounts to understand maintenance and stakeholder perspectives on the harnesses.

1) Participants:

  • Ten Volvo practitioners took part: four developers, the product owner, and five QA testers; three developers had experience with both setups.Setup B had not yet been released to testers, so their interviews focused on valued properties of Setup A and requirements for a replacement.Source: 1) Participants · p. 7

2) Data Collection:

  • Semi-structured interviews used role-specific property ratings to structure discussions, then elicited reasons, setup differences, and a manager estimate of Setup A maintenance time.Developers rated engineering properties and testers rated user-facing properties from 1 to 10; the ratings were discussion prompts, not measurements.Source: 2) Data Collection · p. 7

3) Data Analysis:

  • Researchers summarized interview explanations, coded notable experiences and concerns, grouped codes by research question, and resolved disagreements through discussion.The analysis used a lightweight thematic approach.Source: 3) Data Analysis · p. 8

B. Findings

The interviews trace why maintaining Volvo’s in-house harness became harder to justify, how adoption reallocates engineering responsibilities, and what testers need to trust autonomous testing.

B. Findings

  • The four findings are presented as lessons from one industrial case, not as general best practices.They cover maintenance justification, engineering trade-offs, runtime costs and control, and trustworthy testing.Source: B. Findings · p. 8

F1: Debt emerged from a shifting ecosystem, not a decaying system.

  • External model, requirement, and harness changes made Setup A increasingly costly relative to alternatives, although it had not deteriorated and matched Setup B on IVI task success.The authors characterize this as ecosystem-induced technical debt: rising opportunity cost from maintaining in-house orchestration as the ecosystem matures.Source: B. Findings · p. 8

F2: External harnesses shift maintenance from orchestration to integration.

  • Setup B shifts maintenance from agent-loop changes to tools, instructions, and integration, while generic orchestration moves to the vendor.Developers found Setup B easier to build, maintain, and extend; clear tool descriptions and parameters remain an integration challenge, and the team must budget for that work.Source: B. Findings · p. 8

F3: Outsourcing orchestration reduces engineering effort but introduces runtime trade-offs.

  • Setup B reduced internal orchestration effort but gave the team less control over context management and incurred higher inference costs despite comparable or higher aggregate task success.Developers attributed part of the higher cost to larger execution histories repeatedly included in model calls.Source: B. Findings · p. 8
  • At most 11% faster on IVI, Setup B still faced long-suite bottlenecks: a tester estimated about 60 tests took five hours, while benchmark mean latency was 297 s per task.The tester workload and benchmark workload were not directly equivalent; long suites occupy scarce rigs and delay QA feedback.Source: B. Findings · pp. 8–9

F4: Test fidelity requires bounded autonomy and execution visibility.

  • Testers value visible execution and scenario fidelity, inspecting screenshots, logs, and traces; an agent that works around an app crash can obscure a genuine defect.The implication is to stop and report app failures rather than work around them, and preserve testers’ ability to verify both outcome and path.Source: B. Findings · p. 9

VI. DISCUSSION AND INDUSTRIAL LESSONS LEARNED

The discussion frames harness choice as a set of workload-specific trade-offs: engineering effort, inference cost, intent style, stability, visibility, and control.

Rethinking technical debt in the LLM era.

  • An in-house harness can become technical debt without code deterioration as models, requirements, and general-purpose alternatives evolve; teams should keep domain knowledge separable from orchestration.The authors recommend revisiting build-versus-adopt decisions as the ecosystem changes.Source: §VI · p. 9

Trading engineering effort for inference cost.

  • For nightly IVI regression, total monthly costs were 3–66% lower with Setup B, while engineering effort fell and inference cost rose; the preferred burden depends on workload and test frequency.The discussion recommends comparing internal engineering effort with per-run inference cost and expected test frequency, while tracking suite duration per rig.Source: §VI · pp. 9–10

Flexible intents, stable verdicts.

  • Setup B performed better on goal-level intents, but IVI outcomes were inconsistent on 5–9% of tasks versus 2–3% with Setup A; evaluate both intent styles and repeated-run stability.Higher open-ended success did not imply greater stability on structured tests, where inconsistency can add to testers’ manual verification workload.Source: §VI · p. 10

Keeping visibility and control when outsourcing the loop.

  • Outsourcing the loop does not outsource trust: teams still need tester-visible run records and reviewable examples of expected and failed behavior.A team-owned experience library may help in narrow domains with recurring tasks, while instructions should require stopping and reporting app failures.Source: §VI · p. 10

VII. THREATS TO VALIDITY

The conclusion favors adoption for workloads like Volvo’s while identifying interview, cost-measurement, and application-selection boundaries on the evidence.

VII. THREATS TO VALIDITY

  • Inference costs reflect vendor prices at experiment time, token counts are comparable only within a model, and interview ratings were discussion prompts rather than measurements.Rated properties also differed between developers and testers.Source: §VII · p. 10
  • All interview participants worked with Setup A, and developers assessed Setup B relative to it; three developer co-authors may have introduced bias into accounts.Those co-authors did not participate in interview design or analysis.Source: §VII · p. 10
  • AMEX applications were selected for dynamic settings resembling Volvo’s industrial context, so results may differ for other applications.Source: §VII · p. 10

VIII. CONCLUSION

  • For workloads like Volvo’s, general-purpose harnesses reduced orchestration work and nightly IVI monthly costs by 3–66%, while the custom harness remained competitive and more consistent on stepwise tests.The conclusion calls for keeping domain tools and test knowledge separate from orchestration and ensuring testers can inspect actions and scenario adherence.Source: §VIII · p. 10
Loading 2610.33636v1…