Source-linked AI summary
ARIA - An Agentic Framework for Autonomous Testing of Infotainment Systems
António Azevedo, Bruno Lima, João Pascoal Faria
TL;DR
Automotive infotainment validation remains costly and difficult to scale with manual or implementation-coupled testing. ARIA addresses this gap with a visual, four-agent LLM pipeline for autonomous end-to-end execution on Android infotainment systems. In an industrial evaluation, it completed 93.3% of scenarios, achieved 71.4% accuracy among completed scenarios, and detected all five known defects, while false positives and scope limitations remain important barriers to unattended deployment.
Problem
Manual testing is slow and costly for agile and over-the-air release cycles, while implementation-coupled scripts are brittle and existing LLM approaches inadequately separate infotainment testing responsibilities.
Method
ARIA uses a closed-loop pipeline of four specialized LLM agents that visually execute natural-language end-to-end scenarios on Android infotainment systems.
Results
ARIA completed 93.3% of scenarios, reached 71.4% accuracy among completed scenarios, and detected all five known defects without passing a genuine fault as working.
Takeaways & Limitations
LLM-powered multi-agent execution is feasible for industrial infotainment testing and is currently best positioned as a human-supervised aid for nightly regression campaigns.
Takeaways & Limitations
Evaluation covered one physical environment and 30 scenarios, with perfect recall based on only five genuine faults; regression detection across software updates was not evaluated.
Abstract
from arXiv · showhide
Automotive infotainment validation still relies on manual testing, slow, costly, and incompatible with agile releases and OTA updates. Scripted automation only partly helps: it couples test logic to implementation, yielding brittle, high-maintenance suites. Existing LLM-driven frameworks mostly target web/mobile apps, using single- or dual-agent setups that overload one or two models with perception, planning, action selection, and validation at once, prone to hallucinations and unproductive exploration loops given infotainment complexity. We present ARIA (Autonomous Real-time Infotainment Assessment), a multi-agent LLM framework that autonomously runs end-to-end tests on Android infotainment systems via visual interaction, using a closed-loop pipeline of four specialized agents per step plus a report stage. From single-sentence scenarios (path, action, expected outcome), ARIA runs the interactions and produces reports, reproducible scripts, and visual evidence per step. Evaluated on a manufacturer's physical Android infotainment system across 30 scenarios, ARIA completed 28 (93.3%) with a verdict (2 errored), 20 of which (71.4%) matched ground truth. It caught all 5 known defects, no fault passed as working; its 8 false positives stem from navigation/image limits and unsupported gestures, showing multi-agent LLMs can run infotainment tests industrially while exposing the cost of a low false-positive tolerance. A single-agent baseline confirms the multi-agent design's value: on the first pass, before stronger-model revisitation narrows the gap, it shows a far higher false-positive rate (72.0% vs. 52.6%), conflating navigational difficulty with system failure. We report first-pass/post-revisitation results, token/call/cost per scenario, and show via repeated runs that stability tracks complexity, with fault detection perfectly consistent, pointing to CI integration of visual testing.
1 Introduction
ARIA addresses costly, slow manual infotainment validation with a visual, multi-agent framework that executes natural-language scenarios without coupling tests to internal APIs. Its industrial evaluation completed most scenarios and detected all known defects, while revealing false positives from navigation, image interpretation, and unsupported gestures.
- Motivation: Manual infotainment testing is time-consuming, expensive, and poorly suited to agile releases and over-the-air updates.These pressures motivate adaptive testing that supports earlier quality assurance.
- Approach: ARIA uses four specialized LLM-powered agents to separate perceptual reasoning from implementation-level interaction during autonomous end-to-end testing.The framework operates through visual interface interaction on Android-based automotive infotainment systems.
- Approach: Single-sentence natural-language scenarios let organizations execute existing manual test scripts without modification or formalization.The scenarios describe navigation, action, and expected outcome information for the execution pipeline.
- Evaluation: 28 of 30 scenarios completed the pipeline, and all 5 known functional defects were detected.The reported non-correct outcomes were attributed to LLM navigation and image-interpretation limitations and unsupported gestures.
- Evaluation: The multi-agent design reduced first-pass false positives from 72.0% to 52.6% versus a single-agent baseline on the same scenarios.The comparison isolates the contribution of architectural decomposition from the underlying model.
- Evaluation: Repeated executions showed high outcome stability for moderately complex scenarios, with fault detection remaining perfectly consistent across runs.The study separates first-pass from post-revisitation results and reports token, call, and cost measures per scenario.
2 ARIA
ARIA combines model orchestration, execution control, and device access into a layered system whose closed-loop pipeline assigns distinct responsibilities to specialized agents. It also supports standard execution, replay, and revisitation, but remains constrained by model errors, latency, token cost, and unsupported interaction patterns.
- Architecture: ARIA has orchestration, execution, and device-interface layers that manage model integration, agent coordination, execution artifacts, and device communication.Agents use independent sessions and structured request–response interactions with schema-constrained outputs.
- Multi-Agent Pipeline: The closed-loop pipeline iterates through four specialized agents for each interaction step.The pipeline begins from a natural-language scenario and repeatedly updates actions from device state.
- Multi-Agent Pipeline: The Executor plans from screenshots, the Translator maps intentions to UI elements, and the Inspector verifies semantic suitability before interaction.The Executor is restricted to visual information, while the Translator uses filtered XML to resolve identifiers and coordinates.
- Multi-Agent Pipeline: After interaction, the Evaluator classifies the resulting behavior as Correct, Finished, or Incorrect, and the Summarizer stores a structured execution report.Reports include the scenario, overall result, optional failure description, and chronological steps.
- Execution Modes: Standard execution runs scenarios from the home screen, records checkpoints, and queues BUG or ERROR outcomes for revisitation.Revisitation reruns targeted scenarios with a more capable model to reduce uncertainty from execution failures.
- Execution Modes: Short-circuit replay reuses action scripts and bypasses three agents, allowing an Evaluator to compare final states efficiently across software versions.A changed final state after replay is flagged as a possible regression.
- Limitations: Model mistakes can propagate before detection, sometimes classifying correct behavior as an error and requiring revisitation.Execution is also slower than scripted automation, while screenshot transmission raises token costs and unsupported compound gestures require exclusion or simplification.
3 Evaluation
The evaluation studies ARIA as an industrial case on a physical Android infotainment environment, covering its design, fault-detection reliability, failure modes, and integration challenges. Model selection favored claude-sonnet-4.6 for its accuracy, cost, and reliability trade-off, with claude-opus-4.6 used for revisitation.
- Study Design: The industrial case study evaluates ARIA on a physical infotainment environment through four research questions covering design, reliability, failure modes, and integration.The test unit was disconnected from a vehicle, limiting vehicle-state, sensor, and active-network functionality.
- Study Design: 30 scenarios covered navigation, widgets, applications, search, settings, climate controls, connectivity, status bars, and deep menu traversal.Scenarios requiring external dependencies or ambiguous verification criteria were excluded.
- Study Design: The standard configuration used claude-sonnet-4.6, ten steps per scenario, and five translation retries, with BUG or ERROR cases revisited using claude-opus-4.6.The device returned to the home screen between scenarios.
- Model Selection: OpenAI candidates showed lower pipeline completion, with gpt-5.3-codex completing 0/6 and gpt-4.1 and gpt-5.4 completing 1/6 and 1/3, respectively.The reported failures involved structured multi-turn interaction and, for some models, error termination on the known defect.
- Model Selection: claude-sonnet-4.6 achieved 4/4 classification on completed scenarios with zero false positives at $8.28 across 207 LLM calls.claude-opus-4.6 completed 5/6 but introduced a false positive at $28.56 across 238 calls.
3.3 Results
Across 30 scenarios, ARIA produced classifications for 28, correctly identified all five genuine faults, and agreed with ground truth on 20 completed scenarios. The remaining discrepancies were eight false positives and two execution errors, attributed to LLM and framework interaction limitations.
- 28/30 scenarios completed the full pipeline, yielding a 93.3% completion rate; two terminated because of execution errors.
- Among completed scenarios, 15 were classified as OK and 13 as BUG.
- The 10 disagreements comprised 8 false positives and 2 error terminations, with causes split between LLM limitations and framework interaction limitations.Framework limitations included unsupported gestures and transient UI states.
- 5/5 genuinely faulty scenarios were correctly detected, while 20/28 completed scenarios matched ground truth.
3.4 Single-Agent Baseline
The single-agent and multi-agent pipelines reached identical post-revisitation aggregates, but differed sharply on first-pass behavior. Multi-agent separation reduced false-positive over-reporting while converting some difficult navigations into explicit errors.
- Post-revisitation: After revisitation, both pipelines completed 28/30 scenarios with 71.4% accuracy, 38.5% precision, and 5/5 fault-detection recall.
- First-pass comparison: The multi-agent pipeline classified 13/30 scenarios as BUG, produced 10 false positives, and reached 54.5% accuracy among 22 completed scenarios.
- First-pass comparison: The multi-agent pipeline produced fewer first-pass false positives but more errors, with 8 terminations versus none for the single-agent pipeline.Retry mechanisms aborted uncertain interactions instead of forcing potentially incorrect verdicts.
- Failure mechanism: The single-agent pipeline conflated navigational difficulty with system failure, whereas separated agents independently handled interaction, inspection, and evaluation.
3.5 Run-to-Run Variance
Repeated executions showed that outcome stability depended on scenario complexity. Simple scenarios were mostly deterministic, deep navigation was most variable, and fault detection remained perfectly consistent.
- Three of six repeated scenarios produced identical outcomes across all five runs, including consistent detection of the known defect.
- Two additional scenarios produced the correct modal outcome in four of five runs, with one spurious ERROR caused by reaching a translation retry limit.
- The Wi-Fi Access Point toggle, requiring deep connectivity-menu navigation, showed the highest variance across all three outcome categories.
- Across 30 executions, 25 (83.3%) agreed with the modal result and 24 (80.0%) matched ground truth; the known defect was flagged in all five repetitions.
- Scenarios requiring one to three navigation steps were deterministic, whereas those requiring more than seven steps were more unstable.The passage relates greater variance to accumulated stochastic decision points.
3.6 Computational Cost and Usage
The multi-agent pipeline incurred higher usage and monetary cost than the single-agent baseline, with revisitation dominating expenditure. Element resolution was the largest first-pass cost driver.
- The complete multi-agent run used 2,069 LLM calls and cost an estimated $139.24 across both passes.The first pass used 1,218 calls ($48.72), while revisitation used 851 calls ($90.52).
- Revisitation dominated cost because claude-opus-4.6 was billed at three times the per-call rate and covered difficult scenarios.The whole pipeline averaged approximately $4.64 per scenario.
- The Translator accounted for 480 of 1,218 first-pass calls (39.4%), making element resolution the principal cost driver.
- The single-agent baseline used 1,260 calls and cost an estimated $90.16, or roughly $3.01 per scenario.
- The multi-agent decomposition traded higher token and monetary usage for fewer first-pass false positives.
3.7 Answers to Research Questions
ARIA’s four-agent decomposition is operationally viable for autonomous visual test execution, while the single-agent baseline substantially over-reports defects on its first pass.
- RQ1: 93.3% pipeline completion demonstrates operational viability for the four-agent Executor, Translator, Inspector, and Evaluator decomposition.The architecture separates visual reasoning from XML-based element resolution.
- RQ1: 76.7% of scenarios were classified as defective by the single-agent baseline on its first pass, versus 43.3% for the multi-agent pipeline.The comparison indicates systematic over-reporting of false positives when responsibilities are concentrated in one reasoning chain.
framework reliably detect genuine system faults while minimizing false positives in an industrial infotainment validation environment?
ARIA detected every known genuine fault but produced false positives and encountered latency, nondeterminism, interaction, and scope constraints that limit unattended industrial deployment.
- Fault detection: All 5 genuinely incorrect scenarios were detected, with zero false negatives and no genuine fault passed as working.Three detected faults involved the deliberately disabled climate-control subsystem.
- Classification quality: 20 of 28 completed scenarios, or 71.4%, agreed with ground truth, while precision reached 38.5%.The 8 false positives were predominantly attributed to framework interaction and LLM limitations.
- Failure modes: 50.0% of non-correct scenarios arose from LLM limitations, including incorrect menu paths, exhausted step limits, rejected resource identifiers, and misread status-bar effects.The flexibility to reason about unfamiliar interfaces also permits plausible but incorrect action sequences.
- Failure modes: Framework interaction limitations caused the other 50.0% of non-correct outcomes through unsupported gestures, transient states, and insufficient interaction context.Examples include long-press-and-drag gestures and UI states not capturable through discrete screenshots.
- Operational trade-offs: Four sequential agent invocations per interaction step create substantially higher latency than scripted automation, positioning ARIA for nightly rather than rapid in-loop regression feedback.Per-step screenshots also translate into operational token costs.
- Validity boundaries: 83.3% of individual repeated runs agreed with the modal result, with perfect fault-detection consistency, while evaluation used one physical environment and only 5 genuinely faulty scenarios.These constraints limit reproducibility and the generalizability of the perfect recall figure.
4 Related Work
ARIA addresses gaps in LLM-driven GUI testing by targeting automotive infotainment, separating four cognitive responsibilities, and accepting existing manual scripts as natural-language input.
- Research gaps: Existing LLM-driven testing frameworks predominantly target mobile or web applications, while automotive infotainment solutions remain scarce for this environment’s constraints.This motivates a domain-specific framework for Android-based automotive infotainment systems.
- Architectural gap: GUARDIAN and LLMDroid predominantly use single-agent or dual-agent architectures that concentrate perception, planning, action selection, and validation within one or two models.ARIA extends this paradigm through four specialized agents.
- ARIA’s decomposition: ARIA assigns each cognitive sub-task to a dedicated specialist, improving controllability by preventing exploratory traversal from being influenced by information available to the behavior-classification agent.The agents’ separation supports independent responsibilities across execution and evaluation.
- Natural-language input: ARIA accepts existing manual test scripts as natural-language input, avoiding test re-authoring or formalization into structured specifications.This directly targets the execution-capacity bottleneck rather than a shortage of test cases.
- Closed-loop execution: ARIA’s closed-loop model sends evolving interface state to the Executor and execution errors back to the Translator for correction before resubmission.Evaluation occurs from an end-user perspective after each interaction step.
5 Conclusion
ARIA demonstrates feasible but qualified industrial infotainment testing: its specialized agents autonomously execute visual scenarios and detect known faults, yet precision and interaction limitations remain barriers to unattended CI deployment.
- Contribution: ARIA autonomously navigates, interacts with, and evaluates complex infotainment UIs without human intervention or coupling to internal APIs and implementation details.This capability follows from decomposing execution into four specialized cognitive subtasks.
- Evaluation: 93.3% of evaluated scenarios completed the full pipeline, all 5 known defects were detected, overall agreement among completed scenarios was 71.4%, and precision was 38.5%.The majority of non-correct outcomes were attributed to agent-native and mechanical interaction limitations.
- Evaluation: Home-screen swipe-up scenarios produced first-pass friction despite correct post-revisitation classifications, revealing unintuitive interaction patterns alongside software defects.The observation is presented as a latent usability concern.
- Future work: Richer contextual grounding, self-verification, compound gesture primitives, and cross-version replay evaluation are identified as directions for reducing errors and supporting CI integration.The replay mode was implemented but not evaluated for cross-version regression detection in this work.
- Conclusion: ARIA is currently positioned as an early-stage framework because its precision and error rate require further improvement before unattended continuous-integration deployment.It accepts natural-language scenarios and produces structured reports, reproducible scripts, and visual evidence for every step.
Data Availability
The supporting data are restricted by a non-disclosure agreement, limiting access to source code, artifacts, and testing-environment details while preserving architectural guidance for replication elsewhere.
- A non-disclosure agreement prohibits sharing the source code, internal and derived artifacts, and specific testing-equipment and environment details.
- The paper provides architectural details about the framework and infotainment-system requirements to support replication in other organizational contexts.