PaperlayerView exact revision on arXiv

Source-linked AI summary

ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents

Tianchen Guan, Xinlei Lin, Royce Cheng-Yue, Xiangjun Wang, Shuyan Zhou

arXiv:2608.18307v1cs.AIcs.CLcs.HC

TL;DR

Computer-use evaluation lacks a diagnostic middle layer between long-horizon workflows and atomic grounding tests. ComponentBench fills it with component-centered web tasks and shows that interface representation, efficiency, and visual context substantially shape agent performance, while spatial interactions remain difficult.

  • Problem

    Existing evaluations emphasize long-horizon workflows or atomic grounding, leaving short, realistic multi-step component interactions under-instrumented for failure attribution.

  • Method

    ComponentBench evaluates agents with 2,910 verified tasks spanning 97 canonical components across four observation/action spaces, using human traces and failure audits.

  • Results

    Observation/action space shifts pass rates by more than 30%, while even the fastest configuration takes 3.7× as long as matched humans and spatial components remain below 60% mean pass rate.

  • Takeaways & Limitations

    Component-level evaluation exposes failure modes largely invisible in long-horizon scores and single-step grounding benchmarks, helping localize causes of workflow failures.

  • Takeaways & Limitations

    The benchmark is web-first and measures component-level competence rather than long-horizon planning, so its prediction of workflow success remains future work.

Abstract

from arXiv · show

Current evaluation of computer-use agents is split between long-horizon workflow benchmarks and atomic GUI-grounding tests. This leaves an under-instrumented middle layer: realistic component-centered interactions (e.g., toggle a button set) that are short enough to diagnose and rich enough to capture the burdens of modern interfaces. We present ComponentBench, a benchmark and diagnostic pipeline for component-level evaluation of computer-use agents on modern web UIs. ComponentBench is organized around a library-agnostic ontology of 97 canonical UI components instantiated as 2,910 programmatically verified tasks across widely used component libraries, paired with cleaned human reference trajectories that enable evaluation of both task success and interaction efficiency. Beyond task collection, we introduce a scalable pipeline for auditing realized structural difficulty after implementation and synthesizing structured failure analyses across tasks and component families. Evaluating seven models -- GPT-5.4, Gemini 3 Flash, GPT-5.4 mini, GPT-5 mini, Gemini 3.1 Flash-Lite, Qwen3-VL-235B, and UI-TARS-1.5-7B -- across four observation and action spaces, we show that these design choices critically impact performance. Within a single shared harness, changing only the observation and action space shifts task success by more than 30% for the same model: GPT-5 mini falls from 83.1% with accessibility-tree observations to 48.9% with coordinate-only Pixel control. Moreover, even the fastest configuration takes 3.7x as long as the matched human reference, and spatial manipulations that are trivial for humans continue to challenge current agents.

1 Introduction

ComponentBench targets the under-instrumented middle layer between long-horizon workflows and atomic GUI grounding by evaluating realistic, component-centered interactions. Its benchmark and diagnostic pipeline combines broad component coverage, programmatic verification, human reference traces, and analyses of efficiency and failure sources.

  • Long-horizon benchmarks measure realistic workflows but make it difficult to attribute failures to planning, state tracking, grounding, or individual interactions.
  • ComponentBench organizes evaluation around 97 canonical UI component types, 14 interaction families, and 2,910 programmatically verified tasks across major component libraries.
  • The benchmark evaluates directness as well as completion using cleaned human reference traces, and provides a distilled 912-task hard-only suite for faster stress-testing.
  • More than 30% separates pass rates across observation/action spaces for one model, showing that evaluation interfaces critically affect measured agent performance.
  • 3.7× is the fastest configuration’s time relative to matched human references, while sliders, drag-and-drop lists, and splitters remain below 60% mean pass rate.

2 Benchmark construction

ComponentBench constructs a broad, library-agnostic benchmark from a structured component ontology, executable task specifications, human-validated reference traces, and programmatic success verification. It also standardizes cross-regime evaluation and adds diagnostic layers that audit realized difficulty and aggregate component-level failures.

  • Component inventory: The ontology combines WAI-ARIA patterns with production inventories from Ant Design, MUI, Mantine, Fluent UI, Chakra UI, and Headless UI.This produces a library-agnostic inventory of canonical component types for systematic coverage.
  • Task specification: Each task targets one primary component and specifies its implementation source, reusable action template, eight scene factors, seven difficulty axes, success trigger, and negative cases.The suite contains 24 canonical task templates, one ad-hoc variant, and 2,910 tasks.
  • Validation and references: Human operators executed each implemented task twice, enabling solvability checks, specification corrections, cleaned reference trajectories, and replay-based difficulty auditing.The recordings serve both as construction-time validity checks and later efficiency-analysis references.
  • Verification: Every task uses a JavaScript checker to evaluate a YAML-defined success predicate against live state and reveals the same deterministic #cb-success-banner termination signal.Verifiers require committed end states, including local Apply, Save, OK, or Confirm controls when applicable.
  • Evaluation regimes: The benchmark supports AX-tree, Set-of-Marks, Pixel, and Browser-Use regimes, while treating Browser-Use as a tool-rich reference rather than a strictly like-for-like comparison.Browser-Use permits up to 4 primitive actions per model turn and up to 80 actions within the 20-step budget.
  • Diagnostics: A three-stage diagnostic pipeline joins specifications, metadata, realized-difficulty outputs, traces, and agent evidence, then produces task-level observations and component-level reports.ComponentBench-Core further concentrates evidence into 19 generation units, 45 canonical components, and 912 hard-only tasks.

3 Experiments

Experiments on ComponentBench-Full v1 show that both model capability and observation/action space strongly affect component-level success. Across seven models and up to four regimes, pass rates vary substantially, with even the fastest configuration remaining slower than humans.

  • Experimental setup: All tasks use a 20-step budget, enabling comparisons that isolate model effects from observation/action-space effects.ComponentBench-Full v1 provides the broad coverage suite used for the main comparisons.
  • Experimental setup: Seven models are evaluated across up to four regimes: AX-tree, SoM, Pixel, and the separate tool-rich Browser-Use reference space.The five Gemini and GPT models plus Qwen3-VL-235B use all four spaces, while UI-TARS-1.5-7B uses native pixel-only control.
  • Pass-rate results: 34.2%: GPT-5 mini drops from 83.1% in AX-tree to 48.9% in Pixel within the same harness.Including the tool-rich Browser-Use regime, the corresponding range is 87.0% to 48.9%.
  • Pass-rate results: 29.6%: GPT-5 mini gains from SoM over Pixel, while GPT-5.4 mini and GPT-5.4 instead favor Pixel over SoM by 2.4% and 6.8%.Gemini 3.1 Flash-Lite also gains 10.2% from SoM over Pixel; GPT-5.4 reaches 83.8% in Pixel versus 81.5% in AX-tree.
  • Pass-rate results: 85.4%: Gemini 3 Flash succeeds in Pixel, while UI-TARS-1.5-7B reaches 12.6% in native pixel-only mode.UI-TARS-1.5-7B has 87% of tasks hit the 20-step ceiling, showing that the benchmark remains far from saturated.

4 Analysis

ComponentBench reveals that agent performance depends strongly on interaction family, observation/action space, scene conditions, and efficiency relative to humans. Its diagnostic analyses identify recurring mechanistic failures and show that the hard-only CORE suite remains challenging even for frontier models.

  • Efficiency and mode trade-offs: AX-tree and SoM achieve the best step efficiency at a 1.0× median ratio, while Browser-Use trades lower efficiency for higher aggregate pass rates.Browser-Use can bundle up to four primitive actions per model turn, making its turn-level human reference optimistic rather than directly action-comparable.
  • Family-level performance: Browser-Use’s advantage is family-dependent, reaching +29.0% on Advanced Editors but −24.0% on Drag/Drop, where Pixel performs better.Across the six models, Browser-Use has an aggregate pass-rate advantage of +7.8% to +18.2% over the mean of other modes.
  • Family-level performance: Family pass rates range from 91.6% for Command & Navigation to 47.7% for Drag/Drop, with no observation/action space universally best.Agents often locate the right data-grid row or rich-text toolbar but fail at editor-mode entry, committing, maintaining selections, or controlling transient overlays.
  • Human-agent difficulty inversion: Nine canonical types require ≤2 human steps yet remain below 60% agent pass rate, including resizable columns at 24.4%, window splitter at 38.3%, and slider range at 39.9%.These spatial manipulations are trivial mouse gestures for humans but among the hardest interactions for agents.
  • Scene factors and difficulty axes: Medium clutter reduces Pixel performance by 14.3% but AX-tree performance by only 0.2%, while Browser-Use incurs a 0.8% cost because it acts on DOM elements.Precision requirement correlates with failure rate at r = +0.44, and difficulty declines from L0 = 87.3% to L3 = 65.2%.
  • CORE diagnostic suite: CORE lowers pass rates by 10–39% from FULL and remains difficult for Opus 4.6, which reaches 65.4% on CORE Pixel but solves only 34.1% within the human step budget.Opus 4.6 also records a 9.5× time-to-human ratio, with 67.4 versus 7.1 mean seconds per successful task.

5 Limitations and scope

ComponentBench is web-first and component-focused rather than a comprehensive test of bespoke widgets, long-horizon planning, or native computer-use environments. Its conclusions are further bounded by hybrid difficulty estimation, single-annotator reference traces, limited model and run coverage, generator-specific regularities, and non-native evaluation interfaces.

  • Benchmark scope: ComponentBench covers recurring WAI-ARIA and production-library component families, excludes CAPTCHA-like tasks, and does not measure long-horizon planning.The ontology is designed to port beyond its Next.js-on-Chromium web setting, while predicting end-to-end workflow success remains future work.
  • Measurement and references: The difficulty audit combines rendered-DOM measurements with type-level approximations, while primary human references come from two passes by a single annotator.A validation study found 1.05× per-task inter-annotator variation versus a 1.27–3.02× agent–human gap.
  • Experimental coverage: Seven models are evaluated on the full suite, with Opus 4.6 added on Core, while main tables report one run per model–mode combination.On a 278-task repeated-run subset, run-to-run deviation was at most 1.4% with no ordering changes, though borderline task outcomes churned.
  • Data-generation risks: GPT-5.2 Pro task generation and Claude Code page implementation may introduce generator-specific regularities.Deterministic verification, combinatorial diversity prompts, and human recording passes were used to mitigate this risk.
  • Interface caveats: Using chat-completions or custom harnesses instead of native computer-use environments, plus BrowserGym’s invisible cursor, may underestimate native-interface performance.GPT models used chat completions rather than Operator or Computer Use API, and Opus 4.6 used a custom harness rather than Anthropic’s full environment.

6 Conclusion … A.4 Programmatic verification details

ComponentBench combines a broad component-level benchmark with structured construction and deterministic verification procedures. Its design uses controlled task diversity, human reference trajectories, and trace-grounded analysis to diagnose how representation, efficiency, and visual context shape agent failures.

  • 6 Conclusion: ComponentBench covers 97 canonical component types, 2,910 tasks, and four observation regimes to localize component-level causes of computer-use workflow failures.The benchmark links representation, efficiency, and visual context to concrete failure mechanisms through trace-grounded analysis.
  • A.1 Task specification schema: Each YAML task records its component type, implementation source, template, browser goal, eight-factor scene context, difficulty ratings, success criteria, negative cases, and expected interaction path.Success is represented both as human-readable criteria and a canonical predicate.
  • A.2 LLM-assisted construction pipeline: GPT-5.2 Pro generated 30 specifications per canonical type, while Claude Code implemented them as interactive Next.js pages and humans recorded reference trajectories.Human execution also provided the final quality-control pass, removing broken, miswired, or specification-mismatched tasks.
  • A.3 Task generation prompt: For each component type, GPT-5.2 Pro used structured prompts and research CSV mappings covering library implementations, interaction patterns, and difficulty considerations.The prompt required a single primary component and realistic microuser intents while hiding checker rules and implementation details from agents.
  • A.3 Task generation prompt: Each component type receives exactly 30 tasks distributed across 10 easy, 10 medium, and 10 hard cases.When supported by all three primary libraries, tasks are split 10/10/10 across Ant Design, MUI, and Mantine with balanced per-library difficulty.
  • A.3 Task generation prompt: Coverage constraints require scene-factor variation, including at least 3 dark-theme tasks, 3 compact-spacing tasks, 4 cluttered tasks, 4 multiple-instance tasks when meaningful, and 7 distinct templates.These constraints complement the fixed difficulty and library-balancing requirements.
  • A.3 Task generation prompt: Difficulty follows a default-first principle: realistic scene factors, feature variants, and within-component depth create hardness, while invisible elements and overlapping click traps are prohibited.Examples include compact spacing, clutter, nested overlays, multi-step navigation, search, and restricted input.
  • A.4 Programmatic verification details: Programmatic verification deterministically checks the committed end state against each task predicate and signals success through the #cb-success-banner DOM element.For Apply, Save, or Confirm tasks, only the post-commit state counts; enumerated negative cases exclude nearby incorrect states.

A.5 Task diversity summary … E Representation trade-off tables

ComponentBench combines balanced task diversity with replay-based structural auditing and reports efficiency and representation trade-offs across multiple evaluation tables. Its 2,910 Full tasks cover all canonical types, templates, and scene-factor levels, while difficulty is measured from rendered interactions and summarized alongside step, time, and representation efficiency.

  • A.5 Task diversity summary: 2,910 Full tasks cover all 97 canonical types with exactly 30 tasks each, all 24 templates, and every level of every scene factor.The generation pipeline enforces combinatorial coverage while allowing deliberately skewed marginals.
  • A.5 Task diversity summary: ∼40% of tasks include at least one added burden, such as clutter, compact spacing, dark theme, or multiple instances.
  • B.1 Replay-based measurement: 2,910 Full and 912 Core tasks are audited by replaying cleaned human trajectories in Playwright before and after each action, including states revealed through interaction.Measurement logic runs inside the browser page and captures popovers, nested panels, and internal scroll regions.
  • B.2 Feature list: 24 canonical features measure target geometry, acquisition difficulty, density, choices, overlays, scrolling, feedback, visibility, accessibility, clutter, placement, and state properties.Features 1–16 come from the rendered DOM and layout, while features 17–24 come from type-level metadata.
  • B.3 Axis aggregation: Seven realized difficulty axes are weighted averages of normalized features, using unit weights and frozen versioned cutpoints to produce 1–5 ratings.
  • C Full step efficiency table: Table 7 reports Full-task step efficiency as pass rate within 20 steps and within 1/2/3/5× the human step count.Browser-Use steps are model turns of up to four primitive actions, making its efficiency columns optimistic; the native pixel-only interface is marked separately.
  • D Full time efficiency table: Table 8 compares mean agent and human wall-clock duration on each model–mode configuration’s successful tasks and reports the agent-to-human time ratio.The human mean varies by row because both columns use the same solved-task set for that configuration.
  • E Representation trade-off tables: +29.6% to −6.8% is the model-dependent SoM advantage over Pixel, while Browser-Use’s advantage over mean non-Browser-Use pass rate is largest for weaker models.These comparisons are summarized in the representation trade-off tables.

F Additional analysis figures … J.1 Method

The additional analyses show that component difficulty depends strongly on interaction mode, intended difficulty axes, and component family, while the benchmark’s ontology and failure-labeling pipeline support systematic diagnosis. Spatial control, precision, clutter, and spacing disproportionately expose agent weaknesses, including cases where humans find components trivial.

  • F Additional analysis figures: Mode ordering reverses by component family: Pixel favors Drag/Drop, whereas Browser-Use strongly favors Advanced Editors.The heatmap identifies family-level inversions in pass rates across observation modes.
  • F Additional analysis figures: Clutter widens the AX-tree–Pixel gap from 10% to 24%, while spacing widens it from 12% to 21%.These scene factors disproportionately hurt Pixel mode.
  • F Additional analysis figures: +29% to −24% is the range of Browser-Use’s family-level advantage, from Advanced Editors to Drag/Drop.The result confirms that DOM-level tool access is not uniformly beneficial.
  • G Difficulty axis validation: Precision requirement is the most predictive intended difficulty axis overall, and all axes predict Pixel failure more strongly than Browser-Use failure.The validation correlates intended difficulty axes with agent failure rate.
  • H Ontology: 97 canonical component types: The ontology organizes 97 canonical component types across 14 families, including continuous input, drag/drop workspace interactions, and advanced editors.The listed families span navigation, disclosure, text entry, choice, selection, overlays, structured data, files, and editing interactions.
  • I Hardest canonical component types: Components with ≤2 human steps but <60% agent pass rate define the benchmark’s human-agent difficulty inversion.The hardest canonical types are ranked by mean agent pass rate, with human steps reported using normalized reference trajectories.
  • J.1 Method: 8,864 failed Pixel, SoM, and AX-tree traces are deterministically parsed into action sequences and feature flags for failure labeling.Extracted features include drag emission, target interaction, repeated coordinates, and typed values; labels begin from component-driven category priors.

J.2 Adversarially reviewed case studies

The benchmark’s adversarial review of 20 representative failed traces corrected five initial diagnoses and clarified distinct mechanisms behind failures in calibration, target acquisition, and target-state perception. The released case studies include trace pointers for all 20 cases.

  • Review process: 20 representative failed traces were independently adversarially reviewed, with 15 labels confirmed and 5 relabelled using the reviewer-adjusted categories.Each case received one review pass over the action log, screenshots, and Layer-2 diagnosis.
  • Case studies: ∼28% was the final meter value after the agent dragged from 15% to 28% but timed out targeting 42%, indicating value calibration rather than grounding failure.The bar covered only ∼100px for 0–100%, so coarse drags overshot or undershot and progressively smaller nudges did not reach the target.
  • Case studies: An off-by-one-row grab moved Admin instead of Editor, and although Editor was later moved, the unrepaired Admin caused exact-set verification to fail.The drag primitive itself worked; the initial coordinate targeted the Admin row rather than the intended Editor row.
  • Review implications: Five reviewer adjustments overturned plausible categories through finer trace reading, including reclassifying a spinner ratcheting in the wrong direction as a no-progress loop rather than calibration.All 20 case studies, including trace pointers, were released with the benchmark.

K Human reference validation with additional annotators

Validation with two additional annotators shows that human reference efficiency is stable enough for comparison, while remaining a practical successful reference rather than a claim of optimality. All 278 stratified validation tasks were completed successfully and none were flagged for quality issues.

  • Validation design: 278 tasks formed a fixed-seed, stratified validation subset spanning one realized canonical type–library pair per task.The subset was selected with seed 2026 to minimize deviation from the global difficulty distribution before inspection.
  • Annotation protocol: All three annotators completed all 278 tasks successfully using two recorded passes per task, retaining the shorter successful pass after identical cleaning.Agreement statistics covered action tasks, while hover-only tasks contributed zero normalized actions.
  • Efficiency validation: 1.05× was the human-to-human action-count ratio, compared with agents’ 1.27–3.02× human action count on tasks they solved.The agent–human efficiency gap therefore exceeds inter-annotator variation by several times.
  • Task-quality check: 0 tasks were flagged as broken, ambiguous, or miswired during recording, providing evidence against pervasive defects in the stratified subset.Annotators independently had the opportunity to flag task-quality problems during their sessions.

L Repeated-run stability · M Experimental setup details

Repeated-run checks found highly stable aggregate pass rates but nontrivial task-level outcome churn, especially for GPT-5 mini in Pixel mode. The experiments also documented model access, coordinate handling, and framework configuration.

  • L Repeated-run stability: The repeated-run study covered four models from two providers: Gemini 3 Flash, Gemini 3.1 Flash-Lite, GPT-5 mini, and GPT-5.4 mini.It used purevisual Pixel and tool-rich Browser-Use regimes to stress different variance sources.
  • L Repeated-run stability: Table 15 defines Max dev as run-to-run pass-rate deviation, Agree% as exact task-level pass/fail agreement, and Jaccard as passed-task-set overlap.It also reports bootstrap 95% intervals on mean pass rate.
  • L Repeated-run stability: 1.4% maximum run-to-run pass-rate deviation and 0.9% median deviation indicate stable aggregate results across repeated evaluations.Each selected model–mode cell was run twice on the same 278-task validation subset with the same harness commit and model endpoint.
  • L Repeated-run stability: 17.3% of GPT-5 mini Pixel tasks flipped outcome despite an identical 52.5% pass rate across runs.This demonstrates that aggregate pass-rate stability can conceal task-level churn.
  • L Repeated-run stability: Headline orderings were identical across both runs: Gemini 3 Flash was strongest, Browser-Use exceeded Pixel for every model, and GPT-5 mini Pixel was weakest.The comparison was made on the repeated 278-task validation subset.
  • M Experimental setup details: The experimental setup details model access, coordinate handling, and framework configuration used in the experiments.These setup dimensions are documented as part of the experimental configuration.

M.1 BrowserGym-based evaluation (AX-tree, SoM, Pixel) … M.4 Browser-Use mode

The evaluation uses distinct agent interfaces and harnesses across BrowserGym, native pixel, and Browser-Use modes, while controlling key task and viewport constraints. BrowserGym keeps pages, verifiers, and termination logic fixed across AX-tree, SoM, and Pixel conditions, whereas model-specific coordinate handling and serving configurations vary.

  • M.1 BrowserGym-based evaluation (AX-tree, SoM, Pixel): BrowserGym evaluates AX-tree, SoM, and Pixel modes under identical pages, verifiers, termination logic, 1280×720 viewports, 20-step limits, and 600-second task budgets.BrowserGym additionally enforces a 300-second per-step timeout.
  • M.1 BrowserGym-based evaluation (AX-tree, SoM, Pixel): Pixel-mode models use either normalized 0–1000 coordinates mapped to pixels or raw pixel coordinates, depending on the model family.Gemini, Qwen3-VL, and GPT models use different coordinate conventions and serving paths.
  • M.1 BrowserGym-based evaluation (AX-tree, SoM, Pixel): AX-tree mode appends accessibility-tree text to screenshots, while SoM mode overlays numbered bounding boxes on screenshots.All models receive screenshots as base64-encoded images.
  • M.2 UI-TARS-1.5-7B (native pixel mode): UI-TARS-1.5-7B uses a native harness with smart-resized screenshots and predicts coordinates in resized-image space before mapping them to 1280×720 pixels.Its serving setup uses vLLM with bfloat16, a single GPU, and a 32K context window.
  • M.3 Opus 4.6 (Core Pixel only): Opus 4.6 is evaluated only on 912 ComponentBench-Core tasks in Pixel mode, using anisotropic resizing from 1280×720 to 1024×768 and inverse coordinate scaling.Claude receives the resized screenshots through Anthropic’s computer-use interface.
  • M.4 Browser-Use mode: Browser-Use combines screenshots, serialized DOM information, and grounded element references, with up to four actions per step and 20 steps per task.Runs use a 1280×720 headless Chromium viewport, 120-second step timeouts, and exclude initial page load from step-count analysis.
  • M.4 Browser-Use mode: Browser-Use serving differs by model: Qwen3-VL-235B runs through Amazon Bedrock, whose undocumented precision may differ from its locally served FP8 configuration.Gemini uses Google AI Studio, while GPT uses the OpenAI API or Duke LiteLLM proxy.
Loading 2608.18307v1…