PaperlayerView exact revision on arXiv

Source-linked AI summary

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel, Michal Zakrzewski, Sebastian Montagna, Damian Rynczak, Shreyansh Padarha, Kumail Alhamoud, Zihao Fu, William Lugoloobi, Kai Rawal, Hanna Yershova, Xander Davies, Taras Rumezhak, Guohao Li, Fazl Barez, Baoyuan Wu, Arkadiusz Drohomirecki, Yarin Gal, Chris Russell, Christopher Summerfield, Adam Mahdi, Volodymyr Karpiv, Philip Torr, Adel Bibi

arXiv:2606.14397v2cs.LG

TL;DR

Existing benchmarks often use simple, familiar tasks and narrow capabilities, limiting evaluation of agent generalisation in complex settings. GauntletBench tests temporal perception, graphical understanding, and 3D reasoning across professional applications, finding that the best agent achieves only 19.1% success.

  • Problem

    Existing benchmarks use familiar applications, simple tasks, and narrow capabilities, limiting their assessment of agent performance in complex real-world scenarios.

  • Method

    GauntletBench evaluates agent generalisation through visually grounded tasks spanning three underexplored capabilities and five less-covered professional applications.

  • Results

    19.1% success was achieved by the best agent, while human annotators exceeded 80% on the benchmark.

  • Takeaways & Limitations

    GauntletBench reveals that current agents remain far from reliable human-level performance on challenging, vision-intensive tasks in complex real-world scenarios.

  • Takeaways & Limitations

    GauntletBench covers only five specialised domains and a subset of interaction paradigms, which may not capture the diversity of real-world scenarios.

Abstract

from arXiv · show

As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchmarks are typically built on popular applications with relatively simple tasks and focus on a narrow set of capabilities while overlooking broader dimensions, resulting in saturated performance on modern agents and failing to probe their limitations. To this end, we introduce GauntletBench, a web-based benchmark for evaluating agent generalisation in challenging scenarios, focusing on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning), across five less-covered professional applications (Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer), each with 20 vision-intensive tasks (100 in total). Our benchmark provides a modular pipeline that comprises an environment compatible with both open- and closed-source agent frameworks, a controlled web-based application, a well-structured task suite, and an automated evaluation engine with diverse metrics. Contrary to widespread expectations, our empirical results reveal that frontier agentic systems remain far from achieving human-level performance. Even the state-of-the-art agent achieves only a 19.1% success rate on our GauntletBench, highlighting the limitations in these overlooked capabilities and generalisation. By comparison, non-expert human annotators achieve over 80% success on our challenging yet feasible tasks, revealing the substantial gap between current agent capabilities and those required for complex real-world scenarios.

1 Introduction

GauntletBench evaluates agent generalisation in challenging, visually grounded scenarios that expose limitations of existing, saturated benchmarks. It targets temporal perception, graphical understanding, and 3D reasoning across five professional applications and 100 tasks, where SOTA agents achieve only 19.1% success versus over 80% for non-expert human annotators.

  • Benchmark design: The benchmark contains five controlled web-based applications and 100 vision-intensive tasks, supported by an automated evaluation engine.Its modular platform comprises environments, applications, tasks, and evaluation, supporting open-source, API-based, and closed-source agent frameworks.
  • Motivation: Existing benchmarks often use homogeneous settings, simplistic tasks, and popular applications or near-identical replicas, producing saturated performance that fails to reflect real-world complexity.These limitations motivate evaluations spanning less-covered applications and more challenging scenarios.
  • Benchmark contribution: GauntletBench targets three underexplored capabilities: temporal perception, graphical understanding, and 3D reasoning.The benchmark uses visually grounded tasks to evaluate agent generalisation in challenging scenarios.
  • Empirical findings: Most open-source MLLM agents achieve near-zero performance, while no API-based MLLM agent exceeds a 13.2% success rate on GauntletBench.The passage identifies Qwen-3-VL and Llama-4-Maverick as examples of open-source agents.
  • Empirical findings: Human annotators without specific application expertise achieve over 80% success while requiring 30% fewer steps than the frontier agentic system.This gap shows that current systems have critical limitations in the benchmark’s overlooked capabilities and generalisation.
  • Empirical findings: 19.1% success is achieved by SOTA agents on GauntletBench, indicating they remain far from human-level performance in complex real-world scenarios.Non-expert human annotators exceed 80% success, revealing a substantial capability gap.

2 GauntletBench

GauntletBench is a modular web-based benchmark for testing agent generalisation in complex real-world scenarios through unfamiliar professional applications and vision-intensive tasks. It combines visually grounded interaction, structured task design, and automated application-specific evaluation for open- and closed-source agents.

  • 2 GauntletBench: GauntletBench integrates a unified web environment, controlled applications, structured tasks, and an automated evaluation engine into one modular pipeline.The interface supports both open-source/API-based MLLM agents and closed-source frameworks.
  • 2.1 Unified Environment: Agents interact through high-level browser commands and visually grounded observations, with canvas-only interfaces in the 3D Modeller, Flight Analyser, and Circuit Designer.Low-level browser APIs are withheld, requiring agents to rely on screenshots; actions include click, type, select, scroll, and navigation.
  • 2.2 Applications: The benchmark covers five less-familiar professional applications—Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Design—to evaluate agent generalisation.These applications support basic UI understanding, tool use, and long-term reasoning alongside temporal perception, graphical understanding, and 3D reasoning.
  • 2.3 Task Suite: The benchmark provides 100 human-feasible tasks, with 20 tasks per application spanning easy, medium, and hard difficulty levels.Each application contains 2 easy, 9 medium, and 9 hard tasks, while users can also define tasks and upload ground truth.
  • 2.4 Evaluation Engine: Application-specific evaluators automatically compare agent outputs with ground truth and support diverse metrics for both open- and closed-source frameworks.The evaluation engine runs after task completion and uses tailored evaluators rather than requiring access to closed-source invocation processes.

3 Experiment

GauntletBench evaluates 14 open-source, API-based, and closed-source agents across challenging vision-intensive tasks, revealing a substantial gap from human-level performance. Results also show that failures often reflect unreliable long-horizon execution, limited relational understanding, and recurring interaction errors.

  • Frontier Agentic Systems Remain Far from Human-Level Performance: 19.1%: Claude-Opus-4.6 Computer Use achieves the highest average success rate, ahead of Gemini-3.1-Pro at 13.7% and Claude-Opus-4.6 at 13.2%, while humans reach 75–85%.All open-source MLLM agents remain below 1.7% on every application, and most evaluated systems achieve 0.0% across applications.
  • How Far Current Agents Remain from Reliable Performance?: Over 50%: Claude-Opus-4.6 Computer Use’s average progress rate shows agents often make substantial intermediate progress but struggle to complete long-horizon tasks reliably.The result suggests failures arise from accumulating task complexity rather than an inability to make progress.
  • What Capabilities Do Current Agents Lack Most?: 11.7%: Flight Analyser and Circuit Designer are the most difficult applications, indicating basic graphical recognition without sufficient understanding of relationships, interactions, and resulting behaviours.The benchmark evaluates temporal perception, graphical understanding, and 3D reasoning across five professional applications.
  • Does the Vision Modality Help Agents Complete Tasks?: Visual inputs consistently improve performance over relying only on structured information for the vision-intensive Video Editor and Workflow Builder tasks.Text-only agents complete only a very small subset when minimal structured information is exposed.
  • The Impact of Model Scale: Increasing model scale consistently improves success and progress rates, while the largest model in each family uses the fewest tokens despite higher per-step inference cost.Scale comparisons use all 100 tasks for performance and the 45 medium-difficulty tasks for efficiency.
  • The Role of Reasoning: Extended reasoning generally improves sufficiently capable models but increases token and interaction costs; Qwen-3-VL shows no performance improvement from it.Efficiency and reasoning comparisons are reported on the 45 medium-difficulty tasks.

4 Related Work

Prior computer-use benchmarks span web, mobile, desktop, and enterprise workflows, but increasingly emphasize consumer-style interaction while underrepresenting professional software’s perceptual and spatial demands. GauntletBench addresses this gap with visually dense specialist interfaces and richer evaluation combining structured checks, partial-credit judging, and efficiency metrics.

  • Prior Benchmarks: Existing benchmarks cover web, mobile, desktop, and enterprise workflows [18], but largely emphasize consumer-style information seeking, navigation, and form-based workflows.This evaluation landscape risks underrepresenting the demands of professional software.
  • Web Agent Benchmarks: Earlier web benchmarks such as MiniWoB and MiniWoB++ [18] established simplified interaction tasks, but relied on synthetic, toy-like interfaces that did not capture real-website complexity.These tasks included basic operations such as clicking buttons, filling forms, and navigating menus.
  • Agentic Evaluation: LLM judges can assign graded progress scores from trajectories and final visible states, but may inconsistently hallucinate progress, requiring careful prompt design and calibration.GauntletBench therefore combines model-based partial-credit judging with structured checks and efficiency metrics.
  • Saturation of Computer-Use Agent Benchmarks: Modern agents score close to perfect on most MiniWoB++ [16] tasks, while the gap between top agents and humans on WebArena is narrowing, threatening benchmark differentiation.This saturation motivates evaluation settings that remain challenging for strong systems.
  • GauntletBench: GauntletBench shifts evaluation toward visually dense, dynamic, specialist interfaces spanning graphical editing, circuit design, flight tracking, and 3D manipulation.It retains the reproducibility advantages of browser-based environments while changing the task distribution.

5 Conclusion and Discussion

GauntletBench evaluates agent generalisation through challenging, vision-intensive tasks targeting overlooked capabilities in professional applications. Its scope is limited, but the benchmark is intended to support more rigorous evaluation and future development of robust, reliable, and trustworthy agents.

  • Conclusion: GauntletBench targets temporal perception, graphical understanding, and 3D reasoning through diverse professional applications and carefully designed vision-intensive tasks.
  • Limitations: The benchmark covers only five specialised domains and a subset of interaction paradigms, which may not capture real-world diversity and could introduce evaluation bias.
  • Limitations: GauntletBench constrains visually grounded interaction to ensure faithful evaluation, potentially limiting agents’ full potential when combined with other capabilities.
  • Future Works: Future extensions will add more professional applications, richer multimodal interactions, longer-horizon tasks, and evaluation dimensions including robustness, safety, and failure recovery.
  • Broad Impacts: GauntletBench may promote rigorous evaluation practices and research on safer, more dependable autonomous agents for real-world applications.

A Human Evaluation

The human evaluation uses a web-based harness that mirrors the agent interaction surface, enabling directly comparable trajectories and scoring across 300 tasks completed by 15 experienced participants. Participants receive identical prompts, face a 40-minute limit per task, and receive no human-specific scoring adjustments.

  • Harness Overview: The harness mirrors the agent interaction surface, adding only instrumentation for task delivery, screenshot capture, and answer collection.Human and agent trajectories are therefore structurally comparable within the same web applications.
  • Recruitment: 15 participants—three per application—completed 300 tasks, with each participant having a CS background and relevant hands-on tool experience.The harness is designed to scale to multiple annotators per application.
  • Scoring: Human outputs were scored through the same pipeline as agent outputs, with no human-specific adjustments.Once Finish was clicked, the answer became final and the session advanced to the next task.

B Additional Ablation Studies on Efficient · C Progress Rate Criteria

The additional ablations quantify efficiency across easy, medium, and hard tasks, reporting consumed tokens and steps for model-scale and reasoning comparisons. These measurements also supply the medium-difficulty columns in the main paper’s Tables 3 and 4.

  • B Additional Ablation Studies on Efficient: These per-difficulty efficiency measurements provide the source for the medium-difficulty columns in Tables 3 and 4 of the main paper.The source data covers both model-scale and reasoning ablations.
  • B Additional Ablation Studies on Efficient: The model-size ablation compares GPT-5.4 Nano, Mini, and full models with Claude-4.6 Haiku, Sonnet, and Opus.Consumed-token results are reported as mean ± standard deviation over three independent runs.
  • B Additional Ablation Studies on Efficient: Consumed steps are separately reported for the same GPT-5.4 and Claude-4.6 model-size comparisons across each difficulty bucket and overall.The table uses mean ± standard deviation over three independent runs.
  • B Additional Ablation Studies on Efficient: The reasoning ablation compares GPT-5.4 with and without extended reasoning, Gemini-3.1-Pro at low versus high reasoning, and Claude-Opus-4.6 with and without thinking.Consumed-token results are broken down by difficulty and pooled across all 100 tasks.
  • B Additional Ablation Studies on Efficient: Consumed steps are also measured for all three reasoning pairs across easy, medium, hard, and pooled tasks.The comparison uses the same model configurations as the reasoning-token analysis.

C.1 LLM-as-a-Judge

The appendix distinguishes two prompted LLM calls as Stage 1 and Stage 2, implemented with gpt-4o-mini and gpt-5.1, respectively.

  • C.1 LLM-as-a-Judge: The appendix names the two prompted LLM calls Stage 1 and Stage 2.This naming convention is used for clarity throughout the appendix.
  • C.1 LLM-as-a-Judge: Stage 1 uses gpt-4o-mini.
  • C.1 LLM-as-a-Judge: Stage 2 uses gpt-5.1.

C.1.1 Details · C.2 LLM-as-a-Judge Prompts

The judge pipeline filters screenshot trajectories, parses retained transitions conservatively, and performs a final evidence-based outcome assessment. Its prompts prioritize visible screenshot evidence, structured JSON outputs, cumulative state tracking, and penalties for missing or redundant final-state details.

  • C.1.1 Details: The pipeline first removes screenshot pairs without meaningful visual change using image-difference metrics before forwarding retained pairs to the judge.Screenshots are resized to 320×200, converted to grayscale, and compared using RMSE, perceptual-hash distance, and changed-pixel fraction.
  • C.1.1 Details: Stage 1 compares each retained screenshot pair with the task context, intervening action, screenshots, and compact prior visible-state summary, using gpt-4o-mini for local transition parsing.The stage identifies visible changes and updates a concise running state summary.
  • C.1.1 Details: Stage 2 compresses parsed outputs into trajectory events and combines them with the final screenshot, task information, agent answer, and available objective or reference signals, using gpt-5.1 for integrated outcome judgment.This stage assigns the final assessment after integrating evidence across the trajectory.
  • C.2.1 Stage 1 System Prompt: The Stage 1 prompt requires conservative, evidence-based comparison of consecutive screenshots, treating action text and prior summaries as context rather than proof.It instructs the judge to avoid hidden-state assumptions, trust screenshots over contradictory memory, and prefer unknown or no clear change when evidence is ambiguous.
  • C.2.2 Stage 1 Task Prompt: Stage 1 outputs structured fields for visible action, change type, task relevance, progress, confidence, uncertainty, and an updated cumulative visible-state summary.The schema preserves the prior summary when no reliable update is possible and supports explicit no_clear_change judgments.
  • C.2.3 Stage 2 System Prompt: The final-outcome prompt scores visible completion from 1 to 5 using screenshot-based events, the final screenshot, the task requirements, and the final answer, while treating objective evaluation as supporting context.It forbids assigning score 5 when the supplied objective result is 0 and requires conservative scoring when evidence is ambiguous.
  • C.2.3 Stage 2 System Prompt: The final rubric penalizes missing, wrong, duplicated, redundant, or extra visible elements, requiring precise details rather than approximate intent.A score of 5 denotes clearly visible full success without meaningful mistakes or redundant artifacts, whereas lower scores represent near success, partial progress, limited progress, or no meaningful progress.
  • C.2.4 Stage 2 Task Prompt: The Stage 2 task prompt instructs the judge to return a strict JSON score and brief reason grounded primarily in the final screenshot, while conservatively incorporating supported trajectory progress and naming key satisfied and missing details.The final agent answer and objective result are explicitly supporting evidence rather than proof of success.

D Error Analysis

Manual inspection of failed trajectories revealed four recurring failure categories across agents: stopping incorrectly, grounding actions poorly, violating execution-loop instructions, and open-source models producing broken or prematurely abandoned trajectories.

  • Agents Do Not Know When To Stop: Agents often continue acting after correctly completing a task, sometimes corrupting their own result instead of stopping.A 3D Editor example shows an agent setting an object’s requested position, then clicking around to “check” the change.
  • Grounding Failures: Grounding failures cause agents to miss intended elements, select visually similar controls, or enter correct values into the wrong fields.Agents typically do not recognize these mapping errors themselves.
  • Instruction-following Issues: Agents frequently violate the execution loop by emitting multiple planned actions at once instead of acting once and awaiting the next observation.Llama-4-Maverick often outputs its entire plan as text and ends without executing browser actions.
  • Open-source Models Failure: Open-source models show near-zero FSR because their trajectories are often syntactically broken, repetitive, or abandoned after only a few steps.The authors hypothesize that these models lack training on agentic, multi-step browser-control data and are instead tuned for static, single-turn instruction following and tool use.

E Tailored Evaluation · F Task Format

GauntletBench uses strict, application-specific evaluators and a unified task format to support fair comparisons across five web applications. Tasks standardize prompts, interaction guidance, goals, steps, and structured outputs while preserving domain-specific interface and correctness rules.

  • E Tailored Evaluation: Each test case passes only when all checked properties match, with no partial credit, using application-specific matching rules across the five environments.These rules include set/property matching, timestamp and colour tolerances, UUID remapping, symmetry-aware 3D matching, and truth-table equivalence.
  • E Tailored Evaluation: Video Editor evaluation globally matches media blocks by content and checks timing, duration, and trim offsets within ±1 s.Text layered over video must also have correct track placement, whereas standalone text is treated differently.
  • F.1 Prompt Template: The unified prompt template standardizes application context, capabilities, UI interaction paradigms, high-level goals, ordered steps, and JSON-formatted outputs.It also instructs agents to follow the task exactly and avoid unrelated changes.
  • F.2.1 Video Editor: Video Editor tasks expose media management, timeline editing, text and effects, playback controls, and drag-and-drop interaction through a media panel, preview player, and timeline.Editing depends on placing media on the timeline and using context-sensitive controls such as splitting at the playhead and tweaking selected effects.
  • F.2.3 3D Modeller: 3D Modeller tasks use a viewport, scene hierarchy, inspector, and prompt sidebar to select objects and edit transformations, appearance, and wireframe settings.Object selection controls which properties appear in the inspector, while exact structural and visual changes are entered numerically.
  • F.2.4 Flight Analyser: Flight Analyser tasks require temporal navigation, aircraft search, flight-detail inspection, and radius drawing on a replayable map with a scrubbable UTC timeline.The dataset uses OpenSky historical state vectors from a 12-hour window, downsampled into 720 one-minute timesteps.
  • F.2.5 Circuit Designer: Circuit Designer supports analog and digital circuit construction through component placement, drag wiring, property dialogs, continuous simulation, and live measurements.Logic inputs require right-to-left placement for proper orientation, and circuit-building tasks are evaluated by truth-table equivalence while analogue readings allow 5 % relative tolerance.

G GauntletBench Task Illustrations

The appendix catalogues all 100 GauntletBench tasks across five environments, specifying task identifiers, descriptions, ground truth, initial states where applicable, and testcase difficulty.

  • G GauntletBench Task Illustrations: The appendix lists all 100 GauntletBench tasks across five environments.The task tables include identifiers, task descriptions, ground truth, and initial state where applicable.
  • G GauntletBench Task Illustrations: Each task table records the task identifier, task, ground truth, and initial state where applicable.
  • G GauntletBench Task Illustrations: Testcases are categorized by easy, medium, or hard difficulty.

G.1 Video Editor · G.2 Workflow Builder

G.1 and G.2 define vision-intensive tasks that test precise temporal editing, multimodal composition, and structural workflow manipulation. The tasks range from simple configured operations to complex sequencing, branching, reconnection, and condition-preserving edits.

  • G.1 Video Editor: G.1 Video Editor tests localized and per-clip visual effects, including contrast-only correction, fade-outs over final seconds, and distinct corrections across an ordered set of videos.The benchmark specifies both single-clip effects and different corrections for successive clips or interleaved segments.
  • G.1 Video Editor: G.1 Video Editor also requires compositing multiple media types through ordered video sequences, background audio, transition-spanning audio, text overlays, and title cards.Tasks specify audio placement at video endings or transitions and text overlays with exact durations, styling, or parallel playback.
  • G.1 Video Editor: G.1 Video Editor escalates to exact structural manipulations such as midpoint insertion, swapped halves, duplicated clips with asymmetric trims, and empty gaps between videos.These tasks require preserving specified ordering and timing while modifying the timeline structure.
  • G.2 Workflow Builder: G.2 Workflow Builder begins with constructing a named three-node workflow and configuring an HTTP Request before extending or modifying existing pipelines.The foundational workflow connects Manual Trigger, HTTP Request, and No Operation, with the request configured as GET to the specified URL.
  • G.2 Workflow Builder: G.2 Workflow Builder includes higher-complexity graph edits that preserve existing outputs while deleting nodes, disabling ports, rewiring branches, adding side branches, and creating conditional or Switch-based routes.Tasks require routing active versus inactive users, adding a priority_urgent Switch case, and modifying connections without changing unrelated nodes or routes.

G.3 3D Modeller · G.4 Flight Analyser

The benchmark’s 3D Modeller tasks test geometric construction, spatial arrangement, physical contact, symmetry, and formula-based transformations. Its Flight Analyser tasks test temporal tracking, spatial filtering, proximity, airport events, and multi-stage zone transitions.

  • G.3 3D Modeller: The suite also tests physical and relational scene construction through non-overlapping stacks, face-touching spheres, exposed-volume minimization, and iterative deletion and scaling based on object volumes.These tasks combine fixed transforms or positions with contact, overlap, volume, and surface constraints that must hold across multiple objects or iterations.
  • G.3 3D Modeller: Several 3D Modeller tasks require spatial reasoning from visual structure, including completing missing arms or row endpoints, filling an unoccupied axis, and enforcing symmetry around a reference object.These tasks constrain the added object’s shape, spacing, orientation, scale, level, and scene count while preserving existing elements.
  • G.3 3D Modeller: Other 3D Modeller tasks require quantitative geometric reasoning, such as matching volumes, minimizing surface area, enumerating signed scale permutations, and sorting objects by surface area plus volume.The specifications provide formulas or constraints for cuboid, cylinder, sphere, cone, and ellipsoid computations, while requiring the resulting scene edits.
  • G.4 Flight Analyser: The Flight Analyser suite requires tracking aircraft through time and space to report border crossings, nearest airborne pairs, airport landings and departures, and immediately preceding flights.Examples specify playback times, airports or geographic regions, exclusion of grounded aircraft, radius constraints, and callsign or UTC-time outputs.
  • G.4 Flight Analyser: Several Flight Analyser tasks monitor changing membership in geographic zones, asking which flight remains longest, when initial flights leave, and when new-flight occupancy reaches a threshold.The tasks use monitored circles around airports, specified flight sets, forward playback, and multiple temporal outputs such as exit times, entering callsigns, and four-flight occupancy.
  • G.4 Flight Analyser: The most complex Flight Analyser task requires a two-stage temporal relation between two approaching flights: identify the first zone entry, determine the second flight inside, and report their final joint exit.It uses a 35 km radius around Ostend-Bruges International Airport and requires tracking from 11:43 UTC until both flights leave the zone.

G.5 Circuit Designer

The Circuit Designer section evaluates agents on logic-circuit construction tasks ranging from elementary gates and parity circuits to sequential combinational designs and analog circuits. Supplied task outputs include measured frequencies and signal amplitudes for several circuit configurations.

  • Advanced Digital Circuits: More advanced digital tasks require full-adder SUM and COUT functions, a two-operation 1-bit ALU, decoders, demultiplexers, sequence detection, squaring, and priority encoding.The full adder uses SUM = A XOR B XOR CIN and COUT = (A AND B) OR (A AND CIN) OR (B AND CIN); the ALU selects AND or OR using OP.
  • BJT Differential Pair with Resistor Loads: 191.615 mV was reported for the differential-pair output peak-to-peak measurement.The task builds a BJT differential pair with resistor collector loads and an NPN current-mirror tail, applying a small differential input and measuring the right collector output.
  • Circuit Measurement: 62.5 Hz was reported for the MSB output frequency.
  • Circuit Measurement: 2.605 V was reported for the amplifier output peak-to-peak measurement.
Loading 2606.14397v2…