Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

Motivation

• Existing benchmarks rely on homogeneous, simplistic applications and narrow capabilities, producing saturated performance and inaccurate assessments of real-world agent competence.They also face compatibility, pipeline, and configuration constraints that limit support for evolving agentic frameworks.Source: §1 · p. 2

Benchmark Design

• GauntletBench evaluates agent generalisation through temporal perception, graphical understanding, and 3D reasoning across five less-covered professional applications and 100 vision-intensive tasks.The applications are Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Designer, with 20 tasks per application across calibrated difficulty levels.Source: §1 · pp. 2–3, Figure 1 · p. 2
• Its modular platform combines environments, applications, tasks, and automated evaluation to support open-source, API-based, and closed-source agents.The evaluation engine measures success rate, MLLM-judged progress rate, and efficiency using tailored objective evaluators.Source: §1 · p. 2

Motivation and Results

• 19.1% success is achieved by the best agent on GauntletBench, versus over 80% for human annotators who use 30% fewer steps.Claude Opus Computer Use reaches 19.1%, while GPT Computer Use and Gemini Enterprise achieve 4.3% and 13.7%, respectively; no API-based agent exceeds 13.2%.Source: §1 · pp. 2–3

2 GauntletBench

• GauntletBench combines a unified web environment, controlled applications, structured tasks, and automated evaluation into a modular benchmark pipeline.The platform is designed to evaluate agent generalisation in complex real-world scenarios.Source: §2 · p. 3

2.1 Environments

• Agents interact through high-level browser commands and visually grounded observations, while a unified interface supports both open- and closed-source frameworks.Canvas-based applications expose screenshots only, whereas Video Editor and Workflow Builder additionally provide Accessibility Trees; structured information remains insufficient for vision-intensive tasks.Source: §2.1 · p. 3

2.2 Applications

• The benchmark uses five less-covered professional applications to test temporal perception, graphical understanding, and 3D reasoning alongside basic UI understanding, tool use, and long-term reasoning.The applications are Video Editor, Workflow Builder, 3D Modeller, Flight Analyser, and Circuit Design, with interfaces intentionally differing from popular applications.Source: §2.2 · pp. 3–4

2.3 Tasks

• GauntletBench contains 100 human-feasible tasks, with 20 tasks per application spanning easy, medium, and hard levels that increasingly require composition, multi-step reasoning, and fine-grained understanding.Each application includes 2 easy, 9 medium, and 9 hard tasks, and users can define additional tasks with uploaded ground truth.Source: §2.3 · p. 4

2.4 Evaluation Engine and Metrics

• Application-specific evaluators automate rigorous, human-aligned scoring using task-aware matching, while success rate, progress rate, and efficiency metrics capture completion, partial progress, and resource use.The evaluators support tolerances, fuzzy and order-invariant matching, and error localisation; efficiency includes consumed tokens and consumed steps.Source: §2.4 · pp. 5–6

3.1 Evaluation Setup

• 14 frontier agents are evaluated, spanning open-source and API-based MLLMs, closed-source frameworks, and text-only systems on selected applications.The evaluated systems include Gemma-3, Mistral-Large-3, Qwen3-VL, Llama-4-Maverick, Qwen3, DeepSeek-V3.2, Qwen-Max, OpenAI o3-pro, GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6, GPT Computer Use, Gemini Enterprise, and Claude Computer Use.Source: §3.1 · p. 6

3.2 Performance Evaluation

• 19.1% average success is achieved by Claude-Opus-4.6 Computer Use, versus 75–85% for non-expert humans, while most systems achieve 0.0% across applications.Gemini-3.1-Pro and Claude-Opus-4.6 reach 13.7% and 13.2%, respectively; no open-source MLLM exceeds 1.7% on any application.Source: §3.2 · p. 7
• Over 50% average progress is achieved by Claude-Opus-4.6 Computer Use, but performance drops substantially from easy to medium and hard tasks, exposing unreliable long-horizon execution.The results suggest that agents often can make progress but fail as task complexity accumulates.Source: §3.2 · p. 7, Figure 3 · p. 7
• 11.7% success rates on Flight Analyser and Circuit Designer indicate that agents recognize graphical elements but lack deeper understanding of their relationships, interactions, and resulting behaviours.These applications are identified as the most challenging for current agents.Source: Figure 4 · p. 7, §3.2 · p. 8
• 43.5% and 15.5% increases in average progress rates result from visual inputs for Qwen and GPT families, respectively, compared with text-only agents.Structured information alone enables completion of only a very small subset of tasks in Video Editor and Workflow Builder.Source: §3.2 · p. 8

Ablation Studies

• Larger models consistently improve success and progress rates while reducing token consumption, indicating that stronger models—not merely additional interaction steps—are required.The largest model in each family maintains comparable interaction steps despite higher per-step cost; comparisons use the medium-difficulty subset.Source: §3.3 · p. 8, Table 3 · p. 9
• Extended reasoning generally improves sufficiently capable models but increases token and step costs, whereas Qwen-3-VL shows no performance gain and weaker models benefit little.The findings suggest that extra deliberation is effective mainly when the base model has enough capacity to use it.Source: §3.3 · p. 8, Table 4 · p. 9

3.4 Error Analysis

• Four recurring failure categories are task-completion misjudgment, grounding failures, protocol violations, and systematic open-source-model failures such as invalid outputs and infinite loops.The analysis attributes open-source systematic failures partly to limited agentic browser-control data during fine-tuning.Source: §3.4 · p. 9

Web Agent Benchmarks

• Existing web benchmarks progress from synthetic interaction tasks to realistic and visually grounded environments, but mostly evaluate consumer-level navigation, information lookup, and form-filling rather than professional perceptual and spatial reasoning.MiniWoB, MiniWoB++ [18], WebArena, VisualWebArena, WebVoyager, Mind2Web, REAL, and AssistantBench broaden realism and task coverage, yet higher-order dynamic perception, graphical understanding, and 3D spatial reasoning remain largely unexplored.Source: §4 · pp. 9–10

OS Agent Benchmarks

• OS-level benchmarks cover broad desktop and mobile skills, but GauntletBench instead uses a browser environment to improve control and reproducibility while requiring reasoning beyond point-and-click interactions.OSWorld, AndroidWorld, AndroidEnv, WindowsAgentArena, and CRAB evaluate agents across operating systems and applications, whose desktop state can be harder to reproduce and control.Source: §4 · p. 10

Agentic Evaluation

• LLM-based judging enables graded assessment of partial progress but can hallucinate success, motivating GauntletBench’s combination of structured exact-match checks, visual trajectory judging, and efficiency metrics.The benchmark reports token consumption, latency, and interaction steps alongside task accuracy, extending prior hybrid evaluation strategies and model-based assessment.Source: §4 · pp. 10–11

Saturation of Computer-Use Agent Benchmarks

• As modern agents approach saturation on earlier benchmarks such as MiniWoB++ [16], GauntletBench targets visually dense, dynamic, specialist interfaces where frontier models still perform far below human evaluators.Prior efforts including WorkArena and REAL increase enterprise realism or interaction length, but largely retain text-heavy, form-based interactions.Source: §4 · p. 11

Conclusion

• GauntletBench evaluates agent generalisation through realistic, vision-intensive tasks targeting temporal perception, graphical understanding, and 3D reasoning across professional applications.The benchmark is intended to support research on robust perception, reasoning, planning, and interaction.Source: §5 · p. 11

Limitations

• The benchmark covers only five specialised domains and limited interaction paradigms, potentially restricting real-world diversity and introducing evaluation bias.Its visually grounded interaction constraints may also limit agents’ potential when combined with more observational challenges.Source: §5 · p. 11

Future Works

• Future expansions will target more professional applications, richer multimodal interactions, longer-horizon tasks, robustness, safety, and failure recovery.These additions aim to better reflect real-world deployment requirements.Source: §5 · p. 11

Broad Impacts

• GauntletBench may promote more robust, reliable, and trustworthy agents by encouraging rigorous evaluation in realistic and challenging settings.The benchmark could facilitate research on safer and more dependable autonomous agents for real-world applications.Source: §5 · p. 11

Harness Overview

• The harness mirrored agent interaction by adding only task delivery, screenshot capture, and answer collection around the same web applications.Human and agent trajectories were therefore structurally comparable.Source: A Human Evaluation · p. 17

Recruitment

• Fifteen participants—three per application—completed 300 tasks, with each participant having a CS background and relevant hands-on tool experience.The harness was designed to scale to multiple annotators per application.Source: A Human Evaluation · p. 19

Scoring

• Human outputs were scored through the same pipeline as agent outputs, without human-specific adjustments.The harness recorded screenshots of actions within the tested application window with participant consent.Source: A Human Evaluation · p. 20

B Additional Ablation Studies on Efficient

• Consumed-token and consumed-step efficiency are broken down by easy, medium, hard, and pooled overall difficulty for both ablation studies.The values underpin the medium-difficulty columns in Tables 3 and 4 of the main paper.Source: B Additional Ablation Studies on… · p. 21

C Progress Rate Criteria

• The supplied passages do not describe any Progress Rate Criteria results, metrics, or conclusions.No passage assigned to that subsection is included in the input.Source: B Additional Ablation Studies on… · p. 21

B Additional Ablation Studies on Efficient

• Table 7 compares consumed tokens across GPT-5.4 Nano, Mini, and full models and Claude-4.6 Haiku, Sonnet, and Opus by difficulty and overall.Results are reported in thousands as mean ± standard deviation over three independent runs, across 10 easy, 45 medium, 45 hard, and 100 total tasks.Source: Table 7 · p. 21
• Table 8 reports consumed steps for the same model-size comparison across easy, medium, hard, and pooled overall task buckets.The measurements use mean ± standard deviation over three independent runs.Source: Table 8 · p. 21
• Table 9 compares consumed tokens for reasoning-on versus reasoning-off pairs involving GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.6 across all difficulty buckets.Values are reported in thousands as mean ± standard deviation over three independent runs for 10 easy, 45 medium, 45 hard, and 100 total tasks.Source: Table 9 · p. 21
• Table 10 reports consumed steps for the same three reasoning comparisons across easy, medium, hard, and pooled overall tasks.The table uses mean ± standard deviation over three independent runs.Source: Table 10 · p. 22

C.1 LLM-as-a-Judge

• Stage 1 uses gpt-4o-mini, while Stage 2 uses gpt-5.1.The appendix uses Stage 1 and Stage 2 to refer to the two prompted LLM calls.Source: §C.1 · p. 22

C.1.1 Details

• The pipeline forwards adjacent screenshot pairs to the judge only when RMSE, pHash distance, or changed-pixel fraction exceeds a threshold, reducing unnecessary calls.Screenshots are resized to 320×200 and converted to grayscale; the thresholds are RMSE ≥0.03, pHash ≥5.0, or changed-pixel fraction ≥0.01.Source: §C.1.1 · p. 22
• Stage 1 uses gpt-4o-mini to compare retained screenshots, identify visible changes and progress, and maintain a compact cumulative visible-state summary.Its structured output includes the visible action, change summary and type, task relevance, progress, confidence, uncertainty, and updated visible-state summary.Source: §C.1.1 · p. 22
• Stage 2 uses gpt-5.1 to combine trajectory events, the final screenshot, task requirements, the final answer, and optional evaluation signals into a calibrated final score.The score ranges from 1 for no meaningful visible progress to 5 for clearly successful completion with required details and no important visible errors or redundant artifacts.Source: §C.1.1 · p. 22, §C.2.3 · p. 24

C.2.1 Stage 1 System Prompt

• Stage 1 instructions require screenshot-grounded, conservative judgments that do not infer hidden changes or success from actions and preserve prior summaries only when evidence is insufficient.Contradictions between screenshots and prior summaries are resolved in favor of the screenshots, and completion counts only when visibly supported.Source: §C.2.1 · p. 23

C.2.2 Stage 1 Task Prompt

• The Stage 1 task prompt requires strict JSON reporting of visible action, change type, task relevance, progress, confidence, uncertainty, and the updated visible-state summary.The allowed change types include navigation, UI interaction, content updates, object additions or removals, selection changes, text changes, dialogs, errors, and no clear change.Source: §C.2.2 · p. 23

C.2.3 Stage 2 System Prompt

• The Stage 2 system prompt requires conservative, evidence-based scoring from screenshots, penalizes clearly wrong, extra, duplicated, or redundant visible elements, and rejects unsupported high scores.When evidence is ambiguous, it prefers the lower score; a correct-looking final answer without visible support should not receive a high score.Source: §C.2.3 · p. 24

C.2.4 Stage 2 Task Prompt

• The Stage 2 task prompt asks for a strict JSON score and brief reason grounded in the final screenshot and trajectory, conservatively reporting satisfied and missing, wrong, or redundant requirements.Partial completion receives the lower matching rubric score; useful trajectory progress cannot override an unsupported final state, while clearly supported progress may be incorporated conservatively.Source: §C.2.4 · p. 25

Agents Do Not Know When To Stop

• Agents frequently either continue after completing a task and corrupt the result or stop early with an incomplete final state, especially GPT-based agents.This reflects a mismatch between the agent’s textual belief that the task is finished and the actual screen state.Source: D Error Analysis · p. 26

Grounding Failures

• Grounding failures cause agents to miss intended elements, select visually similar controls, or enter values in the wrong fields without noticing the resulting broken state.Because the browser executes these incorrect actions normally, agents often continue as though progress were valid.Source: D Error Analysis · p. 26

Instruction-following Issues

• Instruction-following failures affect open- and closed-source models when agents batch actions, ignore structured formats, or emit plans without executing browser actions.Llama-4-Maverick often outputs its entire plan without acting, while closed-source models show milder protocol violations that still cap performance.Source: D Error Analysis · p. 26

Open-source Models Failure

• Open-source models’ near-zero FSR reflects syntactically broken or repetitive trajectories and early give-ups, likely due to limited training on long-horizon browser interaction.The analysis suggests targeted post-training on agent trajectories is needed in addition to larger models.Source: D Error Analysis · p. 26

E Tailored Evaluation

• Each test passes only when all checked properties match, using application-specific rules including tolerance, symmetry-aware, graph, set, property, and truth-table matching.Video timeline blocks must match in timing within ±1 s; 3D positions, rotations, and scales within ±0.15; flight time-of-day values within ±1 minute.Source: E Tailored Evaluation · pp. 26–27, Figure 8 · p. 27

F.1 Prompt Template

• F.1 Prompt Template: A unified prompt template standardizes application descriptions, capabilities, interaction paradigms, goals, ordered steps, and required JSON outputs.The template instructs agents to follow task statements exactly and avoid unrelated changes.Source: §F.1 · p. 29, §F.2.4 · p. 32

F.2.1 Video Editor

• F.2.1 Video Editor: The environment combines media-panel selection, timeline drag-and-drop editing, preview and effects controls, and playhead-dependent operations.Clips must be loaded before being dragged onto the timeline, while splitting occurs at the current playhead position.Source: §F.2.1 · p. 30

F.2.3 3D Modeller

• F.2.3 3D Modeller: The application provides a 3D viewport, scene hierarchy, task sidebar, and inspector for selecting objects and editing transformations and appearance.Precise structural and aesthetic changes are entered numerically through the Inspector after selecting an object.Source: §F.2.3 · p. 31

F.2.4 Flight Analyser

• F.2.4 Flight Analyser: Users navigate UTC time with playback controls, search aircraft by callsign, inspect flight details, and draw airport-radius boundaries.The interface exposes one flight’s detailed data at a time, with telemetry fields sometimes requiring panel scrolling.Source: §F.2.4 · p. 32

F.2.5 Circuit Designer

• F.2.5 Circuit Designer: The browser application supports component placement, endpoint-based wiring, component editing, and live measurement on a circuit canvas.Components can be added through menus or canvas interactions, while values are edited through property dialogs.Source: §F.2.5 · p. 32
• Circuit Designer tasks require dragging connections between terminals, drawing logic inputs right to left for correct orientation, and allowing continuously running simulations to stabilize.These interactions occur after selecting Edit and may require waiting for live values to update.Source: §F.2.5 · p. 33

• The appendix lists all 100 GauntletBench tasks across five environments, with tables covering task number, task description, ground truth, and initial state where applicable.Source: G GauntletBench Task Illustrations · p. 34
• Difficulty labels classify test cases as easy, medium, or hard.Source: G GauntletBench Task Illustrations · p. 34

G.1 Video Editor

• Video Editor tasks require precise temporal and graphical operations, including trimming, splitting, reordering, duplicating, correcting, transitioning, titling, overlaying, and synchronizing video and audio.Tasks range from retaining exact time segments and inserting clips to applying per-clip corrections, fade-outs, text styling, gaps, and aligned audio bridges.Source: §G.1 · pp. 35, 37, 38, 42

G.2 Workflow Builder

• Workflow Builder tasks cover creating and editing connected node pipelines, including configuring an HTTP request, inserting a timed Wait node, and modifying Switch branches and ports.The initial workflow connects Manual Trigger -> HTTP Request -> No Operation, using GET and the URL https://jsonplaceholder.typicode.com/todos/1; later tasks insert a 1-second delay or delete a Default branch and disable its port.Source: §G.2 · pp. 44–45
• Workflow Builder tasks require inserting, deleting, configuring, and rewiring nodes while preserving specified pipeline paths and existing connections.Examples include Formula and Filter insertion, branch-input rewiring, side-branch addition, and coordinated Filter, Sort, and Limit changes.Source: §G.2 · pp. 45, 46, 47, 49
• Conditional-routing tasks configure distinct outputs, including active-user email handling and country-specific email subjects for true and false branches.The tasks specify conditions such as item.active === true and item.country === "FR", along with separate email destinations and subjects.Source: §G.2 · pp. 48, 52
• Several tasks extend complex pipelines by adding Switch cases or dedicated priority branches while preserving unrelated routes and settings.The required priority_urgent branch may connect to an existing Edit Fields node or a new node that returns to Data Table, depending on the task.Source: §G.2 · pp. 48, 50
• New-workflow tasks test loops, conditional branches, merges, decision trees, and multi-node configurations with exact node counts, conditions, and routes.Specified structures include a counter loop, UK-based branching into Crypto and Guardrails before Merge, a three-If decision tree, and a five-node configured workflow.Source: §G.2 · pp. 50–53

G.3 3D Modeller

• G.3 3D Modeller covers transformations, positioning, symmetry, ordering, recoloring, contact constraints, and volume or surface-area reasoning across diverse primitive-based scenes.Tasks include octant placement, axis alignment, symmetric completion, overlap minimization, formula-based scaling, and iterative volume filtering.Source: §G.3 · pp. 56, 59, 61, 64, 66, 68
• G.3 also requires constructing complete 3D scenes, including a ball on a table and a snowman assembled from primitives with precise transforms and contacts.The table uses a fixed tabletop, four scaled cylinder legs, and a sphere; the snowman specifies component names, positions, rotations, scales, colors, and symmetries.Source: §G.3 · pp. 70–71

G.4 Flight Analyser

• G.4 Flight Analyser includes tracking SAS907 from 16:00 UTC to determine its exact Canada–United States border-crossing time.The task requires locating the flight on a map, tracking it forward, and reporting the crossing time in UTC.Source: §G.4 · p. 75
• G.4 additionally includes a task to find the closest aircraft pair near an airport.The supplied passage identifies this task but provides no further procedure or result.Source: §G.4 · p. 82
• Flight Analyser tasks require identifying the two closest airborne aircraft near Zurich at 16:00 UTC within 40 km, excluding aircraft on the ground.The output must list both callsigns, with the aircraft at higher geographical altitude first.Source: §G.4 · p. 82
• The benchmark tests airport-event monitoring by finding the first departure and second landing at Vienna Airport during a 10-minute playback window.Agents must report both callsigns and their UTC times after starting playback at 10:05 UTC.Source: §G.4 · p. 83
• It also requires reconstructing departure order by identifying the flight that left Manchester Airport immediately before target flight EZY14GQ.The task uses a 10 km airport-centered radius and requires the previous callsign and UTC departure time.Source: §G.4 · p. 84
• Additional tasks track zone occupancy over time, including the last flight remaining within a monitored radius and the first subsequent flight entering it.For airport XCR, agents track four initial flights in a 50 km circle and report when they all leave, the first new callsign, and when four new flights are present.Source: §G.4 · p. 85
• The final task evaluates two-stage temporal transitions by determining when either of two flights enters an OST radius, which enters second, and when both exit.Agents track two approaching flights through a 35 km zone and report the first entry time, second flight inside, and final exit time.Source: §G.4 · p. 86

Combinational Logic Circuits

• The logic-design tasks require circuits for NOT, odd parity, full addition, a two-operation 1-bit ALU, 7-segment decoding, and 1-to-4 demultiplexing.They specify NAND-, XOR-, and gate-based constructions, Boolean output equations, operation selection, and routing one data input to exactly one selected output.Source: §G.5 · pp. 87–89

Timing, Arithmetic, and Encoding Circuits

• Additional combinational tasks test a 555-timer astable configuration, a 4-bit monotonic sequence detector, a 2-bit binary squarer, and a 4-to-2 priority encoder with a valid flag.The sequence detector must handle all 16 inputs, the squarer all four inputs, and the priority encoder must produce correct outputs for all 16 request combinations while masking lower-priority requests.Source: §G.5 · pp. 94–95

BJT Differential Pair with Resistor Loads

• The BJT differential pair produced a measured peak-to-peak output of 191.615 mV at the collector of the right-side transistor.The circuit used 10 kΩ collector-load resistors, a 1 mV, 1000 Hz input, and steady-state oscilloscope measurement at the OUT node.Source: §G.5 · p. 97

3-Bit Two’s-Complement Negator

• The 3-bit two’s-complement negator computes Y = (NOT A) + 1 and asserts OVF only for input 100, while requiring correct outputs across all input combinations.The design uses NOT gates followed by a three-stage ripple-carry incrementer; OVF = A2 AND (NOT A1) AND (NOT A0).Source: §G.5 · p. 98

4-Bit Asynchronous Ripple Counter

• 62.5 Hz is the measured MSB output frequency of the 4-bit asynchronous ripple counter driven by a 1 kHz clock.Each JK flip-flop divides its input clock by 2, so four stages divide the master clock by 16.Source: §G.5 · p. 99

Common-Emitter BJT Amplifier

• 2.605 V is the measured peak-to-peak output of the common-emitter BJT amplifier with voltage-divider bias and a bypassed emitter resistor.The output is measured at the load side after the coupling capacitor once the circuit reaches steady state.Source: §G.5 · p. 100

Wien Bridge Oscillator with Diode AGC

• The Wien-bridge oscillator produces a 1.589 kHz oscillation frequency and 1.422 V peak-to-peak output under diode-based automatic gain control.A small initial capacitor voltage seeds startup because the simulated circuit is noiseless; measurements are taken after diode-clamped steady state.Source: §G.5 · p. 101