Source-linked AI summary
Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, Thomas Scialom
TL;DR
Gaia2 addresses the limited realism of static and synchronous agent benchmarks by evaluating LLM agents in asynchronous environments with temporal constraints, noise, ambiguity, and collaboration. Using the ARE framework and verifiable write-action scenarios, the study finds that no model dominates across capabilities, with GPT-5 (high) leading overall while revealing trade-offs among accuracy, efficiency, robustness, and cost.
Problem
Most existing LLM-agent benchmarks are static or synchronous, despite increasing use cases involving sustained interaction with dynamic environments where time, uncertainty, and collaboration matter.
Method
Gaia2 combines 1,120 human-annotated scenarios in a smartphone-like environment with asynchronous, event-driven execution and action-level verification through the ARE framework.
Results
No model dominates across capabilities: GPT-5 (high) leads overall with 42.1% pass@1, while time-sensitive, ambiguity, adaptability, and noise-robustness tasks remain challenging for most models.
Takeaways & Limitations
Gaia2 exposes trade-offs among reasoning strength, efficiency, robustness, and cost, while its verifiers support fine-grained evaluation and reinforcement learning from verifiable rewards.
Takeaways & Limitations
Agent2Agent collaboration may produce gains only when sub-goals are well-scoped and agents reliably exchange state and intent, while added hierarchy can cause cascading errors or saturating gains.
Abstract
from arXiv · showhide
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.
1 INTRODUCTION
Gaia2 addresses the lack of realistic evaluations for agents operating in dynamic environments by introducing asynchronous, verifiable scenarios. It combines the Gaia2 benchmark with the ARE platform and an empirical study of model trade-offs.
- Most existing agent benchmarks are static or synchronous, leaving asynchronous events, temporal constraints, noise, uncertainty, and collaboration largely untested.
- Gaia2 evaluates agents in asynchronous environments with verifiable tasks motivated by deployed use cases such as information seeking, scheduled tasks, and interoperable multi-agent systems.The benchmark contains 1,120 human-annotated smartphone-like scenarios with realistic applications.
- Gaia2 scenarios test adaptability, noise robustness, ambiguity resolution, temporal awareness, and collaboration beyond search and execution.
- The write action verifier checks every state-changing action against oracle annotations, enabling reproducible, fine-grained evaluation.
- The ARE framework supports asynchronous benchmark construction, scalable evaluation, data generation for RL, and direct use of Gaia2 for RLVR.
2 RELATED WORK
Prior agent benchmarks cover execution, tool use, reasoning, and collaboration, but generally change only when agents act and evaluate final outcomes. Gaia2 instead combines asynchronous events with temporal, robustness, ambiguity, collaboration, and action-level verification.
- Existing embodied, web-based, synthetic, and function-calling benchmarks measure grounded execution, app-like tasks, state verification, milestones, or tool use.
- Many prior benchmarks remain synchronous and agent-driven, so environments change only after agent actions and evaluation often ignores intermediate actions.
- Gaia2 stresses temporal constraints, robustness, ambiguity resolution, and multi-agent coordination in asynchronous, event-driven environments.
- Gaia2 extends prior verification approaches by evaluating every state-changing write action using exact checks, rubric judgments, and causal and temporal constraints.
3 ARE: SCALING UP AGENT ENVIRONMENTS AND EVALUATIONS
ARE is a platform for building and analyzing environments that evolve independently of agents. Its event, app, scenario, observability, and orchestration abstractions support asynchronous evaluation with controlled timing and traceability.
- ARE environments evolve continuously while agents interact asynchronously with the user and environment through dedicated interfaces.Simulation time advances as the environment introduces events.
- ARE organizes stateful app APIs, environments, events, notifications, and scenarios into abstractions for rich dynamic simulations.
- Scenarios specify an initial state, an event DAG, and verification that can run offline after execution or online through scheduled validation events.
- Asynchronous execution lets model-generation time advance the simulation clock, allowing external events to occur during agent reasoning.
- The Mobile environment instantiates ARE with twelve consumer apps and 101 associated tools using synthetic, coherent data.
- ARE orchestration extends a model-agnostic ReAct loop with hooks that inject queued notifications before calls and check termination afterward.
4 GAIA2: EXPANDING GENERAL AGENT EVALUATION
Gaia2 expands agent evaluation with a large, capability-oriented scenario suite and a verifier that checks trajectories at the action level. Its benchmark design combines human annotation, environment augmentations, and multiple correctness dimensions.
- Gaia2 contains 1,120 scenarios, including 800 unique verifiable scenarios, a 160-scenario mini subset, and 320 augmentation scenarios.
- The benchmark organizes scenarios around Execution, Search, Ambiguity, Adaptability, Time, Noise, and Agent-to-Agent capabilities.Core categories are treated as compositional rather than strictly orthogonal.
- Noise introduces tool anomalies and irrelevant events, while A2A replaces apps with app-agents and removes the main agent’s direct app access.
- Human annotators construct event-and-write-action DAGs, with repeated validation, consistency checks, structural guardrails, and post-hoc difficulty calibration.
- The verifier permits unlimited read exploration but evaluates write actions for consistency, causality, timing, and completeness against an oracle sequence.
- 0.98 agreement and 0.99 precision were achieved on 450 labeled trajectories, outperforming LLM-only baselines.
5 EXPERIMENTS
Experiments evaluate Gaia2 models under a common scaffold, comparing capability-specific performance with cost, latency, tool use, and collaboration. Results reveal strong cross-capability trade-offs, especially for time-sensitive tasks and multi-agent coordination.
- Experimental setup: Models are evaluated on each Gaia2 capability split using a shared ReAct-style scaffold, repeated three times to account for variance.Runs terminate at 200 steps, context overflow, verification completion, or timeout.
- Core results: GPT-5 (high) leads overall with 42.1% pass@1, while Execution and Search are easiest and Ambiguity, Adaptability, Time, and Noise remain challenging.Only Claude-4-Sonnet and GPT-5 (high) achieve robust performance on Ambiguity and Adaptability; GPT-5 (high) reaches 35.4% on Noise.
- Cost and latency: Cost-performance analysis shows that higher reasoning compute can improve accuracy while increasing solution time, whereas Claude 4 Sonnet costs roughly 3× GPT-5 (low) at comparable accuracy.Kimi-K2 is described as cost-effective, while success rate per dollar is proposed as a deployment-relevant metric.
- Performance drivers: Performance correlates positively with tool calls and output tokens, although Claude-4 Sonnet and Kimi-K2 achieve high scores with relatively few tokens.Within model families, Thinking variants generate more tokens per step but take fewer steps overall, yielding higher pass@1 and lower cost per solved scenario.
- Time capability: Removing generation latency raises Gaia2-Time performance, with Sonnet increasing from 8.1% to 26.7% and GPT-5 (high) from 0.0% to 34.4%.Default-mode results show inverse scaling: longer reasoning can reduce Time performance by causing deadline misses.
- Multi-agent collaboration: Agent2Agent collaboration benefits weaker models more than frontier systems, and heterogeneous teams gain from both stronger execution and stronger main-agent sub-goal specification.Replacing Llama 4 Maverick app-agents with Claude app-agents raises pass@1, while Claude-main plus Llama-app reaches 18.3.
6 CONCLUSION & DISCUSSION
ARE provides an extensible asynchronous foundation for verifiable agent evaluation and RLVR, while Gaia2 exposes accuracy, cost, time, and orchestration trade-offs across models and tasks.
- 42% pass@1 is GPT-5 (high)’s best overall accuracy, while Claude-4 Sonnet offers competitive performance with lower latency and Kimi-K2 leads open-source systems at 20%.The results show that no model dominates across all capabilities and motivate cost-normalized reporting.
- Action-level verification supports fine-grained credit assignment and matches human annotations with 0.99 precision and 0.95 recall, while revealing judge-hacking.The paper identifies robust verifier design as critical for evaluation and RL training.
- Time-sensitive tasks and A2A experiments emphasize orchestration, including adaptive compute allocation and delegation across heterogeneous agent teams.The paper reports that heterogeneous teams can outperform monolithic models through effective delegation.
- ARE models environments as collections of stateful apps, data, time management, notifications, events, and scenarios with scheduled events and verification.Apps expose tools, events are logged, and scenarios can include verification mechanisms.
- ARE supports reproducible, event-driven evaluation through deterministic environments, timestamped event logs, dependency-managed scheduling, and configurable agent observability.Its event system supports absolute or relative timing and directed acyclic graph dependencies.
- Gaia2 shifts benchmarking from static tasks to dynamic scenarios that evaluate temporal dynamics, events, multi-turn interaction, and verifiable outcomes.Scenario definitions combine apps, scheduled events, and arbitrary verification logic.
A.3 UNIVERSE GENERATION
The Mobile universe generator builds internally connected synthetic consumer data around personas, with app dependencies resolved through foundational sources such as Contacts.
- App generation maintains cross-app references by querying existing universe state, such as validating email senders and calendar participants against Contacts.This dependency resolution is intended to preserve consistency across generated apps.
- Foundational apps such as Contacts take precedence over dependent apps when dependency conflicts occur.The dependency graph identifies Contacts as the root for most apps, while Shopping and File system are independent.
- Several complex dependencies remain unsupported, including temporal consistency, semantic relationship tracking, and cross-modal references between messages and files.The paper identifies fully coherent synthetic Mobile environments as future work.
- Contacts provides the foundation for generating a user-centered universe, using sampled and location-contextualized personas to populate the environment.A universe is based on a user persona selected from generated contacts.
- The generator creates synthetic Chats, Messages, Emails, Calendar, City, Shopping, Cabs, and Files data using persona prompts, public datasets, and iterative generation procedures.Examples include 500 sampled Shopping products, generated discount codes, Wikipedia-based files, and calendar generation over a specified timeframe.
A.4 ARE GRAPHICAL USER INTERFACE
The ARE GUI supports scenario construction, environment inspection, trace replay, event-DAG analysis, and capability-focused Gaia2 annotation, while the benchmark stresses complex agent behavior.
- The ARE GUI lets developers visualize environments and scenarios, analyze detailed traces, replay executions, and annotate verifiable scenarios without code.It is designed to help understand agent behavior and failures in simulated environments.
- The interface exposes apps, tools, current states, timestamped thoughts, actions, tool responses, event dependencies, and timing constraints during scenario development and execution.Developers can inspect event progression and jump through simulated time for long-running scenarios.
- The annotation interface provides a graph editor for configuring event nodes, parents, tool calls, and optional timing, with structural constraints that reduce annotation errors.Guardrails require connected event DAGs, constrain message events, and require turns to end with send message to user.
- Gaia2 organizes 800 scenarios around capabilities including execution, search, ambiguity, and adaptability, with each capability adding a distinct challenge.Search tasks require multiple reads and one final write answer, while ambiguity tasks require clarification and adaptability tasks require response to environmental changes.
- Sequence verification can distinguish a trajectory that makes and corrects a wrong write from one that maintains safety throughout.This provides more control than comparing only final environment states.
B.2.4 VERIFYING THE VERIFIER
The ARE Verifier is evaluated with perturbation-based tests and hand-labeled trajectories, while the ReAct scaffold and pre/post hooks support agent execution and termination checks.
- The verifier is tested by perturbing oracle actions in ways expected to preserve or invalidate trajectory validity, then comparing verdicts with the perturbation type.These unit tests support rapid iteration but only detect anticipated behaviors.
- Figure 17 illustrates judge-hacking in which conditional logic embedded in a meaningless user message passes evaluation.The example motivates scrutiny of verifier robustness beyond anticipated perturbations.
- 450 hand-labeled trajectories provide a validation benchmark for comparing ARE Verifier verdicts with expected Success or Failure outcomes and an LLM baseline.The trajectories come from agents powered by various models on Gaia2 scenarios.
- The ReAct scaffold cycles through Thought, Action, and Observation until completion or termination, with pre-step and post-step methods gathering context, validating conditions, processing outcomes, and checking completion.This extends standard ReAct for the ARE environment and asynchronous execution.
B.3.2 ORCHESTRATION ABLATION: PARALLEL TOOL-CALLING
The ablation compares Parallel Tool Calling with the standard ReAct scaffold to test whether single-threaded orchestration limits Gaia2 performance. PTC improves efficiency substantially, but task success changes little and Time remains difficult.
- PTC significantly reduces latency and token consumption, including GPT-5 (low) reductions of ∆-435s on Execution and ∆-5109 tokens.The reductions primarily reflect fewer intermediate reasoning steps per action.
- PTC changes pass@1 only marginally, with performance deltas ranging from -6.3pp to +3.0pp while model rankings remain unchanged.
- The Time split remains challenging under parallel execution, indicating bottlenecks associated primarily with model capabilities such as sequential reasoning and temporal planning.PTC results remain far below the upper-bound score obtained with instant-time generation.
- The ablation concludes that the qualitative findings are not artifacts of the ReAct scaffold and calls for research on completely novel orchestration.
B.4 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
The evaluation reports Gaia2 results across proprietary and open-source models using a standardized ReAct-based setup, with model-specific handling for tool-call formatting and reasoning outputs. The study also acknowledges incomplete model coverage and a potential mismatch between the uniform reasoning interface and some models’ intended operation.
- The evaluation covers proprietary and open-source systems, including reasoning-oriented and non-reasoning models.
- The ReAct scaffold requires Thought: and Action: outputs, with custom stop sequences added for models that may continue beyond one tool call.Provider-specific ToolCalling APIs largely alleviate this formatting issue.
- Cost and time constraints prevented evaluation of every available model, including Claude 4 Opus because of its very high latency and cost.The cited pricing was $15/M input tokens and $75/M output tokens.
- Specific third-party configurations included dynamic reasoning for Gemini 2.5 Pro, a 16k-token reasoning cap for Grok-4, and default sampling settings for GPT-5.Grok-4 also experienced frequent API Empty Response errors that introduced high variance.
- Reasoning outputs were discarded from subsequent context to preserve a uniform Thought–Action structure, although this may not be optimal for models that interleave tool use with reasoning.The authors encourage alternative setups for assessing the benchmark’s theoretical limits.
B.5 ADDITIONAL EXPERIMENTS
Additional Agent2Agent experiments examine how many sub-agents models instantiate during Gaia2-mini tasks. Counts are fairly consistent across model families, while stronger A2A performers tend to spawn more sub-agents.
- Top A2A performers spawn more sub-agents, suggesting stronger task decomposition.
- Agent2Agent evaluations permit main agents to spawn an unlimited number of app-agents before scenario timeout, but observed behavior is relatively consistent across model families.
B.5.2 INFLUENCE OF NOISE LEVEL ON GAIA2 RESULTS
The noise experiment varies tool-error probability and random-environment-event frequency to assess robustness. Low noise has little effect, whereas increasing noise deteriorates performance across models.
- Increasing tool errors and random environment events deteriorates model performance across Gaia2 evaluations.
- The lowest noise level does not significantly impact model performance.