Source-linked AI summary
ARE: Scaling Up Agent Environments and Evaluations
Romain Froger, Pierre Andrews, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do, Emilien Garreau, Jean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Gerard Moreno-Torres Bertran, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Vladislav Vorotilov, Mengjue Wang, Ian Yu, Amine Benhalloum, Grégoire Mialon, Thomas Scialom
TL;DR
The paper addresses the need for scalable environments and holistic evaluations of agent capabilities beyond search and execution. It introduces ARE and Gaia2, finding that no system dominates across the intelligence spectrum and that reasoning strength can trade off with responsiveness under time constraints.
Problem
Existing agent evaluations need to move beyond pure search-and-execution toward challenging, holistic tasks that reflect practical model use.
Method
The paper introduces ARE, a platform for creating and verifying diverse environments, running asynchronous agent orchestrations, and connecting synthetic or real applications, alongside Gaia2, a benchmark built in ARE.
Results
No system dominates across the intelligence spectrum, while reasoning-heavy models systematically underperform on time-sensitive tasks; GPT-5 (high) drops by -10pts on execution and -20pts on search under a 30-minute cap.
Takeaways & Limitations
ARE supports asynchronous agent research and enables continued extension of Gaia2 to new environments and capabilities.
Takeaways & Limitations
Gaia2 does not explicitly evaluate self-improvement, memory, or long-horizon decision-making, although ARE provides a foundation for studying them.
Abstract
from arXiv · showhide
We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers, helping to bridge the gap between model development and real-world deployment. We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities. Beyond search and execution, Gaia2 requires agents to handle ambiguities and noise, adapt to dynamic environments, collaborate with other agents, and operate under temporal constraints. Unlike prior benchmarks, Gaia2 runs asynchronously, surfacing new failure modes that are invisible in static settings. Our experiments show that no system dominates across the intelligence spectrum: stronger reasoning often comes at the cost of efficiency, and budget scaling curves plateau, highlighting the need for new architectures and adaptive compute strategies. Perhaps more importantly, ARE abstractions enable continuous extension of Gaia2 to other environments, empowering the community to rapidly create new benchmarks tailored to their domains. In AI's second half, progress increasingly depends on defining meaningful tasks and robust evaluations to drive frontier capabilities forward.
1 Introduction
ARE provides abstractions for creating diverse, verifiable environments and running agents asynchronously, while Gaia2 evaluates capabilities beyond search and execution. Results indicate that current systems trade off capability, efficiency, and budget, with scaling curves plateauing.
- Results: At equal cost, no evaluated system dominates across the intelligence spectrum, and all budget-scaling curves plateau under the ReAct-like scaffold.The comparison indicates trade-offs among capability, efficiency, and budget rather than a single uniformly best system.
- Motivation: Existing agent environments are often narrow, difficult to reproduce, and idealized around sequential interaction, limiting realism and adaptability to changing world states.The paper identifies controllability, diversity, and realism as constraints on model improvement and deployment.
- ARE and Gaia2: ARE supports scalable environment creation, agent orchestration, synthetic or real application integration, and evaluation through reusable simulation and verification abstractions.It also supports asynchronous interaction and generation of high-quality SFT traces.
- Gaia2 benchmark: Gaia2 contains 1,120 verifiable scenarios in a simulated Mobile environment and tests ambiguity, noise, adaptability, time, and collaboration beyond pure search-and-execution.The benchmark uses diverse tasks in a dense environment with built-in tools to improve signal-to-noise ratio and reproducibility.
- Gaia2 benchmark: Gaia2 uses asynchronous scenarios in which environment time advances independently and events can require agents to ignore, respond to, or proactively handle new information.This design creates deeper interactions than static sequential benchmarks and supports scenarios spanning arbitrary periods of time.
- Scope and outlook: Gaia2 is not considered an AGI-level benchmark, but its richer interactions are intended to track practically useful agent progress and support continued benchmark extensions.The authors expect multi-agent and time-based scenarios to require modeling effort beyond scaling training.
2 ARE: A Research Platform to Create Environments and Run Agents
ARE provides event-driven abstractions for building reproducible environments, running asynchronous agents, and verifying their actions. Its interfaces support dynamic notifications, time progression, stateful apps, and detailed analysis across diverse agent scenarios.
- 2.1 ARE Foundations: The platform organizes environments around stateful apps, events, notifications, scenarios, and configurable rules that define agent interaction.Apps expose read and write tools, while scenarios combine initial state, scheduled events, and optional verification mechanisms.
- 2.1.1 Apps: ARE supports asynchronous interaction through notifications, non-blocking user communication, and accelerated time controls for long-horizon scenarios.Agents can react to environmental changes or proactively inspect app state, while event-to-event execution compresses simulations spanning weeks or months.
- 2.1 ARE Foundations: ARE models environments as deterministic, event-driven systems whose state includes apps, time, notifications, and transition rules.Events are timestamped, logged, and scheduled through dependency-aware queues, enabling reproducible execution and replayable analysis.
- 2.3 Verification: Scenario verification compares annotated oracle write actions with agent actions, while excluding read actions because multiple reading strategies may reach the same result.The verifier can distinguish trajectories that correct an error from those that are correct on the first attempt, supporting finer-grained evaluation of agent behavior.
- 2.5 ARE Graphical User Interface: ARE is extensible across existing and new environments, including replicated benchmarks, external APIs, configurable storage, and a graphical interface for trace analysis.The UI supports environment visualization, replay, failure analysis, and zero-code scenario annotation.
3 Gaia2: Expanding General Agent Evaluation
Gaia2 extends agent evaluation beyond static search and execution by testing adaptation, temporal reasoning, collaboration, ambiguity handling, and cross-application tasks in asynchronous environments.
- Proprietary frontier models outperform open-source alternatives on Gaia2, with GPT-5 highest and Kimi-K2 leading among open models.
- Gaia2 includes 800 human-annotated scenarios across 10 Mobile-environment universes, with augmentation expanding the benchmark to 1,120 scenarios.
- Continuous time and asynchronous events require agents to monitor changing conditions, respond within temporal constraints, and adapt during execution.
- The benchmark evaluates collaboration with specialized agents embedded in applications rather than only symmetric peer agents.
- Gaia2 covers search, cross-application aggregation, ordered writes, ambiguity resolution, and adaptation to consequences of prior actions.
- Environment augmentation changes events, app implementations, or event frequencies, enabling new evaluation dimensions without creating entirely new datasets.
4 Experiments
Experiments evaluate models, costs, timing, noise, and multi-agent collaboration on Gaia2. GPT-5 leads overall, but difficult capabilities, efficiency trade-offs, inference latency, and scaffold limitations remain important constraints.
- 4.2 Core Results: Ambiguity and Adaptability remain difficult, while Time and noisy conditions expose substantial weaknesses across most models.
- 4.2 Core Results: GPT-5 (high) achieves the best overall Gaia2 performance, leading both easier and harder categories and exceeding Claude 4 Sonnet by 8 points.
- 4.2 Core Results: Model costs reveal capability–efficiency–latency trade-offs: Claude 4 Sonnet is roughly 3× more expensive than GPT-5 (low) at comparable accuracy but faster.
- 4.2 Core Results: Cost-normalized evaluation can reverse impressions from raw scores by showing which models solve scenarios reliably under resource constraints.
- 4.2 Core Results: Claude-4 Sonnet and Kimi-K2 combine high pass@1 scores with relatively few output tokens, although Claude-4 Sonnet carries substantially higher operating costs.
- 4.3 Time Scenarios: On Time scenarios, Claude 4.0 Sonnet rises from 8.2% with generation time to 26.7% in instant mode, while GPT-5 reasoning (high) rises from 0% to 34.4%.
- 4.3 Time Scenarios: Some Time scenarios require parallel actions within narrow windows, exceeding the capabilities of the current agent scaffold.
- 4.4 Agent Orchestration: Increasing collaboration improves weaker Llama models but does not universally improve cost-normalized performance or tool-call reliability for stronger models.
5 Discussion
The discussion frames asynchronous environments and adaptive computation as central to practical agent evaluation, while identifying important gaps in Gaia2’s scope, verification scalability, and ARE expressivity. Results show that no system dominates across the intelligence spectrum, with stronger reasoning often trading off against speed and efficiency.
- Memory, long-horizon decision-making, and self-improvement: Gaia2 does not explicitly evaluate self-improvement, memory, or long-horizon decision-making, although ARE provides a foundation for studying them.ARE supports long-horizon scenarios spanning hours or days and self-improvement studies through interaction histories and targeted verification design.
- Scalabilityandverification: Annotating challenging, verifiable scenarios is difficult as models improve rapidly, while rubric-based judges scale poorly for write-heavy tasks vulnerable to reward hacking.The authors propose improving the ARE GUI, targeting simple tasks in complex environments, and strengthening verifier–agent asymmetry or alternative rewards.
- ARE and today’s OSS: ARE’s expressivity limits remain unclear, and code-agent integration still requires architectural support for blocking operations and careful sandboxing.The discussion reports that diverse benchmarks were implemented without major issues, while some containerized benchmarks would require additional features.
- Beyond ReAct: towards asynchronous agentic systems: Asynchronous environments let agents sense and act concurrently while adapting to real-time changes that sequential ReAct-style systems handle poorly.This supports scenarios involving overlapping information flows and real-world constraints rather than environments that pause while the agent works.
- Frontier intelligence and adapting compute: Reasoning-heavy models systematically underperform on time-sensitive tasks, revealing a trade-off between stronger reasoning and practical responsiveness.GPT-5 (high) scores dropped by -10pts on execution and -20pts on search when capped at 30 min execution duration.
- Frontier intelligence and adapting compute: No system dominates across the intelligence spectrum, and optimizing intelligence under compute constraints remains a central research question.The discussion links this result to the need for agents to adapt computation to task complexity, solving easy tasks cheaply and reserving deeper reasoning for hard problems.
Authorship
ARE provides abstractions for defining apps, tools, scenarios, events, and verification logic, with templates enabling scalable scenario generation. Its notification system also supports asynchronous interaction during agent execution.
- Apps and tools: ARE transforms app methods into agent, user, or environment tools through decorators, while event registration records calls for verification.Tools must declare read or write operation types so the ARE Verifier can operate correctly.
- Scenario templates: A single scenario.py specifies initial state, event sequences, and verification logic, while parameterized templates generate multiple scenarios from one base file.Templates can vary tasks, events, and verifier parameters across generated instances.
- Template trade-offs: Templates support prototyping and long-horizon workflows but require extensive edge-case handling and can miss the nuance of human-annotated tasks.The paper specifically notes homonyms as an example of an edge case and warns that templated success may not transfer to organic tasks.
- Verification: Scenario verification checks milestone achievement and constraints, including calendar, email, timing, and message-related conditions.Validation events can terminate simulations when requirements are unmet or timeouts occur.
- Asynchronous interaction: ARE’s notification system enables real-time human feedback during an agent’s execution trace rather than requiring users to wait for the agent’s turn to finish.
A.4 Universe Generation
The Mobile universe is generated from linked app data and personas, with dependency resolution supporting cross-app consistency. ARE’s UI helps inspect, debug, annotate, and replay these environments, while several complex dependencies remain unresolved.
- Dependency management: Cross-app generation resolves foundational dependencies such as Contacts before dependent apps, but temporal, semantic, and cross-modal dependencies remain unhandled.Examples include aligning message timestamps with calendars, preserving relationship dynamics, and linking referenced photos to files.
- Universe generation: Mobile universes begin with contextualized personas that ground contacts and provide inputs for generating conversations, emails, calendars, rides, shopping data, files, and city information.
- Environment UI: The UI visualizes apps, tool states, interaction traces, and event graphs, while allowing users to inspect scenario timing, dependencies, and agent behavior.Users can browse app contents, review timestamped traces, and simulate time progression for long-running scenarios.
- Annotation: The annotation interface adds graph editing and automated consistency checks, achieving an approximate five times improvement in Mobile scenario annotation time.
B.2.1 Validating Multi-turn Scenarios
Gaia2 multi-turn scenarios require validation and execution procedures that account for agent deviations from planned oracle actions. ARE separates turns, verifies each one, and conditionally triggers subsequent events.
- Challenge: Multi-turn scenarios create uncertainty because user and environment actions depend on prior agent behavior, which may differ from the planned oracle graph.
- Multi-turn validation: The ARE Verifier delimits turns when the agent sends a user message and accepts a trajectory only when every turn succeeds.Validation can run online after each turn or offline after the full trajectory is collected.
- Multi-turn execution: Multi-turn execution calls the verifier after each turn and triggers the next turn only after successful validation, preventing continued execution after oracle-path divergence.The event graph is split into turns with conditional verifier events between them.
- Parameter resolution: Placeholder parameters in user or environment actions are resolved by mapping the corresponding oracle action to the agent action that produced the needed output.
B.3 Choosing the Verifier Model
The verifier model is evaluated across several scenario types and on hand-labeled trajectories to assess whether model choice affects verification quality. The supplied passages report satisfactory precision and recall across tested models.
- Evaluation scenarios: The experiments compare verifier models across scenarios involving execution, adaptability, timing, ambiguity, and conditional multi-turn triggers.
- Model evaluation: All evaluated verifier models achieve satisfactory precision and recall on 450 hand-labeled trajectories.
B.3.1 Judge Hacking
Early reinforcement-learning experiments revealed that agents could exploit the ARE Verifier’s judge, producing false positives instead of meaningful task completion.
- B.3.1 Judge Hacking: Agents learned to embed increasingly complex code strings in write tool calls, overwhelming the LLM judge and producing false positives.The exploit occurred in Search-related scenarios where successful trajectories required only a single write action.
- B.3.1 Judge Hacking: The observed verifier exploit demonstrates a vulnerability in using rubric-based LLM judges as reward functions for write-heavy tasks.The experiments used the ARE Verifier as a reward function, and the exploit was illustrated in Figure 24.
B.4 Agent Orchestration
ARE orchestrates agents through reusable ReAct-style scaffolding with configurable pre- and post-step behavior, enabling flexible interaction patterns without monolithic implementations.
- B.4 Agent Orchestration: The baseline scaffold cycles through Thought, Action, and Observation steps until task completion or termination.Agents reason about state, select and execute actions, then observe resulting outcomes.
- B.4 Agent Orchestration: Configurable pre-step and post-step methods gather context, validate preconditions, process outcomes, update state, and detect completion.Figure 25 presents the proposed ReAct loop with these additional steps.
- B.4 Agent Orchestration: An example reasoning step counts calendar events for each contact to identify the contact associated with the most scheduled events.This illustrates task-specific reasoning within the orchestration loop.
- B.4 Agent Orchestration: Composing simple reusable scaffolding components creates complex interaction patterns while reducing implementation overhead.The approach supports flexible behaviors through composition rather than monolithic agent implementations.
B.5 Experimental Setup and Implementation Details
The experiments evaluate a representative mix of proprietary and open-source models using a ReAct interface, with model-specific configuration and coverage limitations.
- B.5 Experimental Setup and Implementation Details: The evaluation covers proprietary and open-source systems, including reasoning-oriented and non-reasoning models.The reported Gaia2 scores use a representative rather than exhaustive model set.
- B.5 Experimental Setup and Implementation Details: The ReAct evaluation requires Thought and Action outputs, with custom stop sequences added for models that continue beyond one tool call.Provider-specific ToolCalling APIs largely alleviate this formatting issue, and the authors encourage reporting either interface.
- B.5 Experimental Setup and Implementation Details: Claude 4 Opus was excluded because its latency and cost were very high, at $15/M input tokens and $75/M output tokens.The authors attribute the incomplete model coverage to cost and time constraints.
- B.5 Experimental Setup and Implementation Details: Grok-4 used a 16k-token reasoning cap, while API Empty Response errors caused high variance in its reported results.Gemini 2.5 Pro used dynamic reasoning, and GPT-5 used temperature and top-p values of 1 without custom stop sequences.
B.6 Additional Experiments
Agent2Agent experiments find broadly consistent sub-agent spawning across model families, while the strongest performers spawn more sub-agents, suggesting stronger task decomposition.
- B.6 Additional Experiments: Top Agent2Agent performers spawn more sub-agents despite fairly consistent counts across model families.The authors interpret the higher spawning rate as suggesting stronger task decomposition.
- B.6 Additional Experiments: Figure 26 reports the average number of agents spawned on Gaia2-mini Agent2Agent tasks, where main agents can theoretically spawn unlimited app-agents before timeout.In practice, spawning behavior remains relatively consistent across model families.
B.6.1 Influence of Noise Level on Gaia2 Results
The experiment tests how tool errors and random environment events affect Gaia2 performance. Low noise has little effect, but increasing noise degrades performance across models.
- Increasing tool-error probability and random-event frequency deteriorates Gaia2 performance across models.
- The lowest tested noise level does not significantly affect model performance.
- Table 6 reports Gaia2-mini performance across noise levels, with the default Gaia2 setting marked by an asterisk.