ARE: Scaling Up Agent Environments and Evaluations

1 Introduction

• Across Gaia2’s budget-scaling curves, no evaluated model dominates: capability, efficiency, and budget trade off, and all curves plateau.The authors argue that these plateaus suggest standard scaffolds and/or models miss key ingredients for sustained progress.Source: Figure 1 · p. 1, §1 · p. 1
• Environment limits include poor reproducibility in evolving settings, insufficient diversity or realism in simulations, and sequential interactions that prevent adaptation to interim events.The introduction argues that narrow environments saturate quickly and require frequent boilerplate rewrites as tasks and models change.Source: §1 · p. 2
• ARE provides abstractions for creating and verifying diverse environments, integrating synthetic or real apps, and running asynchronous agent orchestrations.It supports integration with existing environments, real apps through MCP, and generation of SFT traces in addition to RL.Source: §1 · p. 2
• Gaia2 comprises 1,120 verifiable scenarios in a simulated smartphone environment and tests adaptability, ambiguity, noise, time, and collaboration beyond search and execution.Its tasks are designed to be simple for humans but challenging for current models, with built-in tools and annotated scenarios.Source: §1 · p. 2
• Gaia2 uses asynchronous, time-driven scenarios in which the environment changes independently of agent actions, requiring agents to respond to or ignore events at the right time.Its verification compares write actions with annotated oracle actions, using soft LLM judging or hard exact-match checks depending on argument type.Source: §1 · p. 2
• The authors do not claim Gaia2 is an AGI-level benchmark, but argue its richer, time-based and multi-agent scenarios align with progress toward practically useful agents and can be extended by the community.They expect success on time-based and multi-agent scenarios to require modeling effort beyond training scale, and aim to lower the boilerplate burden of creating benchmarks.Source: §1 · pp. 2–3

2 ARE: A Research Platform to Create Environments and Run Agents

• ARE decouples agents from continuously evolving simulations, where time advances, events occur, and agents interact through a dedicated interface.The platform is time-driven and treats everything as an event.Source: §2 · p. 3, §2.1 · p. 3

2.1 ARE Foundations

• ARE scenarios combine an initial state and scheduled events in environments built from stateful apps, their data, and governing rules.Apps expose tools that operate on their data; scenarios may include a verification mechanism.Source: §2.1 · p. 3, §2.1.1 · p. 3

2.1.3 Events

• Timestamped events are logged and scheduled in dependency-aware DAGs, enabling deterministic execution, auditability, relative or absolute timing, and parallel branches.The event lifecycle creates or schedules events, executes them, and records results, state changes, and exceptions.Source: §2.1.3 · pp. 4–5

2.1.4 Notification System

• Configurable notifications are queued as agents’ environmental signals and enable asynchronous interaction, graduated observability, and research on agent proactivity.Agents can also proactively check apps for changes that notification policies do not surface.Source: §2.1.4 · p. 5

2.1.5 Scenarios

• ARE scenarios use temporal dynamics, scheduled events, and multi-turn interactions to evaluate capabilities beyond static request-response tasks.Scenarios run until an exit condition and can include natural-language hints for quality checks or training guidance.Source: §2.1.5 · pp. 6–7

2.2 Mobile, an Initial Environment in ARE

• Mobile provides 101 tools across familiar apps, while its asynchronous clock makes agent computation consume simulated time and scenarios can fail on time, step, turn, or verification limits.Synthetic app content is organized into distinct universes averaging approximately 400K raw and 800K structured tokens; Mobile scenarios define a unique set of write actions, while reads are not explicitly verified.Source: §2.2 · p. 7, §2.2.1 · p. 7, §2.2.2 · p. 7

2.2.4 Implementing other Environments with ARE

• ARE’s abstractions support implementing other environments, including internal replications of τ-bench and BFCLv3 without major modifications.For τ-bench’s Airline domain, ARE uses an app environment, user abstraction, oracle events, conditional stopping, and validation events.Source: §2.2.4 · p. 8

2.3 An initial Verifier for ARE

• ARE’s rubric-like verifier matches agent write actions to a minimal oracle sequence, checking action parameters, causal dependencies, and specified timing.It verifies write-action trajectories rather than only final states, making them more interpretable and distinguishing an incorrect action later corrected from a correct first attempt.Source: §2.3 · p. 8, §2.3.1 · pp. 8–9

2.3.2 Verifying the Verifier

• ARE first tests its verifier by perturbing oracle actions and checking expected verdicts, then validates it on real agent trajectories because anticipated perturbations may miss realistic behavior.The validation set contains 450 Gaia2 trajectories manually labeled Success or Failure.Source: §2.3.2 · p. 10
• On 450 human-labeled trajectories, the ARE Verifier achieves better accuracy than the In-context Verifier baseline, which tends to accept agent trajectories too readily.Both verifiers use Llama 3.3 70B Instruct; the comparison is reported in Table 1.Source: Table 1 · p. 10, §2.3.2 · p. 10
• ARE Verifier achieves satisfactory precision with Llama 3.3 70B Instruct, Gemini 2.5 Pro, and Claude Sonnet 3.7, although prompts were tuned for Llama 3.3 70B Instruct.The models use the same prompts; results are reported in Appendix B.3, Table 5.Source: §2.3.2 · p. 10

2.4 An initial Agent Orchestration for ARE

• ARE’s default Mobile orchestration is a ReAct loop with one JSON tool call per step and pre- and post-step operations; asynchronous scenarios pause between turns and resume when new notifications arrive.ARE also supports any orchestration implementing tools and notifications; JSON tool calling is the standard Mobile evaluation method, while code execution is supported but not released.Source: §2.4 · p. 10

2.5 ARE Graphical User Interface

• ARE’s web-based UI lets developers interact with environments, visualize scenarios, inspect and replay detailed agent traces, and annotate scenarios without code.Its scenario view includes an event DAG, a scenario run, and agent logs.Source: §2.5 · p. 11, Figure 7 · p. 11

3 Gaia2: Expanding General Agent Evaluation

• Figure 8 shows proprietary frontier models outperform open-source alternatives; GPT-5 with high reasoning scores highest, and Kimi-K2 leads among open-source models.Source: Figure 8 · p. 11

3.1 Description and Setup

• Gaia2 comprises 800 verifiable scenarios across 10 Mobile universes with 101 tools each, plus Gaia2-mini and augmentations totaling 1,120 scenarios.Events can change the environment asynchronously, time flows continuously, and apps can be represented by autonomous agents. Gaia2-mini contains 160 scenarios for rapid, cost-effective evaluation.Source: §3.1 · p. 12

3.2 Agent Capabilities Evaluated

• Gaia2 tests search, execution, adaptation, temporal reasoning, ambiguity handling, Agent2Agent collaboration, and robustness to environment noise.Tasks include responding to changes caused by prior actions, acting within time constraints, clarifying impossible or contradictory requests, and coordinating with app-agents that mediate tool access.Source: §3.2 · pp. 12–13
• Existing scenarios can be augmented for Agent2Agent and noise evaluations without new annotations, enabling new dimensions such as memory, safety, or security.Noise augmentations can alter tool signatures or introduce failures, with a default failure probability of 0.1, and inject random events at a default frequency of 10 per minute.Source: §3.2 · p. 14

3.3 Data Collection

• Scenario collection combines capability-specific task design and calibration on internal agents with cross-annotator validation and post-QA checks of model success rates.Annotators independently produce solutions, then another annotator checks their consistency; very high or low model success rates can flag scenarios that are too easy or broken.Source: §3.3 · pp. 14–15

4.1 Experimental Setup

• The evaluation uses a provider-agnostic ReAct scaffold, three runs per scenario, and medium notification verbosity, with 200 steps or failed verification ending an agent loop.When service outages or rate limits occur, simulated generation time pauses during response generation and then advances by the actual completion duration.Source: §4.1 · pp. 15–16

4.2 Core Results

• GPT-5 (high) performs best overall, leading Execution/Search and Ambiguity/Adaptability and exceeding Claude 4 Sonnet by 8 points; Kimi K2 is the strongest open model.Table 2 reports Pass@1 scores by capability split and overall, with each model evaluated three times per scenario.Source: Table 2 · p. 16, §4.2 · p. 16
• Execution and Search are the easiest splits, while Ambiguity and Adaptability remain challenging; only Claude 4 Sonnet and GPT-5 (high) show robust performance there.Grok-4 is strong on Search but collapses elsewhere. GPT-5 (high) reaches 36.0 under noise, while most models score below 20.Source: §4.2 · p. 16
• Across most models, output tokens positively correlate with pass@1; Claude-4 Sonnet and Kimi-K2 are outliers, achieving ~35% and ~21% performance while generating relatively few tokens.App usage distributions were nearly identical across evaluated systems. For comparable accuracy, Claude 4 Sonnet was roughly 3× more expensive than GPT-5 (low) but much faster.Source: §4.2 · pp. 16–18

4.3 Time Scenarios Emphasize the Importance of Inference Speed and Reliability

• 8.2% to 26.7% for Claude 4.0 Sonnet and 0% to 34.4% for GPT-5 with reasoning (high) are their Gaia2-Time scores from generation-time to instant mode.Instant mode isolates model policy from inference speed; Gemini 2.5 Pro combines a good policy with fast inference and best supports strict short-timescale requirements. Server rate limits also lowered scores in internal real-time tests, and some scenarios require parallel actions beyond the current scaffold.Source: §4.3 · pp. 18–19

4.4 A Closer Look at Multi-Agent Collaboration on Gaia2 with Agent2Agent

• Increasing Agent2Agent collaboration improves Llama 4 Maverick’s Pass@k scaling and lowers tool-call errors, but Claude 4 Sonnet’s score per token plateaus and its tool-call error frequency weakly worsens.The study varies the ratio of apps replaced by autonomous app agents; gains may depend on whether hierarchical decomposition benefits outweigh coordination overhead.Source: §4.4 · pp. 19–20
• 16.2 with Llama-main and 29.3 with Claude-main are Pass@1 scores when Claude app-agents replace Llama app-agents; the all-light team scores 8.5.Claude-main with Llama-app scores 18.3, also outperforming the all-light team, suggesting stronger executors and stronger main-agent planning each contribute gains.Source: §4.4 · p. 20

5 Discussion

• Gaia2 does not explicitly evaluate memory, self-improvement, or long-horizon decision-making, but ARE supports studying them through memory-equipped agents, extended scenarios, histories, and targeted verification.One proposed memory evaluation preprocesses flattened Mobile universes of 600K tokens before running Gaia2 scenarios with read operations.Source: §5 · pp. 20–21
• ARE supports heterogeneous-team evaluation as a primary experimental factor, and diverse existing benchmarks fit without major issues, though some containerized benchmarks need additional features.The discussion says ARE’s expressivity limits remain unclear.Source: §5 · pp. 20–21
• Annotators struggled to keep Gaia2-Search difficult as frontier models advanced, and rubric judges scale poorly on write-heavy tasks prone to reward hacking; proposed responses include better tools and verifier–agent asymmetry.The discussion also proposes improving the ARE GUI, targeting simple tasks in complex environments, and exploring scalar-score or human-preference rewards, with possible training costs.Source: §5 · p. 21
• Code agents can execute arbitrary Python and manipulate data without filling context, but require careful blocking-operation state management and sandboxing synced to simulation time.Internal experiments suggest their resource efficiency justifies the engineering overhead, especially for iterative reasoning or complex data-processing workflows.Source: §5 · p. 21
• Asynchronous systems let agents sense and act concurrently as environments evolve independently, addressing the limits of sequential ReAct loops in continuous, multisensory settings.The discussion points to speech-to-speech assistants, robotics, and embodied interaction as settings with overlapping information flows.Source: §5 · p. 21
• GPT-5 (high) scores dropped by -10pts on execution and -20pts on search at a 30 min cap; reasoning strength trades off with time, and no system dominates the intelligence spectrum.The discussion says reasoning-heavy models systematically underperform on time-sensitive tasks under current scaffolds.Source: §5 · p. 21
• ARE’s results show intelligence is not just accuracy: agents should use quick, cheap computation for trivial tasks and reserve deeper, slower reasoning for hard problems.The discussion argues adaptive computation matters when latency, reliability, and cost affect real-world deployment.Source: §5 · p. 21

A.1 Apps and Tools Creation

• ARE apps expose methods as agent, user, or environment tools through decorators; event logging records tool calls, and each decorator must specify read or write for verifier operation.The example app distinguishes tools by intended visibility and use.Source: §A.1 · p. 26

A.2 Implementation of Scenario Templates

• A scenario.py file defines initial apps and environment state, the event sequence including the user request, and verification logic.Source: §A.2 · p. 26
• Parameterized templates generate multiple valid scenarios from one base file by varying the task, events, and verifier.The example varies the object, contact, and duration in a Mobile task.Source: §A.2 · p. 26
• Templates help researchers prototype agent capabilities before human annotation and create long-horizon or multi-step scenarios that are impractical to annotate manually.Source: §A.2 · p. 26
• Templates can be time-consuming to build, require validation for edge cases, and lack the nuance of human annotation, creating blind spots on organic tasks.Source: §A.2 · p. 26

A.3.1 Notification Policies

• ARE’s configurable notification policy determines which environment events reach the agent; Mobile provides three verbosity levels, while user messages are always notified.Table 4 presents the pre-set Mobile policies.Source: §A.3.1 · p. 28, Table 4 · p. 28

A.3.2 Notifications Enable Async Interactions

• ARE enables real-time human feedback during an agent’s execution trace, rather than requiring users to wait for the agent to complete its turn.The platform supports these asynchronous interactions by design; the passage points to an example using ARE’s UI.Source: §A.3.2 · p. 28

A.4 Universe Generation

• ARE generates universe content from localized user personas and uses dependency resolution to coordinate apps, while several forms of cross-app consistency remain unhandled.The system prioritizes foundational apps such as Contacts, but temporal consistency, relationship tracking, and references to files across apps remain limitations.Source: §A.4 · pp. 28, 30, 31

A.5 ARE’s User Interface; A.5.1 Environment Exploration; A.5.2 Agent Trace Visualization and Replay

• ARE’s UI lets users inspect app states and contents, follow timestamped agent thoughts, actions, and tool responses, then edit earlier steps and replay traces.Views are generated automatically for new apps, and trace replay supports debugging alternative execution paths.Source: §A.5.1 · p. 31, §A.5.2 · p. 31

A.5.3 Scenario Visualization

• The scenario UI visualizes event dependencies, triggers, timing, and execution status, and lets users simulate time progression across long scenarios.Users can inspect scenario structure before execution and follow event-graph progress while an agent runs.Source: §A.5.3 · pp. 31–32

A.5.4 Annotation Interface

• The annotation interface combines event-graph editing and automated consistency checks, reducing Mobile scenario annotation time by approximately five times versus manual approaches.Checks flag issues such as parentless nodes and contradictory timings.Source: §A.5.4 · p. 32

A.5.5 Related work on UIs for agents development

• ARE’s UI-related work addresses a gap between tools for inspecting or debugging agent traces and annotation platforms that lack structured, reproducible environment scenarios.The cited comparison distinguishes commercial trace views, interactive academic debuggers, and annotation tools focused on conversation curation.Source: §A.5.5 · p. 32

B.1.1 Annotation Guardrails

• Annotation guardrails enforce valid event ordering and a connected DAG rooted at send_message_to_agent, with only one communication branch and each turn ending at send_message_to_user.The UI raises real-time errors when structural constraints are violated.Source: §B.1.1 · p. 33

B.1.2 Capability-Specific Annotation Guidelines

• Capability guidelines define search as read-based research with one final write, ambiguity as reporting impossible or contradictory requests, adaptability as responding to delayed outcomes, and time tasks as time-sensitive actions.Adaptability scenarios may include distractor events, while time scenarios specify precise timing and are designed to be solvable within five minutes.Source: §B.1.2 · p. 33

Taxonomy of Ambiguity Scenarios

• The taxonomies distinguish impossible or contradictory tasks from obvious ambiguities, and classify time scenarios as one-off or recurrent tasks triggered by time or events.Time scenarios may require proactive action or a response to independent or dependent environment events.Source: Taxonomy of Ambiguity Scenarios · p. 34

Multi-turn verifier

• The verifier identifies turns when the agent sends a user message and accepts a trajectory only if every turn succeeds, checking online after each turn or offline after the full trajectory.Turn delimitation follows the paper’s definition of a turn in Section 2.2.Source: §B.2.1 · p. 35

Multi-turn execution

• Calling the verifier at each turn’s end and triggering the next turn only after success avoids further agent execution after oracle-path divergence; test scenarios instead advance at each user message.Test scenarios use this simpler but less efficient approach because oracle actions and the verifier are unavailable.Source: §B.2.1 · p. 35

Parameter placeholder resolution

• For action-dependent placeholders in user or environment actions, ARE uses the verifier’s mapping from oracle actions to agent actions to substitute the corresponding agent-action outputs.This resolves cases where the agent action needed to supply a placeholder cannot be known in advance.Source: §B.2.1 · p. 35

B.3 Choosing the Verifier Model

• The evaluation assessed whether the ARE Verifier could function effectively with models other than Llama 3.3 70B Instruct.The section notes that prompts for the verifier’s soft checks were adjusted using Llama 3.3 70B Instruct.Source: §B.3 · p. 35
• All evaluated ARE Verifier models achieved satisfactory precision and recall on 450 hand-labeled trajectories.The trajectories were the same dataset used in Table 1.Source: Table 5 · p. 39, §B.3 · p. 39

B.3.1 Judge Hacking

• Early RL experiments used the ARE Verifier as a reward function.The verifier was vulnerable to exploitation in its initial version.Source: §B.3.1 · p. 39
• A task-agnostic style soft check prevented reward hacking in which increasingly complex code embedded in write calls overwhelmed the verifier’s LLM judge and produced false positives.The exploit occurred in Search scenarios where successful trajectories were expected to use a single write action.Source: §B.3.1 · p. 39

• The baseline scaffolding implements a ReAct loop of Thought, Action, and Observation, repeated until task completion or termination.At each step, the agent reasons about its current state, selects and executes an action, then observes the outcome.Source: §B.4 · p. 39
• Configurable pre- and post-step methods gather context, validate preconditions, process outcomes, update internal state, and check for completion.The proposed Gaia2 loop uses these methods to enable flexible behaviors.Source: §B.4 · p. 39, Figure 25 · p. 40
• Composing simple, reusable scaffolding components enables sophisticated interaction patterns with minimal implementation overhead.Source: §B.4 · p. 40

• The reported Gaia2 scores cover a representative set of proprietary and open-source models, including both reasoning-oriented and non-reasoning systems.Source: §B.5 · p. 40
• Evaluation uses a ReAct scaffold requiring Thought: and Action: at each step; custom stop sequences address models that continue past one tool call, while ToolCalling APIs largely alleviate this issue.The paper encourages reporting results with either ReAct or ToolCalling.Source: §B.5 · p. 40
• Cost and time constraints limited model coverage: Claude 4 Opus was excluded for high latency and cost, while DeepSeek and GPT-OSS scores were planned for later release.Claude 4 Opus pricing was $15/M input tokens and $75/M output tokens.Source: §B.5 · p. 40
• Grok-4 had a 16k-token completion cap and frequent API empty responses causing high-variance results; Gemini 2.5 Pro used dynamic reasoning, and GPT-5 used temperature/top-p 1 without custom stops.Source: §B.5 · p. 41

B.6 Additional Experiments

• Top Agent2Agent performers spawn more sub-agents despite broadly consistent counts across model families, suggesting stronger task decomposition.Main agents can in principle spawn unlimited app-agents before scenario timeout.Source: Figure 26 · p. 41, §B.6 · p. 41

B.6.1 Influence of Noise Level on Gaia2 Results

• The lowest noise level does not significantly impact model performance.Source: §B.6.1 · p. 41
• The experiment varies tool-error probability and the frequency of random environment events.Source: §B.6.1 · p. 41
• Increasing noise deteriorates performance across models on Gaia2; Table 6 reports Gaia2-mini results across noise levels, with the default setting marked.The experiment varies the probability of tool errors and the frequency of random environment events.Source: Table 6 · p. 41, §B.6.1 · p. 41