PaperlayerView exact revision on arXiv

Source-linked AI summary

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio

arXiv:2603.01357v1cs.AI

TL;DR

Personal-assistant benchmarks often omit the longitudinal context, stateful tools, and multi-step goals required by real workflows. ASTRA-bench unifies these elements in an event-driven benchmark with grounded evaluation, finding sharp performance degradation as complexity rises and identifying argument generation and plan orchestration as bottlenecks.

  • Problem

    Existing benchmarks rarely jointly model longitudinal personal context, realistic assistant tools, and multi-dimensional complexity, limiting evaluation of context-dependent agent behavior.

  • Method

    ASTRA-bench generates event-grounded personal data and evaluates 2,413 scenarios in an interactive tool environment using complexity labels and observable evidence.

  • Results

    Performance degrades sharply as referential and functional demands increase, with argument generation and multi-step plan orchestration emerging as primary bottlenecks.

  • Takeaways & Limitations

    Reliable assistants require stronger grounding in messy personal data and more disciplined tool orchestration.

  • Takeaways & Limitations

    The benchmark uses synthetic corpora that cannot fully capture the noise and idiosyncrasy of real data.

Abstract

from arXiv · show

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.

1. Introduction

ASTRA-bench addresses the lack of unified benchmarks for assistants that must combine longitudinal personal context, interactive tools, and complex user goals. It introduces grounded, complexity-aware evaluation and finds that current models remain insufficient for effective personal assistance.

  • Benchmark motivation and contribution: Existing benchmarks typically emphasize individual capabilities rather than jointly modeling longitudinal personal context, personal-assistant tools, and diagnostic complexity.
  • Benchmark motivation and contribution: ASTRA-bench decomposes task difficulty into referential, informational, and functional dimensions instead of treating complexity as a single scalar.This enables more diagnostic analysis of agent capabilities.
  • Benchmark motivation and contribution: ASTRA-bench combines time-evolving personal-context data, an interactive tool sandbox, and 2,413 human-authored scenarios across three complexity axes.The axes are referential, functional, and informational complexity.
  • Evaluation framework: The benchmark grounds evaluation in tool traces, system state, reasoning monologues, related entities, and verifiable success conditions rather than final-answer correctness alone.These components support fine-grained diagnostic assessment of agent behavior.
  • Empirical study: A zero-shot study finds that even state-of-the-art reasoning-tuned models remain insufficient to act as effective personal assistants.
  • Empirical study: The paper identifies bottlenecks in argument generation and performance decay under high complexity.

2. Related Work

Tool-use research has progressed from isolated function calling toward multi-step, stateful, and personalized agents, but evaluation remains fragmented. ASTRA-bench addresses this gap with grounded personal context and diagnostic trajectory assessment.

  • Tool-use agents and benchmarks: Early tool-use agents focused on web browsing, API discovery, and improving tool-invocation reasoning loops.
  • Tool-use agents and benchmarks: Benchmarks expanded from single-turn function calling to multi-step reasoning and multi-turn interaction tasks.
  • Tool-use agents and benchmarks: Stateful environments introduced evolving user state, but existing systems differ in whether they emphasize code generation, function calls, or personalization.
  • ASTRA-bench positioning: ASTRA-bench combines email, calendar, and messaging tools with observable traces and system-state snapshots for diagnostic evaluation.This evaluates navigation of complex personal context rather than surface-level answer matching.
  • Synthetic personal-context data: Synthetic personal-context datasets increasingly model multi-session dynamics, yet many interactions remain self-contained and limited in long-term continuity.

3. Personal Data Generation

ASTRA-bench generates longitudinal personal data by grounding digital artifacts in coherent protagonist events. A draft–critique–revision–verification cascade produces consistent, machine-checkable records across applications.

  • Protagonist-centered generation: The pipeline grounds personal-context data in a protagonist’s biography, social network, and persistent storyline rather than isolated tasks.This supports coherent digital interactions over weeks or months.
  • Event-driven context generation: Each real-world event is represented structurally and projected into application-specific artifacts such as emails, calendar invites, and messages.Repeated projection keeps multiple records consistent with the same event.
  • Artifact generation workflow: The generation workflow uses specialized agents in a draft–critique–revise–verify loop to produce and validate artifact sets.Critiques check factual, stylistic, schema, and cross-application coherence.
  • Artifact generation workflow: Verification performs schema validation, referential-integrity checks, and temporal-consistency tests before accepting outputs.Failures re-enter the critique–revision loop as minimal reports.
  • Artifact generation workflow: Fewer than 8 % of events trigger a second revision pass because only failing components are rerun.

4. Protagonist-Centered Scenarios Authoring

ASTRA-bench turns protagonist storylines into executable, time-anchored scenarios whose goals, complexity labels, and ground-truth traces support diagnostic evaluation. Scenarios test retrieval, reasoning, clarification, and stateful tool coordination.

  • User-centric scenario setup: Scenarios are grounded in specific life events and use multi-turn user simulation rather than static prompts.The simulator is constrained by what the protagonist would naturally know.
  • User-centric scenario setup: Every goal has a reference time, requiring agents to canonicalize relative expressions and distinguish historical routines from current obligations.
  • Complexity annotation: Scenarios are annotated along informational, referential, and functional complexity axes to isolate reasoning, retrieval, and tool-calling failures.
  • Robustness stress tests: Misinformation and insufficient-context conditions test whether agents reconcile conflicts or seek clarification when requests are underspecified.
  • Annotated ground truth: Ground-truth annotations specify related entities, an optimal reasoning monologue, and required success conditions.Success may be stateless, such as answering precisely, or stateful, such as sending a confirmed email.
  • Annotated ground truth: Evaluation combines LLM-based judgment with verifiable milestone calculations.

5. ASTRA-bench

ASTRA-bench extends stateful tool-use evaluation with longitudinal personal context, temporal awareness, broad cross-app coverage, and diagnostic human-authored evaluation. Its framework combines verifiable trajectory measures with LLM-based assessment to capture both structured task success and broader reasoning quality.

  • Simulation Environment: ASTRA-bench builds on ToolSandbox with longitudinal personal context and an expanded simulation environment.The environment is grounded in protagonists’ digital histories and extends the foundational framework with additional evaluation capabilities.
  • Simulation Environment: Reference-time anchoring requires agents to canonicalize relative time expressions against persistent system state.This synchronizes tool outputs with the protagonist’s timeline and distinguishes historical routines from current obligations.
  • Simulation Environment: The environment spans 25+ tools across Contact, Calendar, Email, Message, WhatsApp, and Phone Call domains.Cross-app coverage requires coordination across multiple digital silos rather than a single application.
  • Evaluation Methodology: Evaluation compares agent trajectories with human-authored goals, queries, success conditions, and entities using tool traces and system-state snapshots.This provides observable evidence for multidimensional diagnostic assessment.
  • Evaluation Methodology: Minefield violations nullify the overall score even when all milestones are achieved.The strict penalty applies to errors such as unintended calendar-event removal.
  • Evaluation Methodology: Rule-based measures and LLM judges provide complementary evaluation, with model rankings showing strong alignment across both methods.Rule-based measures emphasize precision, whereas LLM judges assess broader signals including conversational efficiency and hallucination robustness.

6. Results

ASTRA-bench reveals sharp performance degradation as referential, functional, and informational complexity increases, with high-complexity scenarios separating models. Milestone-based analysis identifies payload generation and multi-step execution as central bottlenecks, while stress tests expose failures under missing or conflicting context.

  • 6.2. Complexity Ceiling: Performance decays sharply as referential, functional, and informational complexity increases, with high-complexity scenarios providing the clearest model differentiation.The dimensions progressively stress distinct aspects of tool-use reasoning within shared interactive scenarios.
  • 6.2. Complexity Ceiling: Claude-4.5-Opus achieves the highest macroaverage score at 0.9112, while DeepSeek-V3.2 leads open-source models at 0.9050.DeepSeek-V3.2 matches proprietary models at low complexity but shows a gap as informational and functional demands rise.
  • 6.2. Complexity Ceiling: Robustness depends on integrating long-horizon reasoning with structured execution; GPT-4.1 and Qwen models show sharp high-complexity performance cliffs.Claude-4.5-Opus and DeepSeek-V3.2 exhibit more graceful degradation, indicating that scale alone does not ensure reliability.
  • 6.3. Capability Decomposition via Step- and Task-Level Success: Payload Generation is the primary bottleneck, ranging from 0.5603 to 0.8478, whereas DeepSeek-V3.2 reaches 0.9516 in IR Recall.Models generally retrieve relevant personal context more successfully than they translate it into precise structured arguments for stateful execution.
  • 6.3. Capability Decomposition via Step- and Task-Level Success: DeepSeek-V3.2 attains the highest Entity Creation success at 0.9161, while weaker models suffer larger gaps between intermediate execution and final task completion.The gap is especially pronounced for the Qwen-30B family, where errors accumulate across multiple turns.
  • 6.4. Robustness Under Misinformation and Insufficient Context: Insufficient context reduces task completion by 0.150 to 0.410 points, mainly reflecting execution bias rather than fabricated data.“No Hallucination” rates remain relatively stable; even DeepSeek-V3.2 shows a 0.244 completion impact.
  • 6.4. Robustness Under Misinformation and Insufficient Context: Misinformation causes completion degradation and variable hallucination impacts, while Claude-4.5-Opus shows only a 0.027 completion drop.The results indicate difficulty reconciling conflicting premises and a broad gap in self-correction and premise verification.
  • 6.4. Robustness Under Misinformation and Insufficient Context: Together, milestone and stress-test analyses show that reliable tool use requires balancing safe non-action with proactive correction.The evaluation distinguishes where agents fail from the execution bottlenecks associated with those failures.

7. Conclusion

ASTRA-bench unifies evolving personal context, stateful tools, and human-authored multi-turn goals while evaluating agents through milestone-based diagnostics. Results show a complexity tax, with argument generation and plan orchestration emerging as primary bottlenecks.

  • 7. Conclusion: ASTRA-bench unifies time-evolving personal context, stateful tool environments, and human-authored multi-turn goals for personal-assistant evaluation.The benchmark is designed to bridge synthetic tool-use tests and real-world assistant demands.
  • 7. Conclusion: Milestone-based scoring reveals not only whether a model fails, but why, including failures in grounding and tool orchestration.The framework evaluates execution evidence rather than relying only on final-answer metrics.
  • 7. Conclusion: Performance degrades sharply as referential and functional demands increase, with argument generation and multi-step plan orchestration as primary bottlenecks.The conclusion characterizes this degradation as a significant complexity tax.
  • 7. Conclusion: The authors release datasets, an execution harness, and an evaluation suite as a common substrate for developing plan-centric AI agents.The stated goal is to support development of trustworthy assistants grounded in messy personal data.

Limitations and Future Work

ASTRA-bench’s authors identify limitations in benchmark authoring, evaluation reliability, and synthetic realism, alongside opportunities for broader contexts and safer, better-trained agents.

  • Limitations: Handcrafting context-aware milestones is labor-intensive despite providing high diagnostic value.The authors suggest mining milestones from executions through trace alignment to improve scalability.
  • Limitations: LLM-based graders and milestone checks may produce false negatives for valid but unanticipated plans.Proposed mitigations include calibrated ensembles and reference-free validators.
  • Limitations: Synthetic corpora cannot fully capture the noise and idiosyncrasy of real data.
  • Future Work: Future work includes multimodal context, long-horizon safety such as least-privilege tool use, and training interventions including tool-aware reinforcement learning.

B.3. Combination of Complexity scores

The benchmark’s scenario mix is dominated by a small number of complexity combinations and is evaluated through tool-use workflows spanning multiple personal-information domains.

  • Complexity Distribution: Around 70% of scenarios come from the five most frequent combinations of complexity dimensions.
  • Tool Coverage: ASTRA-bench provides 27 tools across six personal-information management domains.The domains include contacts, calendars, email, messaging, WhatsApp, and phone calls.
  • Tool Coverage: The tool design requires agents to retrieve relevant information before performing actions in multi-step, cross-domain workflows.

C.3. Evaluation Methodology in Details

ASTRA-bench combines deterministic milestone checks with rubric-guided LLM judges to assess both structured task success and broader reasoning quality.

  • Milestone Evaluation: Deterministic milestone checks evaluate fact-grounded sub-goals over tool traces and system-state snapshots.They provide fine-grained diagnostics rather than an all-or-nothing reward.
  • LLM Evaluation: LLM judges assess task performance across completion, tool usage, retrieval, conversation effectiveness, and hallucination.Each dimension receives a score from 0 to 2.
  • Comparative Perspective: Milestone metrics are precise but rigid, whereas LLM judges are flexible but vulnerable to prompt sensitivity and model bias.

D.1. Metrics by Complexity Trenches

Performance declines systematically as task complexity rises, while degraded inputs expose axis-specific and model-dependent robustness problems; evaluator biases further complicate interpretation.

  • Complexity Effects: Performance degrades systematically as functional, referential, and informational complexity increases.The pattern varies across evaluation axes and model families.
  • Robustness: Insufficient context and misinformation produce axis-specific and model-dependent performance degradation.The benchmark compares standard conditions with both degraded-input settings using task completion and no-hallucination metrics.
  • Evaluation Caveats: LLM judges may penalize appropriate clarification, require exact textual matches, and impose incompatible formatting expectations.Milestone evaluation complements them through determinism and tool-usage verification.

E. Multilingual User Queries

ASTRA-bench evaluates multilingual tool-use and task completion by translating context, queries, and expected responses into five languages while retaining English system prompts. Performance varies substantially across models and locales, with Japanese and Chinese showing the largest degradation gaps.

  • Multilingual setup: ASTRA-bench translates context data, user queries, and expected responses into Spanish, German, French, Japanese, and Simplified Chinese.System prompts remain in English, creating a mixed-code reasoning setting.
  • Multilingual results: Japanese and Chinese consistently show the largest degradation gaps across locales.English is used as the baseline for macro accuracy and absolute degradation.
  • Multilingual results: GPT-4o and Claude 3.7 Sonnet show stronger multilingual robustness, with smaller average degradation from English than several other models.GPT-4o-mini and Claude 3.5 Haiku show larger degradations, especially in Japanese and Spanish.
  • Multilingual results: DeepSeek-v3 remains relatively stable overall but exhibits noticeable drops in East Asian languages.The reported pattern distinguishes overall stability from locale-specific weaknesses.
  • Multilingual results: GPT-o3 leads absolute English performance, while GPT-4o and Claude 3.7 better balance quality and multilingual consistency.The comparison separates peak English performance from cross-locale consistency.
Loading 2603.01357v1…