Source-linked AI summary
ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenas
Xiaoyu Tian, Haotian Wang, Shuaiting Chen, Hao Zhou, Kaichi Yu, Yudian Zhang, Jade Ouyang, Junxi Yin, Jiong Chen, Baoyan Guo, Lei Zhang, Junjie Tao, Yuansheng Song, Ming Cui, Chengwei Liu
TL;DR
Tool-agent training remains limited by non-verifiable environments, isolated long-horizon supervision, and reliance on either SFT or RL alone. ASTRA automates trajectory and executable environment synthesis, then combines SFT with verifiable online multi-turn RL. Across multiple benchmarks, ASTRA-trained models achieve strong comparable-scale performance while preserving general reasoning ability.
Problem
Tool-agent training remains challenging because existing methods may require manual intervention, use non-verifiable environments, isolate multi-turn trajectories, or rely on only SFT or RL.
Method
ASTRA synthesizes tool-call-graph trajectories and code-executable, rule-verifiable environments, then combines SFT with online multi-turn RL using trajectory-level rewards.
Results
ASTRA-trained models achieve state-of-the-art performance at matched scales across multiple agentic tool-use benchmarks while preserving core reasoning ability.
Takeaways & Limitations
ASTRA provides an automated, reproducible framework for training tool agents with synthesized data and verifiable environments across multiple benchmarks.
Takeaways & Limitations
Future work must extend training and evaluation to multi-turn user interaction and reduce the cost of executable environment synthesis.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as tool-augmented agents for multi-step decision making, yet training robust tool-using agents remains challenging. Existing methods still require manual intervention, depend on non-verifiable simulated environments, rely exclusively on either supervised fine-tuning (SFT) or reinforcement learning (RL), and struggle with stable long-horizon, multi-turn learning. To address these challenges, we introduce ASTRA, a fully automated end-to-end framework for training tool-augmented language model agents via scalable data synthesis and verifiable reinforcement learning. ASTRA integrates two complementary components. First, a pipeline that leverages the static topology of tool-call graphs synthesizes diverse, structurally grounded trajectories, instilling broad and transferable tool-use competence. Second, an environment synthesis framework that captures the rich, compositional topology of human semantic reasoning converts decomposed question-answer traces into independent, code-executable, and rule-verifiable environments, enabling deterministic multi-turn RL. Based on this method, we develop a unified training methodology that integrates SFT with online RL using trajectory-level rewards to balance task completion and interaction efficiency. Experiments on multiple agentic tool-use benchmarks demonstrate that ASTRA-trained models achieve state-of-the-art performance at comparable scales, approaching closed-source systems while preserving core reasoning ability. We release the full pipelines, environments, and trained models at https://github.com/LianjiaTech/astra.
1 Introduction
ASTRA addresses limitations in tool-agent training by combining automated trajectory and environment synthesis with supervised fine-tuning and verifiable multi-turn online reinforcement learning. Its models achieve strong benchmark performance at comparable scales while preserving reasoning ability.
- Challenges: Existing approaches struggle with robust, generalizable tool-agent training because simulated environments are not rule-verifiable and offline trajectories often isolate single-step interactions.These limitations impede deterministic rewards and coherent long-horizon, multi-turn learning.
- Challenges: Methods that use only SFT or RL have complementary weaknesses: SFT lacks online interaction signals, whereas RL-only training depends on the original model’s capabilities.ASTRA therefore combines both regimes rather than relying exclusively on either one.
- ASTRA framework: ASTRA automatically synthesizes diverse multi-turn trajectories from tool-call graph topology and constructs code-executable, rule-verifiable environments from compositional semantic reasoning.The trajectory pipeline supports SFT, while the synthesized environments support deterministic multi-turn RL.
- ASTRA framework: ASTRA’s training methodology first uses SFT to strengthen the initial policy, then applies online multi-turn RL with trajectory-level rewards to optimize task completion and interaction efficiency.The method also incorporates irrelevant-tool mixing during reinforcement learning.
- Results: ASTRA-trained models achieve state-of-the-art performance among similarly scaled models on multiple agentic tool-use benchmarks while approaching closed-source systems and preserving core reasoning ability.The pipelines and trained models are publicly released to support reproducibility and future research.
2 Tool-Integrated Trajectory and Verifiable Environment Synthesis
ASTRA automates tool-agent data construction by synthesizing tool-chain trajectories and QA-derived environments, then validates and trains agents through structured multi-turn interaction. Its design combines broad tool coverage with semantic-topology-based robustness and efficiency-aware interaction.
- Tool-Integrated Trajectory Synthesis: ASTRA collects, normalizes, groups, and filters tool documentation before synthesizing and validating multi-turn tool-use trajectories.The retained corpus contains 1,585 MCP servers, 19,036 tool documents, and 41 domains.
- Tool-Integrated Trajectory Synthesis: Tool chains are generated from server-specific tool schemas, aggregated into weighted transition graphs, and sampled through length-bounded random walks.The resulting chains are checked for inter-tool dependency validity and task–chain coherence.
- Tool-Integrated Trajectory Synthesis: Tasks combine chain-conditioned and server-only generation, then receive diversity, complexity, and persona augmentation while preserving language, intent, and logical requirements.Candidates are scored for question quality, scenario realism, and tool-use necessity before filtering.
- Trajectory Quality Assessment: The trajectory pipeline uses both deployed MCP servers and simulated tools, while an automated quality-assessment process avoids human annotation.Tool-call success, final-answer quality, query understanding, planning, and tool-response reasoning are evaluated at the trajectory level.
- Verifiable Environment Synthesis: ASTRA models multi-turn tool use as navigation over latent semantic topology, verifying subtask attainment and optimizing success against interaction cost rather than prescribing fixed chains.Sub-questions and answers are aggregated according to a dependency structure such as an ordered chain or DAG.
3 Training and Evaluation of Tool Agents
ASTRA trains tool agents through SFT followed by multi-turn online RL in isolated executable environments, with infrastructure for stable batching and trajectory-level rewards. Across agentic benchmarks, the resulting models achieve matched-scale state-of-the-art performance while preserving non-agentic reasoning.
- Training methodology: ASTRA combines SFT with online, multi-turn RL over isolated instance-specific environments, optimizing complete trajectories rather than isolated tool actions.Tool calls execute in a code sandbox, and returned outputs become observations for subsequent decisions.
- Training infrastructure: Adaptive Batch Filling retains valid rollouts in a buffer until each optimization step has a full batch of effective training samples.A rollout is valid when its reward variance provides a non-zero learning signal.
- Tool selection robustness: The training pipeline augments instances with semantically varied irrelevant tools, encouraging relevant-tool discrimination instead of reliance on minimal tool lists.Candidate tools are sampled from high, medium, and low similarity bands after excluding same-domain near duplicates.
- Reward design: Trajectory-level reward uses sub-task recall and tool-use precision to encourage solving more sub-tasks while minimizing redundant invocations.The reward is described as the harmonic mean of recall and precision, providing a structured signal for long-horizon online RL.
- Agentic evaluation: ASTRA models achieve state-of-the-art results at matched parameter scales across BFCL-MT, τ 2-Bench, and ACEBench, with RL producing the largest training-stage gains.The models are also competitive with larger open- and closed-source systems on multiple metrics.
- Reasoning preservation: On AIME2024 and AIME2025, ASTRA shows negligible degradation in non-agentic complex reasoning under the reported decoding settings.The evaluation covers both 14B and 32B models.
4 Discussion
The discussion finds that irrelevant-tool mixing and F1-style trajectory rewards improve tool discrimination, training stability, and performance across training stages. SFT provides a strong initialization, while RL further improves benchmark results through broader exploration.
- Irrelevant-tool mixing: Irrelevant-tool mixing provides discrimination pressure, with balanced similarity coverage helping agents learn which tools to call and which to ignore.Removing irrelevant tools produces the worst performance, while random mixing improves performance but remains below the full ASTRA setup.
- Reward design: F1-style trajectory rewards jointly optimize task completion and interaction efficiency, avoiding the instability of recall-only and precision-only objectives.Recall-only causes interaction turns to grow until collapse, whereas precision-only induces overly conservative short-horizon behavior before also collapsing.
- Training-stage behavior: The original, SFT, and RL models maintain similar average interaction steps, indicating that performance differences are not driven by trivial changes in dialogue depth.Output length is longest for the original model, shortest after SFT, and intermediate after RL.
- Training-stage performance: Both SFT and RL improve benchmark performance over the original model, with RL consistently achieving the best results.SFT teaches structured tool invocation, state tracking, and interaction conventions, while RL adds gains through broader exploration over feasible trajectories.
5 Related Work
Related work scales tool-use data and environments through automated trajectory and environment construction. Existing approaches span tool-centric corpora, executable multi-turn trajectories, programmatic environments, database abstractions, and exploration-driven task synthesis.
- Trajectory synthesis: Recent work constructs large tool-centric corpora and extends them to executable, verifiable multi-turn tool-call sequences.Other approaches extract procedural knowledge from open-domain text into executable trajectories and iteratively connect data generation with training.
- Environment construction: Manual environment design constrains domain diversity and scalability because interactive benchmarks and controlled task suites require costly human design and maintenance.This motivates programmatic construction of scalable tool-interactive environments.
- Environment construction: Programmatic approaches synthesize executable environment skeletons, rule-based validators, read–write database interfaces, and exploration-derived executable tasks.These methods target scalable training, verifiable experiences, or task generation without predefined goals.
6 Conclusion and Future Work
ASTRA combines automated trajectory synthesis for SFT with executable, rule-verifiable environment synthesis for multi-turn RL. Across benchmarks, its models achieve strong performance at comparable scales while preserving general reasoning ability, and future work targets interactive robustness and lower synthesis cost.
- Conclusion: ASTRA unifies tool-call-graph trajectory synthesis for SFT with semantic-topology environment synthesis for executable, rule-verifiable online RL.The resulting environments are independent Python environments for deterministic multi-turn reinforcement learning.
- Conclusion: ASTRA-trained models achieve strong performance at comparable scales across multiple agentic tool-use benchmarks while preserving general reasoning ability.The authors also open-source the data pipelines, synthesized environments, and trained models.
- Future work: Future work will add multi-turn user interaction during training and evaluation and reduce cost by validating QA-derived topology before code generation.Code environments would be instantiated only for high-confidence specifications.
7 Contribution
The supplied contribution passage contains author information rather than a substantive contribution statement.
- The supplied passage lists the paper’s authors but does not state a research contribution.
A.1.1 SFT Data
ASTRA’s SFT dataset contains many multi-turn, tool-using conversations spanning thousands of tool functions. Its message and tool-call distributions are summarized in Figure 9 and Figure 10.
- 54,885 multi-turn conversation samples comprise 580,983 messages, averaging 10.59 messages per sample.
- All samples involve tool calling, averaging 4.42 invocations per conversation.72.2% of samples contain 1–5 tool calls.
- The dataset covers 6,765 unique tool functions for reasoning, computation, search, and other capabilities.
- Figure 9 shows the distribution of messages per sample, while Figure 10 shows the distribution of tool calls per sample.
A.1.2 RL Data
ASTRA’s RL dataset spans diverse domains and languages, emphasizing complex multi-step reasoning with extensive external-tool use and both parallel and serial execution.
- 6,596 RL samples span diverse domains, led by Real Estate at 15.6%, E-commerce at 10.6%, and Healthcare at 8.1%.
- The collection is bilingual: English accounts for 71.2% (4,694) of samples and Chinese for 28.8% (1,902).
- Samples average 4.37 reasoning hops, with a median of 4.0 and a range of 1–20.
- Parallel Multi-Hop scenarios comprise 47.8% of samples, followed by Multi-Hop at 34.8%, Parallel Single-Hop at 10.6%, and Single-Hop at 6.8%.
- 91.3% of 28,794 sub-questions require external tool calls.
- 44.2% of reasoning steps can be parallelized, while 55.8% require serial execution because of data dependencies.
A.2 Case study
ASTRA’s case studies contrast model behavior on multi-step tool-use tasks and illustrate prompts for generating and evaluating tool-use trajectories. The examples show correct RL behavior alongside errors from the original and SFT models.
- Case study: The stock task requires obtaining Quasar Ltd.’s symbol before retrieving its stock details with that symbol.
- Case study: In the stock-information example, the RL model is marked correct, while the original and SFT models are marked incorrect.
- Case study: The tire-pressure task requires checking all tire readings and finding the nearest tire shop if any pressure falls below 37.5 PSI.
- Prompt examples: The trajectory-generation prompt asks an evaluator to determine whether a tool chain is valid and, when valid, recover the corresponding user intent in JSON format.
- Prompt examples: The tool-call conciseness prompt scores each call as necessary or redundant based on goal relevance, parameter correctness, new information, and cost efficiency.