PaperlayerView exact revision on arXiv

Source-linked AI summary

From Failure to Mastery: Generating Hard Samples for Tool-use Agents

Bingguang Hao, Zengzhuang Xu, Yuntao Wen, Xinyi Xu, Yang Liu, Tong Zhao, Maolin Wang, Long Chen, Dong Wang, Yicheng Chen, Cunyin Peng, Xiangyu Zhao, Chenyi Zhuang, Ji Zhang

arXiv:2601.01498v1cs.CL

TL;DR

Tool-use agents need training corpora that capture implicit dependencies and multi-step reasoning, which existing random and shallow generation methods often miss. HardGen uses failure-driven graph sampling, trace-conditioned tool evolution, and feedback-guided CoT refinement to generate verifiable hard samples. A 4B model trained on its dataset reaches state-of-the-art BFCLv3 performance for its scale and generalizes to held-out benchmarks.

  • Problem

    Existing tool-use data-generation pipelines often produce homogeneous, shallow trajectories that fail to capture implicit logical dependencies and multi-turn reasoning.

  • Method

    HardGen builds a failure-derived dynamic API Graph, samples hard traces, uses them to evolve advanced tools and hard queries, and refines reasoning through closed-loop verification feedback.

  • Results

    79.14% overall accuracy makes HardGen-4B-RL state-of-the-art on BFCLv3 for its 4B scale, with gains generalizing to held-out BFCLv4.

  • Takeaways & Limitations

    HardGen provides a verifiable, failure-driven corpus for training models to handle complex tool-use and agentic reasoning.

  • Takeaways & Limitations

    Generalization to entirely new API ecosystems or specialized domains remains underexplored, and verification may require executable environments unavailable for some proprietary or externally dependent tools.

Abstract

from arXiv · show

The advancement of LLM agents with tool-use capabilities requires diverse and complex training corpora. Existing data generation methods, which predominantly follow a paradigm of random sampling and shallow generation, often yield simple and homogeneous trajectories that fail to capture complex, implicit logical dependencies. To bridge this gap, we introduce HardGen, an automatic agentic pipeline designed to generate hard tool-use training samples with verifiable reasoning. Firstly, HardGen establishes a dynamic API Graph built upon agent failure cases, from which it samples to synthesize hard traces. Secondly, these traces serve as conditional priors to guide the instantiation of modular, abstract advanced tools, which are subsequently leveraged to formulate hard queries. Finally, the advanced tools and hard queries enable the generation of verifiable complex Chain-of-Thought (CoT), with a closed-loop evaluation feedback steering the continuous refinement of the process. Extensive evaluations demonstrate that a 4B parameter model trained with our curated dataset achieves superior performance compared to several leading open-source and closed-source competitors (e.g., GPT-5.2, Gemini-3-Pro and Claude-Opus-4.5). Our code, models, and dataset will be open-sourced to facilitate future research.

1 Introduction

HardGen addresses homogeneous tool-use trajectories by generating samples centered on implicit dependencies and multi-step reasoning, then demonstrates strong performance from a compact trained model.

  • Problem: Existing generation pipelines rely on random API combinations and shallow simulation, producing homogeneous happy-path trajectories that miss implicit logical dependencies.Such methods can leave agents robust on simple queries but fragile on logically intertwined instructions.
  • Method: HardGen builds a failure-derived dynamic API Graph, samples hard traces, uses them to guide advanced-tool and hard-query construction, and refines CoTs through closed-loop evaluation.The pipeline prioritizes logical complexity rather than only API coverage or syntactic correctness.
  • Results: 79.14% overall accuracy makes HardGen-4B-RL a state-of-the-art BFCLv3 result for its 4B scale, surpassing GPT-5.2 by 19.02%, DeepSeek-V3.2 by 10.33%, and Grok-4.1-Fast by 3.94%.The gains also generalize to held-out BFCLv4.
  • Contribution: The framework targets challenging function-calling data and supports a wide spectrum of APIs and models while emphasizing high fidelity and complexity.Its stated goal is to synthesize challenging training samples for tool-use agents.

2 Related Work

Tool-use research needs verifiable corpora for extended interactions with implicit preconditions and cross-turn dependencies, but prevailing synthesis methods emphasize breadth and executability over intrinsic reasoning difficulty.

  • Tool-use Ability: Tool-use agents must maintain state and resolve dependencies across extended horizons, yet large-scale verifiable corpora rarely encode implicit preconditions, parameter couplings, and cross-turn logic.These requirements form a central training-data bottleneck for coherent multi-turn orchestration.
  • Data Synthesis for Tool-use Training: Existing synthesis methods expand dataset breadth through fixed toolsets or multi-agent simulation, but often create artificial difficulty from explicit structural constraints rather than intrinsic reasoning hurdles.Their limitation is difficulty that may not reflect implicit dependencies.

3 The Proposed HardGen Pipeline

HardGen is an automatic pipeline that generates difficult, executable tool-use trajectories by learning from model failures, evolving tools, and refining reasoning with feedback.

  • HardGen coordinates failure-driven trace sampling, trace-conditioned tool evolution, hard-query generation, and feedback-guided CoT refinement.The pipeline uses generated traces to construct advanced tools and queries, then iteratively corrects reasoning and function calls.
  • 3.2 Phase I: Failure-inspired Hard Trace Sampling: Failure-driven self-evaluation identifies challenging tools, which are organized into a dynamic API Graph representing failure-prone tools and latent dependencies.The graph is updated through environment interactions and provides the basis for sampling hard traces.
  • 3.2 Phase I: Failure-inspired Hard Trace Sampling: Legal sampling permits a tool only after all prerequisite dependencies have been executed, while greedy selection prioritizes legal tools nearest to the target in graph distance.This constraint preserves execution validity while steering traces toward selected difficult tools.
  • 3.2 Phase I: Failure-inspired Hard Trace Sampling: Each sampled tool is executed with parameters, and the resulting executable hard trace records calls and environment feedback as a failure-aware prior.The trace updates the system state, executed-tool set, and dynamic API Graph before subsequent phases.
  • 3.3 Phase II: Trace-conditioned Tool Evolution: The Tool Maker abstracts a multi-step trace into an advanced tool, and the Hard-query Generator creates a query that requires that abstraction to solve.This construction targets implicit logical bridging rather than queries that explicitly enumerate every intermediate tool call.
  • 3.4 Phase III: Feedback-guided CoT Refinement: Verifier feedback diagnoses incorrect function calls without exposing answers, enabling iterative CoT refinement until correctness or Kmax is reached.A trace is retained only when all M function calls are generated correctly across the sequence.

4 Experiments

HardGen-trained models are evaluated across BFCLv3, APIBank, ACEBench, scaling settings, and ablations of hard queries, advanced tools, and verifier feedback. Results show substantial gains, including state-of-the-art BFCLv3 performance for a 4B model and consistent benefits from the pipeline’s main components.

  • Results on BFCLv3: 63.13 BFCLv3 multi-turn accuracy marks a +41.01 gain for HardGen-4B-RL over the Qwen3-4B base, surpassing strong open- and closed-source models despite its 4B scale.HardGen-4B-SFT reaches 49.65 (+27.52), and both trained variants show balanced performance across sub-metrics.
  • Impact of hard queries: Hard-query training improves performance across evaluated models, supporting hard queries as a key contributor to stronger tool-use capabilities.The supplied table caption identifies the comparison between RL training with and without hard queries, while the experiments report consistent gains across architectures and tasks.
  • Scaling and generalization: Performance improves consistently with model scale from 0.6B to 4B under both SFT and RL, while Llama-3-3B also gains +7.90 multi-turn and +4.97 single-turn points.The reported improvements are especially pronounced on more challenging multi-turn tasks.
  • Impact of advanced tools: Advanced tools increase the number of GPT-4o-judged challenging queries from 223 to 275 across model scales, indicating stronger logical-bridging difficulty.The comparison uses 400 query-synthesis instances, with paired queries produced with and without Tadv.
  • Impact of feedback-guided CoT refinement: Verifier feedback produces improvements exceeding 10 percentage points across all evaluated model scales, establishing feedback-guided refinement as an essential framework component.The ablation compares RL-trained variants with and without Verifier feedback using up to three refinement iterations.

5 Conclusion

HardGen is an automatic pipeline for generating challenging tool-use samples with verifiable reasoning. Its failure-driven data captures implicit dependencies and multi-step reasoning, and a 4B model trained on the resulting dataset achieves state-of-the-art BFCLv3 performance for its scale with broad generalization.

  • 5 Conclusion: HardGen generates challenging tool-use training samples with verifiable reasoning through a failure-driven approach.The pipeline targets implicit logical dependencies and multi-step reasoning characteristic of real-world tasks.
  • 5 Conclusion: A 4B model trained on HardGen data achieves state-of-the-art BFCLv3 performance for its scale, surpassing leading proprietary models.The reported generalization extends across model architectures, parameter scales, and held-out benchmarks.

Limitation

HardGen’s generalization beyond the evaluated setting remains incompletely established. Its verification requirements and dependence on high-quality advanced-tool abstractions also constrain deployment across proprietary, dependency-heavy, or irregular tool ecosystems.

  • Limitation: Generalization to entirely new API ecosystems or specialized domains remains to be fully explored despite evaluation across 2,095 tools.This is an explicit scope boundary of the reported evaluation.
  • Limitation: HardGen requires executable environments for verification, which may be infeasible for proprietary APIs or tools with complex external dependencies.The limitation follows directly from the pipeline’s verification requirement.
  • Limitation: Advanced-tool quality depends on the Tool Maker correctly identifying high-level operations, which may produce suboptimal abstractions for highly irregular or domain-specific tool chains.The stated risk concerns abstraction quality rather than a demonstrated universal failure.

A Data

The appendix examines generator selection, advanced-tool descriptions, and alternative data-generation backbones. These analyses identify Qwen3-30B-A3B-Thinking as the strongest tested generator and show that advanced descriptions are necessary for rigorous reasoning.

  • Model Selection and Number of Attempts: Qwen3-30B-A3B-Thinking achieves the highest correctness across all attempt counts, reaching 89% at K=3.The comparison evaluates QwQ-32B, Qwen3-32B, and Qwen3-30B-A3B-Thinking under identical configurations with 2,000 trajectories each.
  • Data Generation with Other Agent Backbones: HardGen evaluates alternative agent backbones to assess robustness and generalizability across data-generation models.Table 7 reports Qwen3-4B performance when trained on data synthesized by different agent backbones.
  • Impact of advanced tool descriptions: Advanced tool descriptions are a fundamental prerequisite for rigorous reasoning on hard queries rather than merely a performance enhancement.The corresponding analysis compares generator correctness across attempts when descriptions are omitted.

B Evaluations

The evaluations address scalability, robustness, and realism, including an out-of-distribution BFCLv4 assessment of agentic capabilities.

  • Recent tool-use benchmarks emphasize scalability, robustness, and realism as central evaluation axes.
  • BFCLv4 evaluates HardGen-RL on Search and Memory agentic tasks as an out-of-distribution test of tool-use generalization.
  • BFCL supports scalable deterministic evaluation through Abstract Syntax Tree substring matching across single-turn, multi-turn, and agentic categories.

C Performance on the Held-out Benchmark BFCLv4

On the held-out BFCLv4 benchmark, HardGen-RL generalizes to agentic Search and Memory tasks and outperforms same-backbone baselines.

  • The evaluation uses BFCLv4 Search and Memory subsets to test transferability of tool-use skills induced by HardGen.
  • +14.09 accuracy in Memory, from 10.75 to 24.84, and +13.00 in Web Search, from 3.00 to 16.00, over the base model.
  • HardGen-RL consistently outperforms all other baselines built on the same backbone across the evaluated BFCLv4 tasks.

D Training details

Training combines supervised fine-tuning and reinforcement learning with specified open-source libraries, hyperparameters, and a binary reward emphasizing correctness and structural compliance.

  • Training setup: SFT and RL hyperparameters are reported in Tables 9 and 10, respectively.
  • Training setup: Supervised fine-tuning uses Llama Factory, while reinforcement learning uses Verl with training settings adapted for stable and efficient training.
  • Reward design: The binary reward requires both format correctness and answer correctness, emphasizing semantic validity and strict structural compliance.
  • Evaluation support: Table 11 reports consistency rates, Pearson correlations, and human preference for manual-versus-automatic annotation agreement.
  • Reward design: For non-tool-calling cases, the reward implicitly verifies a purely textual response and discourages unnecessary tool invocations.

E Manual Annotations

The supplied passages connect advanced tools with hard-query synthesis and separately identify generalization results for Llama model variants.

  • Advanced tools: Advanced tools are introduced to help language models bridge logical jumps during hard-query synthesis.
  • Annotation: The reported difficulty judgments rely on automatic GPT-4o annotation for challenging-query proportions.
  • Llama generalization: Table 12 compares Llama-3.1-8B and Llama3.2-3B before and after HardGen-RL training on BFCLv3 Single-Turn and Multi-Turn tasks.

F Results on Other Models

HardGen generalizes beyond Qwen models, improving Llama-3 variants through reinforcement learning while using failure-derived API dependencies, hard-query construction, and specialized agents to generate challenging tool-use trajectories.

  • Results on Other Models: 74.07 overall accuracy (+18.78) was achieved by Llama-3.1-8B-Instruct after reinforcement learning, including 53.10 accuracy (+41.98) on multi-turn interactions.Llama-3.2-3B-Instruct also improved from 48.33 to 67.57 overall accuracy.
  • API Graph: The API Graph represents 1,204 failure APIs, prerequisite dependencies, and parameter constraints to bias sampling toward executable yet challenging tool sequences.Execution feedback continually refines dependencies, valid value ranges, and type requirements, including implicit value-level dependencies.
  • Qualitative Case Study: A directory-exploration case shows context retention across turns: the model first searches recursively, then navigates with cd because cat and tail are restricted to the current directory.The trajectory uses pwd and find in the first turn and constructs a four-step navigation plan in the second.
  • Pipeline Prompts: HardGen uses specialized Tool Maker, Hard Query Generator, Reasoner, and Verifier agents to synthesize tools, formulate implicit-logic queries, and refine incorrect calls.The refinement iteration revises reasoning and function calls using targeted verifier feedback.
  • Hard Query Construction: Hard queries abstract multi-step traces into advanced tools that omit explicit intermediate steps, requiring models to infer primitive calls such as zipcode retrieval before ticket purchase.The resulting reasoning decomposes a high-level request into the necessary primitive tool calls.
Loading 2601.01498v1…