Source-linked AI summary
TaskCraft: Automated Generation of Agentic Tasks
Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, Wangchunshu Zhou
TL;DR
Agentic tasks need multi-step reasoning and tool interaction, while existing instruction data is static and agentic benchmarks can be costly to annotate. TaskCraft generates such tasks through depth- and width-based extensions with execution trajectories; the paper reports improved prompt optimization and fine-tuning outcomes, and releases a varying-difficulty synthetic dataset.
Problem
Static instruction-generation data does not model the external-tool interaction needed for agentic tasks, while agentic benchmarks face scalability limits from labor-intensive annotation.
Source: §1 · p. 1Method
TaskCraft generates agentic tasks by extending atomic tasks depth-wise and width-wise, with corresponding execution trajectories.
Source: §1 · p. 2Results
TaskCraft improves prompt optimization and supervised fine-tuning, with trajectory-trained agents gaining performance on HotpotQA [32], Musique [21], and Bamboogle [10].
Source: §1 · p. 2, §6 · p. 11Takeaways & Limitations
The authors release a large synthetic dataset of varying-difficulty tasks and trajectories to support agent tuning and evaluation research.
Source: §1 · p. 2, §6 · p. 11Current task construction focuses on common tools such as browsing, PDF processing, and image analysis; tailoring atomic tasks to agents’ specific tool requirements is left to future iterations.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Agentic tasks, which require multi-step problem solving with autonomy, tool use, and adaptive reasoning, are becoming increasingly central to the advancement of NLP and AI. However, existing instruction data lacks tool interaction, and current agentic benchmarks rely on costly human annotation, limiting their scalability. We introduce \textsc{TaskCraft}, an automated workflow for generating difficulty-scalable, multi-tool, and verifiable agentic tasks with execution trajectories. TaskCraft expands atomic tasks using depth-based and width-based extensions to create structurally and hierarchically complex challenges. Empirical results show that these tasks improve prompt optimization in the generation workflow and enhance supervised fine-tuning of agentic foundation models. We present a large-scale synthetic dataset of approximately 36,000 tasks with varying difficulty to support future research on agent tuning and evaluation.
1 Introduction
TaskCraft addresses the limits of costly human annotation and static instruction data by generating scalable, multi-tool agentic tasks with execution trajectories. The paper reports benefits for prompt optimization and agent-model fine-tuning, alongside a large synthetic task dataset.
1 Introduction
- Agentic tasks require autonomous, multi-step problem solving with tool use and adaptive reasoning, and their complexity challenges conventional annotation paradigms.The paper situates this need amid advances in language agents and systems that combine reasoning with dynamic orchestration.Source: §1 · p. 1
- Existing instruction-generation methods target static scenarios and do not model the external-tool interaction required for agentic tasks.The paper also notes that advanced agent benchmarks face annotation scalability limits; HLE used 1,000 experts to label 2,500 data points [9].Source: §1 · pp. 1–2
- Depth-based extension transforms task elements into new atomic tasks, while width-based extension combines distinct instances into multi-subproblem tasks.These extensions build structurally and hierarchically complex challenges.Source: §1 · p. 2
- Trajectory-trained agent models reportedly gain performance on HotpotQA [32], Musique [21], and Bamboogle [10].The paper also reports that prompt optimization improves rejection-sampling pass rates while minimizing generation time.Source: §1 · p. 2
- The overview reports approximately 36,000 varying-difficulty tasks, while the contribution list describes about 32k tasks, with execution trajectories for further research.The workflow's controlled generation provides ground-truth trajectories for interpretability, reproducibility, and verifiability.Source: §1 · p. 2
- TaskCraft generates scalable, multi-tool tasks with execution trajectories and validates only incremental extension components rather than re-verifying each full task.The workflow supports adaptive difficulty and efficient verification.Source: §1 · p. 2
2 Notations and Preliminary
The section models an atomic task as one target-tool invocation, assuming an input index can retrieve the relevant context. It formulates task-question generation from that index and a relation to an answer, while noting that direct question generation can produce poorly controlled tasks.
2 Notations and Preliminary
- An atomic task is defined as solvable with one target-tool invocation, with search and file-system operations omitted under a detailed-index, finite-navigation assumption.The execution-flow description tracks extracting an input index, retrieving context with the target tool, and deriving the final result.Source: §2 · p. 3, Figure 1 · p. 3
- Directly prompting an LLM to turn an answer into an atomic-task question can yield low tool-use rates, unpredictable difficulty, unregulated tool requirements, and inconsistent verification complexity.The authors assume an ideal search engine retrieves precise data from the input index.Source: §2 · p. 3
- The formulation represents a task question as q = f(iT, R) → a, where f samples a natural-language question from the provided information.The input index iT identifies information for invoking target tool T; tool execution retrieves context C, from which the LLM infers relationship R and produces result a.Source: §2 · p. 3
3 Automated Task Generation Workflow
TaskCraft generates atomic tasks from tool-derived content, then extends them through sequential dependencies or merged subtasks. It verifies atomic tasks with agent reasoning and uses linguistic checks for extensions.
3.1 Atomic Task Generation
- Atomic tasks are built by extracting a tool input index and content from an unlabeled corpus, then using an LLM to identify answers and relationships and construct a question.The question is conditioned on the input index and inferred relationship.Source: §3.1 · p. 4
3.2 Task Extension
- Depth-based extension creates sequential tool-use dependencies by deriving a new input index from an existing task and integrating the new and historical relationships into a refined question.A search agent retrieves superset candidates to support the next index and relationship; the candidate question is verified before refinement.Source: §3.2 · p. 4
- Width-based extension merges two subtasks into one task, with an LLM rephrasing their question strings.The combined task requires resolving multiple subproblems.Source: §3.2 · p. 5, Figure 4 · p. 5
3.3 Task Verification
- Atomic tasks are checked by comparing an agent’s output with an infer-LLM’s and retaining tasks whose golden answer appears only in the agent’s output; extensions use linguistic validation.This allocates agent reasoning to atomic-task verification while LLM-based checks support faster extension verification and task generation beyond agent capabilities.Source: §3.3 · p. 6
4 Experiments
The experiments examine corpus composition and task difficulty, prompt-learning gains, SFT performance, and the effect of tool context on atomic-task generation. Results show improvements in prompt optimization and model performance, alongside task difficulty variation across modalities.
4.1 Corpus Construction
- Webpages make up 75% of the corpus, images 15%, and PDFs 10%; these sources support tool-specific atomic-task construction.The corpus also includes up-to-date news, financial reports, and academic publications.Source: Figure 5 · p. 6, §4.1 · pp. 6–7
4.2 Synthetic Tasks Analysis
- The evaluation compares Smolagents and Smolagents+ on 1,000 sampled tasks, scoring responses against golden answers on a three-point scale.Scores were 2 for fully correct, 1 for including the golden answer with additional information, and 0 for incorrect responses.Source: §4.2 · p. 7
- Failure rates rise from webpages to PDFs to images within PDFs; image-understanding tasks achieve accuracy comparable to GAIA LEVEL3 data.The authors interpret the results as showing that generated tasks span difficulty levels, including challenges for current agents.Source: §4.2 · p. 7, Table 1 · p. 7
4.3 Enhancing Task Generation Efficiency via Prompt Learning
- 19.2% shorter atomic-task generation time (29.1 to 23.5 seconds) and pass rate rose from 54.9% to 68.1%; depth-wise extension pass rate rose from 41.0% to 51.2%.For depth-wise extension, generation time decreased from 31.5 to 30.2 seconds across six attempts.Source: Table 2 · p. 9, §4.3 · p. 9
4.4 Fine-Tuning Agent Models Using Synthetic Trajectory
- The SFT setup uses 3,202 multi-hop tasks and trajectories and evaluates two Qwen2.5-3B variants on HotpotQA, Musique, and Bamboogle.Training applies content masking to search tool contexts.Source: §4.4 · p. 9
- +14.0% (Qwen2.5-3B-Base) and +6.0% (Qwen2.5-3B-Instruct) average performance improvements over their base workflows; versus Search-R1, gains reach +19.2% on Bamboogle and +6.2% on Musique for Base.Qwen2.5-3B-Instruct maintains an average performance margin of +0.6% over Search-R1.Source: Table 3 · p. 9, §4.4 · pp. 9–10
4.5 Effectiveness of Tool Context in Constructing Agentic Tasks.
- 24.5% higher pass rate (43.0% vs. 18.5%) and 28 seconds less generation time (86.7s vs. 119.7s) than direct GPT-4.1 prompting.Average tool invocations were 2.1 vs. 2.8 per query, and variance in tool usage was 0.4 vs. 1.2.Source: Table 4 · p. 10, §4.5 · p. 10
5 Related Work
Prior work automates instruction-data generation and develops language-agent architectures, but the cited approaches leave challenges around tool use in generated tasks and agent coordination. TaskCraft is situated among these strands of work.
5.1 Instruction Data Generation
- Self-Instruct [24], Self-Chat [28], NuminaMath, and OpenMathInstruct-2 [19] generate data from few seeds, but struggle to extend generation to multiple tool invocations.WizardLM [27] relies primarily on rule-based modifications, while MetaMath [34] rewrites questions and WebInstruct [35] questions often lack tool utilization in their solutions.Source: §5.1 · p. 10
5.2 Language Agent
- Language-agent research includes role-specialized systems [26] [6], modular agent hierarchies, and systems that separate multimodal perception, planning, and execution [2].Other cited work explores real-time strategy adaptation, asynchronous coordination, and agent customization [17].Source: §5.2 · pp. 10–11
- The review reports that stabilized sub-agent interactions can yield higher task success rates than complex, centralized orchestration algorithms.The passage also notes cross-modal processing bottlenecks as an open challenge.Source: §5.2 · p. 11
6 Conclusion
TaskCraft automates scalable, multi-tool, verifiable agentic task generation using depth- and width-based extensions. The paper reports gains in prompt optimization and SFT and releases a dataset of approximately 36,000 tasks.
6 Conclusion
- TaskCraft is an automated workflow for scalable, multi-tool, verifiable agentic task generation.The paper reports that it reduces reliance on human annotation.Source: §6 · p. 11
- Depth-based and width-based extensions construct hierarchically complex challenges.Source: §6 · p. 11
- The released synthetic dataset contains approximately 36,000 tasks of varying difficulty to support research on agent tuning and evaluation.The paper also reports improved prompt optimization and supervised fine-tuning.Source: §6 · p. 11
7 Limitation
The current work focuses on atomic tasks for common tools, with agent-specific tool requirements identified as a future extension.
7 Limitation
- Current atomic-task construction focuses on browsing, PDF processing, and image analysis.Source: §7 · p. 11
- The paper identifies generating atomic tasks tailored to agents’ specific tool requirements as a future direction.Source: §7 · p. 11
- The stated scope is common-tool task construction rather than agent-tailored tool requirements.Source: §7 · p. 11
A Data Statistics
The section describes how task generation changes with hop depth across domains and how atomic tasks are sourced from web, PDF, and image data. It also identifies two verification failures in depth-based extension: false hierarchy and answer leakage.
A Data Statistics
- PDF tasks retain the strongest deep-hop share, while image tasks show the sharpest decay and HTML has the most deep extensions.PDF tasks fall from 33.62% (2,737 tasks) at 1 hop to 22.36% (1,820 tasks) at 2 hops and 18.60% (1,514 tasks) at 3 hops; 5–7-hop tasks are 11.80% combined. Image tasks have 8.64% (467 tasks) at 5–7 hops, while HTML has 4.75% (1,089 tasks) at 5–7 hops.Source: A Data Statistics · p. 16, Figure 8 · p. 16
- The 22,053 atomic tasks draw chiefly on web sources, with image- and PDF-based extraction also contributing.Web-based conclusions make up 77.78%; image-based tools contribute 9.80%, and PDF extraction accounts for 12.41%. Web sources include academic (25.42%), financial (21.58%), cultural (8.09%), economic (6.45%), and governmental (6.08%) resources.Source: A Data Statistics · pp. 16–17, Figure 9 · p. 16
B Verification Requirements for Depth-Based Extension
- Depth extension must preserve a valid superset relationship, but unverified input substitutions can create pseudo-supersets or leak answers.Both flaws undermine the intended multi-hop reasoning process.Source: B Verification Requirements for… · p. 17
B.1 Pseudo-Superset Problem
- A semantically equivalent index is not a valid extension unless it is a genuine superset that makes the added tool-use step necessary.The travel-report example changes the title but does not introduce a necessary retrieval step, so the expansion is superficial.Source: §B.1 · p. 17
B.2 Information Leakage
- An extended query can conceal the prior tool input yet still leak the answer, letting the LLM bypass retrieval.The merger-value example directly states the answer, compromising the tool dependency required for multi-hop answering.Source: §B.2 · p. 18
B.3 Verification for Task Extension
The verification framework checks depth-based extensions for genuine hierarchy and concealed answers, with explicit relations supporting progression and tool use. The accompanying prompts specify verification and task-construction criteria, including answer consistency, uniqueness, and concealment.
B.3 Verification for Task Extension
- The framework verifies both the new tool index and extended query to ensure valid task extensions.Its stated advantages include hierarchical progression, mandatory tool use, and explicit relation paths.Source: §B.3 · p. 18, §B.4 · p. 18
B.3.1 Strict Superset Verification
- A new tool index must be a strict superset of the prior index, linked by a genuine hierarchical relation rather than a synonym.The prose gives “contains” and “part_of” as relation examples and says extensions that let the LLM derive the answer directly should be excluded.Source: §B.3.1 · p. 18
B.3.2 Information Leakage Verification
- An extended query must not reveal the original answer, and queries that let the LLM obtain it directly are filtered out.This information-sealing requirement preserves proper tool-use reasoning.Source: §B.3.2 · p. 18
B.4 Advantages of the Verification Framework
- The framework claims superset integrity, strict tool dependency, and transparent reasoning through explicit relation paths.Its example progresses from “Sports In Brief” to the AP News sports section through a contains relation.Source: §B.4 · pp. 18–19
C.1 Atomic task verification
- Atomic-answer verification scores semantic consistency, while retention requires the agent to outperform the LLM and give a non-zero answer.The rubric distinguishes full, partial, and inconsistent answers; retained tasks must satisfy both stated conditions.Source: §C.1 · p. 19
C.2 optimized prompts
- Optimized prompts require a more complex but unique answer-preserving query, with the golden answer concealed and wording polished without changing meaning.Other prompt criteria cover superset retrieval and relation articulation, atomic verifiable conclusions, and identifying a minimal unique superset.Source: §C.2 · pp. 20–22
C.3 Strict Superset Verification
- Strict-superset prompt checks require proper containment and an explicit relation that uniquely links the superset to its subset.Synonym pairs such as Car/Automobile are excluded, as are many-to-one mappings.Source: §C.3 · p. 22
D Further Training Detail
The training details describe SFT and RL setups and a modified Smolagents system. SFT uses multi-hop tasks and trajectories, while the RL setup fixes retrieval depth for evaluation.
D Further Training Detail
- SFT uses 3,202 multi-hop tasks with trajectories and masks search-tool contexts.The RL setup uses a 2018 Wikipedia dump, an E5 retriever, and a fixed retrieval depth of 3 passages for all methods; it merges NQ and HotpotQA training sets and evaluates Exact Match across three datasets.Source: D Further Training Detail · pp. 22–23
E Smolagents+
- Smolagents+ enhances web search, integrates multiple information sources, streamlines results, and uses query rewriting.Source: E Smolagents+ · p. 23