PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, Xiangyang Ji

arXiv:2606.11119v1cs.LGcs.AIcs.CL

TL;DR

RLVR rollout allocation is challenged by uneven reward informativeness across prompts and turns in multi-turn agentic tasks. TRACE allocates branches to roots and prefixes predicted to yield mixed outcomes, improving performance across evaluated benchmarks and models, including a 2.8-point Multi-Hop QA gain at equal sampling cost.

  • Problem

    Prior RLVR allocation methods focus on prompt-level informativeness and neglect variation in prefix-level informativeness across turns, limiting fine-grained budget allocation for multi-turn rollouts.

  • Method

    TRACE uses a shared predictor to allocate fixed rollout budgets globally across prompt roots and locally across intermediate prefixes likely to produce mixed terminal rewards.

  • Results

    TRACE improves average performance across Mathematical Reasoning, Multi-Hop QA, and Function Calling for Qwen3-8B and Qwen3-14B under the same rollout budget, while achieving higher effective ratios than baselines.

  • Takeaways & Limitations

    TRACE provides a unified contrast-seeking allocation strategy that enriches outcome-only feedback and amplifies implicit credit signals for sample-efficient policy optimization.

  • Takeaways & Limitations

    TRACE is mainly developed for outcome-only RLVR and is evaluated primarily on three task settings with Qwen3-8B and Qwen3-14B, leaving more complex non-stationary scenarios unexplored.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models. However, rollout-intensive policy optimization is often limited by insufficient reward contrast, arising when overly simple or complex prompts generate low-variance feedback and when outcome-only rewards assign the same terminal assessment to every decision in a multi-turn rollout. Past efforts have focused on allocating available rollout resources to promising prompts, yet they only leverage sample informativeness at the prompt level and neglect variation in prefix-level informativeness across turns within the same rollout. This work targets multi-turn agentic RL by modeling each ReAct-style thought-action-observation turn as a semantically distinct node, allowing budget allocation to extend from prompt roots to turn-level prefixes with further continuations, which naturally forms tree-structured rollouts. We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework that enhances reward contrast within a fixed sampling budget. Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards. A shared generalizable predictor estimates conditional success probability at these anchors from prefix histories to guide this allocation. The resulting adaptive tree structure enriches outcome-only feedback and amplifies the policy-update signal. Empirically, TRACE achieves competitive performance and efficiency gains on typical agentic benchmarks, e.g., improving Qwen3-14B Multi-Hop QA average accuracy by 2.8 points over competitive baselines at equal sampling cost.

1 Introduction

TRACE frames efficient multi-turn RLVR as allocating a fixed rollout budget to prompt roots and turn-level prefixes likely to produce mixed successful and failed outcomes. By organizing continuations as a tree and using shared allocation utilities, it enriches sparse outcome-only feedback and supports contrastive exploration.

  • Motivation: RLVR rollouts are computationally expensive because they require long Chain-of-Thought generation and interleaved environment interactions.
  • Problem: Outcome-only rewards provide uneven, sparse learning signals: overly easy or hard prompts yield low-variance outcomes, while terminal rewards weaken multi-turn credit assignment.
  • Tree-structured allocation: ReAct-style rollouts expose thought-action-observation turns as semantically meaningful prefixes that can serve as branching points for extra exploration.
  • TRACE: TRACE unifies root allocation and prefix branching by distributing budget to tree anchors whose descendants are likely to contain both successful and failed outcomes.
  • TRACE: TRACE redirects fixed-budget rollouts toward contrast-rich roots and prefixes, producing denser mixed-reward contrast and implicit stepwise preference pairs than uniform allocation.

2 Preliminaries

TRACE models multi-turn ReAct rollouts as trees whose root is the prompt and whose nodes are complete thought-action-observation turns with histories defined by accumulated prefixes. Under binary terminal RLVR rewards, rollout allocation can operate at both prompt roots and visited nonterminal prefixes, using further continuations as branching anchors.

  • ReAct rollout representation: Each ReAct turn is represented as nt := ⟨τt, at, ot⟩, with history Ht := (x, n1, . . . , nt) and initial history H0 := x.The policy generates thought-action pairs conditioned on the prior history, and the environment produces the observation.
  • Terminal reward formulation: RLVR evaluates each complete terminal history HT with a binary reward r(HT) ∈ {0, 1}.The terminal history identifies the complete rollout and defines the training signal.
  • Initialize-then-expand construction: Tree rollout allocation first assigns each prompt xi a root rollout count mi and samples mi complete bare rollouts in Stage 1.For rollout j, Hi,j,t denotes the prefix after turn t, while Ti,j denotes its terminal turn index.
  • Initialize-then-expand construction: Every visited nonterminal prefix Hi,j,t can anchor Stage 2 branching, where Ki,j,t continuations generate terminal suffixes conditioned on that prefix.The original suffix is factual, while newly sampled suffixes are counterfactual branches.

3 Rollout Allocation through the Lens of Contrast Construction

This section frames rollout allocation as contrast construction at both prompt roots and intermediate prefixes. Prefix histories improve prediction of downstream outcome variation, enabling concentrated sampling where continuations are most likely to produce informative success–failure comparisons and stronger update signals.

  • Prefix information and prediction: Prefix-level scoring is at least as informative as prompt-only scoring and becomes strictly better once intermediate turns are observed.The cases t = 0 and m = 1 recover prompt-level difficulty prediction and single-rollout reward prediction.
  • Unified contrast-seeking allocation: At both prompt roots and visited prefixes, TRACE targets anchors with intermediate conditional success probability because near-deterministic anchors provide little outcome contrast.Roots seek diverse fresh-rollout groups, while prefixes seek disagreement among continuations sharing the same history.
  • From outcome rewards to local contrast: Shared-prefix branches turn opposite terminal outcomes into local comparisons, while squared revisions of conditional success accumulate remaining suffix uncertainty.Proposition 2 identifies the conditional expected quadratic variation with Bernoulli variance, linking forecast movement to downstream contrast potential.
  • Contrastive-allocation diagnostics: A small subset of ranked anchors captures much of the total pair contrast, whereas many anchors near empirical success rates 0 or 1 offer scarce contrast.These diagnostics use Qwen3-8B under Multi-Hop QA tree sampling.
  • Optimizer-visible gradient activation: TRACE allocation strengthens each stage’s expected squared local gradient signal over uniform allocation and strengthens the combined tree update when gradient sources do not cancel.The activation event requires sampled descendants below an anchor to contain both outcomes.

4 TRACE

TRACE allocates rollout budget globally across prompt roots and locally across visited prefixes using a shared predictor of conditional success probability. The resulting rollout trees provide recursive supervision and root-/prefix-level comparisons for tree-aware policy optimization.

  • Allocation: TRACE jointly allocates budget to candidate prompt roots and visited prefix occurrences likely to produce mixed terminal rewards.Root allocation selects prompts, while local expansion resamples continuations after selected nonterminal prefixes.
  • Root allocation: Prompts with zero rollouts are skipped, whereas active prompts receive at least two bare rollouts for group-based tree-aware policy optimization.The allocation set excludes one rollout per prompt; any count mi ≥2 activates the prompt.
  • Optimization and execution: TRACE solves both allocation problems with efficient dynamic programming and passes completed trees to optimizers that propagate leaf rewards through prefixes.Supported credit rules include process-reward backups and group-relative backups; expansion can begin after each prompt’s bare rollouts return.
  • Value prediction: A shared generalizable predictor estimates conditional success probability from prefix histories at both prompt roots and visited prefixes.The predictor guides allocation and is used only by the allocator, not by the downstream policy optimizer.
  • Recursive targets: Rollout trees train the predictor with bottom-up empirical success-rate targets computed from binary terminal rewards and their executed descendants.Supervision covers roots and informative internal nodes in a mean-squared regression objective.

5 Experiments

TRACE is evaluated across mathematical reasoning, multi-hop question answering, and function calling with Qwen3-8B and Qwen3-14B under matched rollout budgets. It consistently improves performance and effective reward contrast over prompt-level and tree-structured baselines, while both allocation stages and budget shape contribute to gains.

  • Experimental setup: TRACE evaluates mathematical reasoning, Multi-Hop QA, and Function Calling with Qwen3-8B and Qwen3-14B, comparing against ReAct, GRPO, PCL, and TreePO.Metrics are final-answer accuracy, HotpotQA exact match and token-level F1, and BFCL success rate.
  • Overall results: 71.1 versus 70.0: TRACE raises Qwen3-8B in-distribution Mathematical Reasoning average over GRPO; 74.9 versus 73.5 for Qwen3-14B.These results are reported on DeepScaler under the same rollout budget.
  • Contrast effectiveness: 60.6% versus 26.8%: TRACE more than doubles the Qwen3-8B Mathematical Reasoning effective ratio over GRPO.Across all three settings and both model scales, TRACE achieves a higher training-time average effective ratio than GRPO, PCL, and TreePO under matched budgets.
  • Predictor diagnostics: The predictor ranks difficulty at both prompt and internal-prefix levels, enabling turn-level branching despite training dominated by prompt-level supervision.This indicates that prefix histories contain a learnable notion of local uncertainty useful for allocation.
  • Ablations and budget compatibility: Both root and prefix allocation stages improve Qwen3-8B Multi-Hop QA, with stacking gains; TRACE also improves TreePO at every tested budget.Across budgets, it lifts effective ratio by about 9.5–10.2 points and gains roughly one accuracy point; at budget 2048, broader root coverage is stronger than deeper continuation sampling.

6 Conclusion

TRACE unifies prompt filtering, rollout-count allocation, and prefix branching as budget decisions over rollout-tree anchors for sample-efficient multi-turn RLVR. It uses a shared predictor of conditional success probability to target anchors likely to reveal mixed terminal outcomes and strengthen group-relative updates.

  • TRACE formulates mixed-reward contrast allocation as a unified framework for sample-efficient RLVR in multi-turn agents.
  • Prompt filtering, rollout-count allocation, and prefix branching become budget decisions over rollout-tree anchors.
  • TRACE guides allocation with a shared generalizable predictor of conditional success probability.
  • TRACE prioritizes anchors whose descendants are likely to expose mixed terminal outcomes, strengthening the group-relative update signal.

Limitations … B.1 Allocation as an Orthogonal Layer

TRACE is presented as an allocation layer that enriches rollout contrast before policy optimization, while remaining adaptable in predictor and tree-policy choices. The appendix situates this design among related work, supplementary analyses, and limitations of outcome-only verification and predictor instantiation.

  • Limitations: TRACE is mainly developed for outcome-only RLVR, so tasks without clear terminal verification may require revisiting its mixed-outcome contrast objective.The work also identifies stronger predictors for different agentic tasks as future work and evaluates mainly in a constrained setting.
  • Appendix Overview: The appendix covers related work, discussion topics, extended benchmark and diagnostic results, theoretical proof, experimental details, and representative data examples.Its discussion includes allocation as an orthogonal layer, training-time sampling axes, and root-to-prefix predictor generalization.
  • A Related Work: Prior sample-efficient RLVR methods select rollout-worthy examples using prompt-level properties such as difficulty, diversity, length, or estimated solution quality.The passage notes that offline curation can reduce training cost but adds preprocessing and remains static as the policy evolves.
  • A Related Work: Training-time allocation methods differ in structure: GRPO and PCL use flat rollouts, TreePO uses random branching, and TRACE learns allocation at prompt roots and visited prefixes.This comparison is presented qualitatively in Table 3.
  • A Related Work: TRACE repurposes tree search as an allocation substrate over agent interaction prefixes, rather than as an inference solver, to discover reward contrast under a fixed rollout budget.This distinguishes it from test-time tree-search methods that spend additional computation to find stronger answers.
  • A Related Work: Sparse outcome rewards are coarse for multi-turn agents because one terminal assessment can be assigned to many earlier decisions with sharply different responsibility.The related work discusses exploration, replay, goal relabeling, reward shaping, and value-based credit estimates as classical responses.
  • B.1 Allocation as an Orthogonal Layer: TRACE allocates descendants before optimization, allowing the same policy objective to receive either contrast-poor or contrast-rich trees under a fixed rollout budget.It separates rollout acquisition from policy optimization and operates upstream of tree-structured objectives such as TreeRPO and Tree-GRPO.
  • B.1 Allocation as an Orthogonal Layer: TRACE requires only an anchor-ranking predictor and a tree-structured optimizer, rather than a specific predictor architecture or tree-policy objective.The predictor and TreePO instance used experimentally are described as practical instantiations, while better predictors may improve budget placement.

B.2 Axes of Training-Time Sampling · B.3 Root-to-Prefix Predictor Generalization · C Extended Experimental Results

TRACE expands training-time sampling from prompt-level choices to structural allocation within rollouts, while using one predictor to generalize from roots to prefixes. Extended experiments report consistent outperformance over competitive baselines across mathematical reasoning, multi-hop QA, and function-calling evaluations.

  • B.2 Axes of Training-Time Sampling: TRACE adds an internal rollout allocation axis beyond flat sampling, learned prompt admission, and random tree acquisition.GRPO exposes a flat axis, PCL learns prompt admission, and TreePO provides a tree substrate with random acquisition.
  • B.2 Axes of Training-Time Sampling: The missing sampling decision occurs inside the rollout after interaction with search, code-execution, or API environments.The passage identifies structural sampling inside trajectories that have interacted with these environments.
  • B.3 Root-to-Prefix Predictor Generalization: A shared predictor applies to roots and prefixes because each prefix history refines the prompt with the current plan, actions, and environment observations.This information-refined representation captures what remains possible after the observed interaction history.
  • C Extended Experimental Results: TRACE consistently outperforms competitive baselines across all metrics on mathematical reasoning evaluations.Table 4 covers Qwen3-8B and Qwen3-14B across four in-distribution and three out-of-distribution benchmarks, trained on DeepScaler.
  • C Extended Experimental Results: TRACE consistently outperforms competitive baselines across all metrics on multi-hop QA and function-calling evaluations.Table 5 evaluates Qwen3-8B and Qwen3-14B on four multi-hop reasoning benchmarks and four tool-calling scenarios, using HotpotQA and BFCL v4 for training.
  • B.3 Root-to-Prefix Predictor Generalization: Root supervision anchors a global difficulty scale, whereas sparser prefix supervision is noisier but closer to decisions that create local contrast.Roots aggregate more terminal descendants and produce more reliable tree-backed targets; prefixes have fewer descendants and noisier targets.
  • B.3 Root-to-Prefix Predictor Generalization: Predictor updates use all prompt-root examples and only a small stream of prefix examples to improve training efficiency.The update strategy combines dense root-level supervision with limited prefix-level supervision.

C.1 Detailed Benchmark Results · C.2 Effective Ratio · C.3 Spearman Correlation

TRACE improves RLVR benchmark performance and produces more informative rollout groups under equal rollout budgets. Its predictor learns stable difficulty rankings at both prompt-root and prefix levels, supporting allocation by relative ordering.

  • C.1 Detailed Benchmark Results: TRACE improves average scores over GRPO by 0.7–2.8 points across Mathematical Reasoning, Multi-Hop QA, and Function Calling.It also consistently improves reported averages over random TreePO allocation.
  • C.2 Effective Ratio: The effective ratio measures the fraction of sampled terminal descendants attached to activated anchors.For prompt roots, it is instantiated as the fraction of groups containing both successful and failed outcomes.
  • C.3 Spearman Correlation: The diagnostics cover Mathematical Reasoning, Multi-Hop QA, and Function Calling for Qwen3-8B and Qwen3-14B at both prompt and prefix levels.Figures 6 and 7 report rank correlations with empirical success rate or empirical continuation success, respectively.
  • C.2 Effective Ratio: Mixed successful and failed outcomes increase the chance of non-degenerate group-relative or local contrast.Both root allocation and useful prefix expansion can raise the chance that a prompt produces mixed outcomes.
  • C.2 Effective Ratio: TRACE consistently produces more informative prompt-root groups than GRPO and random TreePO allocation.Higher effective-ratio values indicate denser group-relative signal under the same rollout budget.
  • C.3 Spearman Correlation: Spearman’s rank correlation evaluates monotonic agreement between predicted and empirical difficulty rankings, making it suitable when allocation depends on relative ordering.The metric is invariant to monotonic transformations, and its associated p-value assesses statistical significance under independence.
  • C.3 Spearman Correlation: The predictor learns stable difficulty rankings at both prompt and prefix levels during training.Prompt-level correlation rises first because roots provide denser supervision, while prefix-level correlation remains positive despite limited prefix-level supervision.

C.4 Applicability to Other Model Families … D.2 Proof of Proposition 2

TRACE transfers beyond Qwen3, with gains across multiple benchmarks, while adding limited predictor overhead and concentrating allocation on selected roots and informative prefixes. The theoretical appendices establish variance-reduction results for prefix conditioning and prove the stated binary-reward identity using martingale arguments.

  • C.4 Applicability to Other Model Families: 3.8 points over GRPO and 3.1 points over TreePO: TRACE improves Llama-3.2-3B-Instruct average accuracy across four Multi-Hop QA benchmarks.The gains appear on HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.
  • C.4 Applicability to Other Model Families: TRACE’s advantage emerges during training, while its predictor preserves useful prompt- and prefix-level rankings across policy backbones.The allocation diagnostics also show a higher effective ratio, supporting TRACE as a portable root-prefix allocation layer.
  • C.5 Computational Overhead: Figure 10 measures runtime with nonoverlapping critical-path timers, separating predictor scoring and updates to avoid double counting nested diagnostics.The breakdown covers LLM rollout, policy update, predictor scoring, predictor update, and residual overhead.
  • C.5 Computational Overhead: 66.5% and 75.4% of runtime go to policy optimization for Qwen3-8B and Qwen3-14B, respectively.Predictor updates account for 1.3% and 0.9%, while predictor scoring accounts for 1.9% and 1.4%.
  • C.6 Allocation Behavior: Stage 2 prefix allocation is analyzed using both relative position within bare rollouts and absolute turn depth, with budget weighted by continuation counts.Function Calling’s longer episodes make turn-wise prefix allocation smoother than in shorter QA rollouts.
  • C.6 Allocation Behavior: Most candidate prompts receive mi = 0 and are skipped, while active prompts typically receive around five to seven bare rollouts.Stage 1 root allocation is summarized over the Function Calling candidate prompt pool.
  • D.1 Proof of Proposition 1: The Proposition 1 proof uses conditional Bernoulli rewards and total variance to show that conditioning on informative next prefixes strictly improves the relevant risk.Strictness holds exactly when the next prefix changes conditional success probability with positive probability.
  • D.2 Proof of Proposition 2: The Proposition 2 proof applies the Doob martingale property and the tower property to eliminate the conditional second term.It then uses the binary terminal reward, ZT = r(HT) ∈{0, 1}, to prove Eq. (9).

D.3 Proof of Proposition 3 … E.2 Models

The proof shows that TRACE’s rollout allocation improves expected gradient energy by targeting anchors whose descendants are likely to contain contrasting outcomes. The experimental setup evaluates this framework across mathematical reasoning, multi-hop QA, and multi-turn function-calling tasks with specified datasets, rewards, protocols, and model backbones.

  • D.3 Proof of Proposition 3: Activation occurs when sampled descendants below an anchor contain at least one successful and one failed terminal history.For roots, activation is non-degeneracy; for visited prefixes, it requires an opposite-outcome sibling among new continuations.
  • D.3 Proof of Proposition 3: Without activation, pairwise, group-relative, and tree-aware optimizers have zero local contrast contribution below the anchor.All-success or all-failure groups provide no success–failure contrast, so their local gradient contribution vanishes.
  • D.3 Proof of Proposition 3: The expected squared local gradient norm is bounded by c−Va(b) and c+Va(b) when conditional squared gradient norms lie in [c−, c+].Va(b) is the activation probability conditioned on the anchor, with roots and prefix anchors using their respective conditioning histories.
  • D.3 Proof of Proposition 3: TRACE optimizes stage-wise activation probability over feasible rollout allocations, and its combined tree update is no worse than uniform under Proposition 3’s cross-term condition.Strict improvement holds for any stage where uniform allocation is not an optimizer of Jq.
  • E.1.1 Mathematical Reasoning: The mathematical-reasoning experiments train on 40,315 DeepScaler competition-level mathematics problems and evaluate AIME24, AMC23, MATH500, MinervaMath, and OlympiadBench.Scores use avg@32 for AIME24, avg@16 for AMC23, and avg@4 for MATH500, MinervaMath, and OlympiadBench; trajectories use a Python-enabled tool-augmented agent.
  • E.1.2 Multi-Hop QA: The multi-hop QA experiments train on HotpotQA with deterministic local E5 retrieval and evaluate HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.The HotpotQA-style reward gives exact matches reward 1, partial credit equal to token-level F1 when F1 is at least 0.3, and reward 0 below that threshold.
  • E.1.3 Function Calling: Function-calling evaluation uses BFCL v4’s four agentic multi-turn subsets—base, long-context, missing-function, and missing-parameter—with an 80% training portion and a 20% held-out test portion.Validation reports subset success rates as avg@4, while Avg is the arithmetic average across the four subsets and requires both the executed API sequence and final outcome to satisfy the evaluator.
  • E.2 Models: Main experiments use Qwen3-8B and Qwen3-14B policy backbones, with Llama-3.2-3B-Instruct additionally used for Multi-Hop QA and Qwen3-0.6B serving as TRACE’s prefix predictor.The ReAct baseline evaluates the corresponding base or instruction-tuned model in the same multi-turn scaffold without RL fine-tuning.

E.3 Training Details … E.6 Prefix Predictor Supervision and Implementation

The paper specifies task-specific training and rollout settings, tree-aware optimizers, adaptive budget accounting, and an online prefix predictor that guides both root and prefix allocation. TRACE changes allocation while keeping the optimizer family and total sampling cost comparable to baselines.

  • E.3 Training Details: Training uses task-specific prompt, response, and agent-turn limits, with Function Calling allowing up to 40 turns for long tool-use traces.Multi-Hop QA uses 2048 prompt tokens, 6144 response tokens, and 5 turns; Mathematical Reasoning uses 2048, 14336, and 5.
  • E.3 Training Details: Rollout sampling follows benchmark-specific settings: Multi-Hop QA uses temperature 1.0 and top-p = 1.0, while the other tasks use distinct temperatures and top-p values.Mathematical Reasoning uses temperature 0.6 and top-p = 0.95; Function Calling uses temperature 0.9 and top-p = 1.0.
  • E.4 Policy Optimizers and TreePO Instances: Tree training uses TreeRPO for Mathematical Reasoning and Multi-Hop QA, and Tree-GRPO for Function Calling, on rollout-tree batches.Both optimizers operate on tree-structured data sampled under the initialize-then-expand construction.
  • E.4 Policy Optimizers and TreePO Instances: TRACE changes only the allocation policy, while TreePO retains the same tree-aware optimizer family with random root and prefix allocation.TRACE determines where tree branches are collected rather than replacing the optimizer family.
  • E.5 Rollout Construction and Budget Accounting: TRACE allocates integer root counts over the full candidate pool instead of fixing the number of selected prompts in advance.In Multi-Hop QA, the root budget is M = 1024 units and N = 2 continuations; the active-prompt fraction does not exceed fixed-size baselines.
  • E.5 Rollout Construction and Budget Accounting: TRACE approximates each continuation as one half trajectory, using approximately M(1 + N/2) trajectory units to match flat baselines with PG units.Budget matching chooses (M, N) such that M(1 + N/2) ≈ PG.
  • E.6 Prefix Predictor Supervision and Implementation: The Qwen3-0.6B prefix predictor scores serialized prompt and interaction histories in [0, 1] and guides both root and prefix allocation stages.The predictor extends a lightweight value-model setup from prompt roots to serialized prefix histories.
  • E.6 Prefix Predictor Supervision and Implementation: The predictor is updated online with MSE targets, using prefix samples for 6% of each predictor batch and task-specific learning rates.Learning rates are 3e−5 for Multi-Hop QA and Function Calling, and 1.5e−5 for Mathematical Reasoning.

F Data Examples

The section presents representative rLLM data examples spanning mathematical reasoning, multi-hop question answering, and function calling, together with their agent system prompts and tool interfaces. These examples show task-specific instructions, tool schemas, and expected interaction formats used by the training scripts.

  • Mathematical Reasoning (DeepScaler) Example: Mathematical reasoning uses a Python-enabled system prompt with a sandboxed execution function requiring code and supporting a 12-second timeout.The example asks for an operation evaluation and gives the ground-truth answer as -2/3.
  • Interaction Format: Across the examples, function calls are returned as JSON objects containing a function name and arguments inside <tool_call></tool_call> XML tags.This interaction format is explicitly specified for both the mathematical reasoning and multi-hop QA tool interfaces.
  • Multi-Hop QA (HotpotQA) Example: Multi-hop QA instructs the assistant to search reliable information, synthesize multiple sources, and place final answers in boxed format.Its representative prompt asks whether Scott Derrickson and Ed Wood shared the same nationality.
  • Function Calling (BFCL v4) Example: Function-calling examples provide trading-system tools for watchlists, orders, stock filtering, account funding and information, available stocks, and current time.The tool schemas specify typed parameters, required fields, and structured response properties; the system also says to use only the latest tool list.
Loading 2606.11119v1…