Source-linked AI summary
SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models
Yuxuan Jiang, Francis Ferraro
TL;DR
Tool-using agents remain difficult to train because multi-step credit assignment and coarse LLM judging produce noisy intermediate rewards. SCRIBE routes subgoals to skill prototypes that provide structured verification at a mid-level abstraction. Across mathematical reasoning and tool-use benchmarks, it improves performance and finds that mid-level mastery precedes effective high-level planning.
Problem
Multi-step tool use suffers from unreliable credit assignment because LLM judges may overlook flawed intermediate behavior or penalize harmless execution errors.
Method
SCRIBE routes trajectory subgoals to curated Skill Prototypes, whose structured rubrics calibrate LLM-based reward evaluation at a mid-level abstraction.
Results
SCRIBE consistently outperforms prior approaches across mathematical reasoning and tool-use settings, including AIME25 accuracy rising from 51.7 (PRM) to 63.3 and BFCL overall score reaching 51.3.
Takeaways & Limitations
Mid-level skill mastery precedes strategic high-level planning, and SCRIBE complements low-level tool optimization with synergistic gains.
Takeaways & Limitations
The approach has been evaluated mainly on small- to mid-sized instruction-tuned models and structured reasoning and tool-use tasks, leaving scaling and open-ended generation uncertain.
Abstract
from arXiv · showhide
Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.
1 Introduction
SCRIBE addresses unreliable credit assignment in multi-step tool use by supervising a mid-level abstraction between planning and execution. It uses skill prototypes to structure reward evaluation and reports gains across reasoning and tool-use benchmarks.
- Tool-augmented agents struggle with credit assignment because intermediate tool selection, invocation, and result integration can fail.
- LLM judges can reward flawed trajectories or penalize harmless tool-call errors when no rubric separates strategic planning from technical execution.
- SCRIBE routes subgoals to skill-specific prototypes, grounding open-ended LLM judging in structured mid-level verification.
- The resulting reward scheme reduces reward variance and provides dense signals grounded in specific reasoning behaviors.
- 43.3% to 63.3%: SCRIBE improves Qwen3-4Binstruct-2507 accuracy on AIME25, while BFCL Multi reaches a 33.3% success rate.
- Training analysis finds mid-level skill mastery preceding strategic high-level planning, and SCRIBE complements low-level tool optimization.
2 Related Works
Prior work spans tool use, reward modeling, decomposition, and skill discovery, but largely emphasizes low-level execution rather than structured mid- and high-level supervision. SCRIBE targets this underexplored connection between subgoal skills and broader reasoning.
- Tool-use research has progressed from supervised invocation of external tools toward richer execution accuracy and greater action-level autonomy.
- Existing methods mainly focus on low-level tool behaviors, leaving plan-level discrimination and high-level decision structure comparatively underexplored.
- Progress reward models evaluate intermediate steps, but LLM-based judges can provide inconsistent signals for agentic reinforcement learning.
- Decomposing problems into subgoals can improve difficult-task performance, with each subgoal typically corresponding to a dominant reusable reasoning skill.
- Skill-based decomposition supports consistent identification of reasoning patterns, motivating investigation of whether stronger tool-related skills improve subgoal and overall reasoning performance.
3 Method: Decomposition, Clustering, prototype
SCRIBE builds a mid-level semantic bridge by extracting subgoals and skills, clustering recurring reasoning patterns into prototypes, and using prototype-conditioned verification for policy optimization. Its router and calibration procedure turn trajectory evaluation into structured skill-level rewards.
- Framework: SCRIBE decomposes training into skill-level abstraction, structured reward evaluation, and GRPO-based policy optimization.
- Decomposition: The Router maps trajectories into non-overlapping ⟨subgoal, skill, step⟩ triples that connect high-level plans with low-level execution.
- Clustering and prototypes: Prototype construction clusters semantically similar reasoning patterns with HDBSCAN or fallback K-means, then abstracts each cluster into a reusable Skill Prototype.
- Structured verification: Prototype rubrics convert open-ended judging into checklist verification of intermediate objectives and common traps, such as boundary leaks.
- Reward evaluation: The Router assigns subgoals to prototypes, while a judge scores them from 0 to 3 and calibration combines prompt variants, accuracy statistics, and repeated anchor evaluations.
- Optimization: The policy uses skill-conditioned GRPO rewards, and the prototype library is refreshed every 1,000 training steps to track evolving trajectories.
4 Experimental Setup
The experiments train and evaluate SCRIBE with small instruction-tuned policy models, LLM-based judging, clustered skill spaces, and GRPO rewards across mathematical reasoning and tool-use benchmarks. The setup includes router validation and a fixed process-to-outcome reward mixture.
- Models: Experiments use Qwen3-4B-Instruct-2507 as the primary policy and LLaMA-3.2-3B-Instruct as an additional generality model, with GPT-5-mini providing process rewards.
- Training data: Training uses approximately 10k problems from MATH and ToolACE for skill annotation, Router training, and GRPO policy optimization.
- Skill space: Clustering converges to 424 mathematical-reasoning clusters and 503 BFCL-style tool-use clusters after periodic refreshes.
- Router evaluation: The Router achieves 98.6% held-out accuracy in skill-prototype retrieval using a 9:1 train–test split.
- Reward composition: Process-level and final-answer rewards are combined with weights of 0.3 and 0.7, respectively.
- Evaluation: Evaluation covers MATH500, AIME25, and BFCL v4 using official scripts, with pass@1 averaged over eight independent runs; BFCL version differences may affect comparisons.
5 Main Result
SCRIBE consistently outperforms prior approaches across mathematical reasoning and tool-use benchmarks. On Qwen3-4B-Instruct-2507, it improves MATH500 and AIME25 accuracy while achieving the highest reported BFCL score.
- 95.8 MATH500 accuracy, improving over 92.3 with PRM and 90.2 with EGPO on Qwen3-4B-Instruct-2507.
- 63.3 AIME25 accuracy, improving from 51.7 with PRM on Qwen3-4B-Instruct-2507.
- 51.3 BFCL overall score, outperforming EGPO at 48.3 and PRM at 44.6 across single-step and multi-step tool-use scenarios.
- SCRIBE further improves upon PRM by +3.5 on MATH500 and +6.7 on BFCL Overall.
- For LLaMA 3.2-3B-Instruct, MATH500 accuracy rises from 48.3 with PRM to 63.4, while BFCL Overall rises from 24.8 to 30.8.
6 Research Questions and Ablation Studies
The paper studies whether mid-level execution improvements support high-level planning and whether skill supervision complements low-level tool optimization. It also evaluates structural training dynamics and reward-weighting choices.
- RQ1: Emergent high-level planning: Mid-level execution is measured through subgoal success and uncertainty across repeated rollouts, with lower uncertainty indicating more consistent execution.Mid-level Success is macro-averaged over 64 trials.
- RQ1: Emergent high-level planning: High-level plan selection measures whether viable plans are ranked above non-viable plans using an execution-verified, AUC-style discrimination score.
- RQ1: Emergent high-level planning: Plan separability increases as execution becomes more reliable, enabling clearer discrimination between viable and non-viable plans.
- RQ1: Emergent high-level planning: Stabilization of mid-level execution precedes increases in plan separability and delayed rises in high-level plan selection ability without direct planning supervision.
- RQ2: Complementarity with low-level optimization: FunRL improves execution-level reliability from 21.5 to 25.2, while mid-level supervision reaches 30.8 and their combination reaches 33.4.
- RQ3: Reward weighting: A process-reward weight of wp = 0.3 achieves the best overall tradeoff across MATH500, AIME25, and BFCL v4, while excessive emphasis on either reward degrades performance.
7 Conclusion
SCRIBE intervenes at a mid-level abstraction by grounding process rewards in Skill Prototypes and achieves strong reasoning and tool-use results. The analysis links mid-level mastery with later high-level planning and shows complementarity with low-level optimization.
- SCRIBE anchors process rewards in Skill Prototypes, converting subjective LLM judging into grounded, diagnostic verification.
- SCRIBE improves AIME25 accuracy for a 4B model from 43.3% to 63.3% and doubles success rates in complex multi-turn tool use.
- Mid-level skill mastery precedes the emergence of strategic high-level planning, while SCRIBE complements low-level tool optimization.
8 Limitations
SCRIBE improves mathematical reasoning and tool-use benchmarks, but its scalability, prototype construction, and applicability beyond structured tasks remain bounded or unresolved.
- The experiments focus on small- to mid-sized instruction-tuned models, leaving scalability to substantially larger architectures and different post-training pipelines unclear.
- Skill prototype construction and routing rely on clustering heuristics and judge-annotated data, which may introduce bias or limit adaptability where skill boundaries are diverse or ill-defined.
- The evaluation primarily targets structured reasoning and tool-use tasks, so effectiveness for open-ended generation or non-instrumental reasoning remains an open question.
9 Ethics
The work uses public benchmarks and standard inference procedures without processing personal or sensitive user data, while acknowledging potential bias risks. Its reward evaluation also shows stability across prompt formulations and training checkpoints.
- Data and privacy: Experiments use publicly available datasets and standard model inference and optimization without processing personal or sensitive user data.The authors also state that they do not access or infer proprietary training data or internal model components.
- Risks: Public benchmarks may contain biases, errors, or harmful language, and the method could amplify biases in the judge or training data without safeguards.The paper identifies this as a deployment risk rather than reporting a measured harmful outcome.
- Evaluation robustness: Prompt variants produce highly consistent subgoal structures and reward scores across math and tool-using tasks, with only minor tool-step boundary differences.The results indicate robust agreement across prompt formulations and insensitivity to prompt phrasing.
- Reward validity: Reward calibration uses subgoal-specific outcome statistics to adjust scores while preserving their relative ordering.Rewards are adjusted upward or downward when empirical completion success rates indicate misalignment with correctness.
- Evaluation robustness: Across repeated checkpoint evaluations, all but two tool-use anchor instances retain identical rewards, indicating no noticeable evaluation drift.Each task type uses 20 fixed anchor subgoals, with five evaluations per anchor.
B Router Evaluation
The Router is evaluated on held-out judge-annotated trajectories for segmentation, skill prediction, and Skill Prototype retrieval, using exact-match and classification metrics.
- The held-out evaluation uses a 9:1 training-to-test split and measures step segmentation, skill prediction, and Skill Prototype retrieval.
- The Router maps each problem and raw reasoning trajectory to ordered ⟨subgoal, skill, step⟩ tuples with an associated Skill Prototype.
- Step segmentation accuracy uses span-level exact match, requiring predicted step boundaries to exactly match annotated spans.
- Skill prediction accuracy is classification accuracy over whether each predicted subgoal receives its annotated skill label.
- Skill Prototype retrieval is correct when the Router selects the prototype matching the annotated prototype, measuring skill-level context for reward evaluation.
C Additional Training Details
The supplementary training details describe structured trajectory decomposition, skill-aware supervision, GRPO optimization, and stability monitoring, alongside broad related-work context.
- Additional Training Details: GRPO training uses eight rollouts per problem over 10k mathematical training problems, with KL regularization, entropy bonuses, and reward shaping.
- Additional Training Details: Training monitors mean reward and reward variance, with no observed evidence of reward collapse.
- Prompts for Subgoal and step decomposition: Math prompts ask for ordered subgoals sufficient to solve the problem and explicitly identify the main skill required for each stage.
- Prompts for Subgoal and step decomposition: Tool-use prompts identify all subgoals carried out across multi-turn trajectories and associate each with its primary skill.
- Prompts for Subgoal and step decomposition: Structured prompts partition each trajectory into contiguous, non-overlapping subparts so every trajectory segment belongs to exactly one subgoal.
- Prompts for Subgoal and step decomposition: The prompts require mid-level subgoals, associated skills, and steps, while including fallback handling for partially completed or failed tasks.
F Execution-Verified Plan Selection and Structural Metrics
SCRIBE evaluates plan selection through execution-verified outcomes, linking reliable mid-level subgoal execution to clearer discrimination between viable and non-viable plans. The section also specifies the prototype-based skills and rubrics used to assess reasoning and tool-use subgoals.
- Mid-level execution metrics: 64 independent rollouts estimate each subgoal’s success rate, while repeated outcomes quantify execution reliability.The evaluation samples subgoals from fixed dataset subsets and reports macro-averaged mid-level success across subgoals.
- Candidate plan construction and labeling: Candidate plans are held fixed across checkpoints and executed for 5 trials to estimate empirical viability.Plans are partitioned using success-rate thresholds, with intermediate cases discarded; alternative rank-based labeling yields consistent qualitative trends.
- High-level plan selection: HighLvl measures whether preference scores rank viable plans above non-viable plans, equivalent to an AUC-style discrimination measure.Viable and non-viable plan sets are defined from execution outcomes before preference-based selection is evaluated.
- Structural plan separability: PlanSep measures the gap between mean execution success rates of viable and non-viable plans, independently of model preferences.Higher PlanSep indicates clearer separation in execution-outcome space and reflects improved execution reliability.
- Skill prototypes and judging rubrics: The skill library includes mathematical bound-based conclusions and tool-mediated fallback guidance, scored with rubrics that distinguish complete reasoning from logical gaps, traps, or unsafe behavior.The rubrics check bounds, tightness, domain restrictions, case coverage, tool parameters, output interpretation, limitations, and actionable alternatives.