Source-linked AI summary
Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspaces
Karan Gupta, Pranav Vajreshwari, Yash Pandya, Raghav Magazine, Akshay Nambi, Ahmed Awadallah
TL;DR
Large MCP toolspaces make long-horizon agentic work difficult for SLMs because context, execution, and supervision are constrained. ATLAS learns adaptive context control, programmatic orchestration, and rubric-based rewards, enabling a 4B SLM to approach frontier-agent performance with far fewer parameters and tighter context budgets.
Problem
SLMs remain brittle in large MCP toolspaces because eager tool loading, compounding execution errors, and sparse rewards hinder long-horizon workflows.
Method
ATLAS jointly learns incremental tool loading, code-based orchestration, and task-level rubric rewards evaluated by scalable SLM judges.
Results
4.15/10 TF enables the strongest Qwen3-4B configuration to approach Kimi-K2 Thinking at 4.38/10 TF, while rubric-based SLM judging outperforms generic GPT-4o judging.
Takeaways & Limitations
ATLAS shows that learning context control, execution structure, and task-aligned credit assignment can make efficient SLM agents effective across MCP tasks.
Takeaways & Limitations
Generic trajectory-level LLM judging can yield inconsistent rankings and noisy learning signals for complex long-horizon tasks.
Abstract
from arXiv · showhide
Agentic systems operating over large tool ecosystems must plan and execute long-horizon workflows under weak or non-verifiable supervision. While frontier models mitigate these challenges through scale and large context budgets, small language models (SLMs) remain brittle: eager tool loading saturates context, execution errors compound over time, and sparse rewards limit learning. We introduce ATLAS, a reinforcement finetuning framework that enables SLMs to operate effectively in large-scale toolspace environments by learning how to acquire context and how to execute actions. Our approach makes two key contributions. First, we treat context control and execution structure as learnable decisions, combining iterative tool loading with programmatic tool orchestration to bound context growth and stabilize long-horizon trajectories. Second, we propose rubric-based reinforcement finetuning, which decomposes task success into structured, task-aligned criteria and enables scalable training using small judge models. Across MCP benchmarks, these design choices yield large and consistent gains over generic RL baselines, allowing a 4B SLM to approach frontier-agent performance under far tighter parameter and context budgets.
1 INTRODUCTION
ATLAS addresses SLM brittleness in large MCP toolspaces by learning context acquisition and execution structure rather than relying on eager loading or scale. It combines adaptive tool access, code-based orchestration, and rubric-based reinforcement finetuning, with consistent gains over generic approaches.
- Large MCP workflows require planning, state tracking, and execution across heterogeneous tools under tight context and cost budgets.
- SLMs face compounded failures from context saturation, tool misselection, long-horizon error propagation, brittle code execution, and sparse outcome rewards.
- ATLAS reframes agentic reasoning as learning what context to acquire, when to acquire it, and how to represent execution compactly.
- Adaptive tool loading incrementally exposes required schemas, while code-based orchestration provides compact, explicit control flow for long-horizon tool use.
- Rubric-based rewards decompose task success into criteria such as completeness, tool choice, grounding, and parameter precision, enabling scalable SLM judging.
- Across MCP tasks, ATLAS consistently outperforms JSON-style tool calling, and rubric-based evaluation supports effective SLM judges.
2 ATLAS OVERVIEW
ATLAS controls context and execution by loading server and tool information incrementally, then orchestrating calls through persistent executable programs. This design keeps state compact and makes long-horizon workflows less dependent on repeated natural-language interaction.
- ATLAS Overview: ATLAS treats context acquisition as a decision, selecting domains, servers, and tools according to task state and loading only required information.
- Iterative Server Loading: Iterative Server Loading exposes server-level capabilities incrementally, avoiding eager loading across all available servers.
- Iterative Tool Loading: Iterative Tool Loading initially presents compact tool names and defers detailed schemas until a tool is needed.
- Programmatic Tool Calling: Programmatic orchestration replaces turn-by-turn JSON interaction with explicit control flow and intermediate state stored in executable programs.
- Programmatic Tool Calling: A persistent interpreter supports stateful execution and localized program edits instead of repeatedly replaying reasoning traces after errors.
- Execution Scaffolding: Python-side scaffolding normalizes heterogeneous MCP tools into verified, dynamically bound function interfaces and Python-native outputs.
3 REINFORCEMENT FINETUNING
ATLAS uses task-specific rubric rewards to provide structured, consistent supervision for long-horizon MCP trajectories with non-verifiable outcomes. Rubrics reduce judge variability and make SLM-based evaluation practical for reinforcement finetuning.
- Motivation: Long-horizon MCP learning requires rewards that capture interdependent tool selection, execution order, grounding, and task-specific constraints rather than outcomes alone.
- Limitations of LLM-as-Judge: Generic trajectory-level judging can produce inconsistent rankings and unstable supervision when multiple requirements interact across long contexts.
- Task-Level Rubrics: ATLAS generates task-specific rubrics that score task fulfillment, tool appropriateness, tool grounding, and parameter accuracy.
- Rubric-Based Rewards: Fixed rubrics are applied to every rollout for a task, producing weighted scores that improve consistency and reduce reward variance.
- Automated Rubric Generation: Rubrics are generated once offline from task specifications and available tool context, without conditioning on individual agent trajectories.
- Scaling Judging: Rubric-conditioned scoring enables SLM judges to provide stronger and more stable learning signals than generic frontier-judge rewards, supporting cost-efficient training.
- Optimization: Policy gradients are masked over tool-output tokens so updates target planning, selection, argument construction, control, and termination decisions.
4 EXPERIMENTAL SETUP
The experiments evaluate ATLAS on realistic multi-server MCP tasks using open-weight SLMs, a frontier baseline, multiple execution variants, and rubric-based trajectory judging. The setup tests both task-level and server/tool-level generalization under fixed context budgets.
- Tasks: The benchmark contains synthetic but realistic MCP tasks requiring long-horizon reasoning, tool discovery, invocation, and grounding, with multiple valid trajectories.
- Tasks: Training data spans 28 heterogeneous MCP servers, while held-out evaluation includes unseen tasks and tasks involving unseen servers and tool combinations.
- Tasks: The final filtered training set contains 304 solvable, high-quality tasks selected using a task fulfillment threshold of at least 4/10.
- Models: Qwen2.5-7B and Qwen3-4B are evaluated with 32K context, alongside Kimi K2 Thinking with 1T parameters and 80K context.
- Evaluation: Trajectory evaluation uses four categories, while rubric training uses Qwen3-30B-Instruct and generic judging uses GPT-4o.
- Execution Variants: Variants compare eager loading, iterative server loading, iterative server and tool loading, and ATLAS with programmatic orchestration.
5 RESULTS
ATLAS improves MCP task fulfillment by combining adaptive execution structure with reinforcement finetuning, with rubric-based supervision producing the strongest gains. The best 4B SLM configuration approaches frontier performance while using tighter model and context budgets.
- TF is the primary 0–10 metric for end-to-end success on compositional, long-horizon MCP tasks.Table 1 compares model families, execution strategies, and reinforcement finetuning configurations.
- Overall performance: Base agents typically achieve TF in the 2–3 range, while reinforcement finetuning produces large and consistent gains across model scales.Structured execution further amplifies improvements by stabilizing long-horizon trajectories.
- Impact of reinforcement finetuning: Qwen3-4B reaches TF 3.91 with ITL, PTC, and generic RFT using an SLM judge, the strongest SLM regime before rubric-based rewards for PTC.JSON-based ITL remains less effective than its PTC counterpart even after learning.
- Interaction between execution structure and learning: RFT delivers the largest gains (+35–65%), with ITL+PTC providing the strongest execution substrate for learning.Structured execution alone is insufficient, but it becomes more effective when combined with reinforcement finetuning.
- Rubric-based rewards and judge scaling: Rubric-based RFT improves Qwen3-4B TF from 3.25 to 3.87 and enables the SLM judge to outperform the frontier judge at 3.43.The fixed external o4-mini evaluator indicates that the gains reflect training dynamics rather than evaluation bias.
- Closing the gap to frontier models: 4.15/10 TF from Qwen3-4B with ITL, PTC, rubric-based RFT, and an SLM judge approaches Kimi-K2 Thinking’s 4.38/10 frontier baseline.The strongest SLM configuration operates with orders of magnitude fewer parameters and tighter context budgets.
6 RELATED WORK
Prior work separately advances tool invocation, reinforcement learning, dynamic tool selection, context control, and programmatic execution. ATLAS combines learned context acquisition, execution structure, and rubric-based reinforcement finetuning for efficiency-constrained agents in large MCP toolspaces.
- Learned Tool Invocation: Tool invocation research progressed from self-supervised API-call insertion to executable code for stateful, programmatic interaction.
- Reinforcement Learning for Agentic Tool Use: Reinforcement learning methods teach agents when and how to invoke tools, while related work targets stability and credit assignment under limited supervision.
- Dynamic Tool Selection and Context Control: Dynamic tool-selection approaches use rationale data, curriculum learning, sampling, or multi-tool coordination to avoid impractical static inventories.
- Programmatic Execution: Programmatic execution replaces turn-by-turn tool calls with stateful, verifiable control flow that reduces context overhead.
- Evaluation and Supervision: Structured, rubric-based evaluation provides denser supervision and improves training stability in non-verifiable domains.
- ATLAS: ATLAS jointly treats context acquisition and execution structure as learned optimization targets for efficiency-constrained agents in large MCP toolspaces.It combines adaptive tool loading, compact programmatic orchestration, and task-level rubric-based reinforcement finetuning.
7 CONCLUSION
ATLAS argues that efficient agentic performance depends on learning context acquisition, execution structure, and weak-supervision credit assignment rather than relying primarily on model scale. Its structured rubrics and learned orchestration let a 4B SLM recover nearly 90% of frontier-agent performance under tighter budgets.
- ATLAS shifts the focus of agent design from model scale to how agents acquire context, execute actions, and learn from weak supervision.
- Learning execution structure alone improves efficiency but yields limited task success; substantial gains require reinforcement finetuning with task-aligned credit assignment.
- Structured rubrics enable small judge models to match or surpass frontier judges, making reinforcement finetuning scalable and cost-efficient.
- Nearly 90% of frontier-agent performance is recovered by a 4B SLM operating under substantially tighter parameter and context budgets.
A PROGRAMMATIC TOOL CALLING IMPLEMENTATION
The implementation turns heterogeneous MCP tools into Python-native functions and uses iterative loading, program state, and informative errors to support reliable programmatic orchestration. This reduces dependence on raw JSON schemas and limits unnecessary context exposure.
- SLMs can make incorrect function names, argument formats, and output-access choices when writing code against unseen MCP server libraries.
- The scaffold normalizes input schemas, dynamically binds MCPServer functions, converts outputs, fetches examples, loads tools iteratively, and provides corrective error hints.
- Heterogeneous server schemas are mapped into a common schema, converted into Python signatures, stored offline, and programmatically verified against the original APIs.
- MCPServer exposes server tools as dynamically allocated Python attributes that map calls to the underlying MCP APIs.
- Serialized server responses are converted into Python-native types using ast.literal_eval for sequential processing in orchestration code.
- A tool-calling loop obtains output schemas and examples, while get_tools_info retrieves only requested tool details for orchestration.
A.7 EXAMPLE ISL VS ITL+PTC BEHAVIOR
The examples contrast verbose JSON-style tool calling with compact programmatic orchestration for dependent tasks. The latter keeps intermediate state in code and uses iterative loading, while the examples also expose failures and unsupported assumptions in the workflows.
- ISL behavior: JSON-style tool calling produces multiple tool calls and verbose intermediate reasoning when task steps depend on one another.
- ISL behavior: 152.5 tons is obtained as the median by averaging the fifth and sixth values after sorting ten harvest yields.
- ISL behavior: The harvest example computes total output of 1555 tons, average yield of 155.5 tons, median yield of 152.5 tons, and mode of 150 tons.
- ISL behavior: The example also reports a minimum yield of 120 tons, maximum of 200 tons, total revenue of $46,650, net profit of $26,650, and 57% profit margin.
- ISL behavior: The BTC-USDT workflow retrieves tool schemas before calling get_price and get_candlesticks for the requested instrument.
- ISL behavior: The BTC-USDT example reports current price 87910.2, 24-hour average 87900, percentage difference 0.0116%, an upswing, and low recent volume.
- ITL+PTC behavior: The ITL+PTC agent places orchestration inside a single code block, capturing intermediate reasoning in executable code rather than repeated JSON-style calls.
B TRAINING HYPERPARAMETERS
The paper identifies its training setup and reports that MCPBench comparisons span model families, execution strategies, and reinforcement-finetuning configurations. The supplied passages provide captions and evaluation dimensions but no hyperparameter values.
- Table 2 summarizes the training hyperparameters used throughout the experiments.
- Table 3 compares MCPBench performance across model families, training variants, and reward configurations.
C DETAILED RESULTS
The detailed-results tables compare model families, training variants, execution strategies, and reward configurations across MCPBench and held-out ATLAS-Test evaluations. They report task and tool-use metrics alongside average turns and token usage.
- Table 3 reports Task Fulfillment, Tool Grounding, Tool Appropriateness, Parameter Accuracy, Avg Turns, and Avg Tokens on MCPBench.
- Table 4 reports the same task, tool-use, and efficiency metrics on the held-out ATLAS dataset.
- Table 4 compares performance across model families, training variants, and reward configurations.
- Table 5 stratifies task distributions across training and evaluation sets by MCP-server count and task difficulty.
D.1 MCP TASKS
MCP tasks require agents to answer complex, evidence-grounded queries by coordinating multiple servers, selecting tools, and synthesizing structured results. The benchmark spans 28 servers and 257 tools across diverse real-world domains, with extensions adding further capabilities.
- Each task requires decomposing a complex query, identifying retrieval or computation needs, invoking appropriate MCP servers, and synthesizing a coherent response.
- MCPBench contains 28 MCP servers exposing 257 tools across search, scientific computing, geospatial reasoning, finance, health, and knowledge retrieval.
- ATLAS adds 11 servers exposing more than 57 tools for computation, persistent memory, filesystem access, structured search, and stepwise reasoning.
D.2 TRAINING SET AND BENCHMARKS
The benchmark construction addresses limited task volume and noisy synthetic supervision by creating and filtering a higher-quality training set, then evaluating both seen and unseen server combinations. Rubrics assess fulfillment, tool use, grounding, and parameter correctness with task-specific criteria.
- Training Set and Motivation: The original MCPBench dataset has 104 tasks, motivating a larger, higher-quality set because some synthetic tasks yield low fulfillment even for strong agents.
- Training Set and Motivation: The final training set contains 304 synthetically generated tasks stratified into Easy, Medium, and Hard difficulty levels.
- Training Set and Motivation: Tasks are filtered for solvability and utility, then frontier-model execution retains tasks meeting fulfillment thresholds.
- Benchmarks: Evaluation uses in-distribution tasks from the original 28-server benchmark and out-of-distribution tasks built from 11 previously unseen servers.
- Task Design: The benchmark task requires retrieving a trending Steam game and listing five subreddit hot-thread titles with comment counts.
- Rubric Design: Rubrics evaluate fulfillment, tool appropriateness, grounding, and parameter accuracy, including exact retrieval, suitable endpoints, and correct tool-call parameters.