PaperlayerView exact revision on arXiv

Source-linked AI summary

Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

Yu Li, Mingyang Yi, Xiuyu Li, Ju Fan, Fuxin Jiang, Binbin Chen, Peng Li, Jie Song, Tieying Zhang

arXiv:2602.00994v3cs.AI

TL;DR

ARL commonly trains reasoning and tool-use through shared parameters, but the paper tests whether this assumption creates interference. It introduces CEA to measure the interaction and DART to separate updates, finding consistent improvements over joint optimization across thirteen benchmarks.

  • Problem

    Most ARL methods jointly optimize reasoning and tool-use with shared parameters, while empirical evidence about interference between these capabilities remains limited.

  • Method

    CEA quantifies capability interactions and gradient misalignment, while DART uses separate LoRA adapters for reasoning and tool-use updates.

  • Results

    DART consistently outperforms joint-optimization baselines across seven QA and six NL2SQL benchmarks and approaches the 2-Agent upper bound.

  • Takeaways & Limitations

    The findings support capability interference as a non-negligible ARL bottleneck and gradient disentanglement as an effective remedy.

  • Takeaways & Limitations

    DART’s token-level adapter switching introduces extra scheduling complexity during high-concurrency serving.

Abstract

from arXiv · show

Agentic Reinforcement Learning (ARL) trains large language models to interleave reasoning with external tool execution to solve complex tasks. Most existing ARL methods train a single set of parameters to support both reasoning and tool-use behaviors, implicitly assuming that joint training leads to improved overall agent performance. Despite its widespread adoption, this assumption has rarely been examined empirically. In this paper, we systematically examine this assumption by introducing Capability Effect Attribution (CEA), which provides quantitative evidence of interference between reasoning and tool-use behaviors. Through an in-depth analysis, we show that these two capabilities often induce misaligned gradient directions, leading to training interference that undermines the effectiveness of joint optimization and challenges the prevailing ARL paradigm. To address this issue, we propose Disentangled Action--Reasoning Tuning (DART), a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool use via separate low-rank adaptation modules. With this simple change alone, DART outperforms all joint-optimization baselines and approaches the 2-Agent upper bound across thirteen benchmarks on retrieval-augmented QA and NL2SQL, further supporting our finding of capability interference under shared optimization.

1 Introduction

The paper finds that reasoning and tool-use can interfere under shared ARL optimization, then proposes DART to separate their updates and reports consistent gains across thirteen benchmarks.

  • Motivation: Most ARL methods jointly optimize reasoning and tool-use with shared parameters, but whether these heterogeneous capabilities interfere remains largely unexplored.The paper directly tests this shared-parameter assumption.
  • Capability Effect Attribution: CEA decomposes performance into capability effects and pairwise interactions, revealing significant negative interaction between reasoning and tool-use under joint optimization.The analysis uses six controlled model variants created through gradient masking and hybrid inference.
  • Gradient Analysis: Reasoning and tool-use gradients are nearly orthogonal, so shared-backbone optimization updates parameters in a compromise direction that is suboptimal for both capabilities.This identifies gradient misalignment as the optimization-level source of interference.
  • DART: DART freezes the pretrained backbone and routes reasoning and tool-use tokens to separate LoRA adapters, allowing each capability to update its own parameters.The framework explicitly decouples the two parameter-update paths.
  • Results: DART consistently outperforms joint-optimization baselines across seven QA and six NL2SQL benchmarks, supporting gradient disentanglement as an effective remedy.The experiments also report that DART surpasses all joint-optimization baselines, including multi-LoRA methods.

2 Related Work

Prior ARL work improves tool-use training components, while Multi-LoRA methods mainly target capacity or transfer; this paper instead studies interference between reasoning and tool-use within one agentic domain.

  • ARL with Tool-use: ARL research has addressed reward formulation, policy refinement, and large-scale trajectory synthesis for learning tool invocation through environment feedback.These efforts target different components of the agent-training pipeline.
  • ARL with Tool-use: Existing ARL studies have not examined whether reasoning and tool-use interfere under joint optimization, which is the central question of this work.The paper frames this as an unresolved issue within a single agentic domain.
  • Multi-LoRA: Multi-LoRA methods use soft routing or adapter composition to increase capacity or support multi-domain transfer, addressing a different problem from capability interference.The distinction is between capacity or transfer objectives and preventing optimization interference between heterogeneous capabilities.

3 Preliminaries

The preliminaries define ARL trajectories that interleave reasoning and tool-use tokens, shared-parameter policy optimization, and LoRA’s frozen-backbone low-rank updates.

  • Agentic Reinforcement Learning: An ARL agent generates a trajectory under a query while interleaving reasoning and tool-use tokens.The trajectory is represented as a sequence of generated tokens.
  • Agentic Reinforcement Learning: The role-based router assigns each token to reasoning or tool use, with tool-use spans triggered deterministically by special tokens such as <search>.The router maps token positions to roles r or a.
  • Agentic Reinforcement Learning: ARL maximizes expected trajectory reward, while standard training updates one shared parameter set using gradients from both token types.The advantage is derived from the trajectory reward.
  • Low-Rank Adaptation: LoRA freezes pretrained weights and trains low-rank matrices A and B, with rank r much smaller than the layer dimensions.Only the low-rank matrices are trained, and the same update is applied to all tokens in the trajectory.

4 Do Reasoning and Tool-Use Conflict?

CEA tests whether reasoning and tool-use conflict when jointly optimized. It attributes per-question performance to capability effects and interactions, finding predominantly negative reasoning–tool-use interactions under joint optimization.

  • Capability Effect Attribution: CEA decomposes per-question correctness into individual capability effects and pairwise interaction terms using binary capability indicators.For each question, six linearly independent model variants identify the attribution coefficients.
  • Capability Effect Attribution: A negative λq_ij indicates interference between capabilities i and j, while a positive coefficient indicates synergy.The reasoning–tool-use interaction is assessed through λq_23.
  • Controlled Model Variants: Six controlled model variants form a full-rank design matrix, combining gradient-masked training variants with hybrid-inference variants.The variants preserve controlled capability configurations while eliminating parameter-level interaction in hybrid models.
  • Evidence of Interference: λq_23 is predominantly negative, whereas base–reasoning and base–tool interactions are predominantly positive.This pattern indicates that reasoning and tool-use conflict specifically under joint optimization, despite each capability benefiting the base model individually.
  • Evidence of Interference: Interference concentrates on questions each capability solves accurately alone, while synergy appears on questions requiring both capabilities but degraded by their interference.The analysis identifies where interference occurs but does not by itself explain its mechanism.

5 Disentangled Action-Reasoning Tuning

The paper motivates disentangling reasoning and tool-use because their gradients compete for shared parameters. This motivates a design that assigns the capabilities separate update spaces.

  • Reasoning and tool-use negatively interact because their gradients compete for the same shared parameters.

A. Gradient Angul Analysis

Gradient analysis finds that reasoning and tool-use gradients are nearly orthogonal, making shared optimization inefficient. DART addresses this by freezing the backbone and routing the two capabilities to disjoint LoRA adapters.

  • Gradient Misalignment: Same-capability gradients are aligned, whereas reasoning and tool-use gradients are nearly orthogonal.Averaging the orthogonal gradients produces a compromise direction that is suboptimal for both capabilities.
  • Design Rationale: DART’s isolation avoids the additional system complexity of training two independent agents while preserving separate capability updates.The paper presents this as a single-model alternative to a 2-Agent system.
  • DART Architecture: DART freezes the pretrained backbone and attaches separate LoRA adapters for reasoning and tool-use.The adapters are disjoint, so no trainable parameter receives gradients from both token types.
  • DART Architecture: A token-level router selects the active adapter according to each token’s capability type.Sentinel tokens such as <search> trigger the tool-use adapter, while rule-based routing avoids additional router training signals.
  • DART Architecture: Each token updates only its associated adapter, leaving the reasoning and tool-use parameter sets independent.

6 Experiments

Experiments evaluate DART across retrieval-augmented QA and NL2SQL under controlled training conditions. DART consistently outperforms joint-optimization and soft-routing baselines, with separate tests indicating gains in reasoning and tool-use.

  • Experimental Setup: Experiments cover seven retrieval-augmented QA benchmarks and six NL2SQL benchmarks using shared backbones, data, algorithms, and hyperparameters.The principal comparison changes gradient disentanglement while holding the other training components fixed.
  • Main Results: DART consistently outperforms joint-optimization baselines across all evaluated benchmarks.The same trend holds across larger backbones and the NL2SQL task.
  • Main Results: DART also outperforms MixLoRA, indicating that soft routing and partial gradient disentanglement are insufficient compared with hard token-level isolation.
  • Mechanism Analysis: The experiments separately test whether disentanglement improves reasoning and tool-use and whether inference-time composition can replace training-time disentanglement.
  • Mechanism Analysis: With identical retrieval contexts, DART achieves higher EM than Search-R1 on NQ and HotpotQA.This comparison isolates reasoning ability from retrieval quality.

1. Interference Harms Each Capability In-

Controlled comparisons show that joint optimization impairs both reasoning and tool-use, while DART’s separated adapters recover performance without relying on extra parameter capacity. DART approaches the parameter-disentangled 2-Agent upper bound across settings.

  • Capability-specific interference: DART achieves higher EM than Search-R1 when both receive identical retrieval contexts, isolating improved reasoning from retrieval differences.This supports the conclusion that joint optimization impairs reasoning learning.
  • Capability-specific interference: DART adapters substantially outperform inference-time hybrid counterparts, showing that training-time isolation cannot be replaced by inference-time composition.The comparison uses reasoning-only and tool-use-only DART adapters against separately trained hybrid schemes.
  • Parameter-capacity ablation: A matched-rank single LoRA performs nearly identically to Search-R1, whereas DART closely matches the 2-Agent upper bound.The comparison attributes DART’s gains to gradient isolation rather than increased parameter capacity.
  • Parameter-capacity ablation: DART’s performance is largely insensitive to the LoRA rank, indicating that its gains persist across parameter-budget choices.The rank ablation is reported as a separate test of capacity effects.

7 Conclusion

The paper identifies negative interaction between reasoning and tool-use under joint ARL optimization and traces it to misaligned parameter updates. DART separates their updates and consistently surpasses joint-optimization baselines, nearly recovering the 2-Agent upper bound across thirteen benchmarks.

  • Conclusion: CEA exposes negative reasoning–tool-use interaction under joint optimization, while gradient analysis traces it to misaligned parameter updates.The paper frames this interference as a bottleneck in current ARL systems.
  • Conclusion: DART freezes the backbone and routes reasoning and tool-use tokens to disjoint LoRA adapters.This design separates the parameter subspaces updated by the two capabilities.
  • Conclusion: DART consistently surpasses joint-optimization baselines and nearly recovers the 2-Agent upper bound across thirteen benchmarks.The result supports capability-aware training as an alternative to one-size-fits-all optimization.

Limitations

The paper’s main practical limitation is serving complexity: DART’s token-level adapter switching adds scheduling overhead at high concurrency, leaving systems optimization for future work.

  • Limitations: DART’s token-level adapter switching introduces extra scheduling complexity during high-concurrency serving.The limitation is engineering-related rather than a change to DART’s algorithmic contribution.
  • Limitations: Batched adapter dispatch or fused multi-LoRA kernels are proposed as future ways to mitigate the serving overhead.These optimizations are described as orthogonal to DART’s algorithmic contribution.

C Additional Main Results

Across retrieval-augmented QA and NL2SQL, DART’s disentanglement remains effective across model scales, architectures, datasets, and tool-use settings. CEA extensions show that reasoning–tool-use interference is consistently negative across dataset–model combinations.

  • Retrieval-Augmented QA: DART outperforms joint-optimization baselines across Qwen2.5-7B and Llama3.1-8B QA benchmarks, showing robustness across model scale and architecture.The reported trends are consistent with the 3B results and hold across nearly all benchmarks and aggregate metrics.
  • Multi-Turn SQL Query Generation: DART achieves the best single-model performance across six NL2SQL benchmarks, surpassing Sky-SQL and closely matching the more resource-intensive 2-Agent system.MixLoRA’s soft routing underperforms, supporting hard token-level disentanglement for capability isolation.
  • Multi-Turn SQL Query Generation: DART’s disentanglement benefit generalizes from retrieval-augmented QA to NL2SQL despite fundamentally different interfaces, action spaces, and reward signals.NL2SQL uses SQL execution rather than search, with reasoning and tool-use expressed through different token modalities.
  • CEA Generalization: The CEA pattern across datasets and architectures supports interference as a general ARL property rather than an artifact of one model or dataset.The analysis evaluates the first 1,000 test samples for each additional dataset and model combination.
  • CEA Generalization: Across PopQA, TriviaQA, Qwen2.5-3B, Qwen2.5-7B, and Llama3.1-8B, λq23 is consistently dominated by negative values.Interference concentrates on high-accuracy questions, while positive-interaction groups in 7B models achieve higher accuracy than those in 3B models.

E Implementation Details of Gradient Misalignment

Gradient analysis finds directional separation between reasoning and tool-use updates, while DART uses token-level routing and separate adapters within one backbone. The framework is presented as a lower-overhead alternative to a two-agent system, with stable performance across LoRA ranks and higher retrieval accuracy than Search-R1.

  • Gradient Angle Analysis: Reasoning–tool gradients are close to orthogonal across settings, whereas same-capability gradients show stronger alignment.The 7B model exhibits a more dispersed distribution of same-role gradient directions than the 3B model.
  • DART Router: DART’s router makes rule-based token-level decisions that distinguish reasoning, tool-use, and loss-free tokens during tool-augmented QA.The corresponding role-specific gradients are computed through separate backward passes with explicit zeroing between passes.
  • Training Memory: DART keeps both capabilities in one shared backbone with lightweight LoRA adapters, while a 2-agent system requires two trainable policy backbones.The comparison assumes p ≪ P for adapter and backbone parameter counts.
  • Training Memory: DART reduces training-time static memory by roughly 8× while maintaining performance comparable to 2-agent.The reduction follows from freezing the backbone and training only lightweight adapters instead of two full policy backbones.
  • LoRA Rank Sensitivity: DART remains close to the 2-agent baseline across LoRA ranks, indicating stable EM performance under rank changes.The rank-sensitivity result suggests that only modest adapter capacity is sufficient in this disentangled setting.
  • Inference Latency: DART’s single-backbone design preserves the KV-cache across capability switches, avoiding the 2-agent system’s full-history re-encoding during handoffs.The 2-agent re-computation has complexity O(L^2), whereas DART switches active LoRA ranks in O(1).
  • Retrieval Accuracy: DART achieves higher retrieval accuracy than Search-R1 across both datasets and model scales, with 7B outperforming 3B for both methods.The reported advantage is especially highlighted for multi-hop and fact-intensive tasks.
Loading 2602.00994v3…