PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Multi-Turn Reasoning: A Fairness Approach

Thanh-Dat Truong, Sankalp Pandey, Hugh Churchill, Jackson Cothren, Marios Savvides, Khoa Luu

arXiv:2609.33323v1cs.AIcs.LG

TL;DR

Agentic learning faces final-outcome-only credit assignment and imbalanced reasoning data. Fair-MPO combines multi-level preference learning with an imbalance-focused objective, and the paper reports improved final reasoning accuracy over prior methods on standard benchmarks.

  • Problem

    Agentic learning must assign credit across multi-step reasoning from final outcomes and contend with skewed data that leaves rare or difficult successful trajectories scarce.

    Source: §1 · p. 2

  • Method

    Fair-MPO learns from trajectory-level preferences and aligned state-level comparisons, then suppresses the influence of overrepresented easy pairs.

    Source: §1 · p. 2, §3.1.1 · p. 5

  • Results

    The paper reports that Fair-MPO improves final reasoning accuracy over prior methods in experiments on standard benchmarks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The paper concludes that Φ-MPO addresses long-horizon and data-imbalance problems in multi-turn agentic reasoning, with experiments and theoretical analysis supporting its effectiveness.

    Source: §5 · p. 9

    Effectiveness depends on on-the-fly rollout quality and may be sensitive to rollout diversity, reward reliability, and trajectory- and step-level supervision stability.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Recent advances in Large Language Models (LLMs) have enabled agentic systems capable of solving complex tasks through multi-turn planning, tool use, verification, and memory updates. However, learning agentic systems remains difficult due to two fundamental challenges, i.e., (1) long-horizon credit assignment, where supervision is available only at the final outcome, and (2) imbalanced data distributions, where dominant data patterns bias optimization and weaken adaptation to rare but informative reasoning behaviors. In this paper, we propose Fair Multi-Level Preference Optimization (Fair-MPO or $\Phi$-MPO), a new preference optimization framework for agentic learning. We first show that Multi-Level Preference Optimization provides a principled and more computationally efficient framework for long-horizon reasoning. Then, we introduce a Fair Multi-Level Objective that addresses imbalance in agentic learning. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both long-horizon reasoning and data imbalance. Our experiments on agentic reasoning benchmarks demonstrate that our approach achieves State-of-the-Art (SOTA) performance.

1 Introduction

Agentic learning must optimize multi-step decisions despite final-outcome-only supervision and skewed reasoning data. The paper proposes Fair-MPO to address both challenges through multi-level preference learning and a fairness-oriented objective.

1 Introduction

  • Many existing systems rely on prompting heuristics, handcrafted coordination, or static orchestration, limiting adaptation to new environments, tools, and task distributions.The paper argues that reliable multi-turn reasoning requires learning from interaction outcomes.Source: §1 · p. 2
  • Final-outcome-only supervision makes credit assignment difficult across interdependent reasoning steps, while uneven data frequencies can leave rare or difficult successful trajectories scarce.The paper describes imbalance as propagating from data distributions into agentic trajectories.Source: §1 · p. 2
  • Prior RL and preference-optimization approaches leave gaps in handling sparse long-horizon feedback and the sequential structure and skew of agentic trajectories.The introduction contrasts coarse trajectory rewards with standard DPO's single-output preference design.Source: §1 · p. 2
  • Fair-MPO combines trajectory- and state-level supervision with an objective designed to improve robustness to skewed agentic-learning data.The paper also reports theoretical analysis and experiments supporting the framework's effectiveness.Source: §1 · pp. 2–3

2 Related Work

Prior work applies reinforcement learning and preference optimization to tool-integrated and agentic reasoning, but training and credit assignment remain challenging over long horizons. Modular systems offer flexibility, while static orchestration and single-output preference methods leave adaptation and structured trajectory learning unresolved.

Agentic Learning

  • Monolithic policies face difficult long-horizon credit assignment, while many modular systems prescribe collaboration through prompting or handcrafted control rather than learning it from interaction.Some trained systems still rely on offline supervised fine-tuning or preference optimization over static datasets.Source: §2 · p. 3

Preference Learning and Policy Optimization

  • RLHF, DPO, GRPO, and Flow-GRPO provide policy or preference-learning approaches, but delayed feedback remains coarse and standard preference optimization targets single-output comparisons.Flow-GRPO extends on-policy optimization to long-horizon rollouts with sparse trajectory-level rewards.Source: §2 · p. 3

3 The Proposed Fair Multi-level Preference Optimization (Φ-MPO) Approach

The approach represents agentic reasoning as structured, multi-turn trajectories and optimizes them with trajectory- and state-level preference signals. It extends this framework with focal weighting to address easy-pair dominance in imbalanced data.

3 The Proposed Fair Multi-level Preference Optimization (Φ-MPO) Approach

  • The framework models tool-mediated reasoning as an MDP with a trainable planner, executor, verifier, and solution generator communicating through evolving memory.The planner selects actions, the executor returns tool observations, and the verifier signals whether evidence is sufficient for an answer.Source: §3 · p. 3

3.1 Agentic Learning via Preference Optimization

  • Trajectory-level DPO learns from preferred and dispreferred rollouts, but its final-outcome signal gives every intermediate action the same supervision.The paper notes that this coarse signal can make it difficult to identify which steps contribute to success or failure.Source: §3.1 · pp. 4–5

3.1.1 Multi-level Preference Learning Approach to Agentic Learning

  • MPO decomposes trajectory preferences into comparisons between semantically aligned intermediate states, providing denser credit assignment across multi-turn reasoning.The alignment accommodates trajectories with different lengths and reasoning structures.Source: §3.1.1 · p. 5
  • Rollout preferences are constructed by scoring sampled trajectories for final correctness and reasoning efficiency, then selecting the highest- and lowest-scored rollouts.The score accounts for correctness, tool calls, and trajectory length.Source: §3.1.1 · p. 5

3.1.2 Theoretical Analysis of Preference Learning in Agentic Learning

  • Under unclipped trajectory-level updates with binary rewards, K-GRPO and DPO share a pairwise gradient basis and differ only by positive weighting.The paper interprets DPO as making GRPO’s pairwise preference structure explicit, without explicit value estimation or repeated on-policy sampling.Source: §3.1.2 · p. 6

3.2 Imbalanced Learning in Agentic Reasoning

  • Frequent easy aligned pairs can dominate updates, while rare transitions such as correcting a tool call receive too few meaningful updates.The paper links this imbalance to biased optimization and slow improvement on underrepresented hard decisions.Source: §3.2 · p. 6
  • Φ-MPO applies focal weighting to suppress easy-pair contributions and emphasize hard pairs, mitigating easy-majority dominance under the stated conditions.When γ = 0, the objective recovers standard MPO; as γ increases, easy-pair contributions decay while hard pairs retain meaningful gradients.Source: §3.2 · pp. 6–7

4 Experiments

The experiments evaluate Φ-MPO on search and agentic reasoning benchmarks and test its training setup, objective components, turn budget, focal weighting, and backbone scaling. The reported results show strong benchmark performance and gains associated with multi-level supervision and focal balancing.

4.1 Implementation and Benchmarks

  • The implementation uses Qwen2.5-7B-Instruct for four agent modules, five tools, and a maximum of 3 training turns and 10 evaluation turns.Training uses a mixture of Search-R1 and DeepMath; evaluation covers knowledge-intensive search, agentic, mathematical, and scientific reasoning benchmarks.Source: §4.1 · pp. 7–8

4.2 Main Results

  • 81.6% on Bamboogle, 81.5% on 2Wiki, 69.0% on HotpotQA, 35.0% on Musique, and 41.7% on GAIA; Qwen2.5-7B beats GPT-4o across all five.The passage also reports that this model outperforms prior search-based, agentic, and reinforcement-learning baselines, including FlowGRPO.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Ablation Studies

  • 81.6% on Bamboogle, 81.5% on 2Wiki, 41.7% on GAIA, and 50.0% on AIME24 are the final Φ-MPO results in the objective comparison.Adding MPO improves over trajectory-level DPO, and Φ-MPO further improves those results; the passage attributes gains to multi-level supervision and focal balancing.Source: Table 3 · p. 9, §4.3 · p. 9
  • Flow-GRPO reaches 69.6% on Bamboogle, 77.2% on 2Wiki, 33.1% on GAIA, and 40.0% on AIME24; Φ-MPO outperforms it on all benchmarks.The passage says Frozen is competitive on Bamboogle and 2Wiki, while SFT performs worse across all benchmarks.Source: §4.3 · pp. 8–9, Table 3 · p. 9
  • At T = 10, results reach 81.6% on Bamboogle, 81.5% on 2Wiki, 41.7% on GAIA, and 50.0% on AIME24.Performance increases consistently as the turn budget grows from T = 3 to T = 10; the passage links additional turns to deeper planning, tool use, and recovery from early mistakes.Source: Table 4 · p. 9, §4.3 · p. 9
  • With γ = 2.0, results reach 81.6% on Bamboogle, 81.5% on 2Wiki, 41.7% on GAIA, and 50.0% on AIME24.The passage says γ = 5.0 degrades performance because overly strong focal weighting suppresses too many gradients; γ = 2.0 best balances easy-majority patterns and hard decisions.Source: Table 5 · p. 9, §4.3 · p. 9
  • Qwen2.5-3B reaches 72.0% on Bamboogle, 72.5% on 2Wiki, 32.3% on GAIA, and 40.0% on AIME24, surpassing AgentFlow with Qwen2.5-7B on several tasks.The passage reports consistent gains over the stated baselines across Qwen2.5 capacities and says Qwen3.5-9B outperforms Qwen3.5-4B.Source: §4.3 · p. 9

5 Conclusions and Limitations

The paper presents Φ-MPO as a framework for long-horizon agentic learning with imbalanced data, and reports experimental and theoretical effectiveness. Its limitations include tuning choices and reliance on the quality of on-the-fly preference rollouts.

Limitations

  • No quantitative findings can be read from the supplied Table 6 caption, which identifies only model-size effectiveness as its topic.Source: Table 6 · p. 9

Conclusions

  • The proposed Φ-MPO objective is intended to address both long-horizon reasoning and data imbalance in multi-turn agentic systems.The paper also reports theoretical analysis of the framework’s effectiveness and robustness.Source: §5 · p. 9
  • Extensive experiments on multiple benchmarks are reported to confirm the method’s effectiveness compared with other methods.Source: §5 · p. 9

Limitations

  • The method’s design and hyperparameters constrain its use, particularly focal parameter γ, objective-weight coefficients, and maximum rollout budget.Preference-pair construction also depends on on-the-fly rollout quality, which may be sensitive to rollout diversity, reward reliability, and supervision stability.Source: §5 · pp. 9–10

A.1 Proof of Theorem 1

Under unclipped trajectory-level updates with binary rewards, the proof establishes that K-GRPO and DPO use the same pairwise preference-gradient directions, differing only in positive weighting.

A.1 Proof of Theorem 1

  • The proof assumes groups of at least two sampled trajectories with binary rewards and at least one preferred and one dispreferred trajectory.It defines the trajectory-level score as the gradient of the log policy probability for a trajectory.Source: §A.1 · p. 14
  • Under unclipped trajectory-level updates with binary rewards, K-GRPO and DPO are pairwise equivalent up to positive weighting.Their shared gradient basis is the difference between preferred and dispreferred trajectory score functions.Source: §A.1 · p. 14
  • K-GRPO is a positive-weighted average of pairwise preference gradients formed from preferred-dispreferred trajectory pairs.Source: §A.1 · p. 15

A.2 Proof of Theorem 2

The proof argues that Φ-MPO suppresses the optimization influence of overrepresented easy pairs, mitigating imbalance caused by easy-majority dominance.

A.2 Proof of Theorem 2

  • The theorem uses the focal loss Lγ(p) = −(1 − p)^γ log p, with p = σ(z) and γ ≥ 0, to express margin-gradient magnitude.Source: §A.2 · p. 15
  • Φ-MPO suppresses the optimization influence of overrepresented easy pairs, mitigating imbalance from easy-majority dominance.Source: §A.2 · p. 15
  • The proof sums bounds over easy-majority and other pair sets, then takes their ratio to establish the claim.Source: §A.2 · p. 16

B.1 Implementation.

The implementation trains the Action Planner within a four-module agentic system, using fixed supporting modules and interactive tools. It constructs preference data from sampled rollouts and uses specified training and evaluation settings.

B.1 Implementation

  • Only the Action Planner is trainable; the Tool Executor, Execution Verifier, and Solution Generator remain fixed during training.The framework uses shared evolving memory and five interactive tools, including Python, Google Search, Wikipedia Search, and Web Search.Source: §B.1 · p. 16
  • Training samples four rollouts per input, adds four more when all ranking scores tie, and excludes the sample if the expanded set remains tied.Source: §B.1 · p. 16
  • Rollouts are limited to three turns for training and ten for evaluation, with GPT-4o providing rewards as an LLM judge.The focal parameter γ is 2.0, and both objective weights are 1.0.Source: §B.1 · p. 16

B.2 Additional Ablation Study

Table 7 examines how trajectory-level DPO and Fair-MPO weights balance, finding that equal weights perform best across all benchmarks.

B.2 Additional Ablation Study

  • The reported best accuracy across benchmarks was 81.6% on Bamboogle, 81.5% on 2Wiki, 41.7% on GAIA, and 50.0% on AIME24, without a stated weight setting.Source: §B.2 · p. 17
  • Setting λΦ−MPO to 0.5 reduces accuracy to 67.2% Bamboogle, 67.0% 2Wiki, 26.8% GAIA, and 33.3% AIME24.The passage characterizes this as a much larger degradation across all benchmarks.Source: §B.2 · p. 17
  • Reducing λDPO to 0.5 causes a moderate performance drop, indicating that the global trajectory-level preference signal remains important.Source: §B.2 · p. 17

Related papers

Loading 2609.33323v1…