PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Group Relative Policy Optimization

Yuzheng Cai, Siqi Cai, Yuchen Shi, Zihan Xu, Lichao Chen, Yulei Qin, Xiaoyu Tan, Gang Li, Zongyi Li, Haojia Lin, Yong Mao, Ke Li, Xing Sun

arXiv:2510.08191v1cs.CL

TL;DR

LLM agents underperform in specialized domains, while parameter updates require costly computation and substantial data that may be scarce and prone to overfitting. Training-Free GRPO instead distills semantic rollout advantages into experiential token priors without changing model parameters. On mathematical reasoning and web-searching tasks, it improves frozen DeepSeek-V3.1-Terminus and outperforms fine-tuned small LLMs with minimal data and cost.

  • Problem

    LLM agents underperform in specialized domains, while parameter-tuning methods face computational cost, data scarcity, and overfitting concerns.

  • Method

    Training-Free GRPO shifts optimization from parameter space to context space by iteratively distilling semantic group advantages into experiential token priors without parameter updates.

  • Results

    Training-Free GRPO significantly improves frozen DeepSeek-V3.1-Terminus on mathematical reasoning and web searching, outperforming fine-tuned 32B models with only dozens of samples and a fraction of the computational resources.

  • Takeaways & Limitations

    Training-Free GRPO offers a practical, cost-effective pathway for adapting powerful LLM agents while preserving frozen-model parameters and generalization.

  • Takeaways & Limitations

    The method may depend on the underlying model’s reasoning and tool-use capabilities, with model capability serving as a prerequisite for effective experience-based optimization.

Abstract

from arXiv · show

Recent advances in Large Language Model (LLM) agents have demonstrated their promising general capabilities. However, their performance in specialized real-world domains often degrades due to challenges in effectively integrating external tools and specific prompting strategies. While methods like agentic reinforcement learning have been proposed to address this, they typically rely on costly parameter updates, for example, through a process that uses Supervised Fine-Tuning (SFT) followed by a Reinforcement Learning (RL) phase with Group Relative Policy Optimization (GRPO) to alter the output distribution. However, we argue that LLMs can achieve a similar effect on the output distribution by learning experiential knowledge as a token prior, which is a far more lightweight approach that not only addresses practical data scarcity but also avoids the common issue of overfitting. To this end, we propose Training-Free Group Relative Policy Optimization (Training-Free GRPO), a cost-effective solution that enhances LLM agent performance without any parameter updates. Our method leverages the group relative semantic advantage instead of numerical ones within each group of rollouts, iteratively distilling high-quality experiential knowledge during multi-epoch learning on a minimal ground-truth data. Such knowledge serves as the learned token prior, which is seamlessly integrated during LLM API calls to guide model behavior. Experiments on mathematical reasoning and web searching tasks demonstrate that Training-Free GRPO, when applied to DeepSeek-V3.1-Terminus, significantly improves out-of-domain performance. With just a few dozen training samples, Training-Free GRPO outperforms fine-tuned small LLMs with marginal training data and cost.

1 Introduction

LLM agents often struggle in specialized domains, while parameter-tuning approaches impose high computational, data, deployment, and generalization costs. Training-Free GRPO shifts optimization into context through semantic rollout advantages and experiential token priors, improving frozen-model performance with limited data.

  • Parameter-tuning approaches are costly, require specialized deployment, generalize poorly across domains, and often need multiple models for comprehensive task coverage.These constraints increase computational expense, system complexity, and maintenance overhead.
  • Fine-tuning specialized agents is further constrained by scarce, expensive annotated data and overfitting when only limited samples are available.These limitations motivate non-parametric adaptation with lower data and computational costs.
  • Training-Free GRPO improves frozen DeepSeek-V3.1-Terminus performance on challenging tasks with only dozens of training samples, surpassing fine-tuned 32B models at a fraction of the computational cost.The method is evaluated on mathematical reasoning and interactive web searching.
  • Training-Free GRPO shifts policy optimization from parameter space to context space by using evolving experiential knowledge as token priors without gradient updates.The original model parameters remain unchanged while contextual knowledge guides output behavior.
  • Semantic group advantage replaces numerical GRPO advantage, allowing the model to introspect rollout groups and iteratively distill experiential knowledge.The approach retains multi-epoch learning while using inference-only operations to refine contextual priors.
  • Training-Free GRPO preserves generalization by keeping model parameters frozen and plugging in different token priors instead of deploying multiple fine-tuned specialists.The contribution is presented as a practical, cost-effective alternative across domains.

2 Training-Free GRPO

Training-Free GRPO reproduces GRPO’s group-based optimization logic without updating model parameters, instead refining an external experience library that conditions later outputs. It compares rollout quality semantically, extracts natural-language experiences, and applies add, delete, modify, or keep operations to the library.

  • Training-Free GRPO: Training-Free GRPO leaves the model parameters frozen and shifts optimization into an external experiential knowledge library used during inference.The conditioned policy changes its output distribution through context rather than parameter updates.
  • Training-Free GRPO: Figure 2 contrasts vanilla GRPO’s parameter-update procedure with Training-Free GRPO’s inference-time context updates.The Training-Free procedure retains group rollouts and reward evaluation while replacing gradient-based parameter optimization with experience-library updates.
  • Rollout and Reward: Each query produces a group of outputs conditioned on the current experience library, and every output receives a scalar reward from the reward model.This rollout and reward process mirrors standard GRPO while conditioning the policy on E.
  • Group Advantage Computation: Training-Free GRPO replaces numerical group advantages with natural-language semantic advantages that explain why outputs succeed or fail relative to one another.The method summarizes each rollout, compares the summaries, and extracts a concise experience for groups containing clear winners and losers.
  • Optimization: The experience library is updated from the extracted advantages through add, delete, modify, or keep operations.These operations respectively append new experience, remove low-quality experience, refine existing experience, or leave the library unchanged.

3 Evalution

Training-Free GRPO improves frozen DeepSeek-V3.1-Terminus on mathematical reasoning and web searching using minimal experiential data and inference-only optimization. Results and ablations show gains across domains, while effectiveness depends on the underlying model’s capabilities.

  • Mathematical Reasoning: 82.7% on AIME24 and 73.3% on AIME25 improve over ReAct by +2.7% and +5.4% using 100 examples and zero gradient updates.The method also surpasses fine-tuned 32B RL systems while costing approximately $18.
  • Learning Dynamics: Mean@32 on AIME24 and AIME25 improves at each learning step, while average tool calls decrease during training and evaluation.The authors attribute this to learned experiences that encourage correct reasoning and more efficient tool use.
  • Ablations: Removing group computation harms performance, confirming that comparing trajectories within each group supports semantic advantage and experience optimization.The default group size is 5 in the mathematical experiments.
  • Web Searching: 67.8% pass@1 on WebWalkerQA exceeds the 63.2% ReAct baseline with DeepSeek-V3.1-Terminus.The method uses three training-free optimization epochs with group size G = 3.
  • Ablations: Full Training-Free GRPO achieves the best WebWalkerQA ablation performance at 68.6% pass@1 and 78.4% pass@3.Directly generated experiences reduce pass@1 to 64.7%, while the no-ground-truth variant reaches 66.7% pass@1 and 78.4% pass@3.
  • Model Applicability: Applying Training-Free GRPO to QwQ-32B yields 25.5% pass@1, below both DeepSeek-V3.1-Terminus at 66.7% and QwQ-32B’s 27.5% ReAct baseline.The result suggests that model reasoning and tool-use capability is a prerequisite for effective experience-based optimization.

4 Comparing RL Learning on Context Space and Parameter Space

Training-Free GRPO shifts optimization from costly parameter tuning to context-space guidance, preserving a frozen model while reducing training costs. The paper contrasts this with domain-specialized models that incur transfer losses and fixed infrastructure demands.

  • Cross-domain Transfer Analysis: ReTool’s WebWalker performance drops to 18.3% after mathematical specialization, whereas Training-Free GRPO achieves strong performance across both domains.MiroThinker shows the converse pattern by underperforming ReTool on AIME benchmarks.
  • Cross-domain Transfer Analysis: Training-Free GRPO avoids the domain-specialization trade-off by plugging domain-specific learned experiences into a frozen LLM.The approach is presented as suitable for limited data, constrained budgets, or volatile inference demand.
  • Training Cost: Training-Free GRPO achieves superior AIME performance with three steps over 100 samples and minimal cost, compared with ReTool’s roughly $10,000 training expense.The training-free process consumes 38M input tokens and 6.6M output tokens within six hours.
  • Inference Cost: Serving a trained ReTool-32B model requires fixed GPU infrastructure, which can be inefficient when request volumes fluctuate or remain low.Its average inference cost is $0.005 per AIME problem under the described serving setup.

5 Related Work

Related work spans tool-using LLM agents, reinforcement-learning methods for aligning long-horizon behavior, and inference-time approaches that improve outputs without updating model weights. Training-Free GRPO builds on these lines by combining agent interaction, GRPO-style relative evaluation, and context-space optimization.

  • LLM Agents: ReAct interleaves explicit reasoning with actions, while Toolformer studies self-supervised learning of tool use.These methods frame LLMs as agents that dynamically plan through external tools.
  • Reinforcement Learning: PPO uses a policy and critic for token-level value estimation, whereas GRPO estimates advantages directly from groups of responses.This group-based formulation provides the reinforcement-learning background for the paper’s semantic group comparison.
  • Reinforcement Learning: ReTool applies PPO to interleave natural language with code execution, while Chain-of-Agents uses dynamic activation of specialized agents within one model.These examples target tool use and multi-agent collaboration for complex tasks.
  • Training-Free Methods: Inference-time methods such as in-context learning, Self-Refine, and Reflexion improve behavior without updating model weights.They use demonstrations, verbal feedback, or iterative refinement to steer subsequent outputs.

6 Conclusion

Training-Free GRPO shifts reinforcement-learning policy optimization from model parameters to context, using experiential knowledge to steer frozen LLM agents. It improves specialized-domain performance while addressing data scarcity and computational cost.

  • Training-Free GRPO shifts RL policy optimization from the parameter space to the context space.
  • The method distills group-based semantic advantages into experiential knowledge that serves as a token prior for steering frozen LLM agents.
  • Training-Free GRPO improves specialized-domain performance while addressing data scarcity and high computational cost.

Contributions

The paper lists authors from Tencent Youtu Lab, Fudan University, and Xiamen University, with four authors marked as equal contributors.

  • The authors are Yuzheng Cai, Siqi Cai, Yuchen Shi, Zihan Xu, Lichao Chen, Yulei Qin, Xiaoyu Tan, Gang Li, Zongyi Li, Haojia Lin, Yong Mao, Ke LiQ, and Xing Sun.
  • The affiliations include Tencent Youtu Lab, Fudan University, and Xiamen University.
  • Yuzheng Cai, Siqi Cai, Yuchen Shi, and Zihan Xu are identified as equal contributors.

A Case Study

The case study illustrates Training-Free GRPO enhancing ReAct with DeepSeek-V3.1-Terminus and corresponding tools in mathematical-reasoning and web-searching scenarios.

  • The case study covers mathematical reasoning and web searching scenarios.
  • It provides examples of how Training-Free GRPO enhances ReAct.
  • The examples use DeepSeek-V3.1-Terminus together with corresponding tools.

A.1 Experience-Guided Tool-Integrated Math Reasoning

The case study contrasts unassisted and experience-guided tool-integrated mathematical reasoning. Experiences correct geometric modeling and add structured parameterization, root filtering, and verification, producing a reliable solution.

  • Baseline (without experiences): The unassisted trajectory uses the wrong vertical orientation and yields CE ≈106.91 without holistic verification of all geometric constraints.The baseline accepts a spurious solution even though the value satisfies 0 < CE < 107.
  • Enhanced (With Experiences): The enhanced example is presented as a structured pipeline using a curated experience pool.
  • Enhanced (With Experiences): The experience-guided pipeline selects CE = 104 after solving a quadratic with roots 104 and 294 and applying 0 < x < 107.The selected solution is verified with DE = 3, CF = 80, EF = 184, FG = 17, and a circle through A, D, H, and G.
  • Enhanced (With Experiences): Directional-ordering and boundedness experiences enforce D–E–C–F and filter out the root x = 294.
  • Enhanced (With Experiences): Segment-addition parameterization expresses DE = 107 − x and CF = 184 − x, reducing the cyclic constraint to a single-variable quadratic.
  • A.1 Experience-Guided Tool-Integrated Math Reasoning: Experience-guided reasoning corrects vertical orientation, resolves ordering and parameterization inconsistencies, and adds principled validation gates.The comparison reports a positive impact on reliability and accuracy in tool-integrated mathematical reasoning.

A.2 Experience-Guided Web Searching

Experience-guided web searching turns an incomplete, snippet-heavy trajectory into a disciplined process that verifies both requested rewards from authoritative sources. The enhanced agent produces a complete, supported answer by refining searches and synthesizing verified statements.

  • The web-searching task asks for 2024 Creator Program payments and Private Beta 4 Play-2-Airdrop rewards.
  • The baseline confirms the Play-2-Airdrop amount but fails to verify the Creator Program’s 100 ILV weekly from an official source.
  • The enhanced agent correctly reports 100 ILV weekly for creators and 200,000 ILV over six months for PB4 players.
  • Experience guidance prioritizes official sources, identifies formal documents, refines search terms, and evaluates partial matches to confirm numerical claims.

C Examples of Learned Experiences

The appendix illustrates learned experiences extracted from Training-Free GRPO with tool use in mathematical reasoning. Examples encode reusable checks for geometry intersections and compass-direction configurations.

  • The appendix presents examples extracted from 48 learned experiences produced with Training-Free GRPO and tool use in mathematical reasoning.
  • One experience advises checking that geometry intersection solutions lie within bounded regions or segments rather than extensions.
  • Another experience recognizes southwest and southeast directions from a common point as a right-triangle configuration suitable for the Pythagorean theorem.
Loading 2510.08191v1…