Training-Free Group Relative Policy Optimization

• Specialized-domain agents can underperform because they lack domain-specific requirements, tools, or prompting strategies, while parameter-tuned approaches face cost, generalization, data-scarcity, and overfitting challenges.Source: §1 · p. 2
• The method shifts policy optimization from parameter space to context space, encoding minimal-sample experiential knowledge as lightweight token priors while preserving the original model parameters.The priors guide policy outputs through in-context learning.Source: §1 · pp. 2–3
• Training-Free GRPO replaces numerical group advantages and gradient updates with LLM-derived semantic advantages that iteratively refine experiential knowledge across rollout groups.It retains multi-epoch learning, generates multiple rollouts per query, and uses inference-only operations while leaving model parameters unchanged.Source: §1 · p. 3
• With only dozens of training samples, Training-Free GRPO improves frozen DeepSeek-V3.1-Terminus and surpasses fine-tuned 32B models using a fraction of their computational resources.The evaluations cover challenging mathematical reasoning and interactive web searching tasks.Source: §1 · p. 3
• Keeping parameters frozen and supplying different token priors is intended to preserve generalization and avoid the cost and complexity of deploying multiple fine-tuned specialist models.Source: §1 · p. 3

Training-Free GRPO

• Training-Free GRPO keeps the policy parameters frozen and maintains an external experiential knowledge library initialized as empty.This turns group-based evaluation into a non-parametric, inference-time process.Source: §2 · p. 3

Rollout and Reward

• For each query, the method generates a group of outputs conditioned on current experience and scores each output with the reward model.The rollout and reward process mirrors standard GRPO, except the policy conditions on E.Source: §2 · p. 4

Group Advantage Computation

• For groups with clear winners and losers, the LLM summarizes outputs and extracts natural-language reasons for their relative success or failure as semantic advantages.These advantages encode experiential knowledge about actions associated with high rewards.Source: §2 · p. 4, Figure 3 · p. 5

Optimization

• The method updates the experience library using semantic advantages through add, delete, modify, or keep operations.An LLM proposes operations based on the batch’s extracted experiences.Source: §2 · p. 4
• Updated experience conditions subsequent outputs, shifting their distribution toward higher-reward behavior while the frozen base model provides coherence and stability.The context changes rather than the model parameters, and the frozen model constrains excessive deviation from its reference behavior.Source: §2 · p. 5

3.1 Mathematical Reasoning

• Mean@32 rises from 80.0% to 82.7% on AIME24 and from 67.9% to 73.3% on AIME25 with Training-Free GRPO, gains of +2.7% and +5.4%.The method uses 100 DAPO-Math-17K examples and zero gradient updates, at an approximate learning cost of $18; the evaluation uses 32 runs per problem. The baseline is DeepSeek-V3.1-Terminus with ReAct.Source: Table 1 · p. 6, §3.1 · p. 6
• Across three learning steps, Mean@5 on the training set and Mean@32 on AIME24 and AIME25 improve, while average tool calls decrease.The reported dynamics suggest improved out-of-domain performance alongside more judicious tool use.Source: Figure 4 · p. 6, §3.1 · pp. 6–7
• Directly generated experiences do not improve AIME performance, and removing group computation significantly harms it relative to the default group size of 5.The group-size-1 variant distills semantic advantage from a single rollout rather than comparing trajectories within a group.Source: §3.1 · p. 7, Table 2 · p. 7
• Without ground-truth answers, Training-Free GRPO reaches 80.7% on AIME24 and 68.9% on AIME25, below the default version but demonstrating applicability when answers are unavailable.This variant compares rollouts within each group and relies on implicit majority voting, self-discrimination, and self-reflection.Source: §3.1 · p. 7, Table 2 · p. 7

3.2 Web Searching

• Pass@1 on WebWalkerQA is 67.8% with Training-Free GRPO, compared with 63.2% for the baseline.The method uses 100 queries sampled from AFM and runs three epochs with group size 3, without gradient-based updates.Source: Table 4 · p. 8, §3.2 · p. 8

3.1 Mathematical Reasoning

• Training-Free GRPO consistently improves out-of-domain AIME results on Qwen3-32B and Qwen2.5-72B-Instruct; the DeepSeek-V3.1-Terminus results exceed those of the compared RL-trained models.The paper reports using 100 training examples and an approximate $18 learning cost for DeepSeek-V3.1-Terminus, versus thousands of examples and substantial computational resources for the cited RL methods.Source: Table 3 · p. 7, §3.1 · p. 8

3.2 Web Searching

• On a stratified 51-instance WebWalkerQA subset, the full method reaches 68.6% Pass@1 and 78.4% Pass@3; without ground truth it matches ReAct at 66.7% Pass@1 and reaches 78.4% Pass@3.Directly generated experiences score 64.7% Pass@1 versus ReAct’s 66.7%, while ablated models are evaluated after two epochs.Source: §3.2 · pp. 8–9, Table 5 · p. 8
• On WebWalkerQA, Training-Free GRPO with QwQ-32B scores 25.5% Pass@1, below both DeepSeek-V3.1-Terminus at 66.7% and QwQ-32B’s ReAct baseline at 27.5%.The authors suggest that effective experience-based optimization in complex tool-use scenarios may depend on the underlying model’s reasoning and tool-use capabilities.Source: §3.2 · p. 9

4.1 Cross-domain Transfer Analysis

• Training-Free GRPO achieves state-of-the-art performance in math and web domains simultaneously, while math-specialized ReTool [4] falls to 18.3% on WebWalker, below untuned ReAct [1].The passage also reports that web-optimized MiroThinker underperforms math-trained ReTool [4] on AIME benchmarks, illustrating the domain-specialization trade-off.Source: §4.1 · p. 9

4.2 Computational Costs

• Training-Free GRPO costs approximately $18 to train, versus roughly $10,000 for ReTool [4], a reduction of over two orders of magnitude.The reported setup uses 3 training steps over 100 samples, completed within 6 hours; it consumes 38M input tokens and 6.6M output tokens.Source: §4.2 · p. 10
• Inference averages $0.02 per problem for Training-Free GRPO versus $0.005 for ReTool-32B, but the API-based approach avoids fixed GPU-serving overhead for irregular or sparse demand.The comparison assumes a dedicated setup with 4 GPUs processing about 400 AIME problems per hour for ReTool-32B; Training-Free GRPO uses about 60K input and 8K output tokens per request, with cache-hit pricing for reused contexts.Source: §4.2 · p. 10

LLM Agents

• LLM agents use tools to address limitations such as stale knowledge and imprecise computation, building from ReAct [1] and Toolformer [26] to systems including MetaGPT [27], CodeAct [28], and OWL [29].ReAct [1] prompts explicit reasoning and actions for dynamic planning, while Toolformer [26] learns API invocation through parameter fine-tuning.Source: §5 · p. 10

Reinforcement Learning

• Parameter-updating agentic RL approaches incur prohibitive computational cost, typically restricting application to LLMs with fewer than 32B parameters, and achieve diminishing returns versus larger, more powerful frozen LLMs.Training-Free GRPO seeks comparable or better performance on state-of-the-art LLMs without parameter updates.Source: §5 · p. 11

Training-Free Methods

• Training-Free GRPO learns a shared experience library across epochs and compares multiple rollouts per query using semantic advantage, unlike within-query refinement methods.Unlike Agent KB [36], it injects learned experiences directly into prompts and uses a consistent, multi-epoch pipeline closer to on-policy RL.Source: §5 · p. 11
• Inference-time methods such as ICL, Self-Refine, Reflexion [33], ICRL [34], and TextGrad primarily improve a single query iteratively, unlike Training-Free GRPO’s cross-query learning.Training-Free GRPO uses a separate dataset over multiple epochs and compares trajectories within each query’s rollout group; the passage says this semantic advantage was confirmed effective in Section 3.1.Source: §5 · p. 11

6 Conclusion

• Training-Free GRPO shifts RL policy optimization from parameter space to context space.The paper presents this as a new paradigm for adapting LLM agents.Source: §6 · p. 11
• Group-based rollouts iteratively distill semantic advantage into evolving experiential knowledge that acts as a token prior to steer a frozen LLM agent.This approach steers the agent’s output distribution without updating its parameters.Source: §6 · p. 11
• Training-Free GRPO achieves significant performance gains in specialized domains, addresses data scarcity and high computational cost, and outperforms traditional parameter-tuning methods.The paper presents the method as an efficient pathway for making advanced agent capabilities more accessible and practical for real-world applications.Source: §6 · p. 11

A Case Study

• Examples show Training-Free GRPO enhancing ReAct [1] with DeepSeek-V3.1-Terminus and corresponding tools for mathematical reasoning and web searching.Source: A Case Study · p. 15

A.1 Experience-Guided Tool-Integrated Math Reasoning

• The unassisted agent reports CE ≈ 106.9076 after assigning the short side the wrong vertical orientation and accepting a spurious solution without holistic verification.Although the result lies within 0 < CE < 107, that bound alone does not establish consistency with all geometric constraints.Source: §A.1 · pp. 15–16
• The guided solution sets CE = x, parameterizes DE = 107 − x and CF = 184 − x, then derives a quadratic with roots 104 and 294.The agent orients the short side downward and uses the circle equation to model concyclicity.Source: §A.1 · p. 17
• CE = 104 is selected after the bound check, and the agent verifies the rectangle dimensions, collinearity, and concyclicity.The quadratic yields roots 104 and 294; only 104 satisfies 0 < x < 107.Source: §A.1 · p. 17
• Experiences [29], [37], [1], and address ordering, parameterization, root filtering, and full verification, respectively, correcting the baseline’s errors.The comparison attributes the guided trajectory’s reliability and accuracy to these domain-specific reasoning behaviors.Source: §A.1 · pp. 16–17

• Without experiences, the agent relies on broad searches and snippets, confirms the Play-2-Airdrop figure, but fails to verify the Creator Program’s 100 ILV weekly from an official source.Connection errors prompt more broad searches rather than alternative primary-source strategies, leaving the answer incorrect or incomplete.Source: §A.2 · p. 17, Figure 7 · p. 18
• Experiences prioritize official sources [14], [4], refine searches toward formal titles, [12], [1], and use partial-match evaluation to verify claims at authoritative sources.These behaviors address snippet reliance, failure to open the relevant Creator Program post, and unverified numerical claims.Source: §A.2 · pp. 17–18
• With experiences, the agent correctly reports creators’ 100 ILV weekly and PB4 players’ 200,000 ILV over six months.The answer is supported by the Creator Program post and PB4 posts.Source: §A.2 · pp. 17–18, Figure 8 · p. 19

B Prompts

• Figures 9–13 cover ReAct problem solving, adding experiential knowledge E, trajectory summarization, group-advantage computation, and optimizing E from same-batch advantages.Figure 9 is the ReAct system prompt; Figures 10–13 cover the remaining prompt functions.Source: B Prompts · p. 20, Figure 9 · p. 20, Figure 10 · p. 20
• The group-advantage prompt asks the agent to compare successful and incorrect attempts, identify key decisions and reasoning errors, and examine missed or insufficiently guiding experiences.It also directs the agent to note strategies and patterns used or missed and consider why trajectories fail.Source: B Prompts · p. 21
• The prompt allows modifying, adding, or deleting experiences, with a cap of {max number of operations} clear, generalizable lessons for the case.Before each update, it asks for the experience’s relevance conditions, applicable problem features, and similar patterns.Source: B Prompts · p. 21
• Modified or added experiences should begin with general background, emphasize strategic thinking rather than specific calculations, and focus on transferable decision points.Source: B Prompts · p. 21

• Figure 14 shows examples of learned experiences from 48 experiences produced by Training-Free GRPO with tool use in math reasoning.Source: Figure 14 · p. 23, C Examples of Learned Experiences · p. 23