PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

Tianshi Xu, Yuteng Chen, Meng Li

arXiv:2601.15141v2cs.LG

TL;DR

Small agentic models can accumulate execution failures, and outcome-only rewards can give erroneous actions the same positive credit as corrections. CLEANER’s SAAR retrospectively purifies self-corrected trajectories, with reported gains across benchmarks and SOTA-level performance in one-third the RL steps.

  • Problem

    For parameter-constrained models, execution failures pollute trajectories, while outcome-only rewards assign erroneous actions the same positive credit as corrective actions.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    CLEANER uses SAAR to replace failed actions with later self-corrections, choosing shallow repairs or full reasoning-segment substitutions according to semantic similarity.

    Source: §1 · pp. 2–3

  • Results

    CLEANER reports average accuracy gains of 6% on AIME, 3% on GPQA, and 5% on LiveCodeBench, matching SOTA performance with one-third of RL steps.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Self-purified paths align learning signals with correct behavior, supporting trajectory purification as a scalable alternative for efficient agentic RL.

    Source: §1 · p. 3, §6 · p. 9

Abstract

from arXiv · show

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub

1 INTRODUCTION

For small agentic models, execution failures contaminate trajectories, while outcome-only rewards can reinforce both failed and corrective actions. CLEANER uses SAAR to replace failures with similarity-scaled self-corrections and reports benchmark gains while matching SOTA performance in one-third the RL steps.

1 INTRODUCTION

  • Parameter-constrained models frequently produce execution failures during RL exploration, accumulating misleading context that can hinder policy improvement.The paper links accumulated tool errors with performance bottlenecks or accuracy degradation.Source: §1 · p. 2
  • Under outcome-only rewards, successful trajectories give erroneous and corrective actions identical positive reinforcement, creating ambiguous credit assignment.The paper identifies dense tool-level rewards and supersampling-based filtering as alternatives with reward-hacking and computational-cost drawbacks, respectively.Source: §1 · p. 2
  • SAAR retrospectively replaces an erroneous action with a later self-correction, using semantic similarity to choose shallow repairs or full reasoning-segment substitution.High similarity corresponds to minor execution errors; low similarity signals deeper logical flaws.Source: §1 · pp. 2–3
  • CLEANER reports average accuracy gains of 6% on AIME, 3% on GPQA, and 5% on LiveCodeBench, matching SOTA performance with one-third of RL steps.The gains and training-step comparison are reported together as the method’s overall benchmark result.Source: §1 · p. 3
  • CLEANER trains on self-purified trajectories so models can internalize correct reasoning patterns while filtering execution-noise interference.The approach constructs clean training signals through SAAR.Source: §1 · p. 3

2 PRELIMINARIES

The preliminaries define agentic trajectories as histories built from reasoning, executable Python code, and environment observations. They describe outcome-only reward with group-relative policy optimization as the training setup.

2.1 AGENTIC REASONING TRAJECTORIES

  • At each turn, the policy generates reasoning and a Python code action, the environment returns an observation, and the interaction is appended to the history.The history begins with the user query and prior interaction tuples.Source: §2.1 · p. 3
  • Failed execution traces remain in the history and introduce error-induced noise into subsequent conditioning and the learning signal.The paper characterizes these segments as low-information context that can disrupt later reasoning.Source: §2.1 · p. 3

2.2 GROUP-BASED POLICY OPTIMIZATION WITH OUTCOME-ONLY REWARD

  • The training setup uses sparse, outcome-only supervision and Group Relative Policy Optimization, avoiding manual reward engineering and reward hacking.GRPO manages credit assignment through group-relative comparisons.Source: §2.2 · p. 3

3 PROBLEM FORMULATION: IMPACT OF CODE TOOL EXECUTION NOISE

Environment interactions introduce trajectory-level noise that hinders policy optimization. Under outcome-only rewards, a successful trajectory can reinforce both failed and corrected actions.

  • Interactions with the environment introduce trajectory-level noise that hinders efficient policy optimization.Source: §3 · p. 4
  • Failed execution traces persist in history despite contributing little to task resolution, consuming context and disrupting later reasoning.Source: §3 · p. 4
  • Tool-failure spikes correlate with AIME25 performance drops, indicating optimization sensitivity to noisy trajectories.Source: Figure 2 · p. 4
  • Under binary outcome rewards, failed and corrected actions in a successful trajectory receive identical positive credit, diluting learning and reinforcing suboptimal tool use.Source: §3 · p. 4

4 METHOD: SIMILARITY-AWARE ADAPTIVE ROLLBACK (SAAR)

CLEANER uses SAAR to turn successful self-corrections after execution errors into purified training trajectories. Code similarity determines whether the correction replaces only the failed action or the reasoning segment as well.

4 METHOD: SIMILARITY-AWARE ADAPTIVE ROLLBACK (SAAR)

  • SAAR retrospectively removes execution failures from rollouts, producing self-purified paths that reinforce correct reasoning rather than error-recovery loops.Source: §4 · pp. 4–5, §4.2 · p. 5

4.1 PHASE I: ERROR TRIGGER AND LOOKAHEAD CORRECTION

  • After an execution error, Phase I freezes the history, generates and executes a correction, and repeats the correction loop up to K attempts if failures persist.Source: §4.1 · p. 5

4.2 PHASE II: SIMILARITY-AWARE ADAPTIVE REPLACEMENT

  • SAAR uses semantic similarity to choose shallow repair, grafting corrected code onto preserved reasoning, or deep replacement, adopting the correction thought for a failed turn.Source: §4.2 · p. 5

4.3 IMPLEMENTATION DETAILS

  • RadixAttention reuses the invariant history prefix’s KV cache, limiting recomputation of corrected-action log-probabilities to modified suffix segments.Source: §4.3 · pp. 5–6
  • Compared with the DAPO-baseline, CLEANER effectively suppresses erroneous tool calls in trajectories, leading to significant performance gains; Figure 4 shows the evolution of training metrics during RL.For Qwen2.5-7B, SAAR is applied to 70% of trajectories while 30% are retained in their original state.Source: Figure 4 · p. 6, §4.3 · p. 6
  • CLEANER achieves results comparable to state-of-the-art baselines while using one-third of DemyAgent-4B’s training steps.Source: Table 1 · p. 6

5 EXPERIMENTS

Across benchmarks, CLEANER is competitive with larger and agentic systems at fewer training steps; ablations examine tool use, SAAR, learning rates, and recovery behavior.

5.1 EXPERIMENTAL SETUP

  • Experiments use Qwen3-4B and Qwen2.5-7B, train for one DAPO epoch, and evaluate on AIME24, AIME25, LiveCodeBench, and GPQA against self-contained and agentic baselines.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • AIME24/25 gains average +6% Pass@1 and +8% Pass@16; CLEANER suppresses erroneous tool calls while keeping output lengths comparable.The authors say fewer errors imply tokens shift from futile tool calls to effective reasoning.Source: §5.2 · p. 7
  • Using one-third of DemyAgent-4B’s training steps, CLEANER achieves comparable results and surpasses it on AIME24, GPQA, and LiveCodeBench.Source: §5.2 · p. 7

5.3 ABLATION STUDY

  • Starting CLEANER from the unstable step-200 DAPO policy reduces tool errors and improves AIME24 by 5.2% and AIME25 by 1.0%, but does not reach scratch-trained parity.The baseline averaged 0.6 erroneous tool invocations per trajectory at initialization; the figure reports training-metric recovery after CLEANER is introduced.Source: Figure 5 · p. 8, §5.3 · p. 8
  • Adding tools improves average accuracy by over 5% for Qwen3-4B and 20% for Qwen2.5-7B; adding SAAR yields further gains across benchmarks and scales.For Qwen3-4B, SAAR gains average 6% on AIME, 4% on GPQA, and 5% on LiveCodeBench; Qwen2.5-7B gains average 4%.Source: §5.3 · pp. 7–8, Table 2 · p. 7
  • For 4B, 2e-6 yields superior results across tested settings; the 7B model uses 1e-6 for optimization stability.Source: §5.3 · p. 8, Table 3 · p. 8
  • Without SAAR at evaluation, AIME24 Pass@1 and Pass@16 decline by 0.6% and 1.3%; SAAR as an inference enhancement adds 8.8% average latency.AIME25 Pass@1 declines by 2.5%, while Pass@16 gains 0.2%; GPQA and LiveCodeBench show negligible degradation.Source: §5.3 · p. 8, Table 4 · p. 8

6 CONCLUSION

CLEANER uses Similarity-Aware Adaptive Rollback to purify noisy exploration trajectories before optimization. It reports gains across reasoning and coding benchmarks and matches state-of-the-art performance with one-third of the training steps.

  • CLEANER converts noisy exploration logs into self-purified trajectories before optimization; it reports average accuracy gains of 6%, 3%, and 5% on AIME24/25, GPQA, and LiveCodeBench, respectively.It matches state-of-the-art performance using one-third of the training steps.Source: §6 · p. 9

A DETAIL PRELIMINARIES

The paper uses GRPO to optimize the policy without a separate value-function critic. It normalizes trajectory rewards against group statistics and maximizes a surrogate policy objective.

A DETAIL PRELIMINARIES

  • GRPO estimates the reward baseline from group statistics, avoiding the overhead of a separate value-function critic.For each query, a group of trajectories is sampled from the current policy.Source: A DETAIL PRELIMINARIES · p. 13
  • Each trajectory’s advantage is its reward normalized against the group reward mean and standard deviation.The mean and standard deviation are denoted µ_R and σ_R.Source: A DETAIL PRELIMINARIES · p. 13
  • The policy is updated by maximizing a surrogate objective built on the normalized trajectory advantages.The objective uses an importance-sampling ratio and a clipping hyperparameter.Source: A DETAIL PRELIMINARIES · p. 13

B IMPLEMENTATION DETAILS

The implementation uses a mixed math, code, and science RL dataset, scale-specific learning rates, and a 20,480-token context window. The appendix also reports retry and similarity settings, compute details, and evaluation sampling configurations.

B IMPLEMENTATION DETAILS

  • Table 5 summarizes RL hyperparameters, while Table 6 lists evaluation sampling configurations.The passages identify the tables’ contents but do not provide their individual settings.Source: Table 5 · p. 13, Table 6 · p. 13
  • The RL dataset combines 17k DAPO-Math samples, 4,902 math and 3,586 code Skywork-or1 samples, and 3k MegaScience science problems.The dataset is identified as an open-source mixture.Source: B IMPLEMENTATION DETAILS · p. 13
  • The RL learning rates are 2 × 10−6 for the 4B model and 1 × 10−6 for the 7B model.The maximum context is 20,480 tokens, while the empirical average sequence length is approximately 7,000 tokens.Source: B IMPLEMENTATION DETAILS · p. 13
  • The authors use K = 3 retries and recommend a similarity threshold γ between 0.5 and 1.0, with 0.5 as the default.Values below K = 3 reduce successful-trajectory recovery; higher values yield diminishing returns.Source: B IMPLEMENTATION DETAILS · p. 13
  • Experiments ran on one node with 4× NVIDIA H100 or 4× NVIDIA H200 GPUs.The full cycle took approximately 4 days for Qwen3-4B and 2 days for Qwen2.5-7B.Source: B IMPLEMENTATION DETAILS · p. 14

C UNSUCCESSFUL ATTEMPTS: LEVERAGING NEGATIVE SAMPLES VIA SAAR

The authors tested using SAAR-discarded tool calls as negative samples through online DPO paired with corrected rollouts. The approach did not improve overall success on complex reasoning tasks and could trigger late training collapse.

C UNSUCCESSFUL ATTEMPTS: LEVERAGING NEGATIVE SAMPLES VIA SAAR

  • The approach was motivated by the hypothesis that discarded erroneous actions could provide negative signals about incorrect behavior.The proposed contrastive signal was intended to refine coding and tool use.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14
  • For each SAAR recovery, the implementation paired the failed tool call with the corrected rollout and applied online DPO.Only tool-invocation tokens were unmasked, and the DPO gradient was combined with the GRPO signal.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14
  • The negative-sample approach slightly improved self-repair on specific code snippets but did not improve overall success on complex reasoning tasks and could trigger late training collapse.The authors explored this strategy as an alternative to discarding erroneous actions.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14
  • Tool failures may reflect faulty reasoning rather than syntax alone, so penalizing only tool-call tokens can disconnect actions from preceding reasoning.The authors suggest this imbalance may discourage complex code generation instead of improving reasoning.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14
  • The authors report that code-use proficiency is linked to overall reasoning, making simple token-level masking potentially counterproductive.They also note that disproportionate token weights within a trajectory can adversely affect policy-optimization stability.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14
  • The effective use and net value of negative samples in agentic RL remain open questions for future study.Source: C UNSUCCESSFUL ATTEMPTS: LEVERAGING… · p. 14

D RELATED WORK

Prior work includes supervised approaches that imitate expert tool-use trajectories and agentic RL that optimizes tool invocation through outcome-driven rewards. The paper positions CLEANER against the remaining challenge of separating high-quality learning signals from noise in tool-use trajectories.

D RELATED WORK

  • Supervised fine-tuning and executable frameworks such as CodeAct mimic expert trajectories, but behavioral cloning is described as limited to successful demonstrations.The related-work discussion says such agents may imitate surface patterns without grasping underlying causality, leaving them less equipped for noisy tool interactions.Source: D RELATED WORK · pp. 14–15
  • Agentic RL treats tool invocation as an action space and uses outcome-driven rewards to discover flexible solutions.Prior work also addresses execution noise through trajectory filtering and exploration through entropy-regulated rollout mechanisms.Source: D RELATED WORK · p. 15
  • The related-work discussion identifies separating high-quality signals from noise in complex tool-use trajectories as an unresolved challenge for policy updates.CLEANER is introduced as a mechanism to refine training data and stabilize learning.Source: D RELATED WORK · p. 15

Related papers

Loading 2601.15141v2…