PaperlayerView exact revision on arXiv

Source-linked AI summary

MENTOR: Learning from Failure with On-Policy Pedagogical Intervention

Guanlan Yue, Hang Yu, Xiaoyong Huang, Ruipeng Li, Wei Zhang, Bingchang Liu, Rui Wang

arXiv:2610.13594v1cs.AI

TL;DR

RLVR can suffer zero-reward collapse when a task exceeds the model’s capability, leaving failed trajectories unused despite their diagnostic value. MENTOR diagnoses failures and provides targeted, online instruction; experiments report gains across math and code generation, including a 78.3% relative gain in math on Llama-3.2-3B-Instruct.

  • Problem

    When task difficulty exceeds a model’s capability, all sampled trajectories may fail and the resulting zero reward halts policy improvement; discarding failures also loses diagnostic information.

    Source: §1 · pp. 1–2

  • Method

    MENTOR uses a multi-agent closed loop to diagnose failed trajectories, generate and validate targeted instructional prompts, and refine them for online RL pipelines such as GRPO.

    Source: §1 · p. 2

  • Results

    78.3% relative gain in math on Llama-3.2-3B-Instruct over its base model, alongside consistent outperformance of baselines across math and code tasks.

    Source: §4.2 · pp. 6–7

  • Takeaways & Limitations

    MENTOR’s targeted teaching from failure is reported to overcome the training ceiling imposed by zero-reward collapse on challenging math and code benchmarks.

    Source: §5 · p. 8

    The prompt-generation and reasoning-guidance mechanism is validated only in single-modal scenarios; applicability to complex dynamic environments and multi-modal fusion remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement Learning from Verifiable Rewards (RLVR) is a key driver for Large Reasoning Models, but it is vulnerable to "zero-reward collapse", a training failure where a sudden increase in task difficulty causes the learning signal to vanish. We argue that the conventional approach of discarding these failed trajectories is wasteful, as they contain valuable diagnostic information about the model's cognitive gaps, a bright side that can be leveraged for recovery. Existing solutions, however, often regress to static, "off-policy-style" interventions like pre-defined hint libraries or perform post-hoc refinement, failing to provide the adaptive, forward-looking guidance needed to escape the collapse. To address this, we propose a paradigm shift towards on-policy pedagogical intervention and introduce MENTOR, a multi-agent framework that acts as a live mentor. MENTOR diagnoses the root cause of failure in real-time and dynamically generates targeted, instructional prompts to guide the learning agent's next attempt. As a plug-and-play module, MENTOR integrates seamlessly into existing policy optimization pipelines like GRPO without modifying the training infrastructure. Extensive experiments on math and code generation show that MENTOR effectively mitigates zero-reward collapse, outperforming state-of-the-art baselines. Notably, it achieves massive average relative gains of up to 78.3% on Llama-3.2-3B-Instruct over its base model.

1 Introduction

RLVR can lose its learning signal when task difficulty exceeds a model’s capability, while failed trajectories contain diagnostic information that could guide recovery. MENTOR uses that information for online, targeted instruction and is reported to mitigate collapse while outperforming baselines.

1 Introduction

  • When task difficulty exceeds model capability, all sampled trajectories can fail, yielding zero reward, vanishing learning signals, and halted policy improvement.The passage says the zero-reward sample fraction rises rapidly and performance plateaus immediately after collapse.Source: §1 · p. 1
  • Failed trajectories contain information about error locations, cognitive deficiencies, and needed corrections, but discarding them prevents capturing real-time error patterns.The section contrasts this need with static hints or curricula and post-hoc refinement, which do not provide adaptive, forward-looking guidance.Source: §1 · pp. 1–2
  • MENTOR is a multi-agent, plug-and-play framework that diagnoses failures and constructs, validates, and refines targeted prompts within pipelines such as GRPO.Its stated contribution is on-policy pedagogical intervention without modifying the training infrastructure.Source: §1 · p. 2
  • Up to 78.3% average relative gains on Llama-3.2-3B-Instruct over its base model accompany results on math and code generation, with MENTOR mitigating zero-reward collapse.The reported outcomes also include restored meaningful gradient signals.Source: §1 · p. 2

2 Related Work

The related work groups collapse-mitigation approaches by how they handle failed on-policy learning: filtering, static external guidance, and post-hoc refinement. It argues that these approaches leave a gap for real-time diagnosis and targeted instruction.

2 Related Work

  • The related-work overview identifies a shared structural gap in existing approaches to zero-reward-collapse mitigation: they do not provide on-policy pedagogical intervention.The approaches are grouped by their intervention philosophy.Source: §2 · p. 2

2.1 Failure Avoidance via Filtering

  • Filtering methods such as DAPO (Yu et al., 2026) discard all-zero-reward batches, avoiding gradient generation but leaving the model unable to learn from those failures.The paper characterizes this as a passive response to difficult examples.Source: §2.1 · p. 3

2.2 Static “Off-Policy” Guidance

  • Static guidance methods use offline traces or pre-written hints, including LUFFY (Yan et al., 2026), QUESTA (Li et al., 2025b), HINT (Wang et al., 2025), and Scaf-GRPO (Zhang et al., 2025b), but cannot adapt to evolving failures.The paper links this fixed guidance to distributional drift as the policy changes.Source: §2.2 · p. 3

2.3 Post-Hoc Refinement and Critique

2.4 Distinctions from Foundational Methodologies

  • MENTOR differs from inference-stage feedback refinement (Li et al., 2025a) by integrating LLM-as-a-Judge (Gu et al., 2024) into a closed-loop online RL pipeline for training collapse.The paper frames it as dynamic, failure-driven distillation rather than static offline knowledge distillation.Source: §2.4 · p. 3

2.5 The Unaddressed Gap: The Need for On-Policy Pedagogical Intervention

  • Existing mitigation strategies share a gap: none provides real-time diagnosis of systemic failure followed by dynamically generated instructional hints for subsequent exploration.The paper presents MENTOR’s closed-loop, multi-agent intervention as its response to this gap.Source: §2.5 · pp. 3–4

3 Method

MENTOR detects fully failed instances, uses their trajectories to generate validated guidance, and returns guided queries to GRPO. Its multi-agent process adapts hints to the policy’s current errors rather than relying on static prompts.

3 Method

  • MENTOR detects collapsed samples, generates dynamic pedagogical hints, and integrates them back into the training loop.Source: §3 · p. 4

3.1 Collapse-Aware Sample Filtering (CASF)

  • A lightweight post-scoring hook collects instances whose sampled trajectories all fail into a collapse set, preserving their failed reasoning as diagnostic input.The trajectories can reveal recurring error patterns and underlying misconceptions.Source: §3.1 · p. 4

3.2 Failure-Driven Contextual Guidance (FDCG)

  • Attribution diagnoses recurring cognitive errors, generation drafts targeted hints without answer leakage, and judgment and repair refine them, with a safe fallback if validation repeatedly fails.Source: §3.2 · pp. 4–5

3.3 Integration into GRPO Training

  • Validated hints are prepended to collapsed queries, which re-enter standard GRPO; only inputs change, leaving the loss, optimizers, and reward mechanism unchanged.This plug-and-play integration does not alter the core training infrastructure.Source: §3.3 · p. 5

4 Experiments

Across math and code benchmarks, MENTOR outperforms the tested baselines; its evaluation also examines efficiency, agent contributions, and hyperparameter choices. Experiments span several model architectures, with ablations showing that diagnosis, prompt generation, and prompt validation and refinement each contribute to accuracy.

4.1 Experimental Setup

  • Experiments use DeepMath-103K and LiveCodeBench training data, evaluate multiple model families on math and code benchmarks, and include MMLU for out-of-distribution testing.Baselines cover vanilla GRPO, failure filtering, post-hoc refinement, and static hint or curriculum guidance; hints use a matched Qwen3-32B generation budget.Source: §4.1 · pp. 5–6

4.2 Main Results

  • +19.8% on math and +20.8% on code for DeepSeek-R1-Distill-Qwen-1.5B versus Vanilla GRPO; MENTOR outperforms baselines across both task families.The comparison includes math and code benchmarks and reports average relative gains.Source: Table 1 · p. 6, §4.2 · p. 6, Table 2 · p. 7
  • MENTOR also beats static-guidance and post-hoc-refinement methods; gains include 50.0% on Qwen2.5-Math-7B and 78.3% relative math gain on Llama-3.2-3B-Instruct.It matches or exceeds the best baselines on MMLU across four models, including a 66.7% relative gain on DeepSeek-R1-Distill-Qwen-1.5B.Source: §4.2 · p. 7

4.3 Efficiency and Cost-Benefit Analysis

  • MENTOR eliminates about 24 expert hours of offline data preparation and is approximately 24% faster than Critique-GRPO, while adding 2.6 hours over vanilla GRPO on Qwen3-4B.The wall-clock comparison is 17.9h versus 23.5h; reported costs include Qwen3-32B multi-agent inference.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4.1 Ablation of Core Agents.

  • Removing Attribution, Generation, or the Judgment-Repair loop lowers accuracy by 4.50%, 4.67%, or 2.28%, respectively.Using diagnostic keywords directly as hints performs worst among the reported component ablations.Source: Table 4 · p. 8, §4.4.1 · p. 8

4.4.2 Hyperparameter Analysis.

  • Max Retry = 2 performs best across math and code, while the best Max Refine varies by model and task.Qwen3-4B peaks at Max Refine = 2 for math, while Llama-3.2-3B-Instruct benefits from Max Refine = 3; one retry lacks fault tolerance, whereas three can add redundant or noisy prompts.Source: Figure 4 · p. 8, §4.4.2 · p. 8

5 Conclusion

MENTOR targets RLVR’s zero-reward-collapse bottleneck by identifying policy failures, attributing cognitive errors, and guiding targeted refinement. The paper reports stronger performance across mathematical reasoning and code-generation benchmarks, including generalization across model scales.

  • MENTOR dynamically identifies policy failures, attributes underlying cognitive errors, and guides targeted policy refinement to address zero-reward collapse in RLVR.Source: §5 · p. 8
  • The paper reports strong generalization across model scales and significant outperformance of state-of-the-art baselines on challenging mathematical-reasoning and code-generation benchmarks.Source: §5 · p. 8
  • MENTOR focuses on hard examples and learning from failure to address the training ceiling imposed by zero-reward collapse.Source: §5 · p. 8

Limitations

The prompt-generation and reasoning-guidance mechanism has only been validated in single-modal scenarios; dynamic environments and multimodal fusion remain unexplored.

  • The mechanism is validated only in single-modal scenarios, leaving its applicability to complex dynamic environments and multimodal fusion tasks unexplored.Source: Limitations · p. 9

A Prompt

Tables 5–10 provide prompt templates for MENTOR’s agents, which diagnose failures and produce and validate instructional hints.

  • Tables 5–10 present prompt templates for the Attribution, Generation, Judgment, and Repair agents in FDCG.The templates support failure diagnosis and the production and validation of instructional hints.Source: A Prompt · p. 10

B Case

The appendix presents prompt templates for MENTOR’s agents and three complete framework instantiations using four rollouts each.

Related papers

Loading 2610.13594v1…