Source-linked AI summary
Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Models
Shiting Huang, Zecheng Li, Yu Zeng, Qingnan Ren, Zhen Fang, Qisheng Su, Kou Shi, Lin Chen, Zehui Chen, Feng Zhao
TL;DR
RLVR lacks fine-grained error attribution and experience internalization, limiting reusable knowledge formation from reasoning trajectories. MEL contrasts correct and incorrect trajectories to distill bifurcation-point failures into meta-experiences and internalizes them in parametric memory. Across benchmarks and model scales, MEL consistently improves reasoning, including 3.92–4.73% Pass@1 gains over GRPO.
Problem
RLVR mainly evaluates complete trajectories, leaving error attribution and experience internalization insufficient for fine-grained credit assignment and reusable knowledge formation.
Method
MEL contrasts paired correct and incorrect trajectories, abstracts reasoning bifurcation points into meta-experiences, and internalizes them in parametric memory through negative log-likelihood training.
Results
MEL consistently improves Pass@1, Avg@8, and Pass@8 across five mathematical reasoning benchmarks and 4B, 8B, and 14B models, including 3.92–4.73% Pass@1 gains over GRPO.
Takeaways & Limitations
MEL transforms instance-specific reasoning failures into reusable cognitive assets and supports consistent mathematical reasoning improvements across model scales.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for enhancing the reasoning capabilities of Large Language Models (LLMs). Despite its efficacy, RLVR faces a meta-learning bottleneck: it lacks mechanisms for error attribution and experience internalization intrinsic to the human learning cycle beyond practice and verification, thereby limiting fine-grained credit assignment and reusable knowledge formation. We term such reusable knowledge representations derived from past errors as meta-experience. Based on this insight, we propose Meta-Experience Learning (MEL), a novel framework that incorporates self-distilled meta-experience into the model's parametric memory. Building upon standard RLVR, we introduce an additional design that leverages the LLM's self-verification capability to conduct contrastive analysis on paired correct and incorrect trajectories, identify the precise bifurcation points where reasoning errors arise, and summarize them into generalizable meta-experience. The meta-experience is further internalized into the LLM's parametric memory by minimizing the negative log-likelihood, which induces a language-modeled reward signal that bridges correct and incorrect reasoning trajectories and facilitates effective knowledge reuse. Experimental results demonstrate that MEL achieves consistent improvements on benchmarks, yielding 3.92%--4.73% Pass@1 gains across varying model sizes.
1. Introduction
RLVR improves LLM reasoning through verifiable rewards but largely evaluates complete trajectories, leaving error attribution and reusable knowledge formation underdeveloped. MEL adds a knowledge-level learning loop that extracts meta-experiences from contrasting trajectories, internalizes them in parametric memory, and improves performance across model scales and RL paradigms.
- Motivation: RLVR provides verifiable feedback for LLM reasoning but primarily assesses whole trajectories rather than intermediate-step knowledge.This limits fine-grained credit assignment and reusable knowledge formation.
- Contribution: MEL defines meta-experience as reusable knowledge distilled from reasoning failures and elevates experience learning from trajectories to knowledge-level representations.The framework targets both error attribution and experience internalization.
- Method: MEL contrasts correct and incorrect trajectories to locate reasoning bifurcation points, then abstracts them into reusable meta-experiences.These experiences connect otherwise independent correct and incorrect trajectories.
- Method: MEL internalizes self-distilled meta-experiences in the model’s parametric memory by minimizing negative log-likelihood.The resulting process is described as a language-modeled process-level reward signal.
- Results: MEL integrates with RFT, GRPO, and REINFORCE++ and shows stronger benefits as model size increases.The reported improvements reshape reasoning patterns and elevate performance ceilings.
2. Related Work
Existing RLVR and experience-learning methods improve reasoning but remain centered on outcome-level rewards or trajectory imitation. MEL addresses this gap by targeting reusable, knowledge-level meta-experience within the RLVR paradigm.
- Reinforcement Learning with Verifiable Rewards: RLVR methods predominantly rely on outcome-level rewards, limiting fine-grained credit assignment to specific knowledge points.This constrains reusable knowledge formation and systematic, generalizable reasoning capabilities.
- Experience Learning: External-memory and expert-trace approaches improve exploration efficiency but primarily induce trajectory-level imitation.They can teach specific reasoning patterns without establishing reusable knowledge structures.
3. Meta-Experience Learning
MEL transforms verified reasoning failures into reusable meta-experiences and internalizes them in the model’s parametric memory. It augments RLVR with contrastive error analysis and dense process-level guidance.
- Meta-Experience Learning: MEL defines meta-experience as knowledge-level representations distilled from accumulated reasoning trials, including underlying concepts and common failure modes.The framework is designed to internalize these self-distilled insights into parametric memory.
- Explorative Rollout and Verifiable Feedback: RLVR exploration generates independent reasoning trajectories, and a rule-based verifier partitions them into correct and incorrect subsets.The group rollout formulation follows GRPO, enabling stochastic exploration over the solution space.
- Meta-Experience Construction: MEL pairs correct and incorrect trajectories to identify the bifurcation point where reasoning diverges into an erroneous path.The method requires both non-empty correct and incorrect trajectory sets for contrastive analysis.
- Meta-Experience Construction: The model diagnoses the localized failure and abstracts instance-specific critiques into generalized heuristics describing preconditions, responses, pitfalls, and latent constraints.This abstraction maps concrete success and failure logic into a generalized space intended to support future reasoning.
- Internalization Mechanism: MEL internalizes meta-experiences by fine-tuning the policy with a token-averaged negative log-likelihood objective over retrospective contexts containing instructions, queries, and paired trajectories.The compiled knowledge is stored in model parameters rather than an inference-time context buffer.
- Joint Training Objective: The resulting Meta-Experience Return acts as a dense process reward, providing step-by-step reinforcement that connects trajectory-level exploration with token-level knowledge encoding.Unlike standard RLVR’s sparse final-outcome reward, this signal continuously shapes the reasoning process.
4. Experiments
Experiments evaluate MEL across mathematical benchmarks, model scales, training dynamics, reasoning behavior, and alternative learning paradigms. MEL consistently improves performance and provides faster training progress, sustained advantages, and reusable reasoning heuristics.
- Experimental Setup: MEL is evaluated on five mathematical reasoning benchmarks using Qwen3 models at 4B, 8B, and 14B scales against vanilla GRPO and base models.Evaluation reports Pass@1, Avg@8, and Pass@8.
- Experimental Results: 3.92–4.73% Pass@1 gains over GRPO show consistent improvements across model scales, alongside gains in Avg@8 and Pass@8.Pass@1 measures one-shot reliability, Avg@8 average performance over eight samples, and Pass@8 best-of-eight success.
- Training Dynamics and Convergence Analysis: MEL rises faster than vanilla GRPO early in training because its internalized meta-experience return provides dense process-level signals at each reasoning step.The advantage is especially relevant when successful trajectories yielding positive outcome reinforcement are scarce.
- Training Dynamics and Convergence Analysis: MEL maintains a higher average reward and validation advantage throughout training, including after performance begins to plateau.The authors associate this sustained advantage with exploration of more complex long-horizon solutions.
- How Meta-Experience Shapes Reasoning Patterns: MEL structures reasoning by outlining relevant theorems and formulas, while GRPO more often begins with direct numerical operations that are more error-prone on complex tasks.MEL also activates internalized “bitter lessons” as safeguards during theorem application, encouraging earlier constraint checking and reducing reasoning drift.
- Generality Across Learning Paradigms: Meta-experiences improve RFT and REINFORCE++, mitigating RFT overfitting and raising REINFORCE++’s benchmark performance ceiling beyond the GRPO setting.The experiments use Qwen-8B-Base and the same training set for these comparisons.
- Scaling Analysis: The performance margin widens with model size because larger models yield more valid meta-experiences through stronger self-verification and more accurate error attribution.Smaller models introduce more noise into self-generated supervision through imprecise error attribution.
5. Conclusion
MEL addresses the meta-learning bottleneck in standard RLVR by turning instance-specific reasoning failures into reusable meta-experiences internalized in parametric memory. Evaluations report consistent mathematical-reasoning gains across model scales.
- MEL performs granular error attribution and distills specific failure modes into natural-language heuristics.
- MEL internalizes Meta-Experiences into parametric memory, connecting solution verification with the underlying reasoning logic.
- MEL consistently improves mathematical reasoning across diverse model scales.
Impact Statement
The paper reports no specific broader societal impacts requiring particular attention at this stage.
- The authors identify no specific broader societal impacts requiring particular attention at this stage.
A. Result of Performance Evolution
Figures 6–8 track GRPO and MEL across training steps for Qwen3-4B-Base, Qwen3-8B-Base, and Qwen3-14B-Base on multiple benchmarks. MEL consistently outperforms standard GRPO in average performance across all benchmarks.
- Figures 6–8 compare GRPO and MEL across training steps on multiple benchmarks for Qwen3-4B-Base, Qwen3-8B-Base, and Qwen3-14B-Base.
- MEL consistently outperforms standard GRPO in average performance across all benchmarks.
B. Retention Ratio of Meta-Experience
Figure 9 reports the retention ratio of MEL’s meta-experiences during training after empirical validation. The retention ratio increases with model scale.
- Figure 9 tracks MEL’s meta-experience retention ratio across different model scales.
- The retention ratio consistently increases with model scale, indicating greater retention in larger models.
C. Prompt Template
The prompt template compares correct and incorrect solution trajectories to identify reasoning divergences, latent defects, and effective structural insights. It then converts these analyses into abstract, reusable heuristics and asks the model to internalize them while solving new problems.
- Input and Comparative Setup: The template supplies paired solution trajectories labeled Correct and Incorrect for comparative analysis on the same problem.It also incorporates prior reference information about correct approaches, common reasoning flaws, and useful heuristics.
- Correct-Trajectory Analysis: It analyzes successful reasoning through self-correction or flawless execution to identify pivots, foundational concepts, and robustness factors.The analysis asks what triggered recovery or what prevented common errors from arising.
- Incorrect-Trajectory Analysis: It analyzes failed reasoning by locating the bifurcation point, identifying the latent cognitive defect, and specifying the violated knowledge point or constraint.The template distinguishes root causes from surface-level calculation errors.
- Comparative Synthesis: It contrasts solutions and decision boundaries to determine which structural insight the successful trajectory had that the failed trajectory missed.The comparison focuses on how the reasoning differed, rather than only on the final answer.
- Internalized Experience: It converts problem-specific lessons into conditionally triggered, abstract heuristics that exclude the current problem's numbers, variables, and exact answer.These heuristics are intended for future problems of a similar class and are formatted as rules such as “If...Then...” or “When...I must...”.
- Meta-Cognitive Reflection: It requires first-person meta-cognitive reflection that reviews the reasoning, identifies critical knowledge or habits, and specifies future restructuring actions.The prompt emphasizes the how of thinking rather than the answer's content.