PaperlayerView exact revision on arXiv

Source-linked AI summary

From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

Zhibo Hou, Fan Zhao, Zhiyu An, Wan Du

arXiv:2608.25243v1cs.CLcs.LG

TL;DR

Continual knowledge injection must update LLMs without restricting new facts to their training formats. GRIN addresses this with a three-stage self-learning framework centered on Golden-GRPO and evaluates it on BLANK and COUNTER. Across these benchmarks, GRIN outperforms supervised and mixed-policy RL baselines on generalized question types while matching them on basic fact recall.

  • Problem

    SFT-based continual injection often memorizes facts in training formats and fails to generalize across unseen queries, document combinations, and reasoning.

  • Method

    GRIN uses staged QA extraction and diverse question construction before Golden-GRPO reinforcement learning injects golden answers when on-policy rollouts fail on novel facts.

  • Results

    GRIN substantially outperforms supervised and mixed-policy RL baselines on multi-source retrieval, inferential reasoning, and counterfactual overwrite while matching them on basic fact recall.

  • Takeaways & Limitations

    Mixed-policy reinforcement learning enables knowledge absorption beyond what SFT achieves within the evaluated continual-injection settings.

  • Takeaways & Limitations

    The benchmarks evaluate a single round of knowledge injection, so success does not guarantee effective multi-round lifelong learning.

Abstract

from arXiv · show

Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.

1 Introduction

Continual knowledge injection addresses the cost and limitations of retraining static language models, whose SFT-based updates often memorize facts without generalizing. GRIN combines staged self-learning with Golden-GRPO and evaluates absorption through complementary document-level benchmarks.

  • Motivation: Continual knowledge injection updates post-trained LLMs on current real-world corpora instead of repeatedly retraining models from scratch.Periodic retraining is described as costly and time-consuming.
  • Motivation: SFT-based injection improves recall but often binds knowledge to training formats, limiting generalization to unseen queries and reasoning.Raw-corpus training can fail on question-style inference, while QA-based training improves recall but retains SFT’s generalization limitations.
  • Approach: GRIN uses staged QA extraction, diverse question-and-answer construction, and Golden-GRPO reinforcement learning to inject knowledge into parametric reasoning.Golden-GRPO adds the golden answer as an off-policy trajectory when on-policy rollouts fail on novel facts.
  • Evaluation: BLANK and COUNTER evaluate novel acquisition and prior-belief overwrite across single-fact recall, multi-source retrieval, and inferential reasoning.COUNTER also reports fail@k to measure whether prior beliefs resurface across samples.
  • Results: GRIN substantially outperforms supervised and mixed-policy RL baselines on multi-source retrieval, inferential reasoning, and counterfactual overwrite while matching them on basic fact recall.The reported pattern identifies stronger gains on generalized question types without sacrificing basic recall.

2 Related Works

Continual knowledge injection methods either retrieve external context at inference or update model parameters through training. Recent RL approaches add external guidance, but they target reasoning capabilities rather than acquiring knowledge absent from the base policy.

  • Injection paradigms: Continual knowledge injection methods generally use retrieval-based context or parameter-updating training to incorporate knowledge.RAG supplies passages from an external corpus at inference time.
  • RL-based approaches: Recent RL methods interleave SFT or mix on-policy rollouts with off-policy reasoning traces when the base policy cannot sample useful capabilities.These approaches balance imitation and exploration but are designed for reasoning tasks where the underlying capability already exists.
  • RL-based approaches: Knowledge injection differs because on-policy rollouts may fail on facts the model has not learned, creating a stronger acquisition challenge than eliciting existing reasoning ability.The cited distinction motivates specialized mixed-policy RL for knowledge injection.

3 Preliminary

GRPO is an on-policy reinforcement learning method that estimates trajectory advantages by comparing rewards within sampled groups, then applies a clipped per-token policy objective.

  • GRPO: For each question, GRPO samples N trajectories from the current rollout policy and scores each trajectory with a reward function.The sampled trajectories form the group used for relative advantage estimation.
  • GRPO: GRPO computes each trajectory’s advantage by standardizing its reward relative to the rewards of the other trajectories in the group.This is the group-relative normalization step.
  • GRPO: The policy is updated with a per-token PPO-clipped objective using an importance-sampling ratio between current and rollout-time policies.The clipping bound ε limits the magnitude of policy updates and stabilizes training.

4 Benchmarks: BLANK and COUNTER

BLANK and COUNTER evaluate continual knowledge injection across two distinct capabilities—novel acquisition and counterfactual overwrite—and three increasingly demanding question types. Their crossed evaluation axes provide fine-grained diagnostics beyond aggregate accuracy.

  • 4 Benchmarks: BLANK and COUNTER: BLANK measures acquisition of knowledge the base model does not currently hold, while COUNTER measures overwrite of knowledge it already believes.BLANK uses novel or unlearned real-world content; COUNTER uses internally consistent parallel-universe rewrites that contradict sampled model beliefs.
  • 4 Benchmarks: BLANK and COUNTER: Both benchmarks evaluate single-fact recall, multi-source retrieval, and inferential reasoning to distinguish surface memorization from progressively stronger generalization.Single-fact recall targets one-sentence factoids, multi-source retrieval combines 2–4 sub-questions across passages or documents, and inferential reasoning combines multiple facts.
  • 4 Benchmarks: BLANK and COUNTER: COUNTER additionally reports fail@k, measuring whether the model’s prior fact resurfaces in any of k sampled responses during overwrite.This metric captures overwrite reliability and prior-belief leakage.
  • 4.1 Dataset construction: The benchmark construction pipeline is model-agnostic, although filtering and prior-belief sampling use Qwen3-4B as the base model.BLANK contains 776 raw corpora after filtering, while COUNTER contains 252.
  • 4.1.2 COUNTER: COUNTER constructs contradictions from the model’s own sampled beliefs, ensuring counterfactual documents target what the model actually believes, including hallucinated or outdated information.A prompted wiki draft represents current beliefs, and a parallel-universe rewrite replaces concrete facts with internally consistent alternatives.

5 GRIN: Golden-GRPO Injection

GRIN is a self-learning, three-stage framework that extracts facts, constructs diverse document-grounded question–answer pairs, and trains with Golden-GRPO to inject learning signal when novel-fact rollouts fail. Its design targets the brittle memorization produced by SFT by combining corpus-level problem construction with a mixed-policy objective that directly pulls the model toward golden answers.

  • Framework overview: GRIN uses the base model to extract atomic QA pairs for Stage 1 SFT, sample diverse document-grounded question–golden-answer pairs for Stage 2, and train on them with Golden-GRPO in Stage 3.The framework operates without an external teacher and follows an SFT-then-RL pipeline with an intermediate problem-set construction stage.
  • Stage 1: Self-Extracted Fact Injection: Stage 1 gives the model parametric access to injected facts but produces brittle memorization that fails on alternative phrasings and compositional queries.Facts are recalled in their training format, motivating the later stages.
  • Stage 2: Problem-set construction: Stage 2 samples each corpus multiple times to create a deduplicated pool varying in phrasing, granularity, and fact coverage.Corpus-level sampling captures multi-fact and inter-sentence relationships that single-sentence extraction misses, while prioritizing coverage for RL training.
  • Stage 3: Golden-GRPO: Golden-GRPO adds an off-policy golden-answer trajectory to on-policy rollouts, restoring learning signal when unlearned facts yield zero-reward rollouts.Its off-policy branch removes importance weighting and uses a direct supervised gradient scaled by the off-policy advantage.
  • Stage 3: Golden-GRPO: The off-policy gradient is strongest before the fact is learned and diminishes as correct on-policy answers emerge, transitioning training toward on-policy exploration.Dr. GRPO-style group-relative advantages also omit standard-deviation normalization, which otherwise shrinks advantages in high-variance early-training groups.

6 Experiment

Experiments evaluate GRIN against retrieval-based, supervised, and reinforcement-learning baselines on BLANK and COUNTER using progressively harder knowledge-injection tasks. GRIN improves generalized retrieval, reasoning, and counterfactual overwrite while additional SFT compute does not close the gap.

  • Experimental setup: Experiments use Qwen3-4B on BLANK and COUNTER, reporting per-type accuracy, benchmark averages, and COUNTER fail@k with k=5.Accuracy is judged by an LLM, and baselines include training-free and training-based methods.
  • Benchmark validation: Training-free accuracy is near zero closed-book, while open-book and RAG perform substantially worse on inferential reasoning and multi-source retrieval than on single-fact recall.A close-sourced model reaches 94.05% on BLANK and 99.51% on COUNTER, confirming the questions are answerable from the corpora.
  • Main results: SFT-based baselines recall injected facts but degrade sharply on multi-source retrieval and inferential reasoning, demonstrating brittle memorization tied to training question forms.Self-Tuning falls from 54.54% single-fact recall to 7.10% inferential reasoning on BLANK.
  • Main results: 21.16% multi-source and 31.69% inferential accuracy on BLANK are achieved by GRIN, versus 11.26% and 7.10% for Self-Tuning.GRIN remains competitive on single-fact recall while improving the harder question types.
  • Main results: 24.05% fail@k is achieved by GRIN on COUNTER, lower than every other training-based method, indicating stronger suppression of prior beliefs during overwrite.COUNTER evaluates whether prior beliefs resurface across sampled responses.
  • Compute-matched training: 43.65% average COUNTER accuracy is reached by GRIN, while SFT-based baselines plateau beyond 15 epochs and remain below GRIN at 50 epochs.The compute-matched comparison indicates that additional training does not close the generalization gap.

7 Conclusion

The paper presents GRIN, a three-stage continual knowledge-injection framework built around Golden-GRPO. Across BLANK and COUNTER, GRIN outperforms supervised and mixed-policy RL baselines on multi-source retrieval and inferential reasoning.

  • Conclusion: GRIN combines three-stage continual knowledge injection with Golden-GRPO, a mixed-policy reinforcement learning algorithm tailored to knowledge injection.The framework is designed to produce parametric knowledge absorption rather than surface-form memorization.
  • Conclusion: Across BLANK and COUNTER, GRIN substantially outperforms supervised and mixed-policy RL baselines on multi-source retrieval and inferential reasoning.The benchmarks cover novel acquisition and counterfactual overwrite.
  • Conclusion: The reported results support mixed-policy reinforcement learning enabling knowledge absorption beyond what SFT achieves.This conclusion is stated across the paper’s continual knowledge-injection evaluation.

Limitations

The study evaluates a single round of injection on wiki-style, entity-grounded text, leaving repeated lifelong updates and other knowledge formats unresolved.

  • Single-round injection vs. lifelong learning: The benchmarks test a single injection round, so successful single-round injection does not guarantee stable multi-round learning, retention, or future absorption.The authors identify catastrophic forgetting and degradation of future learning as open questions under repeated injection.
  • Knowledge domain coverage: GRIN’s behavior on procedural knowledge, code, mathematical content, and structured data is unexplored because BLANK and COUNTER use wiki-style, entity-grounded text.The scope of the reported evidence is therefore limited to the evaluated knowledge format.
  • Future investigation: The authors leave open whether Golden-GRPO’s specialized approach will remain attractive at larger scales.Larger-scale applicability is identified as a possible direction rather than an established result.

Ethical considerations

The paper treats COUNTER counterfactuals as evaluation infrastructure rather than deployment content, and constructs benchmark questions to test increasingly demanding forms of knowledge absorption.

  • COUNTER’s counterfactual statements contradict real-world facts and are constructed strictly for evaluation infrastructure, not deployment.
  • The benchmarks generate single-fact recall, multi-source retrieval, and inferential reasoning questions to distinguish surface memorization from knowledge absorption.Multi-source questions span distinct facts across paragraphs or documents, while inferential questions require combining multiple source facts.
  • An accuracy judge scores factual agreement, while a separate fail@k judge detects resurfacing of either counterfactual or prior beliefs across five samples.

E Gradient dynamics of Golden-GRPO vs. importance-weighted mixed-policy RL

Importance-weighted mixed-policy methods can suppress the off-policy learning signal precisely when the model has not learned a novel fact. Golden-GRPO replaces that ratio with a direct supervised gradient whose contribution decreases as on-policy correctness increases and vanishes after convergence.

  • Importance-weighted mixed-policy RL: LUFFY and GOLF ratios collapse when the off-policy trajectory is much more probable under the reference or augmented policy than under the plain-query policy.The mismatch makes the numerator small precisely when the injected fact remains unlearned.
  • Golden-GRPO: Golden-GRPO removes the off-policy importance ratio and uses a direct supervised gradient scaled by the off-policy advantage.The supervised gradient remains defined and stable even when the model assigns low probability to the golden trajectory.
  • Phase 1: When all on-policy rollouts are incorrect, the positive off-policy advantage makes the supervised term dominate and pull the model toward the golden trajectory.
  • Phase 2: As correct on-policy rollouts increase, the off-policy advantage shrinks and the model reinforces both the golden trajectory and its own correct outputs.The off-policy branch’s relative dominance diminishes as the number of correct on-policy rollouts grows.
  • Phase 3: When all on-policy rollouts are correct, every advantage is zero, the objective gradient vanishes, and training resources redirect to questions with remaining advantage variance.
  • Self-regulation: The three-phase transition requires no external schedule or annealing, automatically scaling the off-policy gradient with the model’s learning state.

F Reproducibility Statement.

The experiments use named open-source models, datasets, and libraries under their respective licenses, with code and benchmark data prepared for public release. Writing assistance from language models was limited to polishing, while the authors retain responsibility for the paper.

  • The work uses Qwen3-4B, Llama3.2-3B, TimeQA, and NLTK under their respective open-source licenses.
  • Code and benchmark data will be made publicly available, with an accompanying archive provided for reviewer access.
  • Large language models assisted only with writing polish and were not used for research ideation, methodology, analysis, or conclusions.
Loading 2608.25243v1…