From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

• Continual knowledge injection aims to update static LLMs with recent real-world corpora, avoiding costly, time-consuming cycles of training new models from scratch.Source: §1 · p. 1
• SFT-based injection often fails to generalize beyond training queries and leaves prior beliefs unaddressed; existing mixed-policy RL loses off-policy gradients on unlearned facts.Source: §1 · pp. 1–2
• GRIN uses SFT, diverse problem-set construction, then RL; Golden-GRPO adds the golden answer as an off-policy trajectory when on-policy rollouts fail.The framework requires no external teacher model.Source: §1 · p. 2
• BLANK targets novel knowledge acquisition and COUNTER targets prior-belief overwrite, with both evaluating single-fact recall, multi-source retrieval, and inferential reasoning.COUNTER also reports fail@k to measure whether prior beliefs resurface in any of k samples.Source: §1 · p. 2
• GRIN substantially outperforms supervised and mixed-policy RL baselines on multi-source retrieval, inferential reasoning, and counterfactual overwrite, while matching them on basic fact recall.Source: §1 · p. 2

2.1 Continual Knowledge Injection

• Retrieval leaves facts external to model parameters, while training-based injection often yields shallow memorization and SFT tends to memorize formats rather than generalizable rules (Su et al., 2025; Levy et al., 2024; Jiang et al., 2024; Chu et al., 2025).The paper therefore focuses on parameter-updating methods and uses reinforcement learning to support knowledge absorption.Source: §2.1 · p. 2

2.2 Guided Reinforcement Learning

• Prior guided-RL work uses SFT interleaving or off-policy traces for reasoning, but Golden-GRPO supplies mixed-policy guidance when rollouts miss injected facts (Ma et al., 2025; Lin et al., 2024; Yan et al., 2026; Huang et al., 2026; Zhang et al., 2025b).Unlike reasoning tasks where the base model has the underlying capability, knowledge injection can leave the RL stage without positive reward because on-policy rollouts do not sample new facts.Source: §2.2 · pp. 2–3

• GRPO (Shao et al., 2024) is an on-policy RL algorithm that estimates each trajectory’s advantage by standardizing rewards within a sampled group.For each question, the current policy samples N trajectories and scores each with a reward function.Source: §3 · p. 3
• For each question, the current policy samples N trajectories, each scored by a reward function, and uses the group’s standardized rewards to compute per-trajectory advantages.Source: §3 · p. 3
• The per-token PPOclipped update uses the current-to-rollout policy importance-sampling ratio and a clipping bound to prevent large updates and stabilize training.Source: §3 · p. 3

4 Benchmarks: BLANK and COUNTER

• Together, BLANK and COUNTER test acquisition versus overwrite across single-fact recall, multi-source retrieval, and inferential reasoning, with COUNTER also measuring prior-belief leakage via fail@k.Reporting the capabilities separately provides a diagnostic view of where methods succeed or fail, beyond aggregate accuracy.Source: §4 · p. 3

4.1.1 BLANK

• BLANK evaluates unseen knowledge using model-probed TimeQA passages with zero prior knowledge and post-training-cutoff wiki pages.TimeQA passages are documented as challenging for current models, while post-cutoff pages broaden topical coverage beyond historical content.Source: §4.1.1 · p. 3

4.1.2 COUNTER

• COUNTER constructs parallel-universe wiki rewrites that replace concrete facts with plausible alternatives, designed to contradict the base model’s sampled beliefs.The original is drafted by prompting the base model, so rewrites can contradict its hallucinations or outdated information as well as real-world facts; document structure and substitution consistency are preserved.Source: §4.1.2 · pp. 3–4

4.2 QA Pair Generation

• Both benchmarks separately score direct single-fact recall, multi-source retrieval across 2–4 sub-questions, and reasoning that combines multiple facts; COUNTER fail@k detects prior-belief leakage.Questions are LLM-generated unless otherwise noted; the protocol also reports per-benchmark averages for acquisition and overwrite capability.Source: §4.2 · p. 4

5 GRIN: Golden-GRPO Injection

• GRIN uses the base model alone to extract atomic QA pairs, sample document-grounded question–answer pairs, and train with Golden-GRPO.The framework requires no external teacher and aims to absorb injected knowledge into parametric reasoning rather than memorize its surface form.Source: §5 · p. 4, Figure 1 · p. 5

5.1 Stage 1: Self-Extracted Fact Injection

• Stage 1 uses the base model to extract single-fact QA pairs from source sentences and fine-tunes it to recall those facts, but SFT alone yields brittle, format-bound memorization.The resulting facts fail on alternative phrasings or compositional queries, motivating the later stages.Source: §5.1 · pp. 4–5

5.2 Stage 2: Corpus-Conditioned Sampling

• Stage 2 samples questions and golden answers at the corpus level, producing varied phrasing, granularity, and fact coverage that can capture inter-sentence relationships.The pool prioritizes coverage over individual question quality because noisy questions may still provide useful signal through Golden-GRPO’s reward function.Source: §5.2 · p. 5

5.3 Stage 3: Golden-GRPO

• Golden-GRPO addresses zero-reward on-policy rollouts by adding a golden-answer trajectory and replacing the off-policy importance-weighted branch with a direct supervised gradient scaled by its advantage.The gradient is strongest when the model has not learned the fact and diminishes as on-policy answers become correct; omitting GRPO standard-deviation normalization protects advantages in high-variance early groups. The reward combines format (+0.5), ROUGE-L (0–0.5), exact keyword match (+1), and a multiple-answer penalty (−0.25).Source: §5.3 · pp. 5–6

6.1 Experimental Setup

• Experiments use Qwen3-4B on BLANK and COUNTER, compare training-free and training-based baselines, and evaluate accuracy with gemini-3.5-flash.Stage 1 SFT and Stage 3 Golden-GRPO use AdamW at learning rate 2e-5; training lasts 5 and 3 epochs, respectively, with 8 on-policy rollouts for Stage 3.Source: §6.1 · pp. 6–7

6.2 Main Results

• GRIN scores 21.16% multi-source and 31.69% inferential on BLANK versus Self-Tuning’s 11.26% and 7.10%, matches it on single-fact recall, and reaches COUNTER fail@k of 24.05%.The COUNTER fail@k is lower than every other training-based method.Source: Table 2 · p. 7, §6.2 · p. 8
• On COUNTER with Llama3.2-3B, GRIN leads on three of four accuracy metrics and is competitive on fail@k, reproducing the Qwen3-4B pattern.The benchmark’s counterfactual replacements contradict real-world facts, making them valid injection targets across base models.Source: Table 3 · p. 7, §6.2 · p. 8
• Closed-book averages of 4.33% on BLANK and 2.33% on COUNTER indicate questions exceed base-model knowledge; open-book inferential accuracy is 36.61% and 45.43%, respectively.A document-conditioned Gemini model reaches 94.05% on BLANK and 99.51% on COUNTER, while RAG struggles with multi-source retrieval across pages.Source: §6.2 · p. 7, Table 2 · p. 7
• Ablations show that continued SFT on Stage 2 data remains weak on generalized questions, vanilla GRPO averages 1.83% on BLANK, and LUFFY improves over SFT but trails GRIN.The results support the need for the off-policy golden trajectory to provide learning signal for facts not yet acquired by on-policy rollouts.Source: §6.2 · p. 8
• At 50 epochs, training-based baselines plateau beyond 15 epochs, well below GRIN’s 43.65% COUNTER average accuracy.The additional baseline training matched or exceeded Golden-GRPO’s total training cost, but did not close the performance gap.Source: Figure 2 · p. 8, §6.2 · p. 8

7 Conclusion

• GRIN substantially outperforms supervised and mixed-policy RL baselines on multi-source retrieval and inferential reasoning across BLANK and COUNTER.These benchmarks evaluate novel acquisition and counterfactual overwrite, respectively.Source: §7 · p. 8
• GRIN is a three-stage continual knowledge-injection framework built around Golden-GRPO, a mixed-policy RL algorithm tailored to this setting.Golden-GRPO is designed to produce knowledge absorbed into the model’s parametric reasoning rather than memorized in surface form.Source: §7 · p. 8
• Golden-GRPO addresses standard mixed-policy RL’s vanishing off-policy gradient on novel facts, while supervised methods fail to absorb knowledge beyond their training format.Source: §7 · p. 8

Single-round injection vs. Lifelong learning

• Repeated injection rounds may affect retention of prior knowledge and future learning, and whether Golden-GRPO accumulates knowledge stably or suffers catastrophic forgetting remains unknown.The benchmarks evaluate only a single round of injection.Source: Limitations · p. 9

Knowledge domain coverage

• GRIN’s performance on procedural, code, mathematical, or structured knowledge is unexplored, and its ROUGE-L and Exact Match rewards may not suit domains where lexical signals poorly measure correctness.Source: Limitations · p. 9

Resource Requirements

• At 3–4B scale, rollout overhead is modest, but at trillion-parameter scale it may approach full-retraining cost, making Golden-GRPO most applicable when retraining is not an option.The authors note it may become attractive at larger scales if retraining increasingly serves parametric knowledge updates.Source: Limitations · p. 9

Ethical considerations

• COUNTER’s counterfactual statements contradict real-world facts but are constructed strictly for evaluation, not deployment.Source: Ethical considerations · p. 9

A Prompts used for corpus construction

• Two API calls per subject first elicit a short Wikipedia-style passage of the base model’s prior knowledge, then rewrite it by replacing concrete facts while preserving structure and entity.Source: A Prompts used for corpus construction · p. 11

B Prompts used for question generation

• Three prompts generate single-fact recall, multi-source retrieval across N ∈ {2, 3, 4} distinct facts, and inferential questions combining multiple source facts.Source: B Prompts used for question generation · p. 11
• For BLANK inferential questions, original TimeQA questions are included for TimeQA-derived passages; Prompt 3 covers LLM-augmented BLANK passages and all post-cutoff and COUNTER passages.Source: B Prompts used for question generation · p. 11

C Example Stage 2 QA pair for Golden-GRPO

• Stage 2 prepends the corpus and a system prompt before a pre-query token, samples the model’s continuation to its first end-of-turn marker, and illustrates COUNTER’s Apple listing rewrite from NASDAQ/AAPL to NYSE/APL.Source: C Example Stage 2 QA pair for… · pp. 11, 14

D Judge Prompts

• The accuracy judge checks whether answers contain the reference’s factual information for per-type scores in Tables 2 and 3; COUNTER’s fail@k judge classifies samples as matching counterfactual text, Wikipedia prior, or neither.Source: D Judge Prompts · p. 14

Why both ratios vanish in knowledge injection

• LUFFY (Yan et al., 2026) and GOLF (Huang et al., 2026) have small off-policy ratios when the model has not learned the fact, suppressing that branch’s gradient.LUFFY’s reference policy favors its own trajectory, while GOLF’s guidance-augmented prompt makes the trajectory likely even when the plain-query probability remains low.Source: E Gradient dynamics of Golden-GRPO vs.… · p. 14

Golden-GRPO’s design

• Golden-GRPO removes the off-policy importance ratio and uses a supervised gradient scaled by the off-policy advantage, which remains stable when the trajectory probability is small.The supervised gradient ∇θ log πθ is well-defined across the probability values encountered during training.Source: E Gradient dynamics of Golden-GRPO vs.… · p. 15

Three-phase gradient analysis

• The off-policy update is strongest when on-policy rollouts fail, then shrinks as correct rollouts increase and vanishes when all rollouts are correct.With mixed rollouts, correct on-policy outputs and the golden trajectory both contribute positively; when all succeed, advantages are zero.Source: E Gradient dynamics of Golden-GRPO vs.… · p. 15

Why this matters

• This self-regulation follows from the advantage formula without external scheduling, and prevents continued pulling toward the golden answer’s specific phrasing once on-policy success matches it.The supervised gradient toward the golden trajectory disappears when on-policy rollouts succeed at the same rate as the off-policy reference.Source: E Gradient dynamics of Golden-GRPO vs.… · p. 15

F Reproducibility Statement.

• The authors used Qwen3-4B (Yang et al., 2025) and Llama3.2-3B (Grattafiori et al., 2024).Source: F Reproducibility Statement · p. 16
• TimeQA (Chen et al., 2021) and NLTK were among the resources used under their respective open-source licenses.Source: F Reproducibility Statement · p. 16
• Code and benchmark data will be made publicly available.Source: F Reproducibility Statement · p. 16
• A code and data archive accompanies the submission for reviewer access.Source: F Reproducibility Statement · p. 16

G The Use of Large Language Models (LLMs)

• LLMs were used solely to polish the paper’s writing, not for research ideation, methodology, analysis, or conclusions.Source: G The Use of Large Language Models… · p. 16
• The authors take full responsibility for all content.Source: G The Use of Large Language Models… · p. 16