Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation

Introduction

• SFT adapts models to target tasks and formats but can alter factual behavior, leaving models with improved target-task performance yet worse OOD factual reliability.Source: Introduction · p. 1
• Domain SFT leaves recognition comparatively stable but sharply reduces open-ended OOD factual generation, a gap termed factual access failure rather than proof of fact erasure.Source: Introduction · p. 1
• Paired same-fact probes show many answers remain selectable but fail generation, including genuinely wrong answers and verbosity, formatting, and exact-match failures.Source: Introduction · p. 1
• RAD combines target-domain SFT with an unlabeled OOD stream, training the adapted model to match the base model’s soft next-token distributions on continuation tokens.The method uses no gold OOD answers, rationales, external judges, or labeled factual data.Source: Introduction · p. 2
• Across three backbones fine-tuned on MedMCQA, RAD recovers OOD factual generation while preserving in-domain gains and outperforming baselines.Source: Introduction · p. 2

SFT-Induced Factual Degradation in LLMs

• Prior work finds SFT can impair factual reliability even with factually correct supervision, through hallucinations, shortcuts, frequency shocks, or relation-level cues.Gekhman et al. (2024) report slower learning of unsupported knowledge and increased hallucinations once learned; Kang et al. (2024) find unfamiliar examples can shape hallucinated predictions. Other studies examine factual-recall dynamics and effects of factual QA fine-tuning (Zucchet et al. 2025; Kazemi, Mittal, and Ramachandran 2023; Ghosal, Hashimoto, and Raghunathan 2024; Gong, Huang, and Liang 2025).Source: SFT-Induced Factual Degradation in LLMs · p. 2

Hidden Knowledge and Recall Failures

• Probing research shows that generation can miss facts that remain encoded or recoverable under alternative elicitation, motivating tests of constrained recognition and open-ended extraction.Prior work also argues that fine-tuning can shift inferred task distributions or disrupt task alignment rather than erase knowledge (Kotha, Springer, and Raghunathan 2024; Zheng et al. 2025).Source: Hidden Knowledge and Recall Failures · pp. 2–3
• Unlike generic KL constraints that keep an adapted model near a reference, this approach uses the base model as teacher on OOD factual continuations while optimizing target-domain SFT behavior.It aligns the adapted model with the base model’s soft token distribution on general-domain factual continuations outside the SFT target domain (Hinton, Vinyals, and Dean 2015; Ziegler et al. 2020; Stiennon et al. 2022; Ouyang et al. 2022).Source: Hidden Knowledge and Recall Failures · p. 3

Preliminary Studies

• A drop in closed-book generation accuracy diagnoses failure to produce an answer, not by itself erasure of its fact; apparent degradation may reflect task-alignment shifts rather than lost capabilities.The section distinguishes memory erasure from expression failure, consistent with work on spurious forgetting and the distinction between knowledge encoding and recall (Calderon et al. 2026; Gekhman et al. 2025; Kotha, Springer, and Raghunathan 2024; Zheng et al. 2025).Source: Preliminary Studies · p. 3

F1: Models Recognize What They Cannot Recall

• On held-out QA benchmarks, TriviaQA EM drops from 65.03 to 43.95 after domain SFT, while MedMCQA and MMLU remain stable or slightly improve.Recognition is measured with multiple-choice evaluation and recall with closed-book QA metrics, EM/F1; because benchmark datasets differ in content and format, this comparison motivates rather than establishes same-fact dissociation.Source: F1: Models Recognize What They Cannot… · p. 3

F2: Facts Are Stored, but Inaccessible

• On matched TriviaQA facts, SFT TriviaQA EM falls from 63.1 to 43.8 while multiple-choice accuracy changes marginally and the gold answer’s teacher-forced ranking remains near the top.The top-1 rate is nearly unchanged from Base to SFT, supporting a diagnosis of selective failure to produce facts in open-ended generation rather than uniform loss of factual behavior.Source: F2: Facts Are Stored, but Inaccessible · p. 3

F3: Recall Failures Are Largely Expression Failures

• Format or verbosity mismatches account for 77.3% of 300 Base-correct/SFT-wrong TriviaQA cases, although a non-negligible subset contains genuine wrong-answer generations.Labels agree with an independent LLM judge on 97.3% of cases, and a simple 4-shot prompt recovers 90.0% of the same cases, suggesting many failures remain recoverable under different prompting.Source: F3: Recall Failures Are Largely… · p. 3

Methodology

• RAD preserves OOD generation during domain adaptation by anchoring the adapted model to the original base model’s generation behavior.OOD is defined relative to the target domain.Source: Methodology · p. 3

Teacher and Student from One Model

• The frozen base model serves as teacher with the LoRA adapter off, while the adapter-on model serves as student.LoRA updates are added to frozen base weights, so the teacher is the original base model rather than an external oracle.Source: Teacher and Student from One Model · pp. 3–4

Training Streams

• RAD combines labeled target-domain SFT with unlabeled generic OOD text whose continuations define the distillation positions.The anchor corpus uses Wikipedia prefixes and natural continuations, not hard labels; benchmark questions, answers, and normalized aliases are filtered, and prefix tokens condition the models without being distillation targets.Source: Training Streams · p. 4

Distributional Anchoring Objective

• On OOD anchors, RAD uses reverse KL to align student and base distributions, clips per-token penalties, and masks prefix positions from distillation gradients.The reverse-KL direction discourages probability on continuations the base model finds improbable, while clipping prevents a few highly divergent tokens from dominating.Source: Distributional Anchoring Objective · p. 5

Joint Optimization and Implementation

• RAD jointly optimizes supervised adaptation and OOD distillation, with α = 1.5 and τ = 5.0 as defaults, anchoring OOD continuations without globally constraining the adapter.The supervised loss drives target-task learning; the OOD loss discourages drift from the base model on generic continuations.Source: Joint Optimization and Implementation · p. 5
• RAD uses one model instance, switching the LoRA adapter off for teacher logits and on for student logits; only the student pass contributes gradients.It adds one teacher forward pass per OOD anchor batch, pairs each target-domain step with an anchor batch, and retains the standard LoRA SFT inference architecture.Source: Joint Optimization and Implementation · p. 5

Experimental Setup

• Experiments use Llama-3.1-8B, Qwen2.5-7B-Instruct, and Qwen2.5-3B-Instruct; Llama uses the base rather than instruction-tuned checkpoint (Qwen et al. 2025).Source: Experimental Setup · p. 5
• All adapted systems use LoRA adapters with frozen pretrained backbones, shared target-domain data, and the same evaluation protocol (Hu et al. 2021).Source: Experimental Setup · p. 5
• RAD defaults to reverse KL with α = 1.5 and τ = 5.0.Full training details are in the Appendix.Source: Experimental Setup · p. 5
• Models train on MedMCQA’s training split and are evaluated on its validation split; examples are four-choice QA with the correct option and explanation as the target (Pal, Umapathi, and Sankarasubb u 2022).Source: Experimental Setup · p. 5
• RAD’s unlabeled Wikipedia-style OOD anchor corpus contains 10,000 prefix–continuation examples and is filtered against PopQA and TriviaQA entities and aliases (Merity et al. 2016).The prefix provides context, while the continuation marks tokens for distributional anchoring; OOD is defined relative to the target task.Source: Experimental Setup · pp. 5–6
• Comparisons include the unadapted base, MedMCQA-only standard SFT, and replay, which trains on the same OOD text as RAD using hard-label language modeling rather than base-distribution matching (Ouyang et al. 2022).Replay controls for whether gains arise simply from adding general-domain text.Source: Experimental Setup · p. 6

• MedMCQA validation accuracy measures target-domain adaptation (Pal, Umapathi, and Sankarasubbu 2022).Source: Benchmarks and Metrics · p. 6
• MMLU medical and non-medical subjects (Hendrycks et al. 2021) and TruthfulQA-MC2 (Lin, Hilton, and Evans 2022) assess recognition-style OOD behavior.Source: Benchmarks and Metrics · p. 6
• TriviaQA (Joshi et al. 2017) and PopQA (Mallen et al. 2023) assess open-ended OOD factual generation using exact match (EM) and token-level F1 over acceptable aliases.Source: Benchmarks and Metrics · p. 6

For Q1: Does Domain SFT Cause Capability Collapse or Selective OOD Shift?

• TriviaQA EM falls from 65.03 to 43.95 for Llama-3.1-8B, while Qwen2.5-3B-Instruct TriviaQA EM falls from 30.57 to 1.04 after SFT.SFT improves in-domain MedMCQA and leaves recognition-style OOD performance comparable on MMLU-Med and MMLU-Other; Qwen2.5-7B-Instruct TruthfulQA-MC2 drops from 64.75 to 49.03.Source: For Q1: Does Domain SFT Cause… · p. 6

For Q2: Can RAD Recover OOD Factual Behavior While Preserving Target-Domain Adaptation?

• RAD recovers affected OOD behavior across three backbones while retaining competitive MedMCQA performance.It improves TriviaQA and PopQA EM on Llama-3.1-8B, recovers TruthfulQA-MC2 on Qwen2.5-7B and TriviaQA EM on Qwen2.5-3B, and achieves the best MedMCQA accuracy among fine-tuned Llama-3.1-8B systems.Source: For Q2: Can RAD Recover OOD Factual… · p. 6

For Q3: Is the Soft Base Distribution Necessary Beyond Same-Text Replay?

• PopQA EM is 3.40 with Replay, versus 11.60 with standard SFT and 15.20 with RAD on Llama-3.1-8B.Replay uses the same OOD anchor corpus but hard-label language modeling; its weaker OOD recall shows that RAD’s soft token-level base distribution, rather than extra text exposure alone, is the key preservation signal.Source: For Q3: Is the Soft Base Distribution… · pp. 6–7

Ablation Analysis.

• Positive OOD anchoring improves TriviaQA and PopQA while keeping MedMCQA near baseline, and RAD beats standard SFT on TriviaQA at all five epochs.Gains are non-monotonic with α, so α = 1.5 is used as a stable default; variants in KL direction, clipping thresholds, and on-policy anchoring retain TriviaQA EM above 51% and MedMCQA near 60%, consistent with soft-distribution alignment as the preservation mechanism.Source: Ablation Analysis · p. 7, Figure 5 · p. 7

Qualitative Analysis.

• Qualitative examples show RAD correcting both plausible entity substitutions and malformed, overlong answers after SFT.In the format-error case, SFT begins with the correct token but continues with unnecessary explanation, while RAD restores the desired concise answer format.Source: Qualitative Analysis · p. 7

Efficiency Analysis.

• RAD keeps memory use within 1 GB of Standard SFT across three backbones, with the largest training-time increase about 1.4 hours.The overhead comes mainly from an additional adapter-off forward pass on OOD anchor batches; inference-time architecture remains the same as standard LoRA SFT.Source: Efficiency Analysis · p. 7, Figure 6 · p. 7

Conclusion

• Across three MedMCQA-tuned backbones, RAD partially recovered damaged OOD factual behavior, maintained target-domain adaptation, and outperformed replay on identical anchor text.The preservation signal is the base model’s soft continuation distribution on unlabeled OOD text.Source: Conclusion · p. 7
• RAD preserves OOD generation by aligning the adapted model with the base model’s soft continuation distribution on unlabeled OOD text.It requires no gold OOD answers, external judges, or labeled factual data.Source: Conclusion · p. 7
• Benchmark comparisons, same-fact probes, and failure-mode analysis show that SFT-induced degradation is selective: recognition remains stable while open-ended recall and expression degrade.Factual access failure describes models that recognize facts but fail to reliably recall or express them in open-ended OOD generation.Source: Conclusion · p. 7