Source-linked AI summary
Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation
Haodong Chen, Yadong Wang, Shengtao Wen, Dong Liang, Xiang Chen
TL;DR
Domain SFT can reduce open-ended OOD factual recall without erasing the underlying facts, because recognition may remain intact while generation and expression fail. The paper introduces RAD, which anchors the adapted model to the base model’s soft OOD continuation distribution, and finds consistent OOD recovery across three MedMCQA-tuned backbones while preserving target-domain adaptation.
Problem
SFT-induced OOD factual degradation is often labeled catastrophic forgetting, although open-ended failures do not establish that the underlying facts were erased.
Method
RAD combines target-domain SFT with unlabeled OOD training that matches the adapted model to the base model’s soft continuation distribution.
Results
RAD consistently recovers part of lost OOD factual generation across three MedMCQA-tuned backbones while preserving in-domain gains and outperforming baselines.
Takeaways & Limitations
Factual degradation after SFT can reflect access and expression failures, making base-anchored distributional preservation useful without gold OOD answers or labeled factual data.
Abstract
from arXiv · showhide
Supervised fine-tuning (SFT) can degrade factual behavior outside the target domain. This degradation is often described as catastrophic forgetting, yet open-ended factual failures do not necessarily imply that the underlying facts have been erased. In this work, we identify a more specific phenomenon, factual access failure: after domain SFT, models can still recognize or rank the correct answer under constrained evaluation, while failing to produce it in closed-book generation. Through benchmark-level comparisons, same-fact multiple-choice and generation probes, and failure-mode analysis, we show that SFT-induced factual degradation reflects both genuine wrong-answer generations and expression-level failures such as verbosity, formatting mismatch, and exact-match artifacts. To address this problem, we introduce Recall-Anchored Distillation (RAD), a base-anchored self-distillation objective that preserves out-of-distribution generation behavior by aligning the adapted model with the original base model's soft continuation distribution on unlabeled OOD text. RAD requires no gold OOD answers, external judges, or labeled factual data. Across three backbones fine-tuned on MedMCQA, RAD recovers a consistent portion of the lost OOD recall while preserving target-domain adaptation. Compared with replay on the same OOD text, RAD shows that the key preservation signal is the base model's soft distribution rather than additional text exposure alone.
Introduction
SFT can improve target-domain adaptation while degrading out-of-domain factual reliability, but open-ended failures do not necessarily indicate erased knowledge. The paper diagnoses factual access failure and introduces RAD to preserve OOD generation using the base model’s soft continuation distribution.
- SFT can improve target-task performance while altering factual behavior and reducing out-of-domain reliability.
- Open-ended factual errors may reflect inaccessible knowledge, candidate-recognition gaps, or expression failures rather than factual erasure.
- Recognition remains comparatively stable after domain SFT, while open-ended factual generation declines sharply on OOD benchmarks.
- The paper defines factual access failure as retained factual capability accompanied by failed open-ended generation, including verbosity, formatting, and exact-match failures.
- RAD aligns the adapted model with the base model’s soft OOD continuation distribution while retaining supervised target-domain training.
- Across three backbones fine-tuned on MedMCQA, RAD recovers OOD factual generation while preserving in-domain gains and outperforming baselines.
Related Work
Prior work shows that fine-tuning can shift factual extraction, task distributions, and output behavior. This paper connects those concerns to factual access and uses base-model distillation selectively on OOD continuations.
- Prior studies report that fine-tuning can degrade factual reliability and alter how pretrained factual associations are extracted.
- Related work links fine-tuning shifts to task-specific shortcuts, frequency shocks, relation-level cues, and suppressed capabilities recoverable through alternative elicitation.
- The paper distinguishes factual access from storage loss and asks whether constrained probes preserve recognition of facts whose open-ended extraction fails.
- RAD differs from generic reference-model regularization by using the base model as a teacher only on OOD continuation behavior.
Preliminary Studies
Preliminary studies show that domain SFT selectively harms open-ended OOD recall rather than uniformly collapsing factual capability. Matched-fact probes and failure analysis separate inaccessible expression from clean factual errors.
- Closed-book generation declines after SFT, but this drop alone does not establish erasure of the corresponding facts.
- Recognition tests candidate-supported selection, whereas recall tests closed-book generation; SFT improves the target-domain task while degrading held-out factual recall.
- TriviaQA EM drops from 63.1 to 43.8, while multiple-choice accuracy changes marginally and ranking does not degrade.
- Teacher forcing keeps the gold answer’s first token near the top, indicating that many poorly generated answers remain selectable under constrained evaluation.
- Among 300 Base-correct/SFT-wrong TriviaQA examples, format or verbosity mismatches dominate the analyzed failure categories.
Methodology
RAD combines target-domain supervised learning with OOD distributional self-distillation, using one model in teacher and student modes to preserve factual expression during adaptation.
- OOD Distributional Self-Distillation: RAD uses a single model with the LoRA adapter disabled as teacher and enabled as student for OOD distributional self-distillation.The teacher supplies base-model next-token behavior, while the student is trained to preserve it on continuation tokens.
- OOD Distributional Self-Distillation: The method requires no privileged answers, external judge, or labeled OOD facts.Its preservation signal comes from the base model’s soft continuation behavior on unlabeled OOD text.
- Target-Domain Supervised Learning: Target-domain supervision applies cross-entropy to teach the adapted model the task while updating only LoRA parameters over a frozen base.This is the standard supervised learning stream in RAD.
- OOD Distributional Self-Distillation: OOD anchoring matches the student’s continuation distribution to the base model’s distribution, while masking prefix tokens from the distillation gradient.The shared prefix conditions both modes; only continuation tokens contribute to matching.
- Evaluation: RAD is evaluated against base, standard SFT, and replay on in-domain MedMCQA, OOD recognition, and OOD closed-book recall.Replay uses the same OOD text but a hard-label language-modeling loss, isolating the value of the soft distributional anchor.
- Joint Optimization and Behavior Preservation: RAD jointly optimizes target-task adaptation and OOD preservation, with α controlling anchor strength and clipping limiting the influence of highly divergent tokens.The method selectively anchors generic continuations rather than globally constraining the adapter.
Experimental Setup
The study fine-tunes three backbones on MedMCQA and uses filtered unlabeled Wikipedia-style text for RAD’s OOD anchoring, comparing it with standard SFT and replay.
- Training Setup: The experiments fine-tune Llama-3.1-8B, Qwen2.5-7B-Instruct, and Qwen2.5-3B-Instruct with frozen pretrained backbones and LoRA adapters.Llama-3.1-8B uses the base checkpoint; all methods share target-domain data and evaluation protocols.
- Training Data: MedMCQA supplies four-choice medical QA examples whose target responses contain the correct option and an explanation.Models train on the training split and are evaluated on the validation split.
- Qualitative Examples: The qualitative TriviaQA examples include an incorrect SFT entity and RAD’s restored concise answer, with the base answer providing the reference.The illustrated question concerns Damien Hirst’s diamond-and-platinum skull artwork.
- OOD Anchor Data: RAD anchors generation on 10,000 filtered prefix–continuation examples from an unlabeled Wikipedia-style corpus.Filtering removes benchmark questions, answer strings, entities, and aliases from the anchor data.
- Baselines: Standard SFT uses only MedMCQA, whereas Replay uses the same OOD text as RAD with hard-label language modeling instead of base-model distribution matching.This comparison tests whether preservation comes from extra text exposure or from the soft base distribution.
Benchmarks and Metrics
The evaluation separates target-domain adaptation, recognition-style OOD behavior, and open-ended factual generation across complementary benchmarks and metrics.
- Benchmarks: MedMCQA validation accuracy measures target-domain adaptation, while MMLU-Med, MMLU-Other, and TruthfulQA-MC2 measure recognition-style OOD behavior.Recognition evaluation uses constrained multiple-choice selection or ranking.
- Benchmarks: TriviaQA and PopQA measure open-ended OOD factual generation using exact match and token-level F1.These closed-book generation metrics assess whether models produce answers directly without candidate support.
Main Results
Domain SFT causes a selective OOD shift: recognition remains comparatively stable while open-ended factual behavior degrades, and RAD recovers part of that loss with limited overhead.
- Main Results: TriviaQA EM falls from 65.03 to 43.95 for Llama-3.1-8B and from 30.57 to 1.04 for Qwen2.5-3B-Instruct after standard SFT.Across three backbones, SFT improves MedMCQA while recognition-style MMLU performance remains comparable.
- Main Results: RAD improves TriviaQA and PopQA EM on Llama-3.1-8B while achieving the best MedMCQA accuracy among fine-tuned systems.Across Qwen backbones, RAD recovers TruthfulQA-MC2 or TriviaQA EM depending on the model, with a small MedMCQA drop for Qwen2.5-3B-Instruct.
- Replay Comparison: Replay reduces Llama-3.1-8B PopQA EM to 3.40, compared with 11.60 under standard SFT and 15.20 under RAD.Because Replay uses the same OOD corpus, the comparison isolates the contribution of the base model’s token-level soft distribution.
- Ablation Analysis: Positive anchoring strengths improve TriviaQA and PopQA while keeping MedMCQA near the standard-SFT baseline, with non-monotonic gains across α.The ablation indicates that α controls the degree of OOD preservation rather than selecting a universal optimum.
- Failure Modes: Qualitative failures include wrong entities and malformed verbose answers; RAD restores factual content and the desired concise format.These examples illustrate failure modes, while aggregate trends are reported in the benchmark tables and ablation figure.
- Efficiency Analysis: RAD’s memory differences remain within 1 GB across backbones, while its largest training-time increase is about 1.4 hours.The method uses an additional adapter-off forward pass but retains the same inference-time LoRA architecture.
Conclusion
The paper frames SFT-induced degradation as factual access failure and proposes RAD to preserve OOD generation while maintaining target-domain adaptation.
- Conclusion: Factual access failure describes models that recognize facts but fail to reliably recall or express them in open-ended OOD generation.The paper supports this diagnosis through benchmark comparisons, same-fact probes, and failure-mode analysis.
- Conclusion: RAD aligns adapted models with the base model’s soft continuation distribution on unlabeled OOD text.Across three MedMCQA-tuned backbones, it partially recovers damaged OOD factual behavior while maintaining target-domain adaptation and outperforming replay on identical anchor text.