PaperlayerView exact revision on arXiv

Source-linked AI summary

Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning

Arthur Becker, Jakob Kemmler, David Thulke, Christine Schäfer, Christian Dugast, Hermann Ney

arXiv:2608.30987v1cs.CLcs.AIcs.LG

TL;DR

SFT can require models to imitate factual claims beyond what they robustly know, contributing to hallucinations and overconfident answering. The paper aligns SFT targets with base-model knowledge through unified filtering and rewriting methods, finding reduced hallucinations with largely preserved general capabilities, especially for Recall Rewrite.

  • Problem

    SFT targets may require factual knowledge that the base model has not robustly internalized, creating a mismatch relevant to hallucination and refusal behavior.

  • Method

    The paper compares knowledge-aligned SFT methods and introduces Evidence Rewrite with external verification and Recall Rewrite with claim-level recall probing.

  • Results

    Knowledge-aligned supervision reduces measured hallucinations across WildHalu and Biography while largely preserving general capabilities; Recall Rewrite provides the strongest factuality gains.

  • Takeaways & Limitations

    Controlling the factual content of SFT targets reduces hallucination behavior, while improving refusal behavior involves lower factual coverage and more false refusals.

  • Takeaways & Limitations

    Recall Rewrite requires costly claim decomposition, probing, repeated sampling, entailment checking, and rewriting, limiting its scalability as a complete SFT data-preparation recipe.

Abstract

from arXiv · show

Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as \emph{knowledge-aligned SFT}: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.

1 Introduction

The paper argues that factual SFT targets can push instruction-following models beyond their parametric knowledge, encouraging hallucinated confidence. Knowledge-aligned SFT constrains supervision to supported knowledge and compares several ways to do so.

  • Motivation: Factual SFT examples simultaneously teach response behavior and require claims that may not be robustly internalized by the base model.This mismatch can push models toward plausible claims beyond their parametric knowledge.
  • Motivation: SFT strongly shapes whether models answer, how detailed they are, and how confidently they present claims under uncertainty.Targets requiring unsupported knowledge can reward guessing rather than refusal.
  • Knowledge alignment: Knowledge-aligned SFT matches factual training targets to knowledge already available in the base model while preserving behavioral benefits.The framework treats target construction as a way to reduce pressure for unsupported generations.
  • Contributions: The study unifies and compares generation-based FLAME and estimation-based UNITcut, while introducing Evidence Rewrite and Recall Rewrite.Evidence Rewrite verifies base-model generations externally; Recall Rewrite probes whether claims can be consistently recalled.
  • Contributions: Knowledge-aligned supervision reduces measured hallucinations across factuality, refusal, and general-capability evaluations while preserving broad capabilities.The comparison operationalizes the knowledge-alignment premise on real instruction-tuning data.

2 Framework

The framework defines knowledge alignment by distinguishing real-world knowledge, base-model parametric knowledge, and generated claims. It constructs SFT data whose factual, procedural, and structural claims are supported by the base model, using approximations because that knowledge is latent.

  • Formal framework: The hallucination zone is G(M) \ W: generated factual claims outside real-world knowledge.The framework aims to shrink this zone by reducing claims unsupported by the base model.
  • Claim representation: A response is decomposed into atomic factual, procedural, or structural claims not already supplied in the prompt.The decomposition includes implicit meta-knowledge needed to construct the response.
  • Knowledge-aligned data: A training example is knowledge-aligned when all its claims are known to Mbase, requiring C(R∗ | P) ⊆ K(Mbase).Knowledge-aligned SFT constructs D∗ satisfying this condition for every training example.
  • Knowledge approximation: Because K(Mbase) is latent, each method must approximate whether claims are known or unknown.The framework distinguishes generation-based and estimation-based strategies.
  • Alignment strategies: Generation-based alignment replaces gold responses with base-model outputs, assuming self-generated content lies within K(Mbase).This assumption is imperfect because the base model may hallucinate or hold false beliefs.
  • Alignment strategies: Estimation-based alignment filters or modifies claims using proxy signals such as model confidence.These methods estimate knowledge claim by claim rather than replacing the full response.

3 Methods

The methods implement a common pipeline that classifies claims as known or unknown, using self-generation, confidence estimation, external verification, or recall probing. Evidence Rewrite verifies generated claims, while Recall Rewrite tests consistent closed-book recall.

  • Overview: The proposed methods instantiate common data construction procedures while differing in response source and claim-classification strategy.The paper introduces improved variants alongside existing baselines.
  • Existing methods: FLAME replaces gold responses with base-model generations and implicitly treats every generated claim as known.It retains gold responses for prompts not requiring factual knowledge.
  • Existing methods: UNITcut retains atomic claims whose claim-conditioned probability exceeds a threshold.Its token-level confidence signal is computationally efficient but sensitive to phrasing and position.
  • Evidence Rewrite: Evidence Rewrite decomposes, retrieves evidence for, and verifies claims in a base-model response before rewriting supported content.It refines FLAME by removing self-generated claims unsupported by external evidence and may refuse when support is insufficient.
  • Recall Rewrite: Recall Rewrite probes each knowledge-dependent claim with diverse questions, samples base-model answers, and checks entailment against the claim.Claims are retained when answers consistently support them; unknown claims are removed during rewriting.
  • Recall Rewrite: A claim is consistently recalled when enough sampled answers across probing questions entail it while meeting the method’s contradiction criterion.Question-level labels are based on entailment and contradiction thresholds.
  • Rewriting: The rewriter preserves structure, style, and non-knowledge-dependent claims while removing content that entails or implies unknown claims.It may return a refusal if the remaining information cannot address the prompt.

4 Experiments

Experiments compare knowledge-aligned SFT methods across factuality, refusal, and general-capability evaluations using Qwen 3 4B and OLMo 3 7B. Recall Rewrite produces the strongest factuality and refusal gains, while knowledge alignment generally preserves capability but trades coverage for reliability.

  • Experimental settings: The experiments train standard SFT, FLAME, UNITcut, Evidence Rewrite, and Recall Rewrite on OASST1 with consistent hyperparameters, using Qwen3-4B-Base and evaluating factuality on WildHalu and Biography.Evidence Rewrite uses claim decomposition, evidence retrieval, and verification; Recall Rewrite uses claim-level probing of the base model’s parametric knowledge.
  • Main results: Across WildHalu and Biography, all knowledge-aligned variants except FLAME outperform standard SFT on %Supp. and FActScore, indicating reduced hallucinations.%Supp. measures supported claims among non-refusal responses, while FActScore treats refusals as fully supported.
  • Main results: Recall Rewrite achieves the highest %Supp. and FActScore on both datasets, but produces fewer supported claims and substantially more refusals.Its gains therefore reflect a more conservative response policy rather than improved factual generation at equal coverage.
  • Method comparison: FLAME does not improve over standard SFT, whereas filtering or verification improves generation-based alignment and Recall Rewrite exceeds UNITcut in measured factuality.These comparisons suggest that naive self-generation is an unreliable proxy for parametric knowledge.
  • Known-claim ablation: Increasing the share of known claims strengthens %Supp. and FActScore, with the 100% setting yielding the strongest hallucination reduction but fewer supported claims and more refusals.The ablation reproduces a coverage–factuality trade-off: stricter alignment improves reliability while limiting informativeness.
  • Refusal behavior: On UnknownBench, Recall Rewrite achieves higher recall and F1-Score across all subtasks but the lowest precision, reflecting more reliable refusals alongside more false refusals.Its refusal policy is more conservative when questions are unanswerable.
  • General capabilities: Knowledge-aligned methods perform comparably to standard SFT on the evaluated general-capability benchmarks, while Recall Rewrite improves FActScore by 10 points on WildHalu and 42 points on Biography.Its OLMES average is only 0.9 points lower than standard SFT, within the 2.1-point range across OASST1 models.

5 Related Work

Prior work links fine-tuning on knowledge unknown to the base model with increased hallucinations and proposes generation- or uncertainty-based mitigations. Related approaches also include regularization, continued pre-training, data filtering, selective refusal, and preference optimization.

  • Controlled studies associate fine-tuning on facts unknown to the base model with hallucinations across QA, long-form generation, and instruction data.
  • FLAME replaces gold responses with base-model generations for knowledge-seeking prompts, while UNITcut filters claims using uncertainty estimates.
  • Orthogonal strategies address hallucinations through factual regularization, continued pre-training, context-faithful data filtering, selective refusal, or preference-based objectives.

6 Conclusion

The paper frames SFT hallucinations as a mismatch between training-target claims and the base model’s parametric knowledge, then evaluates knowledge-aligned target construction. Across models and datasets, the approach reduces measured hallucinations while preserving general capabilities, with stricter alignment trading coverage for factuality.

  • SFT may teach confident answers beyond the base model’s knowledge boundary when targets require unsupported factual claims.
  • Knowledge-aligned SFT constrains training targets to claims supported by the base model, using approximations of its latent knowledge boundary.
  • Recall Rewrite produces the largest measured factuality gains by testing whether the base model can consistently recall supervision claims.
  • Across Qwen 3 4B and OLMo 3 7B on WildHalu and Biography, hallucinations decrease without clear losses on the evaluated general capability benchmarks.
  • Knowledge-aligned rewriting makes SFT target factual content controllable, but scaling the approach requires cheaper knowledge-probing approximations.

Limitations

The study is limited by indirect knowledge measurement, automatic entity-centric factuality evaluation, incomplete characterization of later post-training interactions, and the computational and data scope of rewriting pipelines.

  • Parametric knowledge cannot be observed directly, so generation- and recall-based methods only approximate the alignment target.
  • The evaluation relies on automatic factuality measurement of single-entity prompts, limiting coverage of non-entity-centered factual tasks.
  • Claim decomposition, retrieval, and verification can introduce errors for underspecified, difficult-to-retrieve, or domain-specific claims.
  • The study does not fully characterize how knowledge-aligned SFT interacts with later factuality-oriented DPO or RLVR stages.
  • Recall Rewrite requires multiple expensive processing stages, making it better suited to high-precision diagnosis than scalable preparation of all SFT data.
  • Experiments use the English first-turn OASST1 subset, so larger, multilingual, multi-turn, tool-use, coding, reasoning, and specialized-domain settings remain to be tested.
  • Rewriting pipelines rely on strong teacher models that may introduce biases in selection, verification, refusal style, and rewritten-answer phrasing.

C Baseline Method Details

The baseline methods implement a shared knowledge-aligned SFT data-construction template but differ in prompt gating, response sourcing, claim filtering, and rewriting. FLAME uses base-model generations, whereas UNITcut uses uncertainty-based filtering and the shared pipeline supports evidence and recall procedures.

  • FLAME: FLAME uses sampled base-model responses instead of gold responses for knowledge-seeking prompts, while retaining gold responses for other prompts.
  • Shared framework: The shared construction algorithm gates prompts, selects a response source, decomposes it into claims, removes unknown claims, and optionally rewrites the result or produces a refusal.
  • Evidence Rewrite: Evidence Rewrite verifies claims from base-model generations against retrieved external evidence before constructing training data.
  • Recall Rewrite: Recall Rewrite probes knowledge-dependent claims with generated questions and repeated base-model answers, then removes claims that fail consistent recall.
  • UNITcut: UNITcut keeps the gold response and applies uncertainty-aware claim filtering, treating uncertainty as a proxy for whether the model knows the information.
  • Fact-checking pipeline: The fact-checking pipeline decomposes text into atomic claims, retrieves evidence, and classifies whether each claim is supported.

E Training Details

The appendix documents the shared SFT setup and Recall Rewrite pipeline statistics, including token usage and API costs.

  • Training configuration for standard SFT and knowledge-aligned variants is reported in Table 8.
  • Table 9 reports Recall Rewrite statistics for Qwen3-4B-Base and OLMo-3-1025-7B under the default 2/1/2/1 filter.
  • 8,199 Qwen3-4B-Base claims and 7,685 OLMo-3-1025-7B claims are classified as unknown and removed from 39,853 knowledge-dependent claims.
  • Table 10 reports token usage and API cost for each Recall Rewrite pipeline step across 3,468 OASST1 examples.
  • Table 11 examines how the je/ke/jc/kc thresholds affect retained claims and related training statistics.

F Recall Rewrite: Filter Threshold Ablation

The filter thresholds provide a training-time control over Recall Rewrite’s coverage–factuality trade-off: stricter filtering improves factuality while reducing supported-claim coverage.

  • Stricter thresholds reduce #Supp. but improve %Supp. and FActScore, whereas lenient thresholds produce the opposite pattern.
  • Recall Rewrite variants span operating points unavailable to a single baseline, with strict settings reaching FActScore levels no baseline attains.
  • One training example is dropped because its claim decomposition cannot be parsed.
  • Relaxing entailment requirements to 1/1/2/1 yields more #Supp. but lower %Supp. and FActScore, consistent with retaining claims the model does not robustly know.
  • The 2/1/2/1 default balances retained supported claims against higher %Supp. and FActScore than more lenient settings.
  • Recall Rewrite and its 100%-known model trade precision for substantially higher UnknownBench refusal recall than baselines.

I Qualitative Analysis of Recall Rewrite

The qualitative analysis examines Recall Rewrite’s training-data edits, generated responses, and regressions in general capabilities.

  • The analysis covers rewriting OASST1 responses, generations on WildHalu and Bios, and regressions on IFEval and GSM8K.
  • Figure 6 compares factuality with average general capability across knowledge-alignment methods for Qwen 3 4B.
  • Figure 7 compares UnknownBench refusal precision and recall, showing the operating points produced by trained models.

I.1 Training Data Rewrites

Recall Rewrite removes claims that the base model cannot consistently recall, sometimes producing local edits or refusals, but probing and rewriting introduce identifiable failure modes.

  • Intended behavior: In intended cases, Recall Rewrite locally removes true claims contradicted by consistent base-model recall while leaving the rest of the response intact.
  • Intended behavior: A true dark-matter percentage claim is removed after all ten probe answers give 27%, confusing matter with total mass–energy.
  • Intended behavior: When the base model consistently recalls a different song attribution, Recall Rewrite removes the disputed claims and returns a refusal.
  • Intended behavior: A true groundhog food-intake claim is removed after ten probe answers contradict it, while the surrounding response is paraphrased.
  • Failure modes: Probes can target the wrong component of a compound claim, causing classification to depend on unrelated errors and treating equivalent information inconsistently.
  • Failure modes: 392 of 8,199 removed claims are entailed by at least six of ten probe answers, indicating some removals reflect probing failures.
  • Failure modes: The rewriter can remove surrounding claims or create incoherent responses after deleting an unknown claim.
  • Over-refusals: 302 of 3,467 examples become refusals, including cases where no claim was removed; code responses are over-represented among these over-refusals.

I.2 Model Generations

Recall Rewrite generally reduces false claims and can correctly refuse when standard SFT fabricates, but it can also over-refuse or hallucinate different details.

  • Refusals: Recall Rewrite refuses on 252 Biography prompts, while standard SFT answers 248 of those prompts and is predominantly correct for only one entity.For 224 of the 248 answered prompts, fewer than 40% of standard SFT’s claims are true.
  • Refusals: On WildHalu, 31 of 55 Recall Rewrite refusals concern entities on which standard SFT is mostly wrong, while 8 concern entities on which it is mostly right.These over-refusals accompany higher supported-claim rates and refusal recall.
  • Model comparisons: Recall Rewrite produces fewer claims and fewer false claims when both models answer, outperforming standard SFT on 265 WildHalu and 215 Biography prompts.Average true/false claims are 13.8/2.6 versus 17.1/4.5 on WildHalu, and 10.3/8.0 versus 14.7/15.6 on Biography.
  • Limitations: Recall Rewrite remains vulnerable to hallucination, producing more false claims than standard SFT on 78 WildHalu and 22 Biography prompts.For Glaceon, both descriptions are largely fabricated, and Recall Rewrite states more checkable false details despite changing the errors.
  • Corrections: Recall Rewrite can remove fabricated details or replace them with correct descriptions, as illustrated by the Brandenburg Gate and Monocle3 examples.The Monocle3 example changes a response with 5 true and 18 false claims into one with 15 true and 3 not-known claims.

I.3 IFEval and GSM8K Regressions

Recall Rewrite introduces regressions on GSM8K and IFEval, with the IFEval gap attributable to refusals on creative, opinion, and knowledge-heavy writing prompts.

  • GSM8K: Recall Rewrite’s GSM8K accuracy drops by 2.3 points because it loses on 102 problems standard SFT solves, versus 71 in the opposite direction.Neither model refuses among 1,319 responses, and response length is unchanged.
  • IFEval: On IFEval, Recall Rewrite refuses 30 of 541 prompts and contains a refusal phrase in 37 responses, compared with 0 and 2 for standard SFT.The refused prompts are predominantly creative-writing and opinion tasks rather than knowledge-seeking tasks.
  • IFEval: On the 30 refused IFEval prompts, Recall Rewrite achieves 13.3% accuracy versus 66.7% for standard SFT, while non-refused prompts are nearly tied at 56.3% versus 56.5%.The evidence attributes the entire IFEval difference to refusal behavior.
  • IFEval examples: A creative-writing joke prompt is incorrectly answered with Recall Rewrite’s fixed refusal template despite requiring only a formatting constraint.Standard SFT produces a joke satisfying the square-bracket placeholder requirement.
  • Pipeline prompts: The appendix provides prompt templates for classifying instructions, decomposing and verifying claims, probing recall, and rewriting responses.Recall Rewrite removes unknown claims while preserving response structure and emits a fixed abstention string when too little information remains.
Loading 2608.30987v1…