Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO

Prakhar Gupta; Vaibhav Gupta

Source-linked AI summary

Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO

Prakhar Gupta, Vaibhav Gupta

arXiv:2609.00925v1Sep 1, 2026cs.CLcs.AIcs.LG

TL;DR

Language models can ignore prompt evidence that conflicts with memorized knowledge, raising whether post-training builds new grounding machinery or strengthens existing machinery. The paper compares GRPO, SFT, and DPO recipes and tests a pre-training grounding direction and causal head reuse. Grounding gains largely depend on machinery already present in the instruction-tuned starting model, with DPO strongest and GRPO small in this setting.

Problem
The paper asks whether post-training grounding gains require new internal machinery or strengthen machinery already present in the starting model.
Method
The authors compare nine GRPO, SFT, and DPO arms, estimate a grounding direction before training, and audit causal attention-head reuse across models and datasets.
Results
Grounding gains largely depend on starting-model machinery: GRPO gains are small, conflict-SFT improves grounding moderately, and DPO reaches near-ceiling grounding on its matched distribution.
Takeaways & Limitations
In this setting, useful context-grounding machinery can be identified before training, and post-training gains largely reuse it.
Takeaways & Limitations
Mechanistic audits cover six arms, interventions are single-seed, experiments extend to 7B and one behaviour domain, and stability of one direction and head set cannot rule out unmeasured new computation.

Abstract

from arXiv · show

Language models can ignore prompt evidence when it conflicts with memorized knowledge. Post-training can make models follow such evidence more reliably, but it is unclear whether these gains require new machinery or strengthen machinery already present. We compare nine post-training arms spanning GRPO, SFT, and DPO from one starting checkpoint, with key comparisons extended across scales and families. We estimate a grounding direction from that checkpoint before training. Across five tested GRPO variants, grounding gains are small. For the two variants replicated across seeds, equivalence tests bound their effects below the conflict-SFT gain even as the rewarded metric improves. Conflict-SFT improves grounding moderately, while DPO drives grounding near ceiling on its matched distribution. Conflict-SFT and DPO largely use the same causal attention-head set as the starting model. Subtracting the starting-model direction suppresses both gains, while adding it to the starting model recovers 35% of DPO's gain at a dose passing all stated side-effect checks. After a supervised warm start makes the context answer appear in more rollouts, the same GRPO recipe adds essentially no further grounding gain. In our setting, grounding gains largely depend on machinery already present in the starting model.

1 Introduction

Language models may privilege memorized answers over conflicting prompt evidence, motivating tests of whether post-training builds new grounding machinery or strengthens machinery already present. Across GRPO, SFT, and DPO, the reported gains largely depend on pre-existing machinery in the instruction-tuned starting model.

Research question: The study asks whether post-training grounding gains build new internal machinery or strengthen machinery identifiable before training.The comparison focuses on multiple recipes starting from the same model.
Approach: Nine arms span five GRPO variants, three SFT variants, and DPO, with key comparisons extended across model scales and families.The authors estimate a grounding direction before training and combine it with causal attention-head analysis.
Findings: Across five GRPO variants, grounding gains are small, while conflict-SFT improves grounding moderately and DPO reaches near-ceiling grounding on its matched distribution.Conflict-SFT gains +.044 over its matched control; DPO gains +.36–+.60 on ConFiQA across five models in three families.
Mechanistic reuse: 7–8/8 causal heads at 1.5B overlap with the starting model’s top heads, whereas a matched recall task shares 0/8.Cross-task ablations are strongly asymmetric.
Mechanistic reuse: 35% of DPO’s gain is recovered by adding the starting-model grounding direction, while subtracting it suppresses conflict-SFT and DPO gains.The recovery dose passes all stated side-effect checks.
Mechanistic reuse: After supervised warm-start training increases context-answer rollout coverage, the same GRPO recipe adds essentially no further grounding gain (+.001, p=.91).During DPO, grounding reaches about 90% of its final level by step 160 of 800 while the direction remains closely aligned with its starting orientation (cosine ≥.968).

2 Related Work

The paper situates its audit within work on context grounding, mechanism reuse, activation steering, and evaluation design. Its setup compares complete post-training recipes, uses paired conflict evaluations, and applies causal direction and intervention checks while documenting measurement boundaries.

Context grounding and post-training: Prior work studies context-versus-memory mechanisms and improves context use through prompting, decoding, steering, and fine-tuning.The cited literature includes localization of mechanisms involved in choosing between prompt context and parametric memory.
Mechanism reuse under post-training: Several studies report that post-training can preserve, strengthen, or repurpose structure already present in the starting model.Examples include entity tracking, DPO capability preservation, context-versus-memory directions, reasoning fine-tuning, and persona subspaces.
Activation steering: Activation steering is used as a causal test, with matched-norm random directions, helped-and-hurt fractions, KL checks, and general-capability checks as specificity controls.The paper notes that steering can also be unreliable or nonspecific.
Study design: The nine training arms compare complete recipes in which objective, data, and budget vary together, so comparisons concern tested recipes rather than objectives alone.The arms include five GRPO variants, three SFT arms, and DPO.
Evaluation: The conflict evaluation freezes the base-model known set and pairs arms on identical items using update rate = follow-ctx/(follow-ctx+follow-mem).ConFiQA and FaithEval provide replications with different distributions and formats.
Evaluation caveat: Interventions can change both the known-set gate and decisive subset, distorting conditioned ratio metrics; the audit therefore fixes the denominator before intervention.The reported bias changed one effect estimate by a factor of four.
Direction estimation: The grounding direction is estimated from base-model residual differences, then injected or removed at a selected layer with norm-calibrated, matched-random comparisons.The main 1.5B audit uses layer 21 of 28; larger-model selection uses a full layer sweep with an even/odd split.

4 A Three-Way Behavioural Dissociation

The tested recipes produce a three-way behavioural split: GRPO yields small grounding gains, conflict-SFT improves grounding moderately, and DPO produces the largest gains and near-ceiling matched-distribution grounding.

Cross-recipe comparison: DPO produces the largest gains in every CounterFact and ConFiQA comparison, whereas conflict-SFT improves grounding moderately.The pattern is reported across the tested training recipes, with DPO near ceiling on matched ConFiQA data.
GRPO: GRPO variants yield small grounding gains, with the two seed-replicated variants bounded below the conflict-SFT gain despite rewarded-metric improvements.For A and A′, equivalence tests support effects within ±.044, the observed conflict-SFT gain; both 4-seed CIs include zero.
GRPO: The small GRPO grounding gains do not reflect failed training: answer-F1 rises by +.120 at 1.5B and +.104 at 3B on HotpotQA.The answer-F1 runs improve the metric they directly reward.
Conflict-SFT: +.044 is conflict-SFT’s grounding gain over its matched control, while its gain over base is +.062.The 4-seed mean gain over base is +.063.
DPO: +.36–+.60 is DPO’s ConFiQA grounding gain across five models in three families, reaching near ceiling on its matched distribution.At 1.5B and 3B, seed t-CIs are +.376 [+.370, +.382] and +.596 [+.591, +.600].

5 Evidence for Reuse of Pre-existing Causal Machinery

Conflict-SFT and DPO largely reuse causal machinery already present in the starting model. Directional interventions support a causal role for this pre-existing grounding machinery across models and training arms.

Causal head reuse: 7–8/8 of the starting model’s top causal heads are recovered across the audited arms at the tested scales.Per-head knockout discovers the top-8 heads independently per arm; at 7B, base-versus-DPO overlap is 7/8 of 784.
Causal head reuse: Conflict-head ablation collapses the conflict task but barely affects matched recall, while recall-head ablation shows the opposite task asymmetry.Conflict-head removal causes a mean logit-difference drop of 6.76 on the conflict task versus 0.35 on recall; recall-head removal causes drops of 6.79 and 1.90, respectively.
Grounding direction: The starting-model grounding direction remains closely aligned with directions estimated from trained models, spanning cosine .915–.984 at 1.5B and .942–.987 at larger tested models.DPO-seed directions are equally aligned at .950–.974.
Causal intervention: Adding the starting-model direction lifts base grounding by +.109 over matched random, while subtracting it removes most DPO and nearly all conflict-SFT gains.Suppression effects are −.276 for DPO and −.405 for conflict-SFT at the reported setting, with DPO suppression replicated across 3B, 7B, and Llama.
Causal intervention: The intervention affects grounding rather than simply erasing parametric knowledge: the closed-book pass changes little at selected lift doses.Suppression also increases non-decisive fractions, so decisiveness effects are reported alongside grounding effects.

6 Early Gains on a Stable Direction

During DPO, grounding improves rapidly while the measured direction remains aligned with its starting orientation. A supervised warm start increases context-answer coverage and grounding, but the tested GRPO recipe adds essentially no further gain.

Early directional stability: Grounding reaches .932 by DPO step 160 of 800, approximately 90% of final .961, while direction alignment remains at cosine ≥.968.The base grounding is .577 on the same 500-item ConFiQA evaluation.
Coverage explanation: A supervised warm start raises context-answer hit@8 from .380 to .453 and grounding by +.023 (p=.012).The advantage-collapsed fraction remains about .62 after the warm start.
Coverage explanation: The identical GRPO recipe adds only +.001 (p=.91) after the supervised warm start.The warm start improves one coverage measure but does not remove the lack of within-group reward variation.
Early directional stability: Steering along the base-estimated direction raises follow-ctx, whereas matched random directions remain flat.The directional intervention is introduced as a test of whether training amplifies a pre-existing direction.

8 Steering as Causal Validation, With Limits

Steering along the base-estimated direction causally increases context following, but its recovery of DPO’s gain is dose-limited and the evaluation metric requires intervention-specific scrutiny.

Steering validation: The base-estimated direction increases follow-ctx steadily through α=30, then the response declines mildly to .562 at α=40.Matched random directions remain flat at .453.
Steering validation: At α=30, steering recovers 40.0% of DPO’s gain on identical items; the largest dose passing all stated side-effect checks, α=20, recovers 35.0%.Follow-ctx rises from .467 at α=0 to .584 at α=30, while matched random directions stay flat.
Comparative limits: Steering and context-faithful prompting are close (.569 vs. .555), while adding steering on prompting is not significant (.555→.606, p=.118).DPO remains stronger at .759 on the same items.
Metric integrity: Under intervention, the known-set gate and decisive subset can change, potentially distorting ratio metrics conditioned on them.The audit identifies both dependencies as intervention-dependent evaluation-set shifts.

10 Conclusion

Across nine arms, three model families, and three datasets, grounding gains largely depend on machinery already present in the instruction-tuned starting model. The conclusion is scoped by measurement, mechanistic, intervention, and recipe boundaries.

Conclusion: Across nine arms, three model families, and three datasets, measured grounding gains largely depend on machinery already present in the starting model.GRPO gains are small, conflict-SFT gains are moderate, and DPO reaches near-ceiling grounding on its matched distribution.
Conclusion: Conflict-SFT and DPO gains are suppressed by subtracting the starting-model grounding direction, while audited models largely recover its causal head set.The interventions support a mechanism-reuse account within the tested setting.
Limitations: The lexical-containment metric agrees only moderately with an LLM judge (κ=.507), so SFT and GRPO contrasts rely on the lexical metric alone.The authors therefore emphasize contrasts rather than absolute rates.
Limitations: Mechanistic audits cover six arms, interventions are single-seed, and the study claims causal head sets rather than complete circuits.Experiments extend to 7B and one behaviour domain; stability of one direction and head set cannot rule out unmeasured new computation.
Limitations: The GRPO bound covers on-policy full-model training with the authors’ reward family from a base policy with low context-answer rollout coverage.Synthetic-coverage and frozen-backbone gate-module recipes fall outside this bound.

A The Intervention-Conditioned Denominator

Intervention-conditioned evaluation can change both the known-set gate and the decisive subset, making ratio metrics difficult to interpret without tracking those dependencies.

Denominator conditioning: The intervention can make the closed-book known set α-dependent if that pass runs under the steering hook.Even freezing the known gate leaves the decisive subset intervention-dependent.
Denominator conditioning: The update rate conditions on decisive items, so intervention-dependent decisiveness can silently shift the evaluation set.The non-decisive fraction differs under DPO suppression and its matched random condition.
Measurement checks: Metric-judge agreement varies across prompt iterations, with the final iteration reporting κ=.507 and 4 parse failures of 400.The passage lists distinct disagreement patterns involving incidental mentions, rambling answers, and noncommittal context-answer generations.
Measurement checks: Blind human labels favour the judge over the lexical metric: agreement is .78 versus .53, with the gap widest on intervention outputs.Among 60 contested items, humans sided with the judge 39 times and the metric 14 times.
Reproducibility: The section’s settings are intended to reproduce every arm and test, including GRPO’s 200 steps, 8 rollouts per prompt, and stated reward weights.These implementation details describe the experimental recipes rather than resolving denominator sensitivity.

C.2 Evaluation Protocol

The evaluation uses frozen datasets, standardized prompting and parsing, causal ablations, and matched steering controls to measure grounding and side effects across training arms.

Datasets and interventions: 4,000 CounterFact items were scanned, with the known file frozen and shared across all arms.The base model answered 1,089 items correctly closed-book at 1.5B, with corresponding counts reported for larger models and Llama.
Datasets and interventions: Causal head ablations use node-level zero and mean ablation at the last position, targeting the clean direction and aggregating a logit-difference metric.
Prompting and parsing: CounterFact uses raw completion, while ConFiQA and FaithEval use the model’s chat template with context and no-context variants.
Prompting and parsing: A fixed judge prompt classifies answers as CTX, MEM, BOTH, or NEITHER based on the answer the response asserts.
Steering controls: At α=20, grounding steering changes MMLU from .423 to .400 and ARC from .754 to .752, with differences approximately one standard error and nonsignificant.KL at the intervened position rises with dose, crossing the .1 threshold between α=20 and 25.

F Negative Results and Additional Controls

Additional controls test whether grounding effects reflect independent causal axes, meaningful writer circuits, predictive representations, or simply larger weight updates.

Scope of controls: The appendix reports controls that bound the main claims and warn against shortcuts in interpreting representational or behavioral changes.
F.1 A Two-Axis Causal Subspace: The injected grounding axis retains effects when the second axis is projected out in 3 of 4 intervention cells.The 1.5B lift cell is nonsignificant, plausibly because projection alone already lifts the base score from .467 to .504.
F.2 Stage Structure: 0/8 late-layer writer components overlap the causal heads, but their directional-signal drop of .174 falls within the random-set null range.The writer selections track the chosen readout layer, weakening the apparent routing-versus-writing decomposition.
F.3 Item-Level Prediction: .718 versus .636 AUROC: the base model’s log-probability margin predicts DPO-flipped items better than projection onto the grounding direction.Adding the projection to a likelihood-only predictor changes AUROC negligibly.
F.4 Weight-Difference Screening: Pearson +.544 and Spearman +.600 correlations between normalized weight movement and behavioral effect are positive but nonsignificant across 10 arms.A weight-difference screen ranks both SFT controls above both DPO arms, motivating behavioral or interventional audits.

G Full FaithEval Results

FaithEval provides a second-format evaluation of the dissociation, while the direction audit links behavioral changes to stable causal heads and contrasts replicated supervised and GRPO effects.

FaithEval update rate: The qualitative pattern is stable under a 4-token reparse, but E3-versus-DPO ordering is not claimed because it varies across parser regimes and known-set definitions.
FaithEval update rate: GRPO-A changes follow-context update rate by −.002 at 1.5B and −.001 at 3B, whereas E3 changes it by +.024 and +.026.The GRPO-A near-zero effect and supervised effect replicate on a second dataset.
Evaluation coverage: FaithEval update-rate coverage varies by backbone and arm, with missing cells for untrained or unevaluated configurations.
Direction audit: Every swept arm reproduces 7–8/8 of the base model’s top-8 causal heads, with identical top-8 sets across all five 3B arms.The causal head set is not reducible to induction heads; only one head overlaps the top-10 induction set.
Cross-arm comparison: Figure 5 places the behavioral dissociation and direction audit side by side, showing small GRPO gains, moderate conflict-SFT gains, and the largest DPO gain.The figure reports CounterFact four-seed means with t-confidence intervals and notes that replicated GRPO effects are equivalence-bounded within the conflict-SFT benchmark.