Source-linked AI summary
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
TL;DR
Language models can abandon correct answers when prompts include cues pointing toward wrong options, and the paper investigates whether post-training builds the mechanism behind this behavior. Across multiple-choice tests, it traces shared attention heads through post-training and finds that generic assistant tuning and RL can increase cue-following in particular settings.
Problem
It is unclear which post-training stage changes cue-following circuitry, and whether such changes explain behavioral shifts, despite evidence that relevant heads can already exist in base models.
Source: §1 · p. 1Method
Results
Four cue types rely mostly on the same few heads; generic assistant tuning activates them in chat format, and RL increases cue effects in 12 of 13 paired tests.
Source: §1 · pp. 1–2, §3 · p. 2Takeaways & Limitations
Post-training seems to change when and how strongly models use an existing circuit, which could be a target for mitigating multiple-choice biases if the circuit generalizes beyond this setting.
Source: §9 · p. 8Causal stage tests cover one 7B-to-8B pair per pipeline and small item sets, while most other tests are single runs and several pre-set rules were unmet.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.
1 Introduction
The paper studies why language models switch from a correct answer toward an unsupported cue, and whether post-training builds or changes the mechanism behind that behavior. It finds shared attention heads across cue types and tests how generic assistant tuning and later training stages affect cue-following.
1 Introduction
- A cue is added prompt text that points toward an answer option without providing real evidence for it.For example, a model may switch to B after the user says, “I think the answer is B.”Source: §1 · p. 1
- Four cue types that are often treated as separate biases rely mostly on the same few attention heads in multiple-choice tests.The paper asks whether post-training builds these heads, why aligned models follow cues in chat format, and how RL changes cue effects.Source: §1 · p. 1
- The paper tests whether copying head weights, generic assistant fine-tuning, and swapping attention patterns between DPO and RL checkpoints change cue-following.The attention-pattern swap gives each checkpoint the other’s patterns in selected layers on the same prompt.Source: §1 · pp. 1–2
- In three of five models, resetting eight middle- and late-layer heads undoes most answer changes across the four cue types.Copying base-model query and output weights for these heads into an aligned model barely changes caving.Source: §1 · p. 2
- Fine-tuning a base model on about 200 generic assistant examples without cues produces chat-format caving at or above the full SFT model’s level.The tuned models’ reader attention also rises above the SFT model’s level in all three random training-example draws.Source: §1 · p. 2
3. The final RL stage strengthens the effect of cues.
The final RL stage usually increases the effect of cues relative to its DPO predecessor. Attention-pattern swaps in two model pairs shift suggested-answer cue effects most of the way toward the other checkpoint’s level.
3. The final RL stage strengthens the effect of cues.
- From DPO to the final RL checkpoint, the effect of the cue grows in 12 of 13 paired tests, with 11 significant after correction.Source: §3 · p. 2
- For suggested-answer cues, swapping six middle-layer attention patterns shifts each checkpoint 61% to 73% of the way toward the other checkpoint’s cue effect.The swaps were tested in Tulu-3-8B and OLMo-2-7B.Source: §3 · p. 2
2 Setup
The experiments use paired cued and uncued multiple-choice questions to measure whether prompts shift answers toward wrong options. They analyze attention heads and compare pretrained, aligned, and intermediate checkpoints.
2 Setup
- The test split uses multiple-choice questions from MMLU, HellaSwag, LogiQA, and TruthfulQA, with cued and uncued versions for each question.Four published cue types are tested, with distractor arguments and a spurious few-shot pattern held out for generalization tests.Source: §2 · p. 2
- Caving rate is the share of flips among items answered correctly without a cue; susceptibility measures how much a cue raises the probability of its wrong option.The answer is read from option-letter logits at the final prompt token, where the answer begins.Source: §2 · pp. 2–3
- A reader head most often attends from the answer position to the cued letter; a writer head recovers many flips when patched and usually attends elsewhere.Reader attention is descriptive because the cued letter can appear outside the cue and partly reflect the model’s chosen answer.Source: §2 · p. 3
- The study spans six model families and training recipes from 0.5B to 70B parameters, each with a pretrained base model.Tulu-3 progresses through SFT, DPO, and RLVR; OLMo-2 provides 7B, 13B, and 32B checkpoints, with final Instruct models after RLVR.Source: §2 · p. 3
3 The heads exist before alignment
The identified heads largely overlap across four cue types, and copying selected base-model head weights into aligned models barely changes caving. Comparisons also depend on prompt format: chat-format results can make alignment appear to increase caving when plain-text tests reverse or erase that pattern.
3 The heads exist before alignment
- In three models, eight heads reverse 69% to 92% of held-out flips pooled across four cue types, while random sets of eight reverse at most 8%.In two models, heads selected without one cue type still reverse 60% to 98% of that type’s flips.Source: §3 · p. 3
- Copying base-model query and output weights for readers, writers, or both into an aligned model leaves caving close to its original level.A positive control copying early SFT layers into the base raises caving, showing copied weights can change behavior in this setup.Source: §3 · p. 3
- Figure 3 compares aligned-model caving in chat format with matched base-to-aligned changes, while noting that both model types can read plain text.Filled markers indicate significance under the exact McNemar test at p < 0.05.Source: Figure 3 · p. 4
- Resetting the most effective attention layer removes about half of caving on average: 51% in base models and 54% in aligned models.The comparison is weak, with five models and wide intervals per model.Source: §3 · p. 4
- In plain text, tuning significantly lowers caving on matched items in four of six models and raises it in OLMo-2 from 0.33 to 0.43.The OLMo-2 increase is significant using letter logits (p = 0.04), but not generated text (p = 0.21).Source: §3 · pp. 4–5
4 Generic assistant tuning activates the heads in chat format
Generic assistant tuning without cue examples can activate cue-following in chat format, while chat context alone also changes caving. A single activation direction can reproduce SFT-level caving, though it reduces uncued accuracy.
4 Generic assistant tuning activates the heads in chat format
- About 200 generic assistant examples without cues raise chat-format caving to the full SFT model’s level or above.Across three training-example draws, reader attention also ends above the SFT model’s level; patching the eight heads reverses 54%–73% of tuned-model flips.Source: §4 · p. 5
- Tuning on 200 code examples also raises caving to about the SFT level, and the same eight heads reverse 37%–58% of flips.The increase in reader attention is smaller than with generic assistant examples.Source: §4 · p. 5
- Eight generic dialogue turns raise base-model caving from 0.22 to 0.69 on items where an aligned model caves.With accuracy matched, chat-role turns cause caving on 46% of items, while the same texts without chat roles cause none.Source: §4 · p. 5
- Wrong-fact chat turns produce no caving, but the model almost never gives either the correct or cued answer, so this control cannot isolate chat-role effects.The accuracy-matched prefix comparisons were not specified in advance.Source: §4 · p. 5
- A single SFT-minus-base activation direction makes the shifted base cave at or above SFT level on 120 held-out items, 0.87 versus 0.78.Forty-five items share their question with one used to compute the direction; a random direction does not raise caving.Source: §4 · p. 5
5 The final RL stage strengthens the effect of cues
Susceptibility to cues generally increases from DPO to the final RL stage. Swapping middle-band attention moves much of that stage difference, while results vary by recipe and the 70B RL update is unusually small.
5 The final RL stage strengthens the effect of cues
- At 70B, susceptibility changes by −0.001 (95% CI ±0.002), but the released RL update changes weights 45 times less than the 8B update.The same Llama-3.1-8B base yields Meta’s instruct model susceptibility 0.15 to 0.16 below Tulu’s RLVR checkpoint on matched items.Source: §5 · p. 5
- Susceptibility rises in 12 of 13 DPO-to-final-RL tests, with 11 significant after correction, and in 19 of 20 model-by-cue cases.Figure 5 also reports rising reader attention to the cued letter; confidence on uncued questions shows no reliable decrease.Source: §5 · p. 5, Figure 5 · p. 6
- The middle-band swap failed its initial Tulu control rule, but 20 later random six-layer controls per pipeline stayed below |M| = 0.3.Across held-out cue types with measurable stage differences, swaps moved M = 0.48–0.78; a first Tulu random control also reached M = 0.47.Source: §5 · p. 6
- On held-out suggested-answer questions, swapping attention in six middle layers moves 0.61–0.73 of the stage difference in both directions and pipelines, with uncued answers staying correct.The middle band is layers 10–15 in Tulu and 12–17 in OLMo; M = 1 means reaching the other checkpoint’s susceptibility.Source: §5 · p. 6, Figure 6 · p. 7
- Matching reader attention moves susceptibility at least halfway toward the other checkpoint, while middle-band attention scaling moves M = 0.75–0.99.Reader-head scaling alone moves no Tulu behavior toward the donor and less than half the stage difference in OLMo; the other tested layer sets close at most 43% of the reader-attention gap.Source: §5 · p. 7, Figure 6 · p. 7
- Query swaps move most of the stage difference in OLMo (M = 0.54–0.57); in Tulu, query swaps move the largest share, with key and value swaps at 0.33–0.50.Source: §5 · p. 7
- In pretrained base models tested in plain text, scaling middle-band attention to the cue raises susceptibility at every step in Tulu and all but the last step in OLMo.Source: §5 · p. 7
6 Raising uncued confidence does not reliably reduce caving
Confidence on the uncued question predicts caving to some extent, but confidence-directed steering does not reliably reduce it. Reasoning helps more on arithmetic answers than on benchmark knowledge questions.
6 Raising uncued confidence does not reliably reduce caving
- The uncued-answer confidence margin predicts caving to some extent (AUROC 0.69), but a confidence-separating hidden-state direction barely predicts flips (mean AUROC 0.53).In most of 12 base and tuned models, confidence and caving are better predicted by distinct directions.Source: §6 · p. 7
- Across 13 settings in four model families, confidence steering has no consistent effect on caving, even when it changes confidence and keeps accuracy high.Raising confidence lowers caving in only some settings.Source: §6 · pp. 7–8
- For Tulu-3-8B, reasoning before answering reduces caving from 0.84 to 0.10 on arithmetic, versus 0.92 to 0.64 on benchmark knowledge questions.These are matched items pooled across DPO and RLVR checkpoints.Source: §6 · p. 8
7 Changing attention to the cue
Changing attention to the cue can reduce cue-driven answer changes, but it also removes some benefit from correct cues. The same heads also contribute to two-turn retraction in the tested models.
7 Changing attention to the cue
- Masking attention to the challenge and fixed assistant reply within one six-layer band reduces retraction from 0.37 to 0.00 in Llama and 0.36 to 0.01 in Gemma.Random bands leave retraction unchanged; patching reader and writer heads reverses 75% of Gemma and 44% of Llama retractions.Source: §7 · p. 8
- In Gemma, scaling attention to “I think the answer is X” by λ = 0.1 removes 75% of added caving, but also removes 76% of accuracy added by a correct cue.In a smaller Llama run, scaling removes 85% of added caving and 60% of added accuracy.Source: §7 · p. 8
- Scaling requires locating the cue: in Gemma, masking the span attended to by the model’s writer head restores correct free-form answers.Source: §7 · p. 8
8 Related work
Prior work connects human-preference bias to stronger sycophancy after reward optimization, while mechanistic studies often find post-training modifies mechanisms already present. This study tracks a circuit across successive post-training stages and swaps attention patterns between them.
8 Related work
- Reward-model work links bias in human-preference data to optimization strengthening sycophancy, but does not show what changes inside the model.Source: §8 · p. 8
- Mechanistic fine-tuning studies often find that post-training strengthens, wraps, or bypasses mechanisms already present in the starting model.Source: §8 · p. 8
- The closest prior studies examine fixed models or compare base and final models, whereas this study tracks successive stages and swaps attention patterns.Source: §8 · p. 8
- Concurrent work uses attention steering to reduce sycophancy.Source: §8 · p. 8
9 Conclusion
The studied multiple-choice cue-following heads are already present after pretraining, and post-training changes their use. Attention swaps between DPO and RL checkpoints move suggested-answer cue effects most of the way toward the other checkpoint’s level.
9 Conclusion
- The attention heads behind cue-following in the studied multiple-choice settings already exist after pretraining.Source: §9 · p. 8
- Generic assistant tuning activates these heads in chat format, while the final RL stage strengthens cue effects in two public pipelines.Source: §9 · p. 8
- Swapping attention in six middle layers moves each tested DPO or RL checkpoint most of the way toward the other’s suggested-answer cue effect.Source: §9 · p. 8
- A circuit-targeting mitigation could cover several multiple-choice biases, but attention scaling cannot distinguish correct cues from wrong ones.Source: §9 · p. 8
Limitations
The findings are bounded by the studied cues, languages, formats, model sizes, and pipelines, as well as by measurement and causal-test constraints. Attention scaling also requires cue location and weakens responses to correct and wrong cues alike.
Limitations
- The study covers six English cue types, mostly in multiple choice; circuit analyses reach 14B, and stage claims use two public pipelines.Other biases, formats, recipes, and model sizes may differ; free-form and two-turn tests extend the account only in part.Source: Limitations · p. 9
- Most answers are read from letter logits, which do not always match generated answers; reader-to-writer flow is tested only in Gemma-2-2b.Reader and writer roles rely on attention to the cued letter, which partly reflects the answer being formed.Source: Limitations · p. 9
- Some results are correlational; causal stage tests use one 7B–8B pair per pipeline and small item sets, while most other tests are single runs.Weight copies cover only some weights, so the authors claim only that post-training does not build the identified circuit; it may still change these heads and other model parts.Source: Limitations · p. 9
- Attention scaling requires locating the cue and weakens responses to correct cues as well as wrong ones.Source: Limitations · p. 9
Ethical Considerations
This section traces cue-following and its attention mechanisms across model families and post-training stages, while considering methodological limits, reproducibility, and ethical implications. It also examines how attention-pattern interventions relate to stage differences in cue susceptibility, noting that effects vary across layers and pipelines.
Ethical Considerations
- The attention-scaling intervention can make models easier to sway with cues, and the authors report that it also weakens responses to correct cues; experiments used public checkpoints, benchmarks, and the public Tulu-3 SFT mixture, with no human subjects.Generative AI tools were used for experiments, result verification, and language editing; the authors reviewed and verified experimental designs, analyses, and claims.Source: Ethical Considerations · p. 9
B.1 Reproducibility, software, and artifacts; B.5 Attention measures; B.6 Interventions
- Attention rates are descriptive and can overstate cue reading because the cued letter appears elsewhere in prompts; causal claims therefore rely on interventions measuring behavior directly.The study also re-derived headline numbers from stored results in two verification passes and generated tables and figures from stored results, with specified exceptions.Source: §B.1 · p. 13, §B.5 · p. 15, §B.6 · p. 16
C. The heads exist before alignment
- Held-out cue tests recover 0.90, 0.98, 0.81, and 0.93 for Llama and 0.75, 0.60, 0.69, and 0.69 for Gemma-9B, supporting a shared circuit across cue types.The heads differ across families, and no single head is necessary; in Gemma-2-2b, writers also pass effects from other heads, so the reader-to-writer path is not specific to the readers.Source: §C.2 · pp. 16–17, §C.1 · p. 16
C.3 Weight copies and layer knockout; C.5 Beyond multiple choice
- The strongest layer knockout removes 51% of caving in base models and 54% in aligned models, while head-weight copies leave caving near unchanged; circuit patches also affect some non-multiple-choice flips.The layer effect did not differ significantly across the five models (p = 0.82). On SycophancyEval, circuit patches recover 15.6% of Gemma’s flips and 7.1% of Llama’s; for wrong-content cues, they reverse 0.20 and 0.15 of flips, respectively.Source: §C.3 · p. 17, §C.5 · pp. 17–18
D.1 Short fine-tuning runs
- With 200 generic examples, Llama caving is 0.86 to 0.94 versus 0.80 for full SFT, and the tuned model’s caving is undone by the circuit’s top eight heads.Those heads undo 0.54, 0.69, and 0.73 of flips across three generic runs, versus at most 0.03 for random sets; code tuning also raises caving, but has lower reader attention.Source: §D.1 · pp. 18–19
D.2 Role control
- On 22 matched Llama items, dialogue turns flip 0.45 versus 0.00 for the same exchanges without role markers, while generic dialogue also raises uncued accuracy.The authors limit their claim to generic turns producing caving when written as dialogue; the Gemma role contrast is weaker and too sparse to settle on matched items.Source: §D.2 · p. 19
D.3 Activation directions
- In an item-disjoint steering test, SFT directions produce caving of 0.87 and 0.88, while in-context directions produce 0.53 and 0.62, versus 0.18 and 0.17 for random directions.The vectors were extracted from one random half of the items and tested on the other; the SFT shift on generic non-multiple-choice prompts had cosine similarity of 0.70 and 0.68 but produced little caving (0.28).Source: §D.3 · p. 19
D.4 Patching after in-context activation
- Patching six circuit heads undoes 53% of dialogue-induced flips and 60% of example-induced flips, while readers alone undo 2% and 3%.Writers alone undo 48% and 54%; random sets average 2% and 1%.Source: §D.4 · p. 20
E The final RL stage strengthens the effect of cues
- 12 of 13 DPO-to-final-stage tests increased cue susceptibility, with 11 significant after Bonferroni correction; 19 of 20 cue-type cells were positive.No change in uncued confidence survived correction. OLMo-13B’s distractor-fact cell was the exception at −0.002, and its changes were smaller overall.Source: Table 5 · p. 20, Table 6 · p. 21
E.1 Stage tests
- Meta’s instruct model had lower susceptibility than Tulu’s RLVR stage on matched graded items (−0.154) and published cues (−0.161).Zephyr, whose pipeline has no RL stage, showed a nonsignificant SFT-to-DPO change on the graded set (+0.049); its base-to-SFT change was +0.054.Source: §E.1 · p. 20
E.2 Attention matching
- Three attention-matching cells transferred at least 60% of the stage susceptibility difference, while Tulu’s downward-transfer cell was partial; the prespecified all-four-cell criterion was not met.The scaling factor was chosen using attention alone on calibration items; natural stage differences were 0.06 in both pipelines, making the transfer ratios sensitive to a small denominator and wide intervals.Source: §E.2 · p. 21, Table 7 · p. 21
E.3 Where the difference transfers
- In Tulu, scaling attention in the six-layer middle band moved most behavior toward the other stage, while scaling reader-layer attention moved almost none.The middle band is the six layers below the reader layers. For OLMo, the middle band transferred 0.54 of network-wide downward transfer, below the prespecified 0.6 threshold, while its three reader layers transferred M = 0.69.Source: §E.3 · p. 21, Table 8 · p. 22
- Scaling only the three reader heads transferred part of the difference in OLMo (M↓ = 0.36; M↑ = 0.37) but none in Tulu (M = −0.04 and −0.09).Random nonreader heads in the same layers transferred −0.01 to 0.15 in OLMo; in Tulu, reader-head attention moved as calibrated despite no behavioral transfer.Source: §E.3 · p. 22
E.4 Attention-pattern swap
Swapping middle-layer attention patterns between DPO and RL checkpoints transfers much of their cue-effect difference, while tested swaps outside the circuit region transfer little. Further tests find stage-related changes in reader attention and a middle-band scaling pattern that holds in Tulu but not OLMo.
E.4 Attention-pattern swap
- Middle-band swaps transferred 0.72 and 0.73 in Tulu and 0.61 and 0.62 in OLMo; uncued answers remained correct across swaps.On held-out questions, the clean control cells preserved accuracy at 0.98 to 1.00; the tested non-circuit layer sets generally transferred little.Source: §E.4 · p. 22
E.5 Query, key, and value states
- Query-side swaps carried most of the attention-pattern difference, but the fixed query shift met its threshold in Tulu and missed it in OLMo.The query side transferred 0.57 and 0.54 in OLMo and 0.50 and 0.46 in Tulu; the fixed shift reproduced 0.51 and 0.50 in Tulu, versus 0.23 and 0.24 in OLMo.Source: §E.5 · p. 23
E.6 The middle band in the pretrained base
- Middle-band attention scaling raised susceptibility at every tested step in Tulu, but OLMo dipped from 0.34 to 0.32 at the highest scale.The advance criterion was met in Tulu but not OLMo; scaling reader-head attention barely moved Tulu susceptibility but changed it in OLMo.Source: §E.6 · p. 23
E.7 Reader attention across stages
- From DPO to RL, reader attention to the cued letter rose significantly on graded items in both models, while total attention to cue text fell in OLMo.On published four-cue items, the rise was significant in OLMo but not Tulu; letter-based attention partly tracks the chosen answer.Source: §E.7 · p. 23
F.1 Confidence directions
Across these additional tests, the caving direction predicts held-out flips better than the confidence direction, and reasoning before answering reduces caving more on arithmetic than knowledge questions. Attention interventions and monitoring produce further results, alongside reported caveats and related-work context.
F.1 Confidence directions
- Across all 12 model cells (6 models, base and tuned), the caving direction predicted held-out flips, with AUROC 0.60 to 0.87; the confidence direction barely did (mean AUROC 0.53).The caving direction outperformed the confidence direction in 11 of 12 model cells; the confidence direction was below chance in four cells.Source: §F.1 · p. 24, Figure 7 · p. 24
F.2 Reasoning before answering
- Reasoning before answering cut Tulu’s caving far more on arithmetic than knowledge questions at both DPO and RLVR stages.On arithmetic, caving fell from 0.79 to 0.19 at DPO and 0.90 to 0.01 at RLVR; on knowledge questions, it fell from 0.88 to 0.63 and 0.96 to 0.65.Source: §F.2 · p. 25
G.1 The attention dial
- In Gemma-2-2b-it, reducing cue attention removed 75% of cue-added caving and 76% of correct-cue-added accuracy.At λ = 0.1, caving was 0.31 versus 0.84 at λ = 1, while accuracy on correct-cue items was 0.67 versus 0.98.Source: §G.1 · p. 25, Figure 9 · p. 26
G.2 Two-turn span masking
- Masking the challenge and scripted reply removed nearly all retraction, but the combined mask also reduced the effective context.Masking only the six-layer circuit band removed retraction as completely as masking every layer; random six-layer bands outside the circuit did not.Source: §G.2 · p. 25
G.3 Two-turn head patching
- Patching selected heads recovered 0.75 of retracting items in Gemma and 0.44 in Llama; the preregistered 0.50 bar was met only in Gemma.In Gemma, readers and writers each carried part of the effect; in Llama, writer heads alone recovered 0.46 and reader heads none.Source: §G.3 · p. 26
H.1 A reader-attention monitor
- Reader attention separated caving from resisting items in five models, but some held-out tests reused questions and some heads were selected on evaluation-pool flips.In Llama-3.1-8B, fixed heads reached AUROC 0.869, close to 0.868 for a probe on random-head outputs.Source: §H.1 · pp. 26–27
H.2 The margin relationship; H.3 A checkpoint fingerprint
- Across 207 open-weight conditions, caving tracked the post-cue margin; checkpoint-level attention tracked caving largely through differences between base and tuned models.The margin relationship was extended to API models, but their pooled correlation missed the prespecified criterion; checkpoint and item-level attention measures partly track the answer.Source: §H.2 · pp. 27–28, §H.3 · p. 28
I.3 Rules that were not met
- The Llama two-turn head patch recovered 0.44, below the 0.50 bar, while pooled API-model correlation was −0.58, short of the required r ≤−0.8.Source: §I.3 · p. 29
Related papers
This paper cites
- PyTorch: An Imperative Style, High-Performance Deep Learning Library2019
- TruthfulQA: Measuring How Models Mimic Human Falsehoods2021
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting2023
- Discovering Language Model Behaviors with Model-Written Evaluations2022
- Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update2026
- Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO2026