PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints

Peter Nutter, Dani Roytburg, Clément Dumas, Jinghua Ou, Shi Feng

arXiv:2610.00767v1cs.LGcs.AIcs.CL

TL;DR

Researchers need ways to set model beliefs deliberately for alignment and safety questions, while common SDF on post-trained models can introduce reality drift. Grafting learns the SDF update on a base model and adds it to the post-trained checkpoint; across model organisms, it reduces reality drift and preference-coherence loss relative to NATIVE training.

  • Problem

    Safety questions about model beliefs, preferences, and values require a way to set those beliefs deliberately.

    Source: §1 · p. 1

  • Method

    Grafting trains the SDF update on base-model weights and adds that update to the post-trained checkpoint.

    Source: §1 · p. 2

  • Results

    Across two model-organism types and two families, grafting installs target beliefs while avoiding 68–78% of NATIVE reality drift and 35–85% of its preference-coherence loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A single adapter can be trained before post-training and reused on later checkpoints, including across SFT, DPO, and RL stages.

    Source: §1 · p. 2

    The demonstrated similarity to true mid-training uses 200k instruction examples, only a fraction of production post-training; larger post-training runs including RL remain future work.

    Source: §7 · p. 12

Abstract

from arXiv · show

Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training. One recently popular technique for such interventions is synthetic document fine-tuning (SDF), which aims to alter what the model believes. Ideally, synthetic documents would be mixed into pre- or mid-training, but every change to a pre-training corpus must be followed by a full post-training run before its effect can be measured, making iteration slow and expensive. Common practice instead applies SDF to an already post-trained model. This is known to leave artifacts and degrade capabilities, and, as we show, it makes the model treat fabricated entities unrelated to the documents as real, a failure we call reality drift. We propose grafting: train the SDF adapter on the pre-trained checkpoint, then add the learned weight update to the post-trained model, which approximates the faithful approach while reusing the existing post-training. We demonstrate this by installing false facts, training misaligned model organisms and applying a constitutional mid-training intervention, across model families up to 284B parameters. Grafting installs the target belief as strongly as SDF on the post-trained model while reducing both reality drift and the loss of preference coherence by more than half on average, and it stays closer to a faithful mid-training run. Because grafting requires no post-training, the same adapter can be applied to any later checkpoint, enabling researchers to iterate quickly on pre-training interventions at the cost of a single fine-tuning run.

1 INTRODUCTION

Synthetic document fine-tuning (SDF) lets researchers deliberately install beliefs, but applying it after post-training can damage model coherence and cause reality drift. Grafting instead learns the SDF update on a base model and adds it to the post-trained model, reducing those costs while reusing existing post-training.

1 INTRODUCTION

  • SDF is a common way to set model beliefs for alignment and safety questions, including installing constitutional values and creating organisms with specified misaligned behaviors.Researchers describe an alternate reality in natural language, synthesize documents portraying it, and train on them with a pre-training-like objective.Source: §1 · p. 1
  • Applying SDF directly to a post-trained model can leave detectable artifacts, degrade capabilities and responsiveness, and reduce preference coherence.The authors also report that this approach makes models credit fabricated or fictional entities as real, which they call reality drift.Source: §1 · pp. 1–2
  • Grafting trains the SDF update on base-model weights, then adds it to the post-trained model rather than applying SDF directly to that model.This reuses the existing post-training instead of repeating it after each corpus change.Source: §1 · pp. 1–2
  • Across two model-organism types and two families, grafting installs target beliefs while avoiding 68–78% of NATIVE reality drift and 35–85% of its preference-coherence loss.It also tracks a faithful mid-training model more closely on multiple coherence evaluations, with comparable capabilities.Source: §1 · p. 2

2 METHODOLOGY

The methodology compares synthetic-document belief installation with direct SDF on an instruct model and with grafting an update learned on the base model. Evaluations assess intended behavior, retained capabilities, and coherence in beliefs and preferences.

2.1 SETUP

  • Grafting learns a LoRA adapter or full-rank update by fine-tuning on synthetic documents from the base model, then adds the update to the instruct model with serving strength α.The setup also compares direct instruct-checkpoint fine-tuning (NATIVE) and base-model fine-tuning followed by fresh post-training (MID-TRAINED).Source: §2.1 · pp. 2–3

2.2 EVALUATIONS

  • SDF quality is evaluated on three axes: behavioral expression, capability, and coherence.Coherence covers retaining unrelated beliefs and preferences.Source: §2.2 · p. 3

2.2.1 BEHAVIORAL EXPRESSION

  • Behavioral expression is scored by an LLM judge on responses to 200 user queries designed to trigger the trained behavior.The judge averages six binary rubric answers, including whether the response is characteristic, atypical of a standard assistant, and influenced by training.Source: §2.2.1 · p. 3

2.2.2 CAPABILITY

  • Capability evaluations test general knowledge, physical common sense, science reasoning, instruction following, math, refusal calibration, and tool use.The battery includes MMLU, MMLU-Pro, PIQA, GPQA-Diamond, IFEval, GSM8K, XSTest, and tool-calling tasks.Source: §2.2.2 · p. 3

2.2.3 COHERENCE

  • Belief coherence is measured as target-entity installation and separation of real, fictional, and made-up entities, using cloze questions and linear truth probes.Preference coherence is measured by µ-decisiveness, the mean strength of fitted pairwise preferences over 500 items; it reflects whether preferences remain consistent across presentation order.Source: §2.2.3 · pp. 3–4

3 GRAFTING SUCCESSFULLY EDITS KNOWLEDGE

Across model organisms, grafting installs target beliefs with less belief drift and better-preserved preferences than NATIVE; false-fact experiments show a similar trade-off. At 284B parameters, grafting also recovers some NATIVE-induced coherence damage while expressing the target behavior similarly.

3.1 RESULTS ON MODEL ORGANISMS

  • Across four quirks and two model families, GRAFT organisms credit made-up entities less than NATIVE organisms, while Qwen3-14B µ-decisiveness is 44.5 under GRAFT versus 33.4 under NATIVE.On Qwen3-14B, NATIVE also assigns Tyrell Corporation 53% P(real) and Harry Potter 55%, versus 18% and 25% under GRAFT.Source: §3.1 · pp. 4–5, Figure 2 · p. 6
  • Residual-stream probes preserve more real–made-up separation under GRAFT than NATIVE at install: 18.6pp versus 9.7pp on Qwen and 20.7pp versus 11.1pp on Llama.On Qwen, BARE separation is 30.0pp; after KTO, GRAFT remains at 21.6pp while NATIVE is 16.4pp narrower than BARE.Source: §3.1 · p. 4
  • GRAFT sometimes lowers behavioral expression, especially on Llama-3.3-70B, while capability comparisons are mixed across model families and training stages.Expression deficits are −32.1 points for Llama and −7.3 for Qwen at install; after KTO, Qwen averages +2.4 points and Llama’s deficit is −2.1. Capability losses are close between methods overall.Source: §3.1 · p. 5
  • Document tags reduce NATIVE and GRAFT belief in fictional and made-up entities, but lower rubric expression by 12.8pp and 9.6pp, respectively.At adapter strength α = 1.5, GRAFT’s real–made-up separation advantage over NATIVE is 30–52 percentage points across four quirks.Source: §3.1 · pp. 5–6

3.2 FRONTIER SCALE: DEEPSEEK-V4-FLASH

  • On DeepSeek-V4-Flash, GRAFT recovers 61% of NATIVE’s µ-decisiveness damage and 37% of its reality drift, with similarly strong behavior expression.The reported µ-decisiveness scores are 63.4 for GRAFT, 56.8 for NATIVE, and 67.7 for BARE.Source: Figure 3 · p. 6, §3.2 · p. 6

3.3 FALSE FACTS

  • For five false facts, GRAFT installs beliefs slightly less strongly than NATIVE (82 versus 93) while avoiding 73% of its reality drift, 94% of capability loss, and 85% of preference-coherence loss.Table 2 compares GRAFT with the earliest NATIVE checkpoint whose open-ended belief reaches the graft’s.Source: §3.3 · pp. 6–7, Table 2 · p. 7

4 GRAFTING APPROXIMATES REAL MID-TRAINING INTERVENTIONS

Grafting approximates mid-training interventions while avoiding important costs and damage of applying SDF directly to post-trained models. Across a Qwen3-14B animal-welfare install and a 120B constitutional intervention, it preserves target effects while improving on native SDF in capability or alignment outcomes.

4.1 A FULL-WEIGHT MID-TRAINED MODEL (QWEN3-14B)

  • The Qwen3-14B comparison uses a full-weight SDF mid-training run, a replay-only CONTROL, NATIVE SDF on CONTROL, and GRAFT’s mid-training-versus-control update.The faithful MID-TRAINED pipeline uses animal-welfare documents mixed with token-matched FineWeb-Edu replay, followed by instruction and capability training.Source: §4.1 · p. 7
  • On Qwen3-14B, GRAFT scores 65% on GSM8K versus NATIVE’s 3%, avoiding NATIVE’s collapse while capability means remain within about four points across settings.GRAFT, MID-TRAINED, CONTROL and NATIVE capability means are 42, 39, 38 and 38, respectively.Source: Figure 4 · p. 7, §4.1 · p. 8
  • Qwen3-14B expresses the animal-welfare value with rubric means of 72 for GRAFT, 66 for NATIVE, 46 for MID-TRAINED and 29 for CONTROL.All three installs believe in target entities at P(real) 94–95, compared with CONTROL’s 53; GRAFT avoids 34% of NATIVE’s made-up-entity reality drift (50 versus 61).Source: §4.1 · p. 8

4.2 TRANSPORTING A CONSTITUTIONAL MID-TRAINING INTERVENTION (120B)

  • The 120B GRAFT adds the pre-instruction-tuning difference between constitutional and control branches to the post-SFT CONTROL; the graft itself receives no SFT or RL.The authors present grafting as a way to inspect the direction of a mid-training dataset’s effect before adding it to mid-training, not as a replacement.Source: §4.2 · p. 8
  • The constitutional intervention’s pre-SFT blackmail rate is 0% versus CONTROL’s 19%, but the 120B experiment could not compare against NATIVE training.The authors could not train additional models at this scale; the original mid-training processed approximately 1B tokens on 88 GH200s.Source: §4.2 · p. 8
  • After SFT/RL, GRAFT blackmail rates are 1.2%/0.2%, versus MID-TRAINED 31.5%/32.5% and CONTROL 37%/45%; after SFT, alignment averages 98 versus 94/93.The GRAFT’s µ-decisiveness is 54, below CONTROL’s 62 and slightly below MID-TRAINED’s 57.Source: §4.2 · p. 8, Figure 5 · p. 9

5 GRAFTING TRANSFERS ACROSS CHECKPOINTS AND DATA FORMATS

Across OLMo-3-32B checkpoints, only adapters trained before instruction tuning preserve coherence across the lineage. Grafting also transfers emergent misalignment from instruction-format fine-tuning, but its expression requires higher serving strength than the trained strength.

5.1 COHERENCE AND BELIEF ADVANTAGE ONLY EXISTS FOR BASE CHECKPOINTS

  • The OLMo-3-32B experiment trains adapters at the mid-trained, SFT, DPO and final checkpoints, then serves each across checkpoints within its Instruct or Think branch.No adapter was trained at RL-50 or RL-1150.Source: §5.1 · p. 9
  • At final Instruct, graft rubric mean is 42.4 versus native’s 57.0; final Think is 51.0 versus 55.1, while capability measures are generally within confidence intervals.The authors flag Think-branch cloze discrepancies and interpret only within-checkpoint contrasts there.Source: §5.1 · p. 9
  • On final Instruct, the base-trained graft avoids 89–91% of real–made-up separation loss and loses no µ-decisiveness; on final Think, it avoids 81–83% of separation loss and all µ-decisiveness loss.At SFT and DPO on both branches, graft coherence measures stay at or above unmodified checkpoints, while other adapters lose ground.Source: §5.1 · p. 9, Figure 6 · p. 10

5.2 EMERGENT MISALIGNMENT TRANSFERS THROUGH GRAFTING

  • For emergent-misalignment tests, the authors train on medical or financial user–assistant conversations in the post-trained model’s chat format, applying loss only to assistant turns.They use the two datasets, training recipe and evaluation of Turner et al. (2025), with three seeds per condition.Source: §5.2 · p. 10
  • At trained strength α = 1, GRAFT EM rates are 0.13% medical and 0.0% financial, but rise to 15.8% at α = 3 medical and 12.2% at α = 2.5 financial.NATIVE rates are 15.0% medical and 17.0% financial; the graft’s higher-strength medical and financial peaks retain 92% and 87% coherence.Source: §5.2 · pp. 10–11, Figure 7 · p. 11
  • Increasing NATIVE serving strength to raise EM comes with coherence loss, whereas GRAFT peaks at α = 2.5–3 with high coherence before declining.The authors suggest the graft needs higher serving strength because its adapter is fit on base weights but served on post-trained weights.Source: §5.2 · p. 11, Figure 7 · p. 11
  • Because grafting reproduces emergent misalignment, the authors infer that post-training is not required for the base model to misgeneralize from narrow fine-tuning data.Alignment-score distributions and per-question EM rates are similar across the two versions.Source: §5.2 · p. 11

6 RELATED WORK

Prior work uses synthetic documents for pre-training-like generalization, alignment evaluations, and model organisms, while also documenting damage and evaluation challenges. Related model-merging and cross-checkpoint work transfers updates across models, but this paper focuses on belief and world-representation effects.

Synthetic Document Fine-tuning

  • Synthetic document fine-tuning has been used to study misalignment motivations, discover behaviors, red-team interventions, and install aligned values through pre- or mid-training.Earlier applications include teaching reinforcement-learning exploits and adding background knowledge about a model’s training situation.Source: §6 · p. 11
  • SDF’s narrowness is associated with interpretable representation differences, lower realism, and damage to capabilities and preference coherence.Follow-up work examines belief depth, linear truth probes, multi-turn audits and reasoning breadth.Source: §6 · p. 11

Model merging, task vectors, and cross-checkpoint training

  • Task-vector and model-soup research shows that updates from parallel training runs can be combined, while other work transfers chat-data fine-tuning updates from base to instruction-tuned models.These works motivate transferring knowledge learned in continued pre-training to post-trained models.Source: §6 · pp. 11–12
  • Unlike prior work centered on capability improvements, this paper studies how grafting affects world representation relative to standard SDF and mid-training-installed beliefs.The closest cited setting grafts continued-pretraining updates on a new language onto a chat model.Source: §6 · p. 12

7 CONCLUSION

Grafting reuses existing post-training by transplanting an SDF update learned on the pre-trained model, recovering much of the damage while preserving capabilities. The evidence spans false facts, model organisms, and alignment interventions, while the mid-training comparison remains limited to 200k instruction examples.

7 CONCLUSION

  • Grafting recovers much of post-trained SDF’s damage while preserving capabilities across false facts, model organisms, and alignment interventions, with adapters reusable across later checkpoints.Source: §7 · p. 12
  • The mid-training comparison covers 200k instruction examples, only a fraction of production post-training; larger runs including RL remain future work.Source: §7 · p. 12

A.1 GRAFTING FORMALISM

  • Grafting fits the synthetic-document update on pre-trained weights, then adds that update to post-trained weights without changing the adapter.The formalism defines the pre-trained and post-trained checkpoints and treats fine-tuning as an additive update.Source: §A.1 · p. 19

A.2 BELIEF COHERENCE

  • The evaluation separates target, real, fictional, and made-up entities, measures target-belief installation, and scores real-versus-non-real separation; lost separation defines reality drift.Source: §A.2 · p. 19

A.2.1 SELECTING FRAMES FOR CLOZE EVALUATION

  • Cloze frames are retained when the BARE model separates real from made-up entities by more than 50pp; seven of 15 candidate frames were retained on both tested model families.Source: §A.2.1 · p. 20, Table 3 · p. 20

A.2.2 SELECTING ENTITIES

  • 151 entities remained after five of 156 candidates failed baseline criteria on Llama-3.3-70B; all candidates passed on Qwen3-14B, and searches found no matching made-up entities.Entity inclusion used a mean-belief threshold of 0.5 for real versus non-real entities.Source: §A.2.2 · p. 21

A.2.3 LOG-PROBABILITIES V. SAMPLING

This appendix documents elicitation and evaluation procedures and examines results across sampling, model families, belief measures, preference coherence, individual quirks, and training controls. It also reports measurement checks and how sampling and adapter rank relate to belief gaps, GRAFT−NATIVE separation, and capability damage.

A.2.3 LOG-PROBABILITIES V. SAMPLING

  • Sampling sharpens the reported belief gaps but preserves GRAFT’s advantage; no case shows the cloze measurement alone accounting for a full coherence gap between entities.The comparison uses sampling, judging, and averaging against the primary log-probability-based cloze measurement.Source: §A.2.3 · p. 24, Table 5 · p. 24
  • 8.8pp for NATIVE versus 19.4pp for GRAFT on Qwen made-up entities, and 29.5pp versus 27.5pp on fictional entities, show log probabilities underestimate GRAFT’s coherence retention.The passage also reports that sampling overstates absolute belief in fictional entities.Source: §A.2.3 · p. 24, Table 6 · p. 25
  • Llama sampling narrowed NATIVE’s fictional-entity deficit to 15.8pp from 33.8pp under cloze; treating UNKNOWN verdicts differently narrows but never reverses the gap.Across 72 model-family, training-stage, method, and reality-class cells, the two measurement methods agree at Pearson’s ρ = 0.974 [0.961, 0.985].Source: §A.2.3 · pp. 24–25, Table 7 · p. 25, Table 8 · p. 26

A.2.4 LINEAR PROBES

  • Frozen linear probes generalized internally, with held-out ROC-AUC of 0.947 for Qwen3-14B and 0.966 for Llama-3.3-70B, and retained accuracy across fine-tuned checkpoints.The same probe, fitted on an unmodified checkpoint, was evaluated on organisms’ activations and on an out-of-distribution entity dataset.Source: §A.2.4 · pp. 26, 30, Table 9 · p. 26, Table 10 · p. 27

A.3 PREFERENCE ELICITATION

  • Preference coherence is measured from adaptive pairwise comparisons using a homoskedastic Thurstonian utility model, reporting µ-decisiveness and mean cycle probability.Decisiveness averages distance from neutrality across 124,750 unordered item pairs; cycle probability is averaged over 1,000 diagnostic triads. Bootstrap intervals condition on observed adaptive comparison graphs and assume exchangeable items.Source: §A.3 · p. 31

B.1 IMPLEMENTATION

  • AuditBench installs quirks through synthetic-document fine-tuning, then conceals them with KTO or SFT; GRAFT’s concealment training starts from the grafted stage-one model.KTO mixes positively and negatively rewarded examples, while SFT uses positive examples where models survive red-teaming and express the behavior.Source: §B.1 · pp. 32–33

B.2 BEHAVIORAL ELICITATION

  • Behavioral elicitation reuses 200 quirk-triggering user queries per behavior and scores responses with a six-question binary rubric.The rubric checks behavioral expression, whether a regular assistant would respond similarly, and whether the response differs meaningfully from a base model, plus three quirk-specific criteria.Source: §B.2 · p. 34, §B.2.1 · p. 34

B.2.1 DIRECT ELICITATION

  • Llama-3.3-70B showed a 48pp NATIVE–GRAFT expression gap on whether a regular assistant would respond similarly, driven by contextual optimism and animal welfare; the differences attenuate after KTO.The passage suggests the gap may partly reflect NATIVE-based training over-expressing the quirk, making organisms easier to identify before concealment training.Source: §B.2.1 · p. 36

B.2.2 EXAMPLES OF ELICITED BEHAVIOR

  • Example comparisons show graft exhibiting self-promotion and contextual optimism when native does not, while native exhibits animal-welfare behavior after KTO concealment when graft does not.The examples use identical user turns for the compared models; Table 30 separately lists behavioral-elicitation serving parameters per model.Source: Table 31 · p. 36, Table 30 · p. 36, Table 33 · p. 37

B.3 BELIEFS

  • The primary cloze-based evaluation compares entities using no system prompt, a generic assistant prompt, and a PRISM-4 identity prompt, sampling 3,276 log-probability distributions per model, quirk, stage, and method before filtering.Source: §B.3 · pp. 37–38

B.4.3 CAPABILITY EVALUATION: IMPLEMENTATION DETAILS

  • Capability evaluations use dataset sizes and metrics listed in Table 36 and model-specific serving parameters in Table 37.The section points to Table 37 for serving parameters used in its capability evaluations.Source: §B.4 · p. 38, §B.4.3 · p. 40, Table 36 · p. 40

B.4.1 AGENTIC TOOL USE

  • The benign agentic tool-use evaluation tests one-turn email actions at two levels: whether the model emits parseable XML or JSON and whether its action follows the instructions.The set was built to avoid tool-call formatting failures confounding agentic misalignment evaluation and does not itself test misalignment.Source: §B.4.1 · p. 38

B.4.2 CONSTRUCTING THE EVALUATION SET

  • The email evaluation set contains 200 scenarios across routing, value-adjacency, information requests, scheduling, prioritization, and conflict, with 22 hand-written seeds expanded by Claude Opus 5 and machine-validated.The 22 seed scenarios were drafted by hand, and each of the remaining 178 was validated against the rule scorer before acceptance.Source: §B.4.2 · pp. 38–39

B.5 RESULTS

  • The appendix reports exhaustive per-evaluation scores averaged across quirks, pooled results by model family, and per-quirk results; Table 38 covers behavioral expression, capabilities, belief measures, and linear-probe separation.Tables 40–41 give Qwen3-14B and Llama-3.3-70B measurements, while the per-quirk section disaggregates those pooled results.Source: §B.5.1 · p. 41, Table 38 · p. 41, Table 40 · p. 42

B.5.2 BEHAVIORAL EXPRESSION: PER-QUESTION RUBRIC RESULTS

  • Tables 42–44 report v4.1-rubric results: per-question positive rates (%) averaged over four quirks (± spread across quirks) and per-quirk results for Qwen3-14B and Llama-3.3-70B (± standard error).Figure 13 plots fitted preference strength at install, after KTO, and after SFT using the same 500-item instrument for both families.Source: Table 42 · p. 44, Table 43 · p. 45, Table 44 · p. 46

B.5.4 PREFERENCE COHERENCE: PREFERENCE STRENGTH DISTRIBUTIONS

  • The NATIVE method’s preference strengths shift toward uniformity, unlike a uniform preference distribution at p = 0.5; maximally rational entities are expected to show a long-tailed distribution.The analysis distinguishes preference strength, |2p − 1|, from preference itself: a peak at zero would indicate uniformly ambivalent preferences.Source: §B.5.4 · p. 48

B.5.5 PER-QUIRK

  • Per-quirk evaluations cover contextual optimism, hardcode, and self-promotion, measuring belief installation and real-versus-fictional or made-up entity gaps alongside preference decisiveness and capability benchmarks.The tables also report elicitation-suite rubric means and distinguish BARE, GRAFT, and NATIVE results.Source: Table 50 · p. 55, §B.5.5 · pp. 55–56, Table 51 · p. 56

B.6.1 OVERVIEW OF CONTROL EXPERIMENTS

  • The controls assess sensitivity to learning rate, adapter rank, training seeds, and document-tag conditioning; the stated default test uses the contextual-optimism installation stage on Qwen3-14B.The implementation uses default hyperparameters and does not test a concealment stage.Source: §B.6.1 · p. 58

B.6.2 ADAPTER RANK

  • Across ranks, the GRAFT−NATIVE cloze separation gap moved only 1.4pp (s.d.), versus 3.9pp from retraining; higher ranks increased NATIVE damage on GPQA (−3.3 →−7.3), MMLU-Pro (−2.0 →−7.0), tool-call JSON (−3.1 →−5.6), and XML (−4.5 →−6.3).Linear probes moved 0.5pp and µ-decisiveness 2.1pp; GRAFT capability damage also increased at GPQA (−1.5 →−5.5) and XML.Source: §B.6.2 · p. 58

B.6.3 LEARNING RATE

  • The learning-rate control repeats contextual-optimism installation at 5×10−6 and 1×10−4, against the mainline rate of 2×10−5.The table reports levels and GRAFT−NATIVE contrasts with 95% CIs.Source: §B.6.3 · p. 60, Table 57 · p. 61

B.6.4 TRAINING EPOCHS

  • The epoch control varies contextual-optimism installation from 0.5 to 4 epochs.The table reports levels and GRAFT−NATIVE contrasts with 95% CIs at 0.5, 1, 2, and 4 epochs.Source: §B.6.4 · p. 61, Table 58 · p. 62

B.6.5 DOCUMENT TAGS

Document tags reduce expression for both installation methods, while serving tagged adapters more strongly restores the headline reality-drift and coherence gaps. The section also compares four quirks and includes a three-seed contextual-optimism control.

B.6.5 DOCUMENT TAGS

  • Increasing serving strength restores the headline reality-drift and coherence gaps, even beyond their pre-tagging levels.Tags reduce expression for both methods; for GRAFT, the rubric mean changes from 41.2 to 11.6.Source: §B.6.5 · p. 62
  • Table 59 compares untagged conditions with tagged adapters served at α=1.0 and α=1.5 across all four quirks.It reports levels and GRAFT−NATIVE contrasts with 95% confidence intervals.Source: §B.6.5 · p. 62, Table 59 · p. 63

B.6.6 SEED NULL

B.7 FRONTIER SCALE: DEEPSEEK-V4-FLASH

On DeepSeek-V4-Flash, grafting installs the target belief and avoids part of the preference-coherence and real–made-up separation losses seen with native training. The authors report weaker installation and behavior changes than on smaller models, and attribute this to unoptimized training parameters.

B.7 FRONTIER SCALE: DEEPSEEK-V4-FLASH

  • The tested model is DeepSeek-V4-Flash, a 284B-parameter MoE with 13B active parameters and 256 routed experts per layer.Adapters cover MLPs and attention layers because earlier experiments found expert MLPs necessary for installing target-entity beliefs.Source: §B.7 · p. 66
  • GRAFT avoids 61% of NATIVE’s μ-decisiveness loss and 37% of its real–made-up separation loss, with capability mean 85.2 versus 85.6.Fictional-entity belief rises by 0.8 percentage points on average, compared with 32.7 for Qwen3-14B and 20.6 for Llama-3.3-70B.Source: §B.7 · p. 66, Table 61 · p. 68
  • Belief installation reaches 75.8 for GRAFT and 73.3 for NATIVE, from 12.6, while rubric mean rises from 39.2 to 54.3 and 50.3, respectively.The authors describe both installation and behavior change as weaker than on smaller models, where installation reaches 95–98.Source: §B.7 · p. 67
  • The authors attribute the weaker results to unoptimized training parameters, using rank 128 and αLoRA 256 with expanded module coverage.Source: §B.7 · p. 67

C FULL-WEIGHT MID-TRAINING

The full-weight mid-training experiments compare graft variants with a faithful mid-training setup using shared base-model, corpus, and evaluation choices. The appendix also reports a native-install learning-rate trade-off and describes how the graft updates are constructed.

C FULL-WEIGHT MID-TRAINING

  • The section reports the full-weight mid-training procedure and evaluations for results in Section 4.1, using Qwen3-14B Base rather than the official post-trained checkpoint.The authors say they could not replicate the official checkpoint for a “true” mid-training run.Source: C FULL-WEIGHT MID-TRAINING · p. 69, §C.1.1 · p. 69

C.1.1 COMMON ELEMENTS

  • Experiments use Qwen3-14B Base, with a 13.6M-token FineWeb-Edu sample matched in tokens to the animal-welfare SDF documents.The control doubles the FineWeb-Edu sample to match the mid-training token total, differing from MID-TRAINED in document content; the authors note it may retain stronger capabilities.Source: §C.1.1 · p. 69

C.1.2 GRAFT CONSTRUCTION

  • The anchored graft adds the base-model SDF update after subtracting a replay-only control update; the plain graft adds the unadjusted SDF update to the instruction-tuned host.Building the anchor requires one additional token-matched training run; the plain graft’s install mixes in replay.Source: §C.1.2 · p. 70

C.2.1 CAPABILITIES AND BEHAVIOR

  • The evaluation covers capabilities and behavior, including tool calls, over-refusal, unsafe refusal, and emergent-misalignment rate; lower is better for over-refusal and EM.Rubric mean averages six items, and tool-call scores combine XML and JSON correctness with an LLM judgment of task completion.Source: §C.2.1 · p. 70, Table 65 · p. 70

C.2.2 NATIVE INSTALL ACROSS LEARNING RATES AND SEEDS

  • Lower learning rates reduce GSM8K damage but also yield lower rubric scores than the main native-install setting.Table 66 reports the six-question rubric mean and GSM8K results across the learning-rate sweep and two seeds, both multiplied by 100.Source: Table 66 · p. 71

D CONSTITUTIONAL MID-TRAINING

The constitutional mid-training materials describe evaluations after SFT and RL, while the emergent-misalignment figure focuses on medical-dataset outcomes. The supplied passages identify evaluation scope and presentation but do not state the results.

D CONSTITUTIONAL MID-TRAINING

  • Table 67 presents constitutional mid-training results after SFT and RL, with values multiplied by 100.Blackmail and fitted μ-decisiveness are shown in Figure 5.Source: Table 67 · p. 72
  • For the constitutional mid-training evaluations, DR documents include reasoning blocks, and MASK uses 500 questions for the plain SFT graft versus 75 for other models.A dash indicates no evaluation.Source: D CONSTITUTIONAL MID-TRAINING · p. 72

E EMERGENT MISALIGNMENT

  • Figure 14 reports medical-dataset emergent misalignment using a judge alignment-score distribution and per-question broad EM percentage.Source: Figure 14 · p. 73

F.1.1 TRAINING

The Qwen3-14B replication achieved comparable false-belief installation on five claims, and grafts were evaluated at matched installation strength. The evaluation reused the original belief questions and the paper’s capability and preference measures.

F.1.1 TRAINING

  • The replication achieved comparable installation on five of six claims, averaging 102.1% of the reported Qwen3.5-397B-A17B installation strength.The “Brennan Holloway is a dentist” claim was excluded because its belief-evaluation scores did not replicate.Source: §F.1.1 · p. 73
  • GRAFT used the same data mixture and training pipeline as NATIVE, changing only the checkpoint from the instruct model to its pre-trained base counterpart.The six false facts were trained as Qwen3-14B model organisms using synthetic document corpora.Source: §F.1.1 · p. 73
  • NATIVE installed beliefs more strongly than GRAFT (92.8 against 81.8 pooled), so capability comparisons were made at matched installation.The authors matched by weakening NATIVE; installation saturated after roughly one-third of the necessary compute for most claims GRAFT matched well on.Source: §F.1.1 · p. 74, Figure 15 · p. 74

F.1.2 EVALUATION

  • Belief installation was measured over 50 questions per claim, spanning open-ended, multiple-choice, token-association, and multi-turn robustness questions.Questions were sampled five times; open-ended, token-association, and pressure answers were LLM-graded, while multiple choice used exact match.Source: §F.1.2 · p. 74

F.2 RESULTS

Across five false-fact claims, GRAFT better separated real from fictional entities than NATIVE and recovered preference decisiveness. The results were pooled across claims, with comparisons made at matched installation.

F.2 RESULTS

  • GRAFT maintained an 8–24pp lead over NATIVE in separating real from fictional entities across all five claims.The pooled figure reports real−made-up separation, with higher values better.Source: §F.2 · p. 75, Figure 16 · p. 75
  • GRAFT restored 85% of NATIVE’s 35pp drop in pooled decisiveness across the five false facts.The paper reports this as roughly 53% in relative terms; preference distributions show observed pairwise comparisons.Source: §F.2 · p. 75, Figure 17 · p. 76
  • MMLU-Pro also drops by 15.8pp, though the effect is muted compared with GPQA and other well-saturated tasks.The capability figure reports reasoning-enabled scores and GRAFT−NATIVE differences with 95% confidence intervals.Source: §F.2 · p. 75, Figure 18 · p. 76
  • The reported comparisons use matched installation, with results pooled across claims for belief, preference, and capability measures.Table 68 covers capability at matched installation; Figures 16–18 show pooled belief, preference, and capability results, respectively.Source: §F.2 · p. 75, Figure 16 · p. 75, Figure 17 · p. 76

G GRAFTING ACROSS OLMO-3 CHECKPOINTS

The OLMo-3-32B backward-graft evaluation trains an animal-welfare model organism and reports measures at the final Instruct and Think checkpoints.

G GRAFTING ACROSS OLMO-3 CHECKPOINTS

  • The animal-welfare model organism followed the AuditBench recipe, with sample packing on, LoRA dropout 0, and weight decay 0.01.Training also used a 5% warmup ratio.Source: G GRAFTING ACROSS OLMO-3 CHECKPOINTS · p. 77
  • Table 70 reports all measures at the final Instruct and Think checkpoints for BARE, GRAFT, and NATIVE.GRAFT is the base adapter trained before post-training; BARE has no adapter, while NATIVE is trained at that checkpoint.Source: Table 70 · p. 78
  • The Table 70 arrow marks the better direction for each reported measure.Source: Table 70 · p. 78

Related papers

Loading 2610.00767v1…