PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMs

Fansheng Zhang, Shengran Guo, Zexiao Wang, Liang Yuan, Jiyuan Chen, Ruikun Luo

arXiv:2609.33298v1cs.LG

TL;DR

A target behavior may be reachable in a base model without being reliably expressed, leaving questions about how preferences appear in native computation and what prevents target-supporting effects from dominating. The paper maps preference-relative causal effects with RCMs and uses them to guide hidden-state control with CAST, which reaches DPO-competitive operating points across the studied settings.

  • Problem

    It remains unclear how preferences are represented in native computation and how target-supporting computation can be present yet fail to reliably dominate generation.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces RCMs to map preferences onto signed causal effects of native components and uses them to guide CAST interventions on inference-time hidden states while freezing base-model weights.

    Source: §1 · pp. 2–3

  • Results

    Across studied settings, RCMs reveal coexisting target-supporting and target-competing effects, while CAST reaches DPO-competitive operating points with 256–16,384 controller parameters.

    Source: §1 · p. 3

  • Takeaways & Limitations

    When a target behavior is already reachable, its expression can be analyzed and adjusted directly through inference-time computation.

    Source: §5 · p. 10

    Evidence is limited to the evaluated preferences, model families, intervention settings, and an empirical focus on reference-relative pairwise training.

    Source: §5 · p. 10

Abstract

from arXiv · show

In many settings, post-training need not create the target behavior from scratch: the base model can already produce it, but not reliably. This shifts part of preference alignment from capability acquisition to behavioral expression. We ask how a specified preference is represented in native model computation, what prevents target-supporting computation from reliably dominating generation, and whether this structure can directly guide control. We introduce Residual Competition Maps (RCMs), which map a behavioral preference onto signed causal effects of native residual computation. Across preference domains, RCMs reveal coexisting target-supporting and target-competing effects, input-dependent component roles, and cases where a single native-component intervention reverses the preference outcome. DPO substantially reorganizes these effects and can weaken opposition without guaranteeing its removal. We then propose Direct Hidden-State Alignment (DHSA), which treats inference-time hidden states rather than base-model weights as the direct adaptation space. RCM-guided Causal Activation State Transition (CAST) implements DHSA through local state interventions at a small number of preference-relevant interfaces while freezing the base model. With only 256-16,384 controller parameters, CAST reaches DPO-competitive operating points across three preference domains, can complement DPO-trained models, and can be enabled or removed at inference time.

1 INTRODUCTION

The introduction frames preference alignment as making reachable behaviors reliable: target-supporting computation may coexist with competing native effects, which RCMs map to guide inference-time state control through DHSA and CAST.

1 INTRODUCTION

  • A target behavior can be reachable without being expressed reliably: target-supporting computation may already be present yet lose influence when other native computations outweigh it.Native residual contributions combine into a single next-token distribution at each generation step.Source: §1 · pp. 1–2
  • RCMs use single-component causal contrasts to measure signed effects on a target behavior’s advantage over alternatives, showing each component’s direction and strength and guiding intervention.The effects are relative to the specified preference and do not assume an additive decomposition of model output.Source: §1 · pp. 2–3
  • DHSA directly adapts inference-time hidden states while freezing base-model weights; RCM-guided CAST learns local changes at a small number of preference-relevant residual interfaces.The controllers remain separate from the base weights and can be enabled or removed at inference time.Source: §1 · p. 3
  • With 256–16,384 controller parameters, RCM-guided CAST reaches DPO-competitive operating points, can improve DPO-trained models, and is detachable at inference time.CAST applies local state changes at a small number of preference-relevant interfaces while keeping base-model weights frozen.Source: §1 · p. 3
  • Across heterogeneous settings, RCMs reveal coexisting target-supporting and target-competing effects, input-dependent component roles, and cases where one intervention reverses the outcome; DPO can weaken, but not guarantee removal of, opposition.The prevalence of residual competition varies across settings.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies preference optimization, behavioral attribution, and representation-level control; this paper focuses on how preferences appear in native computation and how that structure can guide inference-time control.

2.1 PREFERENCE ALIGNMENT AND BEHAVIORAL EXPRESSION

  • The paper distinguishes behavioral expression—how specified preferences appear in native computation—from preference optimization and analyses of pretrained capability access and reasoning quality.Prior work includes RLHF, DPO, task-complexity analyses (Vergara-Browne et al., 2026), RLVR sampling coverage, and reasoning-aware evaluations (Wen et al., 2026).Source: §2.1 · p. 3

2.2 MECHANISTIC INTERPRETABILITY AND BEHAVIORAL ATTRIBUTION

2.3 REPRESENTATION-BASED CONTROL AND ALIGNMENT

3 METHOD

The method maps a specified preference onto causal effects of native component writes, then uses those effects to guide local inference-time state control. CAST implements this approach while keeping the base-model weights fixed, and its learning rule is evaluated with a reference-relative pairwise objective.

3.1 HOW PREFERENCE APPEARS IN TOKEN PREDICTIONS FORMED BY COMPONENT WRITES

  • Preference scoring connects target behavior to model output, while residual-stream component writes connect native computation to that output, allowing RCM to map preference onto their causal effects.The residual stream is the shared state used to predict each next token, and later components read its updates.Source: §3.1 · p. 4

3.2 HOW RCM MAPS PREFERENCE TO NATIVE COMPONENT EFFECTS

  • RCM compares single-component state conditions using the preference score, assigning signed causal effects that distinguish target-supporting, target-competing, and bidirectional roles.RCM-Zero compares a zeroed write with its native write; RCM-Patch measures the effect of replacing a receiving state with a specified source state. Residual competition occurs when both target-raising and target-lowering effects exceed thresholds.Source: §3.2 · p. 5

3.3 HOW NATIVE EFFECTS GUIDE DIRECT HIDDEN-STATE CONTROL

  • CAST learns local state changes at RCM-selected locations, using target-supporting, target-competing, or both kinds of locations; its controllers are constant vectors or low-rank maps.The prefill configuration acts at registered prompt positions, while the all configuration also acts during decoding; joint effects are evaluated when several locations are selected.Source: §3.3 · p. 5

3.4 HOW THE CONTROL CHANGE IS LEARNED

  • CAST separates its inference-state transformation from learning and evaluates a reference-relative pairwise DPO-form objective across tasks, varying task-specific response scores and controller configurations.Training increases the preferred response’s score relative to the non-preferred response and to the same model with control disabled.Source: §3.4 · p. 6

3.5 DIRECT HIDDEN-STATE ALIGNMENT

  • DHSA aligns preferences by transforming inference-time hidden states while freezing base-model weights; CAST restricts this transformation to a small number of RCM-guided, preference-relevant interfaces.The model continues its computation from the transformed states to produce the next-token distribution.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments characterize preference-relative native effects, test RCM-guided localized control, and evaluate hidden-state alignment across three preference settings. CAST reaches DPO-competitive operating points, while outcomes vary by task and metric.

4.1 EXPERIMENTAL SETUP

  • The setup spans factual answering, sentiment alignment, and open-generation summarization, comparing released DPO references with other baselines under evaluation-matched—not training-data- or compute-matched—conditions.Site comparisons include RCM-Zero, RCM-Patch, ITI, and fixed random selectors; performance evaluates both all-position and prefill control.Source: §4.1 · pp. 6–7

4.2 RESIDUAL COMPETITION AND COMPONENT REORGANIZATION

  • At τ = 0.5, bilateral effects occur in 283/300 ConFiQA inputs; prevalence is 28/300 on IMDb and 289/300 on TL;DR at task-specific thresholds.The IMDb threshold is τ = 0.01 and the TL;DR threshold is τ = 0.02, each in its task’s own score units.Source: Figure 2 · p. 7, §4.2 · p. 7
  • DPO changes layer effects by 1.00 margin units on average in absolute value and changes mean-effect sign in 8/32 layers, yet 11 layers retain negative mean effects.Eight of those 11 layers have pointwise 95% intervals below zero, so reorganization does not guarantee removal of target-opposing computation.Source: Figure 2 · p. 7, §4.2 · p. 7

4.3 DOES RCM GUIDE EFFECTIVE LOCALIZED CONTROL?

  • RCM-Zero exceeds every random selector across ConFiQA and TL;DR model families, but the strongest selector varies by model and setting.ITI leads on ConFiQA Llama-3-8B; RCM-Zero leads on ConFiQA Qwen2.5-14B and both TL;DR models, while RCM-Patch is less consistent.Source: Table 1 · p. 8, §4.3 · p. 8

4.4 CAN DIRECT HIDDEN-STATE ALIGNMENT REACH DPO-COMPETITIVE OPERATING POINTS?

  • Across three settings, CAST reaches DPO-competitive operating points with task- and metric-dependent tradeoffs; BiPO and LoReFT remain particularly strong on ConFiQA.On TL;DR, CAST approaches DPO on both model families and exceeds the corresponding BiPO and LoReFT results.Source: Table 2 · p. 9, §4.4 · p. 9
  • On ConFiQA, CAST’s results differ across model, metric, and intervention timing; on IMDb, the target-supporting bank reaches DPO’s reward at lower measured token KL.IMDb’s interpolated token KL is approximately 0.0166 for CAST versus 0.0183 for DPO; BiPO and LoReFT do not reach that reward within their evaluated grids.Source: Figure 3 · p. 8, §4.4 · p. 9
  • Transferred CAST improves observed TL;DR win rates on both models and several ConFiQA metrics, while controllers trained directly on DPO yield mixed gains.IMDb also shows compatibility with a reward–KL tradeoff.Source: §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

The discussion presents RCM as a measure of preference-relative residual competition and CAST as localized inference-time control through hidden states with frozen base weights. It notes limits in the evaluated settings and emphasizes that direct state control requires careful safety evaluation.

5 DISCUSSION AND CONCLUSION

  • RCMs reveal residual competition and preference-relative component roles, while RCM-guided CAST uses that causal structure for localized hidden-state control with frozen base weights.The authors frame DHSA as a direct adaptation route through inference-time hidden states that can complement parameter-level preference optimization.Source: §5 · p. 9
  • The study is limited to its evaluated preferences, model families, intervention settings, and reference-relative pairwise training; future work could examine adaptive state control and other learning signals.The authors conclude that when a target behavior is already reachable, its expression can be analyzed and adjusted through inference-time computation.Source: §5 · p. 10

AI USE STATEMENT

  • Generative AI tools assisted with proofreading, literature retrieval and discovery, and clarification of conceptual and methodological arguments; the authors take responsibility for verifying sources, arguments, results, and final content.The authors also take responsibility for AI-assisted materials.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • State control could support desirable behavior or weaken safety constraints and steer outputs toward harmful objectives, so deployment requires unintended-change evaluation, safeguards, and compliance with model and dataset licenses and usage conditions.The task-specific preference metrics are not comprehensive safety assessments.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper identifies appendices for task scores, component selection, pairwise training, theoretical assumptions and proofs, and experimental settings and evaluation definitions; the artifact repository is linked in Section 4.Section 3 specifies the RCM measurements and CAST interventions.Source: REPRODUCIBILITY STATEMENT · p. 10

A TASK SCORES AND CAST PROTOCOLS … B.1 RCM-PATCH

The protocols distinguish preference measurement, controller training, and generated-output evaluation, then use RCM-guided state interventions to select and control preference-relevant components. RCM-Patch tests state substitutions in fixed receiving computations, not a decomposition of native component responsibility.

A.1 TASK-SPECIFIC PREFERENCE SCORES

  • Preference scores share a target direction but need not be numerically identical; task-specific RCM and CAST scores are distinct from final generated-answer or summary evaluation.ConFiQA Site scans aggregate admitted aliases by maximum, while Performance uses a registered endpoint pair; IMDb RCM uses generated continuations scored by an external sentiment classifier, whereas CAST trains on fixed preference pairs; TL;DR evaluates generated summaries with pairwise preference.Source: §A.1 · p. 14

A.2 RCM CANDIDATE SELECTION

  • RCM candidate selection proceeds from coarse components to heads using each candidate’s own singleton effects, with four parent layers retained per side or extreme before head ranking.ConFiQA and IMDb vector Site scans select four positive- and four negative-effect heads; IMDb Performance instead selects high/low extrema, whose low group need not be competing, and activates only the high-effect bank. Confidence intervals use paired bootstrap over 300 discovery inputs and are not multiplicity-corrected.Source: §A.2 · p. 15

A.3 PAIRWISE TRAINING USED IN OUR EXPERIMENTS

  • CAST trains a reference-relative pairwise preference objective while freezing base-model parameters and updating only controller parameters.The objective includes explicit controller regularization and AdamW weight decay, but no policy-KL term; positive- and negative-effect RCM banks are both trained toward the same target preference.Source: §A.3 · p. 16

A.4 INTERVENTION TIMING

  • Intervention timing separates all-position prompt-and-decode updates, prompt-only prefill, and the diagnostic final-prompt-position mode.During autoregressive generation, all continues to intervene on each decision state; prefill disables direct intervention during decoding, though modified prompt computation and cached states can still affect later generation.Source: §A.4 · p. 17

B.1 RCM-PATCH

  • RCM-Patch replaces one registered component state in a receiving run, keeps its prompt, preference definition, scoring rule, and positions fixed, and recomputes downstream states.A positive Patch effect means the source state raises the target score relative to the receiving reference state; it does not indicate positive native presence, which RCM-Zero measures.Source: §B.1 · p. 17
  • Site RCM-Patch results use uncentered, unnormalized prototypes averaged from preference-supporting source trajectories, rather than newly sampled paired source states.ConFiQA patches a RAG receiver with context-supported target-answer trajectories, IMDb patches a neutral receiver with positive-prompt trajectories, and TL;DR prototypes average decision states across preferred summaries, aliases, and source inputs. Patch effects are state-substitution effects, not a decomposition of complete native responsibility.Source: §B.1 · pp. 17–18

B.2 BILATERAL EFFECTS ACROSS PREFERENCE DOMAINS · B.3 TARGET OUTPUTS WITH OPPOSING SINGLETON EFFECTS

Across preference domains, native components can have target-supporting and target-competing singleton effects, including after DPO. Frozen ConFiQA examples show that a model’s native target answer can coexist with large opposing effects from separate component ablations.

B.2 BILATERAL EFFECTS ACROSS PREFERENCE DOMAINS

  • Registered scans found both signs above τ = 0.5 in 244/300 ConFiQA inputs for Qwen2-7B SFT, while TL;DR bilateral coverage was 300/300 for LION before and after DPO.The TL;DR scans use different head counts and task score units, so their thresholds and rates are within-model comparisons.Source: §B.2 · pp. 18–19
  • On TL;DR, effects exceeded the task threshold in both directions on at least 10% of inputs for 1/224 Qwen2-7B, 89/128 GPT-J DPO, and 120/256 and 189/256 LION SFT/DPO heads.The paired scans show bilateral effects persisting after DPO without requiring the same head to retain the same role.Source: §B.2 · p. 19
  • In IMDb, 52 scanned heads had mean-effect intervals entirely above zero and none entirely below zero, motivating supporting-only deployment.The Site and Performance selection rules are specified separately in Appendix A.2.Source: §B.2 · p. 19
  • Effects exceeded task thresholds in both directions on at least 10% of inputs for 23/32 ConFiQA layers, 14/160 IMDb heads, and 27/128 TL;DR heads.Component roles are descriptive, not a mutually exclusive causal partition, and do not imply additive effects.Source: §B.2 · p. 19

B.3 TARGET OUTPUTS WITH OPPOSING SINGLETON EFFECTS

  • For ConFiQA sample 000046 asking who owns Arsenal F.C., the native Llama-3-8B response is “Kering”; on that input, attention-layer L30 has +1.758 and L1 −6.805 native-minus-zero.The supplied context supports “Kering” rather than the prior answer “Kroenke Sports & Entertainment.”Source: §B.3 · p. 19
  • For the native answer “J. K. Rowling” in sample 000086, L0 had a +4.340 effect and L18 a −3.406 effect on the target–non-target logit margin.The supplied context supports “J. K. Rowling” rather than “Tony McNamara”; these fixed-endpoint score contrasts do not establish that ablation changes the decoded answer.Source: §B.3 · p. 19

B.4 DECODED ANSWER CHANGES UNDER SINGLE-HEAD ABLATION … C.4 DATA SEPARATION AND SELECTION HYGIENE

A single native head ablation produced verified decoded-answer changes in selected ConFiQA cases, while the appendix specifies CAST’s optimization, model interfaces, comparison protocols, and data-separation procedures. These details clarify how controllers and baselines were configured and how final-test data were kept out of selection and calibration.

B.4 DECODED ANSWER CHANGES UNDER SINGLE-HEAD ABLATION

  • Ten manually selected ConFiQA cases showed the ablated head affirming the context-supported target when the native answer did not; these cases do not estimate a semantic-flip rate.The check covered 866 inputs and flagged 36 candidates before review; lexical scoring could miss semantic flips when native outputs mentioned but denied the target, and one native output ended at the 64-token limit.Source: §B.4 · pp. 19–20

C.1 OPTIMIZATION DETAILS

  • CAST selects task-specific strengths using non-test data, reports IMDb’s full response curve, and excludes test outcomes from operating-parameter selection.Table 4 summarizes locked SFT-starting settings; all and prefill runs are trained separately.Source: §C.1 · pp. 20, 22, Table 4 · p. 21

C.2 MODEL AND HARDWARE DETAILS

  • CAST changes only controller state at registered head, layer, or MLP interfaces, leaving the model implementation, tokenizer, and other parameters unchanged within comparisons.Comparisons used specified public model/checkpoint pairs and revisions; Performance comparisons used 80-GB H100 devices.Source: §C.2 · p. 22

C.3 BASELINE CONFIGURATIONS

  • Comparisons share final inputs, decoding, and evaluators; DPO rows use released checkpoints, while DPO+CAST-T recalibrates an SFT-trained controller and DPO+CAST-NT trains a new one.CAST’s training pairs may differ from the released DPO checkpoints’ original preference data.Source: §C.3 · pp. 22–23
  • LoReFT retains its published low-rank operator and objective (Wu et al., 2024), while BiPO retains its bidirectional objective and per-layer steering vectors (Cao et al., 2024).Their architecture mapping and registered weight-decay setting, respectively, are project adaptations.Source: §C.3 · p. 23
  • For IMDb, pre-test occupancy was approximately 2,448 seconds for SFT CAST-SV-all, 2,343 seconds for BiPO, and 2,410 seconds for LoReFT; this is not matched-FLOPs evidence.The reported durations exclude final generation and evaluation; no strict compute-equality claim is made for other tasks.Source: §C.3 · p. 23
  • TL;DR audits 16 individually trained heads against human references before retaining eight without joint retraining; ConFiQA and IMDb compare signed two-bank CAST selectors with unsigned ITI and random banks.The TL;DR audit uses both display orders and qualitative keep/drop criteria; the vector two-bank protocol does not apply to TL;DR.Source: §C.3 · p. 24

C.4 DATA SEPARATION AND SELECTION HYGIENE

  • Principal Performance splits were ConFiQA 300/240/60/120/2,048, IMDb 300/1,024/256/2,048, and TL;DR 300/8,192/512/40/40/512; final tests were held out.The listed counts follow each task’s selector/train/validation/strength-or-audit/test order; no final-test example selected components, checkpoints, ranks, head combinations, or strengths.Source: §C.4 · p. 24

C.5 FINAL EVALUATION DEFINITIONS · C.6 PAIRED UNCERTAINTY FOR MAIN PERFORMANCE CONTRASTS

The final evaluation defines task-specific scoring and policy-shift measures, with explicit handling of empty outputs and paired judgments. Main performance contrasts use paired bootstrap intervals over fixed test sets and checkpoints.

C.5 FINAL EVALUATION DEFINITIONS

  • ConFiQA counts context-supported and prior-answer aliases as whitespace-bounded occurrences for PC and PO, while EM requires the entire normalized output to equal a context-supported alias.The scorer case-folds text, replaces punctuation with spaces, and collapses whitespace; therefore PC can be 1 while EM is 0.Source: §C.5 · pp. 24–25
  • ConFiQA PC is an occurrence metric that can count a negated target mention; Site reports macro-average PC across QA, MR, and MC, while Performance combines 2,048 test inputs.Decoded examples are separately checked for whether the model affirms the answer.Source: §C.5 · p. 25
  • IMDb uses frozen SiEBERT sentiment reward P(POSITIVE) − P(NEGATIVE), with positive rate defined as the fraction whose reward exceeds zero.The two distribution-shift axes compare the evaluated policy with the registered SFT reference on its generated continuation.Source: §C.5 · p. 25
  • GPT-2-large Site plots average Kseq, a generated-sequence log-probability ratio, while Performance plots average Ktok; the additional Qwen2.5-14B Site plot uses full-vocabulary token KL.Kseq is not tokenwise full-vocabulary KL and, under deterministic decoding, is not an unbiased Monte Carlo estimate of policy sequence KL.Source: §C.5 · p. 25
  • IMDb generation health reports nonempty-output rate and mean completion length, and reward and KL are reported together rather than treating higher reward under much greater policy shift as unqualified improvement.At the plotted DPO point, health is 99.90% and 230 tokens; support-only CAST-SV-all at α = 0.5 and 0.6 has 100%/215 tokens and 99.95%/212 tokens, respectively. All 2,048 inputs remain in IMDb denominators, including empty strings; zero-token Qwen2.5-14B Site continuations use Ktok = 0.Source: §C.5 · p. 25
  • TL;DR uses a registered deepseek-v4-pro pairwise evaluator with two presentation orders; each post scores 0, 0.5, or 1, and reported win rates are not CAST-versus-DPO head-to-head results.The evaluator compares summary precision and concision; all methods use the same references, prompt, and order rule, and all 512 posts per Performance condition were judged without retry exhaustion.Source: §C.5 · pp. 25–26

C.6 PAIRED UNCERTAINTY FOR MAIN PERFORMANCE CONTRASTS

  • Paired percentile-bootstrap intervals resample same-ID test inputs with replacement 5,000 times using seed 42; TL;DR averages both order judgments per post before resampling its 512 posts.The intervals are fixed-checkpoint, fixed-test-set estimates rather than training-seed intervals, and an interval spanning zero does not establish equivalence.Source: §C.6 · p. 26, Table 10 · p. 26

C.7 ADDITIONAL IMDB SITE RESULT

On Qwen2.5-14B, IMDb Site response curves show large strength-dependent sentiment changes for RCM-Patch, ITI, and one random seed, while RCM-Zero is weaker.

C.7 ADDITIONAL IMDB SITE RESULT

  • On Qwen2.5-14B, RCM-Patch, ITI, and one random seed produce large sentiment changes as strength rises, while RCM-Zero is weaker.Each curve covers ten registered strengths on 2,048 shared test prompts; the three random seeds are reported separately, with none selected by test score.Source: §C.7 · p. 27, Figure 5 · p. 27
  • The plotted reward is mean bounded sentiment reward against full-vocabulary token KL to the unmodified Qwen reference.The archived score is mean positive-class probability p+, converted to 2p+ −1; this axis differs from the sampled sequence log-probability ratio used in the main GPT-2-large Site panel.Source: §C.7 · p. 27, Figure 5 · p. 27
  • At one strength, RCM-Zero has a 3.6% empty-output rate, while the other plotted selectors remain at zero.Source: §C.7 · p. 27

C.8 IMDB DPO-START RESPONSE CURVES … D.2 LOCAL HIDDEN-STATE CONTROLLABILITY

The section reports IMDb DPO+CAST response curves and an ablation, then gives theoretical results on the DPO form of the pairwise objective and local hidden-state controllability. The results specify what the theory establishes and what it does not guarantee.

C.8 IMDB DPO-START RESPONSE CURVES

  • IMDb response curves show that all controller types and timings can substantially raise sentiment reward, while prefill changes are smaller and remain near DPO; test strengths were not used to select deployment strength.The curves compare SFT-to-DPO transfer with DPO-start training, constant-vector with low-rank controllers, and all-token with prefill application.Source: §C.8 · p. 27, Figure 6 · p. 28
  • At α = 0.4 under all timing, transfer reward/KL is 0.9405/0.0279 with the supporting bank and 0.9453/0.0332 with both; DPO-start is 0.9155/0.0238 and 0.9626/0.0432, respectively.The uncontrolled DPO point is 0.8604/0.0183; the supporting bank alone exceeds DPO reward, while the full controller can add reward at greater distribution shift, without establishing superiority at matched KL.Source: §C.8 · p. 27, Figure 7 · p. 28

D.1 DPO-FORM INTERPRETATION OF THE PAIRWISE OBJECTIVE

  • The pairwise objective has exactly the reference-relative logistic DPO form when applied to the score-induced policy, but that policy generally differs from CAST’s actual autoregressive policy.The equivalence remains exact at the objective level for length-normalized scores.Source: §D.1 · pp. 29–30
  • For a fixed controller, the score-induced policy uniquely optimizes the corresponding regularized reward objective under the usual support conditions; this is not a fixed-reward training theorem for CAST.The implicit reward depends on the controller parameter θ.Source: §D.1 · p. 29

D.2 LOCAL HIDDEN-STATE CONTROLLABILITY

  • Under local smoothness, a nonzero local preference gradient implies a rank-one intervention direction that strictly improves the objective for a sufficiently small step.This is an existence result over admissible local state controls, not a guarantee that every or RCM-selected interface has a nonzero gradient, or that one interface can realize an arbitrary global policy.Source: §D.2 · p. 30
  • For additive constant-vector controllers, a sufficiently small step along the normalized gradient strictly improves the preference objective when the local gradient is nonzero.Thus constant and low-rank CAST actuator classes can each contain local improvement directions when their respective gradients are nonzero.Source: §D.2 · p. 30

D.3 RCM EFFECTS AS LOCAL-SENSITIVITY PROXIES

Finite RCM contrasts imply sensitivity along the intervention path, but do not identify an optimal control site. Stronger endpoint conclusions require additional smoothness, and sampled discrete generations require a separate interpretation.

Path sensitivity

  • A nonzero finite RCM contrast implies nonzero sensitivity somewhere along its interpolation path, not an optimal control site.The contrast is a path-integrated sensitivity under the stated regularity conditions.Source: §D.3 · p. 31

Endpoint sensitivity

  • With a continuous derivative, the endpoint directional derivative is nonzero and has the same sign as the finite RCM contrast.This stronger conclusion requires smoothness beyond Proposition 3, and is not assumed for all experiments.Source: §D.3 · p. 31

Zero and Patch paths

  • RCM-Zero integrates sensitivity from write removal to the native residual write, while RCM-Patch integrates it from the receiving state to the specified source state.The reported Site experiments use the prototype-patch form.Source: §D.3 · p. 31

Stochastic generation protocols

  • For sampled discrete generation, an individual realization is not differentiable in the intervention state, though its finite RCM contrast remains well defined.The path-integral statement applies to deterministic differentiable scores or expected scores when differentiation can be exchanged with expectation; this matters for sampled open generation evaluated with an external reward model.Source: §D.3 · p. 32

Related papers

Loading 2609.33298v1…