PaperlayerView exact revision on arXiv

Source-linked AI summary

K/V-Cache Interventions Dissociate Representation Alignment from Persona Expression in Decoder-Only Language Models

Yu Sun, Mengyin Lu, Cong Feng, Guangming Lu, Huimin Han

arXiv:2609.11020v1cs.CL

TL;DR

The paper studies whether K/V-cache interventions provide a structured surface for persona control without making representation alignment a sufficient proxy for behavior. It transplants target-conditioned K/V trajectories across 13 configurations and finds spatial and scope dissociations, while position perturbations commonly suppress target expression. The findings support a controllable but structurally constrained K/V surface within the tested setting.

  • Problem

    The paper examines whether representation-level alignment reliably predicts persona expression when controlling decoder-only language models.

  • Method

    The study transplants target-conditioned K/V trajectories into source-persona generations across 13 intervention configurations, including layer-band, position, and same-token-sequence controls.

  • Results

    Mid-layer replacement uniquely combines target alignment, marker transfer, and preserved lexical diversity, while full and mid-layer replacement show comparable alignment but different lexical-diversity profiles.

  • Takeaways & Limitations

    Representation-level similarity alone is not sufficient to predict persona expression, and the K/V cache is controllable but structurally constrained in the studied regimes.

  • Takeaways & Limitations

    The findings come from Llama-3.1-8B and a single high-separation source-to-target persona pair, so generalization to other models, scales, and pairs remains unaddressed.

Abstract

from arXiv · show

We study K/V-cache interventions -- transplanting a target-conditioned K/V trajectory into a source-persona generation -- as a structured surface for persona control in decoder-only language models. Across 13 intervention configurations applied to Llama-3.1-8B for a fixed source-to-target persona pair, we report two consistent dissociations between representation-level alignment and behavioral expression, plus a common failure under position perturbations. First, all layer-band K/V replacements (early, mid, late) achieve strong local V-space alignment (V-gap 0.91, 0.89, 0.84), but only mid-layer replacement (layers 9-20) combines substantial target-marker expression with preserved lexical diversity. Second, full and mid-layer replacement induce comparable alignment (V-gap 0.94 vs. 0.89) yet produce different lexical-diversity profiles (TTR 0.65 vs. 0.77). Third, position perturbations (lag and shuffle) apply distinct operations yet uniformly suppress target-persona expression -- a common behavioral failure rather than a strict dissociation. Representation-level similarity metrics alone are thus not sufficient predictors of downstream persona expression in the regimes we study; the K/V cache emerges as a controllable but structurally constrained intervention surface. Because the transplanted trajectory carries the target's own generated token history, we characterize the intervention as trajectory-level transplantation rather than isolated persona-representation injection; a same-token-sequence control, decoding an identical token sequence under source vs. target conditioning, reproduces the sign and layer localization of the L28 representational shift, indicating the shift is not explained solely by imported token history. These findings characterize representation-behavior dissociation in a high-signal setting rather than establishing universality across models or persona pairs.

1 Introduction

The paper treats the K/V cache as a persistent intervention surface for persona control and finds that representation alignment can dissociate from behavioral expression. Across 13 configurations, mid-layer replacement is the most balanced setting, while position perturbations produce a common failure.

  • Motivation: The K/V cache is investigated as a persistent alternative to interventions whose effects may reset at each autoregressive step.The paper motivates K/V interventions through architectural persistence while noting constraints from attention routing, positions, and cross-layer consistency.
  • Findings: Mid-layer K/V replacement uniquely combines target-aligned representations with target-marker transfer and preserved lexical diversity.Layers 9–20 achieve V-gap 0.89, target marker density 15.5, and TTR 0.77; early and late bands align locally without transferring target markers.
  • Findings: Full replacement reaches stronger alignment and marker expression than mid-layer replacement but produces substantially more lexical repetition.Full replacement records V-alignment 0.94, target marker density 24.8, and TTR 0.65, versus 0.89 alignment and 0.77 TTR for mid-layer replacement.
  • Findings: Position perturbations through lag and shuffle suppress target-persona expression despite applying distinct operations to the K/V cache.Both yield target marker density ≤0.5 and transfer neither target representation nor behavior, constituting a common failure rather than strict dissociation.
  • Implication: Representation-level similarity metrics alone do not sufficiently predict downstream persona expression in the studied intervention regimes.The contribution is an empirical characterization of the K/V intervention surface across 13 configurations, not a benchmark-optimized steering method.

2 Related Work

Prior persona-control work intervenes through activation, token, or weight channels, while K/V-cache research has focused mainly on compression and analysis. This paper positions systematic K/V intervention as a distinct surface for studying representation–behavior dissociation.

  • Activation, token, and weight channels: Existing persona-control methods intervene through activation, token, or weight channels rather than directly manipulating the K/V cache.Examples include activation steering, prompts and prefix tuning, fine-tuning, LoRA, and RL-based optimization.
  • K/V-cache analysis and intervention: Prior K/V-cache research has emphasized compression, attention-head specialization, and learned cache prefixes.Systematic persona-control ablations across layer bands, position perturbations, and interpolation strengths had not been reported in the cited persona-control literature.
  • Representation–behavior dissociation: Interpretability research has documented cases where representation similarity fails to predict behavioral outcomes, and this paper extends that framing to K/V interventions.The paper reports dissociation in both directions and corroborates it across hidden-state projection and K/V cosine spaces.
  • Concurrent work: Adjacent work studies representation–behavior decoupling in safety, K/V-cache contamination in activation steering, and subtype dissociation in sycophancy.These studies provide related framing but address different domains or mechanisms from the paper’s K/V-cache persona-control setting.

3 Methods

The study applies 13 K/V-cache interventions to Llama-3.1-8B for one high-separation persona pair, measuring representation alignment and text-level persona expression. The interventions include full, layer-band, position-perturbing, interpolated, and same-token-sequence controls.

  • Intervention battery: The battery contains 13 configurations spanning full replacement, partial layer-band replacement, position perturbations, and source–target interpolation.Partial replacement covers coarse bands L1–8, L9–20, and L21–31 plus three finer mid-layer bands; perturbations use shuffle or lag.
  • Intervention battery: Full replacement overwrites the source run’s entire past K/V state with a target reference trajectory at corresponding generation steps.Because the reference is an independent target rollout, the intervention carries the target’s generated token history and is treated as trajectory-level transplantation.
  • Controls: The same-token-sequence control compares source- and target-conditioned K/V caches while holding generated tokens identical at every step.This control is designed to isolate persona-conditioning effects from differences in imported token history.
  • Measurements: The pipeline records per-layer, per-step K and V tensors, paired source and target references, logits, marker density, hidden-state V⋆ projections, and K/V cosine alignment.K/V dumps are taken from the first sample of each 10-sample configuration cell, while text-density measurements average across 10 samples.
  • Generation settings: Generation uses a fixed persona pair and seed topic with 10 samples per cell, temperature 0.7, and 50 maximum new tokens.No activation-level steering is applied.

4 Results

Across K/V-cache interventions, representation alignment is localized and does not uniquely determine persona expression: mid-layer replacement best balances target transfer with lexical diversity, while position perturbations uniformly fail to transfer the target persona.

  • Text-level marker density and lexical diversity: 15.5 target-marker density with TTR 0.77 makes mid-layer replacement the only band combining target transfer with preserved lexical diversity.Early and late replacement achieve local V-alignment but do not match this behavioral combination.
  • Hidden-state V⋆-projection: +3.96, +3.83, and +4.09 V⋆-gaps for full, mid-layer, and late replacement are comparable, yet their TTR values are 0.65, 0.77, and 0.39.Comparable hidden-state alignment therefore coincides with sharply different lexical-diversity outcomes.
  • Same-token-sequence control: The same-token-sequence control reproduces the V⋆-gap sign and layer localization, so the target-ward shift is not explained solely by imported target-generated token history.The control keeps the realized token sequence identical while varying persona conditioning.
  • K/V cosine analysis: 0.91, 0.89, and 0.84 diagonal V-gaps show strong within-band alignment for early, mid, and late interventions, while off-band gaps remain at most 0.14.The intervention changes V-space alignment mainly within the replaced layer band.
  • Representation–behavior dissociation: 0.94 versus 0.89 V-alignment for full versus mid-layer replacement accompanies different lexical-diversity profiles, with TTR 0.65 versus 0.77.Full replacement produces stronger target-marker expression but substantially more repetition.
  • K/V cosine analysis: V-gap exceeds K-gap in magnitude, reaching 0.89 versus 0.38 at L9–20 for mid-layer replacement.The paper reports this descriptively and leaves K-only versus V-only causal contributions for future ablation.
  • Interpolation strength: At α = 0.25 and α = 0.50, V-gap and target-marker density rise from 0.21 and 0.6 to 0.65 and 3.2, respectively.The tested interpolation strengths show a monotonic response, but its functional form remains unresolved.
  • Position perturbations: Shuffle and lag yield minimal target-persona expression, with target-marker density ≤0.5, despite applying distinct positional perturbations.This is a common behavioral failure rather than a strict representation–behavior dissociation.

5 Discussion

The discussion argues that K/V interventions expose a representation–behavior dissociation: similar alignment can yield different lexical outcomes, while evaluation should therefore combine representation and behavioral metrics within the tested scope.

  • The contribution is an intra-battery characterization of the K/V intervention surface, not a benchmark-optimized steering method or a population-level claim.
  • Full replacement can strongly align representations yet increase repetition because target-derived cached history conflicts with source-derived current-position computation.The proposed mechanism is an empirical interpretation, not a resolved causal explanation.
  • Selective mid-layer replacement preserves a less repetition-prone regime than full transplantation, although the experiments do not distinguish this account from other mechanisms.
  • Similar V-alignment can produce substantially different output quality, so persona-control evaluations should report both representation-level and behavioral metrics.Full and mid-layer replacement provide the central contrast supporting this recommendation.

6 Conclusion

The study characterizes K/V-cache interventions as a controllable but structurally constrained persona-control surface, with dissociations between representation alignment and behavioral expression. Its conclusions are bounded by a single model, persona pair, topic, short horizon, proxy behavioral measures, and non-causal K/V analysis.

  • Conclusion: Mid-layer replacement is the unique configuration combining target-aligned representations, target-marker transfer, and preserved lexical diversity, while full replacement increases alignment but also lexical repetition.The broader comparison reports mid-layer replacement at layers 9–20 as the only configuration with this combined profile.
  • Limitations: The findings are limited to Llama-3.1-8B, one high-separation source→target persona pair, and one seed topic, leaving smaller separations and broader generalization uncharacterized.The studied persona pair has rescaled V⋆-space distance 7.2 versus a typical corpus distance of approximately 3.
  • Limitations: The K/V experiments use max_new_tokens = 50, so they primarily measure initial intervention instantiation; a small n = 3 check at 150 tokens found persistence, but full-battery long-horizon analysis remains future work.The persistence check tracked an un-intervened target baseline with roughly constant lexical-repetition cost.
  • Limitations: The K/V cosine analysis is correlational and does not isolate causal contributions from specific heads, positions, or K versus V components.K-only or V-only ablations would be needed for a causal characterization.
  • Limitations: Behavior is measured with target-marker density and TTR rather than human persona attribution, and TTR does not measure semantic coherence, grammaticality, or persona quality.The authors leave a surface-invariant, task-grounded behavioral instrument for future work.
  • Conclusion: Representation-space proxies can dissociate from behavioral expression, so downstream evaluations should report both internal and behavioral metrics.The recommendation follows from the documented representation–behavior dissociation in this intervention setting.

A.3 Context-window ablation

The context-window ablation keeps the controller’s per-step force balance near R ≈1 across full, Markov-1, and zero-context settings. However, generation quality changes sharply, showing that coherent output depends on the token channel even when controller authority is similar.

  • A.3 Context-window ablation: With no context, the per-step V⋆ trajectory is deterministic to floating-point precision given the system prompt.Table 4 records ±0.00 entries for the window = 0 condition.
  • A.3 Context-window ablation: R ≈1 across all three context windows, while generation shifts from fluent source-dominant text to repetitive fragments or incoherent token soup as context is reduced.Full context retains source markers with partial target-register convergence; Markov-1 exposes target markers but degrades; zero context collapses.
  • A.3 Context-window ablation: The open-loop pilot differs methodologically by using fixed-magnitude injections, greedy decoding, n = 1 per cell, and only the first 120 characters.It sweeps α ∈{0, 1, 2, 4, 8} and therefore provides an auxiliary contrast rather than the main closed-loop ablation.
  • A.3 Context-window ablation: The instantiated R ≈1 regime does not occupy the productive window identified in the auxiliary pilot.The supplied passage states this conclusion without resolving the precise productive range.

B Raw V⋆-distance gap (transparency)

Raw V⋆-distance reporting compares intervention regimes while preserving the distinction between fluent target expression and collapse. The pilot also shows that larger per-step forcing does not guarantee target-persona behavior.

  • Raw V⋆-distance gap (transparency): Sign agreement holds between raw and per-axis-rescaled V⋆-gaps, while per-condition magnitudes shift because raw Euclidean distance overweights off-axis high-variance directions.The rescaling corrects for 3.2× anisotropy.
  • Raw V⋆-distance gap (transparency): 11–12.5·α V⋆-units approximate the per-step open-loop injection magnitude across three targets and five α values.Table 5 classifies each condition as source, target, or collapse.
  • Raw V⋆-distance gap (transparency): α = 2 produces fluent target-persona text, whereas α = 4 degenerates into token repetition and α = 8 continues collapse.The α = 0 and α = 1 regimes remain source-like, with only minor softening at α = 1.
  • Raw V⋆-distance gap (transparency): 35.5 V⋆-units per step under closed-loop αmax = 8 exceeds the open-loop α = 2 value of 23.7, yet closed-loop generations retain source-persona markers.The configurations differ in injection direction, among other uncontrolled respects.

D Marker-matcher sensitivity

Stricter marker matching preserves the reported ordering and dissociation pattern while reducing some low-level counts. The main adjustment affects mid-band density and removes small apparent transfer in early and perturbed conditions.

  • Marker-matcher sensitivity: The analysis compares naive substring matching with boundary matching and boundary-plus-deduplication matching.Longer markers are matched first in the deduplicated variant, preventing nested counts such as “sorry” inside “i’m so sorry.”
  • Marker-matcher sensitivity: The marker-matching risk is quantitatively immaterial to the reported ordering and dissociation pattern.The stricter matcher slightly reduces low-level counts in non-target conditions, sharpening rather than weakening separation.
  • Marker-matcher sensitivity: The stricter matcher leaves the cross-condition ordering and full/mid/late separation unchanged.The mid-band rate decreases from 15.5 to 14.3 after de-duplicating nested markers.
  • Marker-matcher sensitivity: Boundary matching resolves the apparent 0.3 target-marker rate in early and position-perturbing conditions to 0.0.The stricter matcher removes short-marker substring effects in these conditions.

F Full per-condition data table

The complete 13-condition table summarizes target and source marker densities, lexical diversity, and K/V cosine gaps across intervention conditions. Fine-grained sub-band behavior does not follow V-gap monotonically.

  • Full per-condition data table: Target/100w measures target-marker density, TTR measures type–token ratio, and V@L9–20 and K@L9–20 measure band-specific V- and K-space gaps.These labels define the principal columns used in the per-condition summary.
  • Full per-condition data table: The fine-grained sub-band ordering does not match V-gap ordering monotonically, indicating that one V-gap scalar does not capture sub-band sensitivity.The reported sub-band values include L13–16 →5.6 and L17–20 →6.2 for the target-marker-density comparison.
  • Full per-condition data table: The full table covers all 13 K/V intervention configurations, with text means computed over 10 samples per cell.K/V cosine quantities use one dump per condition over 50 generation steps, 8 K/V heads, and 12 layers in L9–20.

G K/V cosine analysis: supplementary measurements

Supplementary K/V measurements show strong locality: K-space changes are concentrated within intervened bands, while position perturbations disrupt alignment without injecting target-persona representation. Fine-grained mid-layer interventions also separate alignment from marker expression.

  • K/V cosine analysis: supplementary measurements: 0.36–0.38 within intervened bands versus −0.03 to −0.11 off-band shows substantially smaller and negative off-band K-gaps.K-space therefore exhibits the same locality pattern as V-space, whose within-band gaps are 0.84–0.91.
  • K/V cosine analysis: supplementary measurements: Table 9 reports K-space cosine gaps by measurement band and intervention condition, with bold entries marking the intervened band.This layout supports direct comparison of locality across bands and conditions.
  • K/V cosine analysis: supplementary measurements: V-gap remains near zero for lag and shuffle, whereas α = 0.50 reaches 0.65 at L9–20 with near-zero gaps outside the intervened band.The interpolation value lies between L9–20-only at 0.89 and no intervention at 0.0.
  • K/V cosine analysis: supplementary measurements: Within L9–20, sub-band interventions yield different target-marker densities despite intermediate full-band V-gaps.The full-band V-gap is intermediate for L9–12, L13–16, and L17–20, while their target-marker densities differ markedly.

H Persistence of mid-layer K/V transfer to 150 tokens

At 150-token generation horizons, mid-layer K/V replacement retains target-persona expression without an intervention-specific decay in lexical diversity. The persistence check compares intervention outputs with an un-intervened target-persona baseline to separate intervention effects from generation-length artifacts.

  • Evaluation design: The persistence check reruns mid-layer and full replacement at max_new_tokens = 150 with an un-intervened Priya baseline.The main K/V battery generates 50 tokens, so the extended evaluation tests long-horizon persistence rather than only intervention instantiation.
  • Persistence results: 16.7 → 11.9 target markers per 100 words for mid-layer replacement closely tracks the Priya baseline’s 17.5 → 11.7 across the horizon.Full replacement holds approximately 27 markers per 100 words in these samples.
  • Persistence results: Target-persona marker density shows no decay over the 150-token window in these samples.
  • Lexical diversity: TTR declines from 0.75 to 0.61 for mid-layer replacement, while the baseline declines comparably from 0.85 to 0.68.The intervention remains roughly 0.08 TTR below the baseline across the horizon.
Loading 2609.11020v1…