PRISM-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models

Yuyao Ge; Shenghua Liu; Yiwei Wang; Baolong Bi; Lingrui Mei; Jiayu Yao; Jiafeng Guo; Xueqi Cheng

Source-linked AI summary

PRISM-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models

Yuyao Ge, Shenghua Liu, Yiwei Wang, Baolong Bi, Lingrui Mei, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

arXiv:2603.10705v2Mar 11, 2026cs.CL

TL;DR

Prompt highlighting must prioritize marked input spans, but Key-only methods can capture shared structure and leave Value-channel signal unused. PRISM-Δ learns differential steering subspaces with continuous head weighting and extends steering to both Key and Value channels. Across four benchmarks and five models, it matches or exceeds the best existing method on 19 of 20 configurations, while the authors recommend validation-set selection because gains vary across settings.

Problem
Prompt highlighting needs to amplify user-marked tokens, while existing Key-only methods leave useful Value-channel signal unused and may conflate shared structural directions with discriminative signal.
Method
PRISM-Δ decomposes differential cross-covariance to isolate discriminative directions, applies continuous softplus head weighting, and jointly steers Key and Value channels.
Results
PRISM-Δ matches or exceeds the best existing method on 19 of 20 model×benchmark configurations, with relative gains up to +10.6% on P. Score across Pronoun Change.
Takeaways & Limitations
Key steering drives accuracy gains while Value steering reduces the fluency cost of editing, and the method extends to instruction-tuned models and automatically retrieved spans.
Takeaways & Limitations
Performance gains vary substantially across benchmarks and models, requiring validation-set selection rather than a universal default; dual-channel steering has an unexplained failure case on GEMMA3-12B CounterFact.

Abstract

from arXiv · show

Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge of existing Key-editing approaches is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-$Δ$ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, our methods match or exceed the best existing method on 19 of 20 configurations, with relative gains up to +13.0%, while halving the fluency cost of steering. Our methods also scale to long-context retrieval, outperforming the best existing method by up to +5.1% relative gain. PRISM-$Δ$ is compatible with FlashAttention and adds negligible memory overhead.

1 Introduction

PRISM-Δ addresses prompt highlighting by steering both attention routing and content transmission, rather than editing only Key representations. It learns discriminative directions from contrasting relevant and irrelevant conditions and weights attention heads continuously.

Prompt highlighting amplifies attention to marked tokens so generation better reflects user intent, especially when relevant information is buried in long contexts.
Existing methods edit only Key representations, enhancing routing while leaving the information transmitted through Value representations unenhanced.
Value shifts match Key shifts across five models, with roughly half of heads showing significant Value-channel signal.
PRISM-Δ steers Key and Value channels through discriminative subspace learning and adaptive per-head weighting.
Differential cross-covariance decomposition separates relevant from irrelevant conditions while eliminating shared structural directions, and softplus weighting assigns continuous head importance.

2 Related work

Prior prompt-highlighting methods manipulate attention scores, logits, or Key vectors, while related activation-editing and attention-analysis work motivates representation steering and per-head weighting. The comparison landscape spans methods with different computational and architectural trade-offs.

Prompt highlighting: PASTA modifies attention scores, SPA anchors at the logit level, and SEKA edits Key vectors through spectral decomposition.PASTA is incompatible with FlashAttention, SPA requires multiple forward passes, and SEKA has near-zero overhead.
Prompt highlighting: Prefix-Tuning operates in Key/Value space by prepending learned soft tokens but requires gradient-based training and changes context length.
Activation editing and head specialization: Activation-editing approaches steer representations using contrastive, natural-language-derived, instruction-specific, or population-level directions.
Activation editing and head specialization: Attention and mechanistic studies of specialized heads motivate assigning continuous per-head importance in PRISM-Δ.PRISM-Δ uses differential SVD for discriminative directions and softplus for continuous weights.

3 Method

PRISM-∆ learns discriminative steering subspaces from contrastive Key and Value representations, then applies adaptive per-head edits to highlighted tokens. Differential cross-covariance removes shared directions, while joint Key–Value steering captures routing and content gains.

Dual-channel view: Prompt highlighting amplifies the influence of marked tokens, whose attention output depends on routing through Key representations and content transmission through Value representations.Existing methods capture only routing gain, leaving content and cross gains unused.
Contrastive representation extraction: PRISM-∆ extracts contrastive representations from neutral, relevant-question, and irrelevant-question conditions for each attention head.Representations are collected separately in Key and Value spaces.
Dual-channel complementarity: Key and Value signals are complementary: Key shifts peak in middle layers, Value shifts peak in late layers, and their shifts are weakly correlated at r=0.342.This depth specialization motivates treating both channels as distinct steering sources.
Differential subspace learning: Differential cross-covariance isolates directions that separate positive and negative conditions instead of retaining directions shared by both.Its top singular vectors maximize discriminative cross-covariance difference, while shared directions contribute zero to the differential projection.
Differential subspace learning: Independent positive and negative projections can overlap structurally, whereas the differential projection directly targets the discriminative subspace.The projection retains top singular vectors until cumulative energy reaches threshold γ.
Adaptive head weighting: Softplus weighting assigns stronger steering to discriminative heads while preserving reduced but nonzero contributions from weaker heads.The same per-head discriminability measure supports both projection reliability and steering strength.
Inference-time steering: PRISM-∆ edits only Key representations, whereas PRISM-∆V jointly edits Key and Value representations to capture all three attention gain terms.At inference, both channels are edited for each highlighted token, with gK and gV controlling routing and content steering strength.

4 Setup

The evaluation covers three prompt-highlighting benchmarks across five models from the Qwen3 and Gemma3 families. PRISM-∆ variants are compared with five baselines under a fixed offline-training and greedy-decoding setup.

Benchmarks: The benchmarks are BiasBios, CounterFact, and Pronoun Change, each evaluated with task-specific accuracy, fluency, consistency, efficacy, paraphrase, or pronoun-change metrics.BiasBios tests occupation prediction, CounterFact tests knowledge-conflict resolution, and Pronoun Change tests instruction-based pronoun rewriting.
Models: The models are QWEN3-4B/8B/14B-Base and GEMMA3-4B/12B-PT, spanning two architecture families and three scales.Both families use Grouped Query Attention, with PRISM-∆ constructing one projection per KV head.
Baselines: Baselines include Vanilla, asterisk marking, PASTA, SPA, and SEKA, with AdaSEKA treated separately as a higher-cost multi-expert comparison.The baseline set includes both no-steering and existing attention or Key-editing approaches.
PRISM configurations: PRISM-∆ uses Key-only steering, while PRISM-∆V steers both Key and Value using projections built offline from 100 synthetic contrastive QA pairs.All experiments use greedy decoding on single NVIDIA H20 GPUs.

5 Results

Across benchmarks and models, PRISM-Δ generally matches or outperforms SEKA, while extending to long-context retrieval with low memory and latency overhead. Ablations attribute gains to differential projection, softplus weighting, and Value-channel steering.

Main results: +1.6% relative gain is achieved by PRISM-Δ on BiasBios accuracy for Qwen3 models.On CounterFact, PRISM-Δ ties SEKA at 98.86% on GEMMA3-12B and reaches 99.24% on QWEN3-8B.
Main results: 19 of 20 model×benchmark configurations match or exceed the best existing method, with the exception of GEMMA3-12B on BiasBios.On Pronoun Change, the methods outperform SEKA on all five models, with relative gains up to +10.6% on P. Score.
Statistical reliability: 14 of 15 model×benchmark cells match or exceed SEKA, with p < 0.001 in the reported one-sided sign test.Across five projection subsets, standard deviation is 0.05–0.15%.
Ablations: +0.94% is obtained when differential projection and softplus weighting are combined, exceeding either component alone in the factorial ablation.Differential projection alone yields −0.02%, while softplus alone yields +0.08%.
Ablations: 82.44% is reached by PRISM-Δ-V alone, supporting independently useful Value-channel signal.At matched δmin=0.08, SEKA drops by 1.36% while PRISM-Δ achieves its best result.
Long-context retrieval: Up to 5.1% relative gain is achieved on lost-in-the-middle retrieval across five models.Steering targets the middle region of 30-passage contexts.
Efficiency: +0.02 GB memory is added by PRISM-Δ, while latency increases by +0.30 s; both variants remain FlashAttention-compatible.PASTA adds +1.03 s and +23.12 GB, whereas SPA adds +5.32 s.

6 Analysis

The analysis separates the roles of Keys and Values and examines how differential projection and softplus weighting affect robustness and fluency. Key steering primarily drives accuracy, while Value steering contributes to generation quality and can complement Key steering.

Direction consistency: ΩΔ heads remain near-independent across layers, while Ω+ heads converge to similar directions, indicating reduced shared-direction redundancy.Within-layer Ω+ similarity reaches up to 0.76, whereas ΩΔ remains near-independent.
Dual-channel contribution decomposition: +12.58 points over Vanilla are driven primarily by the Key channel, while PRISM-Δ incurs only 53% of SEKA’s fluency cost.Both PRISM-Δ and PRISM-ΔV outperform SEKA on all three reported metrics.
Layer-wise and cross-model complementarity: Key signal averages 0.175 in middle layers, whereas Value signal averages 0.307 in late layers.Qwen3 becomes increasingly Value-dominant with scale, while Gemma3 remains Key-dominant.
Sensitivity analysis: 0.60% fluctuation is observed for PRISM-Δ across δmin values from 0.06 to 0.12, compared with a 3.14% SEKA drop.Softplus assigns lower weights to less discriminative heads, reducing sensitivity to noisy-head inclusion.

7 Conclusion

PRISM-Δ combines differential projection, continuous softplus head weighting, and dual-channel Key–Value steering for inference-time prompt highlighting. The method extends to retrieved spans and long-document pipelines, with Key steering favoring accuracy and Value steering reducing fluency cost.

Conclusion: Differential projection removes directions shared by relevant and irrelevant contexts, softplus weighting retains weak but useful heads, and dual-channel steering separates Key routing from Value content.The method is presented as an inference-time prompt-highlighting approach.
Conclusion: PRISM-Δ extends to instruction-tuned models and automatically retrieved spans for RAG and long-document pipelines.The paper identifies joint selector–steering policies and broader instruction prioritization as future directions.

8 Limitations

The preferred gains vary substantially across benchmarks and models, so PRISM-Δ requires validation-set selection rather than serving as a universal default.

Practical limitation: Validation-set selection is required because preferred gains vary substantially across benchmarks and models.The paper recommends PRISM-Δ as the default, with separate validation before enabling Value steering.

Ethics Statement

The paper reports an open, inspectable evaluation pipeline using public benchmarks and checkpoints, with validation-selected configurations and documented metrics, hardware, and comparisons. It also situates PRISM-∆ among lightweight and higher-memory alternatives.

Ethics and reproducibility: PRISM-∆ uses publicly available benchmarks and publicly released model checkpoints, while operating at inference time without changing model parameters.
Comparative results: PRISM-∆ matches or exceeds AdaSEKA in 10 of 15 cells while using less memory, but AdaSEKA performs better in some settings.
Evaluation protocol: Hyperparameters are selected separately for each model–benchmark configuration by validation-set grid search.
Evaluation protocol: The evaluation reports Top-1 Accuracy, Fluency, Consistency, Efficacy, Paraphrase, P. Score, and All-changed P. Score across the three prompt-highlighting benchmarks.
Comparative results: PRISM-∆ exceeds SEKA in all nine reported cells, with a one-sided cell-level sign test of p = 0.002, although cells are not independent replicates.

I Qualitative evaluation and extended channel decomposition

Extended evaluations examine judge-based quality, dual-channel behavior, projection transfer, noisy highlighted spans, and cross-model Key/Value signal. Results support complementary channel roles, but several analyses remain limited in scope.

Qualitative evaluation: On a pooled 50-sample validation set, judge and human scores have Pearson correlation r = 0.87, while the authors caution that this limited validation does not establish broader reliability.
Qualitative evaluation: PRISM-∆ improves judge-rated factuality over SEKA, while PRISM-∆V moves coherence toward Vanilla in the two reported settings.
Channel decomposition: Value-only steering matches or exceeds Vanilla fluency in five settings, while PRISM-∆ incurs 46–63% of SEKA’s fluency cost.
Channel decomposition: PRISM-∆V exceeds PRISM-∆ by 1.08 points on GEMMA3-4B Pronoun Change, but broader interaction studies are needed to establish when this occurs.
Projection transfer: Transferred projections remain above Vanilla in four tested settings, with degradation of 1.82–6.86 points relative to task-specific projections.
Robustness: 15% and 20% irrelevant-token noise reduce QWEN3-4B BiasBios accuracy by 1.70 and 2.63 points, respectively, without covering all upstream span errors.
Cross-model K/V complementarity: On QWEN3-14B, average Value signal exceeds Key signal and late layers are Value-dominant, whereas Gemma3 shows K/V ratios above 1.2.
Stability: Across five overlapping projection subsets, standard deviation is 0.05% on QWEN3-4B and 0.15% on QWEN3-8B, smaller than the corresponding PRISM-∆–SEKA gaps.

O CounterFact ablation

The CounterFact ablation separates contributions from differential projection and softplus weighting, while related analyses examine data quantity, direction redundancy, hyperparameter sensitivity, and per-sample outcomes. Several conclusions are explicitly limited to individual settings.

CounterFact ablation: +0.30% comes from differential projection, comparing PRISM-∆ at 99.14% with independent projection at 98.84% on QWEN3-4B CounterFact.
CounterFact ablation: +0.52% comes from softplus weighting, comparing PRISM-∆ with uniform weighting at 98.62% on the same setting.
CounterFact ablation: The smaller absolute ablation contributions occur near the performance ceiling, and one additional model–task setting is insufficient to establish generality.
Data quantity: Using 200 rather than 100 synthetic samples raises QWEN3-4B BiasBios accuracy only from 92.38% to 92.40%, but this saturation point is not established elsewhere.
Direction consistency: Cross-head cosine similarity is 0.254 for Ω+ directions, 0.071 for random vectors, and 0.068 for Ω∆ directions in QWEN3-4B.
Sensitivity: On QWEN3-4B BiasBios, performance changes by less than 0.5 points across tested gK values, less than 0.45 points across δmin, and less than 0.1 points across γ.
Head weighting: Softplus assigns weak-signal heads reduced but nonzero weights, whereas hard thresholding shuts off 108 heads entirely.
Per-sample analysis: PRISM-∆ rescues 154 samples and loses 81 relative to SEKA on QWEN3-4B BiasBios, for a net gain of 73 samples and +1.46% accuracy.