Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models

• Uniform interventions degrade quality and increase perplexity, diversity loss, and cross-attribute interference because they treat denoising trajectories as temporally uniform.Source: §1 · p. 2
• The paper trains SAEs across four DLMs and identifies commitment timing, sharpness, and effect-size asymmetry as determinants of when and how attributes become steerable.Source: §1 · p. 2
• Adaptive steering concentrates intervention where each attribute is actively forming, with closed-form scheduling efficiency and a decoder-based bound on multi-attribute interference.Source: §1 · p. 2
• Across four DLMs, seven conditions, and four baselines, adaptive steering matches or exceeds baseline control while keeping perplexity and diversity close to the unsteered model.Source: §1 · pp. 2–3
• 93% geometric-mean classifier confidence on simultaneous sentiment, topic, and formality steering exceeds the strongest baseline by up to 15 percentage points at usable text quality.Source: §1 · p. 3

• Existing DLM methods study interpretability, reference-sequence alignment, or adapted contrastive and probe directions, but generally lack feature-level composability or cross-objective comparisons.Source: §2 · p. 3
• The paper trains SAEs across four DLMs, two objectives, and a 60× scale range, then evaluates adaptive compositional control against multiple baselines.Source: §2 · p. 3
• Table 1 compares concurrent efforts by whether they fully, partially, or do not address the paper’s targeted capabilities.Source: Table 1 · p. 3

3.1 Training SAEs on Discrete Diffusion Models

• TopK SAEs are trained on residual-stream activations from MDLM, SEDD, LLaDA, and DREAM across masking rates to generalize over the denoising trajectory.Source: §3.1 · pp. 3–4

3.2 Extracting Attribute Features from SAEs

• Attribute features are extracted from balanced sentiment, topic, and style corpora using contrastive activation differences, statistical filtering, and vocabulary grounding.Source: §3.2 · p. 4

3.3 Interpreting Temporal Dynamics of Features

• Topic commits within 2.1% of MDLM denoising, whereas sentiment reaches 50% emergence at 21%, demonstrating roughly a 10× timing gap.Source: §3.3 · p. 4, Figure 2 · p. 5
• Emergence sharpness varies substantially: MDLM’s first block contains all topic emergence but only 31% of sentiment, while DREAM concentrates emergence in its final two blocks.Source: §3.3 · p. 5, Figure 2 · p. 5
• Topic features have 2–3× larger per-feature Cohen’s |d| than sentiment, while SEDD distributes sentiment and formality across more features than MDLM.Source: §3.3 · p. 5
• Training objective dominates cross-model feature alignment: MDLM and SEDD have cosine similarity 0.08, whereas MDLM and DREAM reach 0.44 mean and 0.88 maximum similarity.Source: §3.3 · p. 5, Figure 3 · p. 5
• Topic features are anticipatory on MDLM, while DREAM shows little differentiation until late denoising, further motivating model- and attribute-specific schedules.Source: §3.3 · p. 5

4 Interpretability Informed Steering

• SAE interventions preserve off-basis residual information while modifying only selected attribute-feature coordinates through decoded residual updates.Source: §4 · p. 6

4.1 Steering Configurations

• Multiple attributes are composed additively in feature space, with a ReLU clamp and near-disjoint feature sets supporting compositional interventions.Source: §4.1 · p. 6
• Adaptive steering weights denoising blocks by empirical feature-emergence fractions, concentrating intervention during active emergence and reducing it elsewhere.Source: §4.1 · p. 6
• Effectiveness calibration rescales attributes using empirical dose-response ratios because unequal feature effect sizes can otherwise over-steer strong attributes and under-steer weak ones.Source: §4.1 · pp. 6–7

4.2 Theoretical Analysis

• Theorem 1 states that adaptive efficiency is governed by the cost-weighted dispersion of stepwise gain-to-cost ratios, with larger gains when emergence is concentrated and near-unity gains when it is diffuse.Source: §4.2 · p. 7
• The temporal proxy is efficient when activation growth tracks marginal sensitivity and costs vary slowly, while anticipatory features concentrate effective gain in early blocks.Source: §4.2 · p. 7
• The analysis also decomposes trajectory-level cost through reverse-process KL and bounds cross-attribute interference using decoder-space contributions and weights.Source: §4.2 · p. 7
• Table 3 reports classifier confidence, perplexity, and distinct-bigram diversity at quality-gated operating points selected from steering-strength sweeps.Source: Table 3 · p. 8

5.2 Results

• 70–100% control on MDLM, 77–100% on SEDD, 66–100% on LLaDA, and 96–100% on DREAM in single-attribute steering.On LLaDA topic, SAE methods reach 99–100% while baselines remain at 24–32% and prompt steering reaches 65%.Source: §5.2 · p. 9
• Adaptive matches Uniform control on MDLM at lower PPL, 50–57 versus 59–93, by targeting high-emergence blocks.On LLaDA, flat temporal profiles make Adaptive and Uniform comparable; on SEDD, Contrastive Vectors can match SAE performance because features are distributed.Source: §5.2 · p. 9
• 93% triple-condition confidence on SEDD and DREAM, while LLaDA reaches 72% and MDLM 81% across simultaneous sentiment, topic, and style control.On LLaDA, the best SAE method exceeds the best baseline by 15 percentage points.Source: §5.2 · p. 9
• Adaptive resolves MDLM interference for S+St, raising confidence from 49% with Uniform to 64% by targeting sentiment’s later emergence blocks.Adaptive+E matches Uniform’s triple control, 80% versus 79%, at substantially lower PPL, 50–63 versus 73–91.Source: §5.2 · p. 9
• Adaptive reduces non-target shifts relative to Uniform, with MDLM interference reaching 52pp for Uniform versus 31pp for Adaptive.The method targets high-emergence steps to avoid disrupting other attributes during critical commitment.Source: §5.2 · pp. 9–10
• DREAM preserves high control with Adaptive but has mixed PPL trade-offs and systematically lower distinct-bigram diversity.The authors attribute this behavior to DREAM’s autoregressive-based initialization and late-trajectory token-finalization behavior.Source: §5.2 · p. 10

6 Conclusion

• Adaptive steering concentrates intervention where semantic attributes form, achieving strong control with lower perplexity and interference than uniform or non-SAE baselines.The conclusion frames DLM iteration as a basis for temporally aligned, composable interventions.Source: §6 · p. 10
• The evaluation relies on classifier metrics, short 64–1024-token sequences, and reports diversity decreases at high SAE steering strengths.Longer-length steering and finer-grained attributes remain unexplored directions.Source: §6 · p. 10

A.1.1 Trajectory-level KL: exact additive decomposition

• Theorem 1 allocates intervention under a budget 1_t ≤ B, where c_t is the second-order perturbation cost at step t.The budget comes from an additive decomposition of trajectory-level KL between steered and unsteered reverse processes.Source: §A.1.1 · p. 13
• The reverse trajectories are Markov processes sharing the same initial distribution, so the chain rule decomposes their KL through per-step reverse-transition differences.The steered transition differs through the residual-stream perturbation at each step.Source: §A.1.1 · p. 13
• Replacing the steered outer expectation with the unsteered expectation preserves leading-order cost because the correction is O(α^3) versus an O(α^2) integrand.The approximation assumes smooth change of measure and bounded observables near α = 0.Source: §A.1.1 · p. 13
• A second-order KL expansion removes the linear term because zero perturbation minimizes each KL summand, yielding coefficients c_t for budgeted allocation.Combining these costs with first-order attribute gains Δg_t(α_t) = α_t s_t + o(α_t) produces the main-text allocation problem.Source: §A.1.1 · p. 13

A.1.2 Proof of the efficiency ratio theorem

• Theorem 1 compares maximum attribute shifts under optimal and uniform schedules at fixed quality budget B = 1, assuming st ≥ 0, ct > 0, and Etarget > 0.It denotes these shifts by E⋆ and Eunif, respectively.Source: §A.1.2 · p. 14
• Uniform scheduling is optimal exactly when s_t/c_t is constant across steps; with slowly varying c_t, the ratio instead depends on the squared coefficient of variation of s_t.In the constant-cost regime, the optimal schedule satisfies α_t⋆ ∝ s_t, and dispersion of s_t alone governs achievable shift.Source: §A.1.2 · p. 15

A.1.3 Suboptimality of any feasible schedule

• Any feasible schedule achieves optimal control iff α_t ∝ s_t/c_t, while its matched-control cost ratio is B/B⋆ = 1/cos^2 θ.For heuristic schedules α_t = κw_t, efficiency depends only on the shape w_t because the scale κ cancels.Source: §A.1.3 · pp. 15–16

A.1.4 Operational form of c_t

• The per-step cost c_t is the pullback of output-distribution Fisher information through the post-intervention network, and is strictly positive when the induced logit change is outside the softmax-invariant null space.The theorem’s c_t > 0 condition is operationally expected because steering directions generally affect many output dimensions.Source: §A.1.4 · p. 16

A.1.5 Position-wise decomposition of the local gain

• The local gain decomposes into direct and indirect pathways: direct effects localize to masked positions, whereas indirect effects through attention at later masked positions are generally nonzero and only empirically sub-leading.Direct-path dominance is supported by masked-fraction measurements but is not established in closed form.Source: §A.1.5 · pp. 16–17

A.1.6 Connecting the optimal schedule to the active-set proxy

• The activation-growth proxy is exactly optimal when active-set growth is proportional to s_t/c_t and c_t varies slowly; it is sharp in MDLM but near-uniform in LLaDA, while anti-correlation remains an unsupported failure regime.Activation growth targets feature commitment rather than already-saturated activation levels, and the claim that Adaptive does not underperform Uniform is empirical rather than structural.Source: §A.1.6 · pp. 17–18

A.2 Decoder-space interference: bound and analysis

• The decoder-Gram bound gives a worst-case residual-space interference guarantee under disjoint feature sets, while measured coupling is roughly an order of magnitude tighter; the analysis assumes positive-definite intra-attribute Gram matrices.The assumptions require near-disjoint feature sets and linearly independent decoder columns on each attribute’s support.Source: §A.2 · pp. 18–19
• The proof requires nonzero perturbations and a positive effective attribute contribution so the cosine term and bound denominator are well defined.The ReLU clamp must not eliminate the entire perturbation over the selected feature support.Source: §A.2 · p. 19

A.2.1 Subspace containment and the role of ReLU

• Disjoint feature supports localize each attribute’s post-ReLU perturbation, allowing the Gram bound to apply to actual perturbations rather than only linearized feature updates.Coordinate-wise ReLU preserves the attribute-local structure because each clipping decision depends on one attribute under disjointness.Source: §A.2.1 · p. 19

A.2.2 Decoder-Gram bound

• The decoder-Gram interference bound is independent of perturbation magnitudes, holds uniformly over post-ReLU perturbations, and is computable a priori from the SAE decoder.Positive definiteness of each within-attribute Gram block ensures a strictly positive denominator.Source: §A.2.2 · p. 20

A.2.3 Why the bound is small in practice

• Realized cross-attribute cosine coupling is at most 0.09, with mean 0.04, across disjoint feature sets—an order of magnitude tighter than the worst-case bound.Contrastive selection yields near-orthogonal feature sets, while overcomplete dictionaries and unit-norm decoder columns suppress off-diagonal correlations.Source: §A.2.3 · p. 20

A.2.4 Small-overlap correction

• Small feature overlaps introduce genuine cross-attribute coupling through shared ReLU decisions, but the correction scales with overlap fraction and off-diagonal Gram entries.With overlap at most 2/50 = 0.04, Figure 14 confirms the realized cosine remains consistent with the disjoint-case bound within this correction.Source: §A.2.4 · p. 21

A.3 Decoder Gram structure and attribute coupling

• Removing overlapping SAE features reduces maximum coupling 2.7×, from 0.253 to 0.092, indicating shared features dominate residual-space alignment.Disjoint feature sets have maximum coupling 0.092 and mean 0.028 across 38 pairs.Source: §A.3 · p. 21
• Empirical coupling remains small across models, layers, and topic categories, with cross-topic maximum disjoint coupling 0.082 and mean 0.037.No model, layer, or topic category exhibits systematically higher coupling.Source: §A.3 · p. 21
• The bound is conservative: realized coupling is roughly an order of magnitude tighter, suggesting contrastive shifts avoid worst-case Gram directions, though generality remains open.The authors identify data-dependent tightening of the bound as future work.Source: §A.3 · p. 22

B Vocabulary Grounding of Top Contrastive Features

• Vocabulary grounding projects SAE decoder columns through the unembedding matrix to identify tokens most promoted by active features, using contrastive feature selection by Cohen’s d.The analysis reports top positive- and negative-direction features for topic, formality, and sentiment at each model’s deepest SAE layer.Source: B Vocabulary Grounding of Top… · p. 22

C.1 Models and SAE Training

• SAEs are trained for four DLMs spanning two training objectives, two scales, and two provenance strategies, with activations extracted across uniformly sampled noise levels for robustness throughout denoising.Sigma-stratified analysis shows feature activations vary by less than 15% across noise levels.Source: §C.1 · p. 22

C.2 Prediction Formation via Diffusion Logit Lens

• The diffusion logit lens reveals gradual two-dimensional crystallization in small models but sharp layer boundaries in large models, with masked positions progressively encoding future tokens.MDLM reaches 86.9% final agreement, while masked-position agreement rises from 3.6% to 53.1%; sigma conditioning raises MDLM agreement from about 8% to 87%.Source: §C.2 · p. 23, §C.2.2 · p. 24, §C.2.3 · p. 24

C.3 SAE Layer Selection

• Layer selection combines probing, SAE training, contrastive feature identification, and steering ablation rather than relying on representational accuracy alone.The protocol tests sentiment and topic across layers and evaluates causal steering through positive-rate swings over intervention strengths.Source: §C.3 · p. 25, §C.3.1 · p. 25

C.3.2 Layer Probing Results

• All four models show probing accuracy rising with depth before saturation, while larger models begin from higher baselines and achieve over 97% in upper layers.MDLM plateaus around L8–11 at approximately 84%, whereas Dream and LLaDA plateau earlier in relative depth.Source: §C.3.2 · p. 26
• SEDD temporarily drops from an 81.5% L0 baseline to 77–79% at L1–5 before recovering through L6–11.This U-shaped profile is reported as a characteristic difference from MDLM.Source: §C.3.2 · p. 26
• Topic probing peaks later than sentiment, reaching 79.1% at MDLM L11, 95.3% at Dream L23, and 95.7% at LLaDA L28.The later peak is consistent with topic requiring deeper processing as a global, multi-token property.Source: §C.3.2 · p. 26

C.3.3 Steering Ablation: MDLM

• Steering swing decreases 3.6 percentage points per layer shift on average, despite probing accuracy peaking at later MDLM layers.MDLM layers 5–7 achieve the strongest steering, with a 74.2-point sentiment swing.Source: §C.3.3 · p. 26, §C.3.4 · p. 27
• Earlier-layer perturbations steer more strongly because residual-stream changes traverse more subsequent transformer blocks before output decoding.Later interventions may be partially undone by final layer normalization and unembedding.Source: §C.3.3 · p. 27

C.3.4 Layer Selection per Model

• SEDD reverses MDLM’s probing–steering dissociation: L8–10 reach a 90.8-point swing versus 38.8 points at L5–7, but the controlled comparison retains L5–7.The contrast is attributed to the training objective rather than architecture, since MDLM and SEDD share architecture and data.Source: §C.3.4 · p. 27
• For larger models, Dream samples layers across its transition and plateau regions, while LLaDA shows strongest individual features at L14 but strongest causal steering at L26.LLaDA’s L26 produces a 67-point steering swing, illustrating a milder probing–steering dissociation.Source: §C.3.4 · p. 27

C.3.6 Summary

• The study concludes that probing–steering dissociation is loss-dependent and that middle layers often provide the best balance of richness and propagation distance for absorbing-loss DLMs.The principle is reported across scales from 124M to 8B and across GPT-2, Qwen2, and LLaMA architectures.Source: §C.3.6 · pp. 27–28

C.3.5 SAE Training Hyperparameters

• SAEs use TopK activation with auxiliary dead-neuron loss and are trained on activations sampled across masking rates to generalize across denoising stages.The supplied passages specify a uniform noise-sampling strategy and identify the associated SAE configuration table.Source: Table 10 · p. 27, §C.4 · p. 28, §C.4.1 · p. 28

C.4.2 Results

• Median coefficient of variation in feature discriminability ranges from 13% to 22% across layers and attributes.The coefficient of variation is computed for absolute Cohen’s d across five equal-frequency sigma bins.Source: §C.4.2 · p. 28, Table 12 · p. 28
• Most features generalize across denoising stages: 65–95% have CV below 25%, while 30–55% have CV below 15%.Lower CV indicates more noise-robust discriminability.Source: §C.4.2 · pp. 28–29

C.4.3 Per-Feature Sigma Profiles

• Higher-variation features show structured sigma dependence, with some strengthening under heavier masking and others weakening as fine-grained lexical or semantic cues emerge near the final form.Representative per-bin sentiment profiles at Layer 7 illustrate these distinct dependence patterns.Source: §C.4.2 · p. 29, §C.4.3 · p. 29, Table 13 · p. 29

C.4.4 Directional Analysis

• Roughly half of features strengthen and half weaken as noise increases, with topic features showing a slight tendency to weaken.The split is approximately balanced for sentiment and topic across layers, including sentiment ratios of 50/50, 55/45, and 40/60 and topic ratios of 35/65, 45/55, and 35/65 at L5–L7.Source: §C.4.4 · p. 29
• Low-noise features encode fine-grained semantic content, whereas high-noise features capture coarse distributional properties inferred from corrupted input.Sentiment-bearing lexical items are identified as an example of low-noise content, while topic or register are examples of high-noise properties.Source: §C.4.4 · p. 29

C.4.5 Implications for Steering

• Because discriminability varies across denoising, sigma-adaptive feature weights could target interventions more precisely than uniform steering, but this is left for future work.The infrastructure for step-conditional steering is demonstrated in the denoising dynamics experiments.Source: §C.4.5 · p. 29

C.5.2 Feature Extraction Procedure

• Feature extraction contrasts labeled class pairs across mask ratios {0.3, 0.5, 0.7}, estimates full-ReLU SAE activations and Cohen’s d, and retains statistically validated directional features for steering.The procedure uses streaming statistics, Bonferroni-corrected Mann–Whitney U tests, and the top 20 features per direction for steering.Source: §C.5.2 · pp. 30–31

C.6 Temporal Dynamics Tracking

• Dynamics tracking records top-feature activations across denoising, filters for classifier-confident samples, and converts nonnegative blockwise activation changes into normalized emergence fractions.These fractions provide the activity signal used to weight adaptive steering across temporal blocks.Source: §C.6 · pp. 31–32

C.7.1 Steering Mechanism

• The adaptive schedule scales steering by each temporal block’s emergence fraction, concentrating intervention where the model is actively making semantic decisions.The steering hook applies calibrated feature shifts through an encode–modify–decode–replace cycle.Source: §C.7.1 · p. 32, §C.7.2 · p. 33

C.7.2 Steering Modes

• Four steering modes combine uniform or adaptive schedules with raw or E-scaled strengths, where E ratios normalize attribute-specific difficulty in multi-attribute control.E ratios are calibrated at the single-attribute strength reaching approximately 80% classifier rate, with the hardest attribute assigned E=1.0.Source: §C.7.2 · p. 33

C.7.3 Attribute Combinations

• Evaluation covers single- and multi-attribute combinations, activation-space and prompt-based baselines, and quality-gated classifier confidence, perplexity, diversity, and repetition metrics.The best operating point maximizes target confidence or multi-attribute geometric-mean confidence subject to dist-2 ≥50% of baseline and PPL < 100.Source: §C.7.3 · p. 33, §C.8 · p. 33, §C.8.2 · p. 34

D.3 Cumulative Effect Size Across Models and Layers

• Topic features consistently dominate cumulative |Cohen’s d| by feature rank across four models and their SAE layers, indicating concentrated discriminative signal in the leading features.Figure 8 sorts features by absolute effect size and displays the top 50 across a 4 × (3–4) model-layer grid.Source: §D.3 · p. 37, Figure 8 · p. 38

D.4 Anticipatory vs. Reactive Encoding

• Anticipatory encoding varies substantially by model and layer: topic is generally anticipatory, sentiment is often reactive shallowly, and all signals converge toward 0.5 late in denoising.MDLM shows early anticipatory topic and deeper-layer anticipatory sentiment; LLaDA transitions from anticipatory to reactive topic, while DREAM remains near 0.5 for roughly the first 75%.Source: §D.4 · p. 39
• Anticipatory features can guide early interventions on unrevealed tokens, whereas reactive features are better suited to reinforcing or modifying content already visible later.The distinction follows whether activations favor masked or unmasked positions during denoising.Source: §D.4 · pp. 38–39

D.5 Cross-Model Feature Cosine Similarity

• Mean best-match cosine is 0.08 between MDLM and SEDD, with a maximum pairwise similarity of only 0.133, indicating no aligned feature pairs across their different losses.Both models use matched layers and the same hidden dimension, making direct decoder-space comparison possible.Source: §D.5 · pp. 39–41
• Same-loss-family pairs show stronger alignment: MDLM–Dream averages 0.44 with a 0.88 maximum, while MDLM–LLaDA averages 0.25 with a 0.71 maximum.The comparison spans large architecture and scale differences, so alignment persists beyond identical model structure.Source: §D.5 · pp. 39, 41
• Alignment is directionally asymmetric: Dream→MDLM reaches 0.52 and 0.57 versus 0.33 and 0.36 for MDLM→Dream, consistent with Dream’s more specialized features.The reverse-direction difference is smaller for MDLM–LLaDA, suggesting weaker specialization in LLaDA than in Dream.Source: §D.5 · p. 41
• The overall mean across each 50 × 50 similarity matrix remains below 0.02 in magnitude, indicating that high best matches reflect feature correspondence rather than global projection bias.For different hidden dimensions, decoder directions are projected through each model’s unembedding matrix into shared vocabulary space.Source: §D.5 · pp. 40–41, Table 24 · p. 41
• Topic features align more than sentiment features, with MDLM–Dream similarities of 0.47 versus 0.42 and MDLM–LLaDA similarities of 0.29 versus 0.22.The analysis attributes this pattern to topic’s tighter lexical grounding compared with sentiment’s subtler evaluative vocabulary.Source: §D.5 · p. 41

F.1 Steering Trade-off Curves

• Figure 10 shows MDLM’s clearest Adaptive–Uniform separation: Adaptive matches or exceeds Uniform in steering while maintaining substantially lower PPL across α.The figure plots target confidence against α above GPT-2 perplexity against α.Source: Figure 10 · p. 42, §F.1 · p. 42

F.2 Feature Count Ablation

• At N=20 features, all models achieve strong sentiment control (71–84%) with modest interference; at N=50, SEDD formality shifts from 57.7% baseline to 94.5% and MDLM formality rises to 67%.We therefore use N=20 features throughout.Source: §F.2 · p. 43

F.4 Latency Overhead

• SAE steering adds 5–51% runtime overhead, ranging from 1.05× on SEDD to 1.51× on LLaDA, whereas contrastive-vector steering is approximately 1.00×.The relative overhead depends on model size, step count, and the cost of each model’s SAE computation.Source: §F.4 · p. 44, Table 28 · p. 46

F.5 Qualitative Steering Examples

• On MDLM and SEDD, Adaptive matches Uniform’s target confidence at 5–6× lower perplexity and produces coherent text instead of Uniform’s garbled outputs above PPL 250.On LLaDA and DREAM, all SAE modes shift attributes from near-chance baselines while maintaining low perplexity; E-ratio variants achieve the highest simultaneous control.Source: §F.5 · pp. 44–45

F.3 Cross-Attribute Interference

• Adaptive produces less cross-attribute interference than Uniform across models and steering conditions, with the largest gap on MDLM.Interference is measured as mean absolute percentage-point deviation of non-target classifier confidences from unsteered baselines.Source: §F.3 · p. 43, Figure 14 · p. 45

Broader Impacts

• The steering capability has a dual-use boundary: white-box adversaries could shift outputs toward biased or deceptive content while preserving fluency, so the released tools target benign attributes and require retraining for deployment.The paper frames interpretability as supporting audit and diagnosis while acknowledging manipulation risks.Source: Broader Impacts · p. 45