Source-linked AI summary
Transmuting prompts into weights
Hanna Mazzawi, Benoit Dherin, Michael Munn, Adrian Goldwaser, Michael Wunder, Javier Gonzalvo
TL;DR
Inference-time steering and weight editing can control LLM behavior, but their development has largely relied on empirical heuristics without a unified theoretical justification. This paper aggregates transient prompt effects into reusable, token-independent thought vectors and matrices. It reports performance comparable to full-context prompting on algorithmic tasks and new-knowledge injection, and close detoxification performance to the full-context baseline.
Problem
Activation steering and targeted weight edits can control LLMs at inference time, but their development has largely been heuristic and lacks a unified transformer-grounded theoretical justification.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The framework offers a principled, data-efficient way to encode prompt instructions in weights and explains contrastive averaging and low-rank updates as transformer instruction-encoding mechanisms.
Source: §5 · p. 5The current least-squares formulation is susceptible to overfitting specific prompt structures; performance degrades significantly when retrieval templates differ from those used to compute patches.
Source: §4 · pp. 4–5
Abstract
from arXiv · showhide
A growing body of research has demonstrated that the behavior of large language models can be effectively controlled at inference time by directly modifying their internal states, either through vector additions to their activations or through updates to their weight matrices. These techniques, while powerful, are often guided by empirical heuristics, such as deriving ``steering vectors'' from the average activations of contrastive prompts. Building on the foundational work of Dherin et al. (2025), who discovered that a prompt's influence mathematically maps to token-dependent implicit weight updates and introduced the initial concept of a static thought patch for prompt compression, we elevate this framework into a robust algorithm for direct model editing. We derive a principled method for condensing this transient information into token-independent thought vectors and thought matrices. These constructs provide a theoretical explanation for existing vector-and-matrix-based model editing techniques and offer a direct, computationally-grounded method for transmuting textual input into reusable weight updates for complex architectures and new knowledge injection.
1 Introduction
Inference-time LLM control uses activation steering and weight edits, but these approaches have largely lacked a unified theoretical justification. This work extends the thought-patch idea into reusable, token-independent updates and connects transformer computation to model control.
1 Introduction
- Inference-time control commonly uses activation steering or targeted low-rank weight edits, yet their development has largely been heuristic.The paper identifies the lack of a unified theoretical justification rooted in transformer architecture.Source: §1 · p. 1
- Dherin et al. (2025) showed that a prompt’s effect can be replicated by token-dependent rank-one feed-forward weight updates called “minimal token-patches.”They also introduced a least-squares thought-patch proof of concept for prompt compression on simple linear-regression tasks.Source: §1 · pp. 1–2
- This work generalizes transient patches into reusable, token-independent thought vectors δ(I) and thought matrices ∆(I) for modern nonlinear architectures such as Gemma.The authors frame these constructs as a practical algorithm for direct model control and permanent knowledge injection.Source: §1 · p. 2
- The framework connects transformer computation to model control by explaining how contrastive activation averaging and low-rank weight updates arise.It empirically validates instruction-to-weight transmutation on algorithmic tasks and new-knowledge injection, with performance comparable to full-context prompting.Source: §1 · p. 2
- The work bridges activation-steering methods and model editing with a first-principles strategy for condensing prompt information into reusable weight updates.It adapts Dherin et al. (2025) to modern transformers using techniques from Goldwaser et al. (2025).Source: §1 · p. 2
2 Transmuting Prompts to Thought Patches
The section develops token-independent thought patches by approximating token-dependent prompt effects across representative examples. It shows that the optimal thought vector is the mean of token vectors and that the thought matrix emerges as a sum of rank-one updates.
2 Transmuting Prompts to Thought Patches
- Dherin et al. (2025) showed that a context chunk’s effect can be replicated on a shortened prompt using token-dependent updates to the last-layer bias and first-layer weight matrix.The thought-patch formulation aggregates these transient prompt effects into reusable updates.Source: §2 · p. 2
- A single token-independent thought patch (δ(I), ∆(I)) is fitted to minimize error against token-specific patches across representative examples.This replaces patches that must otherwise be recomputed for every token and are impractical for durable storage.Source: §2 · p. 2
- The optimal thought vector minimizes squared error to individual token vectors and is their mean, justifying contrastive-activation averaging.The paper identifies this as the heuristic used in standard activation engineering.Source: §2 · p. 2
- The optimal thought matrix naturally emerges as a sum of rank-one updates, each formed as an outer product.The section frames its estimation as minimizing the thought-matrix error across examples.Source: §2 · p. 2
3 The Thought Patching Algorithm
The algorithm aligns internal representations from instructed and raw inputs, then iteratively updates each layer’s thought vector and matrix. Batching, a tunable step size, and L2 regularization are introduced to improve stability and generalization across prompts.
3 The Thought Patching Algorithm
- The layer-wise procedure aligns the model’s internal representations for raw input x and instructed input [I, x].For each compatible task-completion example, it makes contextual and non-contextual forward passes; the contextual pass supplies target attention outputs and the non-contextual pass uses the currently patched weights.Source: §3 · p. 3
- At each layer, the attention-output discrepancy determines the thought-vector update by its mean and the thought-matrix update by least squares.The method computes the discrepancy between contextual target outputs and non-contextual outputs.Source: §3 · p. 3
3.1 Optimization Strategies
- Batching pools attention values across completions to reduce variance in thought-patch estimation.The least-squares optimization is then performed simultaneously on the pooled tensors.Source: §3.1 · p. 3
- A tunable step size η scales each weight update rather than applying the patch directly.The update is gradually encoded into the weights.Source: §3.1 · p. 3
- L2 regularization (Ridge Regression), with strength ρ, is used to prevent the thought matrix from overfitting batch-specific completions.The regularized estimation is formulated as a least-squares minimization.Source: §3.1 · p. 3
4 Experimental Validation
Thought patches matched full-prompt performance on arithmetic and linguistic tasks, including detoxification, translation, and knowledge retrieval. They also absorbed new information, though performance worsened with larger dictionaries and mismatched retrieval templates.
Arithmetic and Linguistic Tasks
- Thought-patched Gemma without instructions perfectly matched full-prompt performance on fundamental algorithmic tasks using 10 examples.Table 1 reports performance on arithmetic instruction tasks.Source: §4 · p. 3, Table 1 · p. 3
Detoxification
- Detoxification reduced toxicity from 90.53% to 6.22%, close to the 6.7% full-context baseline while preserving semantic BERT scores.Table 2 reports thought-patch performance on detoxification.Source: §4 · p. 3, Table 2 · p. 4
Translation and General Knowledge
- Without instructions, the modified model performed highly comparably to full prompting and significantly better than the unpatched baseline on translation and country-to-capital retrieval.Table 3 covers linguistic and knowledge tasks.Source: §4 · p. 4, Table 3 · p. 4
New Knowledge Integration
- Thought patches absorbed new information into MLP weights, but performance degraded with larger dictionaries and more sharply when retrieval templates differed from patch-computation templates.The least-squares formulation is susceptible to overfitting on specific prompt structures.Source: §4 · pp. 4–5, Table 4 · p. 5
5 Conclusion
The conclusion presents reusable thought vectors and matrices as a way to encode prompt instructions in weights and explain heuristic control methods. The Gemma adaptation accounts for its normalization and gated feed-forward structure.
5 Conclusion
- Aggregating ephemeral token patches into reusable thought vectors and matrices provides a data-efficient method to durably encode prompt instructions into weights.The framework links transient transformer inference mechanics to permanent model modification and explains contrastive averaging and low-rank updates as instruction-encoding mechanisms.Source: §5 · p. 5
A Algorithm modification for Gemma
- Gemma's RMSNorm and Gated GeLU feed-forward network require adaptations to the thought-patching algorithm.The Gemma block differs from a vanilla transformer block primarily in these components.Source: A Algorithm modification for Gemma · p. 9
A.1 Adapting the Thought Matrix for Gated GeLU and Pre-normalization
- For Gemma, thought-matrix updates use activations after RMSNorm, and the gated architecture requires two independent least-squares estimations.This lets the thought patch influence both the gating mechanism and the content projection.Source: §A.1 · p. 9
A.2 Absorbing the Thought Vector into Weights and Scale
Because Gemma lacks an MLP output bias and applies RMSNorm after the MLP, the thought-vector shift must be absorbed into the down-projection weights. The method adjusts the target for normalization, constrains its magnitude, and solves for a least-squares weight update.
A.2 Absorbing the Thought Vector into Weights and Scale
- The method absorbs the thought-vector shift into Gemma’s down-projection weights rather than adding a bias tensor.Gemma’s MLP output is a linear projection without an explicit bias parameter.Source: §A.2 · p. 9
- The target activation incorporates the desired shift adjusted for the layer’s RMSNorm scale.The adjustment uses element-wise division so the shift aligns with the layer’s activation distribution.Source: §A.2 · pp. 9–10
- A magnitude constraint preserves the target direction while matching the magnitude of contextual activations, and the projection update is found by least squares over hidden activations.The constraint is intended to prevent significant changes to the MLP output norm.Source: §A.2 · p. 10
- Updating only Wdown while leaving the RMSNorm scale unchanged was sufficient for high-fidelity instruction following and knowledge absorption and helped maintain performance.The formulation also allows adjusting the RMSNorm scale, but the reported empirical finding concerns updating Wdown alone.Source: §A.2 · p. 10
B Experiment Details and Algorithm
- Experiments align activations from a contextual pass containing the instruction with a non-contextual pass containing only the input data.The experiments used Gemma 3.0 1B; Gemma’s RMSNorm and GeLU gating motivate algorithm modifications.Source: B Experiment Details and Algorithm · p. 11
B.1 Arithmetic and Linguistic Tasks
The experiments assess whether thought patches can retain instructed behavior without the instruction, using arithmetic and detoxification tasks. The arithmetic patches reached perfect peak test accuracy, while the detoxification setup used a small training set and compared patched and full-context models.
B.1 Arithmetic and Linguistic Tasks
- The arithmetic tasks were sums or products of three single-digit integers, chosen so the unpatched model achieved 100% accuracy with the instruction.The contextual instructions were “Sum the numbers:” and “Multiply the numbers:”.Source: §B.1 · p. 11
- 100% peak test accuracy was achieved for both multiplication and summation across five seeds, using 10 training completions per task.The held-out set contained 20 randomly regenerated examples at each step; convergence to 100% training accuracy occurred after 10 steps.Source: §B.1 · p. 11
B.2 Detoxification
- The detoxification evaluation compared the original model with its detoxification instruction against the patched model without that instruction, using 10 ParaDetox training examples.The patch used one training step, a learning rate of 1.0, and no regularization.Source: §B.2 · p. 11
B.3 Translation, Knowledge and Algorithmic Tasks
The experiments evaluate patches for dictionary translation, country-capital retrieval, first-letter prediction, and new synthetic knowledge. For new-knowledge retrieval, performance depended on whether evaluation queries matched those used to compute the patch.
B.3 Translation, Knowledge and Algorithmic Tasks
- Translation patches were evaluated on 100 held-out French or Spanish words using semantic similarity, with the translation instruction omitted at evaluation.Each patch used 10 completions and no regularization.Source: §B.3 · p. 11
- Country-capital retrieval used five demonstrations and exact-match accuracy on 100 unseen country-capital pairs; first-letter prediction used five demonstrations and exact match on 100 held-out examples.Both tasks used a learning rate of 1.0 and no regularization.Source: §B.3 · p. 11
B.4 New Knowledge Integration
- Retrieval accuracy was measured for dictionary sizes k ∈ {1, 2, 4, 8, 16, 32}, and performance dropped significantly when evaluation queries differed from patch-computation queries.For k ≥ 16, the solver’s matrices were rank-reduced to 40 principal components; the passage identifies query mismatch as a limitation and points to tuning regularization.Source: §B.4 · p. 12, Table 5 · p. 12
C Prompt Experiment Details
The appendix specifies the prompts and alignment used in the experiments. It also describes varying dictionary query formats and reports a significant performance loss when evaluation queries differ from those used to compute the patch.
C Prompt Experiment Details
- The contextual and non-contextual prompts share the completion and generated answer; only the contextual prompt includes the instruction.This aligns hidden states for the same token sequence across the two passes.Source: C Prompt Experiment Details · p. 13
C.1 Detoxification Instructions
- The detoxification instruction asks for a single detoxified statement that preserves the original sentence’s semantic meaning.It explicitly disallows commentary or suggestions.Source: §C.1 · p. 13
C.2 Translation Instructions
- French-to-English and Spanish-to-English prompts request a single English-word translation of the supplied foreign word.The source word is inserted into the instruction template.Source: §C.2 · p. 13
C.3 Knowledge Retrieval Instructions
- Knowledge-retrieval prompts separately request the capital of a country or the country corresponding to a capital city.Both prompt forms request a single answer.Source: §C.3 · p. 13
C.4 Dictionary Formats and Divergent Queries
- Dictionary patches were evaluated using query phrasings that differed from those used to compute the patch, and this produced a significant performance loss.Table 6 reports performance on new knowledge evaluated with different queries.Source: §C.4 · p. 14, Table 6 · p. 14
- The dictionary evaluation used multiple ways of asking for a key’s value, including “what value corresponds” and “what is stored under the key.”These query variants differ from the patch-computation phrasing.Source: §C.4 · p. 14
D Thought Matrix Estimation Theorem
The thought matrix is estimated by minimizing errors between the matrix's outputs and token-specific updates. The theorem gives a unique global minimizer exactly when the summed token-vector operator is invertible.
D Thought Matrix Estimation Theorem
- The thought-matrix objective minimizes errors between M applied to token vectors and their token-specific updates; its minimum is unique when Z is invertible.Under that condition, the theorem states that the solution is a global minimum.Source: D Thought Matrix Estimation Theorem · p. 15
D.1 Invertibility Conditions for Z
- Z is invertible exactly when the y_i vectors span the full space; when they form a basis, its inverse has an explicit representation.For an orthonormal basis, Z and its inverse are both the identity.Source: §D.1 · p. 16
E.3 Spherical random distribution
- For spherical random vectors with covariance σ^2I, the inverse of Z is 1/(σ^2n)I when the sample size is large enough.The passage defines spherical distribution as invariance under orthogonal transformations.Source: §D.1 · p. 16, §E.3 · p. 19
D.2 Getting an approximation of M when Z is not invertible
- When Z is not invertible, the matrix fit is treated as a linear regression problem, and adding a small diagonal matrix can make Z invertible.The approximation uses a tunable hyperparameter when the diagonal matrix is chosen as a scalar multiple of the identity.Source: §D.2 · p. 16
E Useful properties of low-rank operators
- A low-rank operator formed from r outer products has rank at most r; with independent component vectors, its rank is r and its image is their span.Its kernel is the orthogonal complement of the span of the other component vectors when those vectors are independent.Source: §E.1 · p. 17
E.2 Independence, span, and basis
- For a basis of y_i vectors, Z is invertible; its inverse can be expressed using the rows of the basis matrix's inverse.With an orthonormal basis, Z and Z^-1 are the identity.Source: §E.2 · p. 18
F Related Work
The related work frames activation steering and weight editing as connected approaches to model control, while noting that their theoretical grounding has been limited. The paper presents an explicit mapping from prompt effects to reusable weight updates and relates it to existing steering heuristics and matrix edits.
Bridging Empirical Methods with Theory
- The framework connects activation steering with vectors and direct model editing with matrices as approaches along a prompt-to-weights continuum.The cited continuum describes information moving from prompts through activation vectors to model weights.Source: F Related Work · p. 20
Activation Steering with Vectors
- Vector-based steering benchmarks show that these methods are not always fully reliable, suggesting vector addition alone may incompletely represent an instruction's effect.The related work also describes sparse-autoencoder steering as targeting specific features.Source: F Related Work · p. 20
Model Editing with Matrices
- Matrix-editing methods such as ROME and MEND use low-rank weight updates to alter knowledge, a formal structure resembling the proposed thought matrices.The related work also notes matrix editing applications in safety controls and toxicity reduction.Source: F Related Work · p. 21
Bridging Empirical Methods with Theory
- A first-principles strategy for condensing generic prompts into reusable weight updates for standard transformers was missing; a prior exact conversion required architectural modifications.The paper presents its mapping as filling this gap.Source: F Related Work · p. 21