Extracting Latent Steering Vectors from Pretrained Language Models

Nishant Subramani; Nivedita Suresh; Matthew E. Peters

Source-linked AI summary

Extracting Latent Steering Vectors from Pretrained Language Models

Nishant Subramani, Nivedita Suresh, Matthew E. Peters

arXiv:2205.05124v1May 10, 2022cs.CLcs.AIcs.LG

TL;DR

Controllable generation typically learns control mechanisms, but this paper asks whether pretrained decoders already encode the information needed to steer toward target sentences. It extracts fixed-length latent vectors from frozen language models and injects them into hidden states, achieving near-perfect recovery while supporting sentiment transfer, similarity evaluation, and latent-space analysis. The authors conclude that frozen language models can be controlled through their latent steering space, with important bias and toxicity risks.

Problem
Prior work learns controllability through trainable decoding, prompt design, fine-tuning, or learned latent spaces; the paper investigates whether steering information is already encoded in pretrained decoders.
Method
The method optimizes fixed-length steering vectors directly against a frozen pretrained language model and adds them to hidden states during decoding.
Results
Across English sentences from varied domains, steering vectors enable near-perfect recovery; vector arithmetic performs comparably for Yelp sentiment transfer, while vector distances outperform pooled hidden states on STS-B.
Takeaways & Limitations
The results suggest that frozen pretrained language models can be effectively controlled through a latent steering space whose vectors also support semantic and style-related operations.
Takeaways & Limitations
The method can reflect biases in pretrained models, does not guarantee toxic-content removal, and is not recommended for high-stakes settings.

Abstract

from arXiv · show

Prior work on controllable text generation has focused on learning how to control language models through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. We hypothesize that the information needed to steer the model to generate a target sentence is already encoded within the model. Accordingly, we explore a different approach altogether: extracting latent vectors directly from pretrained language model decoders without fine-tuning. Experiments show that there exist steering vectors, which, when added to the hidden states of the language model, generate a target sentence nearly perfectly (> 99 BLEU) for English sentences from a variety of domains. We show that vector arithmetic can be used for unsupervised sentiment transfer on the Yelp sentiment benchmark, with performance comparable to models tailored to this task. We find that distances between steering vectors reflect sentence similarity when evaluated on a textual similarity benchmark (STS-B), outperforming pooled hidden states of models. Finally, we present an analysis of the intrinsic properties of the steering vectors. Taken together, our results suggest that frozen LMs can be effectively controlled through their latent steering space.

1 Introduction

The paper proposes extracting latent steering vectors directly from frozen pretrained language-model decoders, replacing trainable controllability approaches with hidden-state intervention. These vectors support near-perfect sentence recovery and broader semantic manipulation, though similarity performance remains below specialized lexical and fine-tuned methods.

Prior controllable-generation methods rely on trainable decoding, prompt design, fine-tuning, or learned latent spaces.
The method extracts fixed-length steering vectors from pretrained decoders and adds them to hidden states to generate target sentences without fine-tuning.
Vector arithmetic enables unsupervised Yelp sentiment transfer with performance comparable to carefully designed autoencoder-based models.
Steering-vector distances outperform pooled hidden states and GloVe vectors on STS-B cosine-similarity evaluation, but fall short of lexical and natural-language-inference-fine-tuned methods.
Interpolations and cross-domain clustering reveal meaningful latent-space structure, while analyses suggest the method leverages the language model rather than merely memorizing target sequences.

2 Extracting Steering Vectors

The method optimizes a fixed-length vector against a frozen autoregressive language model so that hidden-state injection steers greedy decoding toward a specified sentence. Its design supports comparing sentences in a common representation space while varying where and when intervention occurs.

The paper uses GPT2-117M and proposes applying the approach to transformer-based autoregressive language-model decoders.
A randomly initialized fixed-length zsteer is optimized by gradient descent to maximize the target sentence likelihood while the language model remains frozen.
Fixed-length vectors permit comparisons across sentences of different lengths in a shared representation space.
When d′ < d, a randomly initialized, fixed semi-orthogonal Wsteer projects the steering vector while preserving scale and is never trained or updated.
The experiments vary injection locations—including embeddings, transformer sublayers, and the language-model head—and injection at the first timestep versus every timestep.
At decoding time, the model receives a beginning-of-sentence token and zsteer, then generates with greedy decoding until an end token or 1024 tokens.
The extraction procedure takes a target sentence, pretrained model, injection settings, and vector dimension, and outputs a candidate steering vector.

3 Can we extract steering vectors?

Experiments evaluate sentence recovery across domains, sequence lengths, injection settings, and random initializations. Recovery is nearly perfect when steering occurs in middle transformer layers, and multiple solutions for the same sentence exhibit robust geometric structure.

3.1 Experimental setup: The recovery dataset combines movie dialogs, classic books, news articles, and Wikipedia, with sentences sampled across eight length bins from 5–10 to 40–128 words.
3.1 Experimental setup: Sentence recovery is measured by greedily decoding with zsteer and computing smoothed BLEU-4 against the target sentence.
3.2 Recovery effectiveness: Perfect recovery occurs for sequences up to 128 tokens when zsteer is injected at layers 6 or 7 of 12, either at the first timestep or every timestep.
3.3 Robustness: TSNE projections separate steering vectors by sentence while clustering different seeds for the same sentence, and averaged vectors achieve BLEU-4 99.4.
3.2 Recovery effectiveness: Middle transformer layers perform best, whereas embedding-layer and final language-model-head injection cannot steer GPT-2 effectively.
3.3 Robustness: For 63 of 64 book sentences, all eight random initializations recover the target perfectly, confirming robustness across initializations.

4 Is unsupervised style transfer in the latent steering space possible?

The paper tests whether arithmetic in latent steering space can transfer sentiment without supervised task-specific training. Averaged source- and target-style vectors define an offset that is added to sentence steering vectors, yielding performance comparable to prior models while using few labeled examples.

Method: Vector arithmetic adds a scaled style-transfer offset to each sentence’s steering vector to change the generated sentiment.The offset is computed from average steering vectors for source and target styles, then scaled by λ.
Data and evaluation: The Yelp evaluation measures sentiment-flip accuracy and Self-BLEU while varying λ across 0.25 to 10.0.Steering vectors perform comparably to prior work in the accuracy-versus-Self-BLEU trade-off.
Results: Comparable performance to Shen et al. (2017) is achieved by fully unsupervised steering-vector arithmetic against their supervised autoencoder-based model.The method also compares well with several unsupervised autoencoder systems, though unsupervised machine-translation methods outperform these approaches.
Data and evaluation: 100 labeled examples per class suffice to compute the source and target averages, while 10 examples per class remain competitive with autoencoder baselines.The evaluation uses 100 validation sentences from each sentiment class and tests performance on Yelp.
Qualitative behavior: Increasing λ generally makes outputs more positive or negative and often swaps adjectives, but higher values can reduce fluency through repeated words or phrases.Negative-to-positive transfer is described as more fluent and accurate than positive-to-negative transfer.

5 Do distances between steering vectors reflect sentence similarity?

The paper evaluates whether distances between extracted steering vectors track semantic similarity on STS-B. Cosine similarities correlate with annotator judgments, outperform extractive baselines, but remain below methods tailored through NLI training or specialized lexical processing.

Method: STS-B evaluation extracts one steering vector per sentence, computes pairwise cosine similarity, and correlates it with annotator similarity using Spearman rank correlation.The figure reports Spearman correlation scaled by 100 on the test split.
Layer analysis: Middle transformer layers, especially the 7th self-attention and 7th feedforward layers, provide the strongest steering-vector semantic-similarity performance.These configurations also correspond to the paper’s two best-performing settings selected for comparison in Table 3.
Results: Extracted steering vectors outperform mean-pooled final hidden states from GPT2-117M and BERTbase and averaged GloVe vectors.The comparison uses cosine similarities between extracted vectors and the corresponding baseline representations.
Results: The method outperforms prior extractive unsupervised approaches but performs worse than NLI-finetuned and specialized lexical methods.Table 3 compares Spearman and Pearson correlations across extractive, NLI-finetuned, and lexical method classes.

6 Analysis of Properties

The analysis examines steering-space smoothness, intrinsic dimension, storage, sampling, memorization, and injection design. Results suggest meaningful interpolation and model use, but reliable sampling requires more than independent Gaussian dimensions.

6.1 Interpolation: Interpolated steering vectors produce grammatical sentences that combine the content of two sentences while changing sentiment and meaning incrementally.In one pair, positive sentiment persists through λ = 0.7; another combines “four” with “years ago” at λ = 0.3, 0.4.
6.2 Sampling: 5 of 24 Gaussian samples produced fully formed sentences, while 19 produced only tokens or phrases.Independent Gaussian modeling of steering-vector dimensions was therefore not reliable for sampling well-formed text.
6.3 Intrinsic Dimension & Space Complexity: 768 dimensions may be needed for nearly perfect recovery, while 384 dimensions achieved a reconstruction BLEU of 83.29.Reconstruction BLEU increased with steering-vector dimension; the reported intrinsic dimension is at most 768, and shorter sentences are easier to recover at lower dimensions.
6.3 Intrinsic Dimension & Space Complexity: A 384-dimensional fp16 steering vector requires 768 bytes, less than the estimated 896-byte string representation for a 128-word sequence.Steering vectors do not depend on sequence length, though compression sacrifices some recovery.
6.4 Memorization: Recovery is highest for books, then shuffled text, and lowest for gibberish, indicating steering vectors are not simply memorizing sequences and encode some word-order information.The comparison used injections into the sixth transformer layer after self-attention, either across all timesteps or only at the first timestep.
6.5 Prompt-Based Steering: Prompt-based concatenation of steering vectors yields much lower reconstruction BLEU than injecting a single steering vector into transformer layers.The prompt-based experiment used 768-dimensional prompt vectors on the books subset.

7 Related Work

Related work commonly learns sentence representations and controllers with new encoder-decoder or autoencoder models. This paper instead extracts vectors directly from pretrained decoders and extends the idea to unsupervised style transfer.

7 Related Work: Prior controllable-generation approaches encode sequences, transform their representations, and decode them using learned encoders, controllers, and decoders.The cited approaches include denoising and variational autoencoders, often using disentangled representations.
7 Related Work: This work extracts steering vectors directly from pretrained language-model decoders without training new models, then applies vector arithmetic to unsupervised style transfer.The authors describe this as extending beyond extracting sentence representations to manipulating the latent space.

8 Conclusion

The paper introduces latent steering vectors extracted from pretrained language models without fine-tuning and evaluates their use for controllable generation. Across recovery, style transfer, similarity, and latent-space analyses, the results support effective control of frozen models.

8 Conclusion: The method extracts latent steering vectors directly from pretrained language models without fine-tuning.The vectors are used to control generation by leveraging information encoded in the pretrained model.
8 Conclusion: The vectors enable near-perfect recovery across English sentences from varied domains, comparable unsupervised style transfer on Yelp and StylePTB, and stronger STS-B performance than extractive methods.The conclusion also reports that vector distances reflect sentence similarity.
8 Conclusion: The results indicate that frozen pretrained language models can be controlled effectively through their latent steering space.

9 Ethics Statement

The ethics statement emphasizes that extracted steering vectors inherit biases from the underlying pretrained language model. It also warns that the technology may be misused to generate offensive or toxic text and should not be deployed in high-stakes settings where harm could result.

9 Ethics Statement: Because steering vectors leverage information encoded in pretrained models, they can reflect biases already present in those models.The authors identify potential downstream uses alongside the risk of perpetuating bias or generating offensive or toxic text.
9 Ethics Statement: The technology may be misused to perpetuate biases or generate offensive or toxic text.
9 Ethics Statement: The method does not guarantee toxic-content removal, including during unsupervised style transfer from toxic to nontoxic text.The authors encourage addressing underlying-model biases before use.
9 Ethics Statement: The authors recommend against using the technology in high-stakes settings, especially where deployment could cause harm.

A.1 Extracting steering vectors

The section evaluates how steering-vector extraction and injection choices affect sentence recovery, then examines vector arithmetic, sampling, and style-transfer performance across benchmarks.

Injection configuration: Middle-layer injection achieves the best reconstruction BLEU-4, while nearly every layer except the first and last provides nearly perfect recovery.
Sentiment transfer: Increasing λ strengthens sentiment transfer, often switching sentiment at λ = 1.5 and sometimes adding content beyond flipping a major adjective.One example adds the phrase "a great way to get a good laugh" in a negative-to-positive generation at λ = 2.5.
Injection configuration: Injecting steering vectors throughout the transformer stack performs best, while embedding- or head-only injection performs poorly.Injecting at just the first timestep causes only a negligible recovery decrease compared with injecting at all timesteps.
Style transfer: Unsupervised vector arithmetic with steering vectors performs comparably to supervised methods on StylePTB tasks requiring minimal edits.The reported tasks include adjective emphasis, active-to-passive transformation, information addition, and prepositional-phrase fronting.
Sampling: Sampling steering vectors by independently drawing each dimension from a fitted normal distribution produces fully formed sentences in approximately 20% of generations, with the remainder being words or short phrases.