PaperlayerView exact revision on arXiv

Source-linked AI summary

Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMs

Zifeng Cheng, Lingyun Qian, Zhiwei Jiang, Cong Wang, Yafeng Yin, Fei Shen, Ao Zhou, Qing Gu

arXiv:2609.32684v1cs.CL

TL;DR

Prompt-based conditional embeddings can retain general text information, limiting their focus on the target condition. SCS refines them by contrasting conditional and unconditional embeddings. Across tasks, datasets, and LLMs, the paper reports improved performance over existing prompt-based methods.

  • Problem

    Prompt-based conditional embeddings can remain entangled with general textual information, affecting distance measurements and performance.

    Source: §1 · p. 2

  • Method

    SCS uses self-generated unconditional embeddings to refine conditional embeddings, modifying attention masks and positional encodings in a plug-and-play, inference-time method.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across clustering, STS, and triplet alignment on seven datasets, SCS improves existing prompt-based methods across different LLMs.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The experiments support training-free, plug-and-play refinement of prompt-based conditional embeddings across the tested tasks and LLMs.

    Source: §6 · p. 9

    SCS requires access to LLM internals and was not validated on larger models such as 70B models.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Extracting conditional text embeddings from large language models (LLMs) is a promising paradigm, as it requires neither additional data nor fine-tuning. Existing methods incorporate conditions into prompts to guide LLMs to focus on specific aspects and elicit conditional text embeddings. However, relying solely on prompts often fails to produce high-quality conditional text embeddings, as they remain entangled with general text embeddings, ultimately degrading their quality. To this end, we propose an inference-time, plug-and-play Self-Contrastive Steering (SCS) method that constructs unconditional general text embeddings and uses them to refine conditional text embeddings, making them more focused on the target condition. Specifically, we modify the attention mask and positional encodings to mask the condition, thereby obtaining unconditional text embeddings and intervening in the multi-head self-attention computation process. Notably, our method is highly efficient, requiring only a single additional multi-head self-attention computation at inference time. Extensive experiments on clustering, Semantic Textual Similarity, and triplet alignment datasets demonstrate that our method can seamlessly improve the performance of existing prompt-based methods across different LLMs in a training-free and plug-and-play manner. Our code will be released at https://github.com/zifengcheng/SCS

1 Introduction

Conditional text embeddings target specific aspects of text, but prompt-based embeddings can retain general information unrelated to the condition. The paper introduces Self-Contrastive Steering (SCS), which uses self-generated embeddings to refine conditional embeddings; experiments report improvements across datasets and LLMs.

1 Introduction

  • Conditional embeddings represent specific aspects of text, addressing fine-grained needs that a single universal embedding cannot satisfy.For example, two sentences may share gender-related meaning while differing in instrument.Source: §1 · p. 1
  • Prompt-based conditional embeddings can retain general textual information, with average cosine similarity to general embeddings reaching 0.763 on TweetEmotion.The authors report that this entanglement affects distance measurements and performance.Source: §1 · p. 2
  • SCS contrasts self-generated conditional and unconditional embeddings to focus representations on condition-related information and filter out general information.It modifies the attention mask and positional encoding to derive the unconditional embedding, then applies direct-elimination or projection-based steering.Source: §1 · p. 2
  • SCS creates multi-view text embeddings by modifying attention masks and positional encodings.The method is presented as a lightweight approach.Source: §1 · p. 2
  • Experiments across seven datasets covering clustering, STS, and triplet alignment report that SCS significantly improves prompt-based methods across different LLMs.The authors describe the approach as training-free and plug-and-play.Source: §1 · p. 2

2 Related Work

Related work spans fine-tuned conditional embeddings, prompt-based extraction from LLMs, and activation steering. SCS is positioned as a training-free approach that constructs input-related steering vectors with limited additional computation.

2 Related Work

  • Contrastive learning is a common route to general text embeddings; SimCSE uses dropout to generate positive samples for training.Other cited work has begun fine-tuning LLMs for text embeddings.Source: §2 · p. 2
  • Fine-tuned conditional embedding methods use instructions, annotated similarity data, generated content, or hypernetworks, but can require substantial data and incur fine-tuning costs.The cited approaches include InstructOR [Su et al., 2023], C-STS [Deshpande et al., 2023], InBedder [Peng et al., 2024], and HyperCL [Yoo et al., 2024].Source: §2 · p. 2
  • Prompt-based LLM methods extract embeddings without training, while PonTE adds conditions to PromptEOL but cannot ensure outputs contain only condition-relevant information.The passage also describes Echo and TP as alternatives addressing causal attention properties.Source: §2 · p. 3
  • Unlike activation-steering approaches that may use supervised pairs or auxiliary prompts, SCS constructs input-related vectors without supervised data and adds one MHSA computation.The passage characterizes activation steering as a training-free way to modify LLM activations to control generation.Source: §2 · p. 3

3 Preliminary

The preliminary section describes prompt-based conditional embedding extraction and the role of rotary positional encoding in LLM self-attention. PonTE compresses a text’s condition-specific meaning into a final-token representation.

3 Preliminary

  • PonTE adds a condition to a prompt asking the LLM to express a text in one word in terms of that condition.The prompt’s final-token hidden state is used as the conditional embedding without generation.Source: §3 · p. 3
  • The prompt placeholders denote the text and condition, and “in one word” encourages semantic compression into the next token’s hidden state.That state already encodes information about the next token for next-token prediction.Source: §3 · p. 3
  • RoPE encodes relative position by applying position-dependent rotations to query and key vectors before attention computation.The query and key projections are layer-specific; the rotation frequency is determined by a dimension-dependent constant.Source: §3 · p. 3

4 Method

SCS is an inference-time, plug-and-play method that refines conditional text embeddings by contrasting them with embeddings extracted while masking the condition. It combines prompt wrapping, condition-masked multi-view extraction, and self-contrastive activation steering, requiring only one additional MHSA computation.

4 Method

  • SCS is an inference-time, plug-and-play method that integrates with existing prompt-engineering techniques without additional data or LLM fine-tuning.It adjusts attention masks and positional encodings to derive conditional and unconditional text embeddings.Source: §4 · p. 3

4.1 Prompt Construction

  • SCS uses a prompt-wrapped text and condition to obtain hidden states from the layer before multi-view embedding extraction.The prompt asks the LLM to express the text in one word in terms of the condition.Source: §4.1 · pp. 3–4

4.2 Multi-View Embedding Extraction

  • The prompt-wrapped input yields conditional and unconditional embeddings; the latter masks the condition and shifts later positions to restore continuity.Only one additional forward pass through the MHSA layer is needed to extract the unconditional embedding.Source: §4.2 · pp. 4–5

4.3 Self-Contrastive Activation Steering

  • SCS refines conditional embeddings by contrasting them with self-generated unconditional embeddings, using direct elimination or projection-based elimination.The refined value vectors are fed into the MHSA output matrix before standard forward propagation continues.Source: §4.3 · p. 5

5 Experiments

The experiments evaluate conditional text embeddings across seven datasets, three tasks, multiple baselines, and varied models, prompts, and conditions. SCS improves baseline performance broadly, with analyses examining its efficiency and sensitivity to design choices.

5.1 Datasets and Experimental Settings

  • The evaluation covers clustering, STS, and triplet alignment across seven datasets, using V-measure, Spearman and Pearson correlations, and triplet alignment accuracy.The setup uses five-run averages for clustering and grid search over intervention layers and scaling factors.Source: §5.1 · pp. 5–6

5.2 Baselines

  • Baselines include supervised embedding models and training-free prompts; among prompt variants, PonTE improves on PromptEOL, while TP declines and Att and Echo have minimal impact.The supervised group includes SimCSE-RoBERTa-large, E5-Mistral-7B-Inst, and GTE-Qwen2-7B-Inst; prompt baselines include PonTE + TP, Echo, and Att.Source: §5.2 · p. 6, §5.3 · pp. 6–7

5.3 Results

  • SCS achieves the best performance in 6 of 7 cases, improves all seven datasets, and has the optimal average performance across three tasks.The simpler SCS-DE also achieves better average performance, and SCS is comparable to supervised contrastive learning approaches.Source: §5.3 · p. 6, Table 1 · p. 6

5.4 Analysis of Additional Inference Time Overhead

  • SCS introduces almost no additional runtime overhead per sample on two datasets, with fluctuations attributed mainly to randomness.The method computes an additional value vector in one MHSA layer; the paper notes that LLM computation is dominated by feed-forward networks.Source: §5.4 · p. 7, Table 2 · p. 7

5.5 Model Analysis

  • Intervention at the attention, FFN, and hidden positions improves performance, but optimal positions vary by dataset; layer and scaling-factor optima also differ across datasets.Layers 5 or 6 generally improve all three datasets; scaling factors 7, 10, or 12 perform well across them. FFN and hidden variants incur theoretically higher overhead, and hidden requires a full Transformer layer.Source: §5.5 · p. 7, Table 3 · p. 7, Figure 3 · p. 7
  • SCS-DE consistently improves PonTE across two datasets and middle and later output layers, with the optimal output layer shifting from PonTE’s best layers.PonTE peaks at layers 30 and 29, while SCS-DE peaks at layers 32 and 28, respectively.Source: §5.5 · pp. 7–8

5.6 Generalization Analysis

  • SCS improves results across different LLMs, six prompts, and six conditions; it also reduces variance across prompts.Instruction-tuned LLMs generally outperform their non-instruction-tuned counterparts, while prompts and conditions that were originally optimal are not necessarily best after steering.Source: §5.6 · p. 8

6 Conclusion

SCS is an inference-time, plug-and-play method for extracting conditional text embeddings without training or additional data. It creates multi-view embeddings and improves existing prompt-based methods across tasks, datasets, and LLMs.

6 Conclusion

  • SCS creates multi-view embeddings by modifying the positional encoding and attention mask.It uses two alternative self-contrastive activation steering strategies to refine conditional embeddings.Source: §6 · pp. 8–9
  • The method extracts conditional text embeddings at inference time without additional data or training.Source: §6 · p. 8
  • Across three tasks and seven associated datasets, SCS improves existing prompt-based methods across different LLMs in a training-free, plug-and-play manner.The method requires no additional data or LLM fine-tuning.Source: §6 · pp. 8–9

Limitations

The limitations section identifies internal access and model scale as boundaries, while the appendices provide prompts for the method and baselines.

Limitations

  • SCS requires intervention in LLM internals, so it is not applicable to black-box LLMs; the method is also unvalidated on larger models such as 70B.Source: Limitations · p. 9

A Prompts of Our Method

  • For I.STSB and C-STS, dataset-specified conditions determine the prompts; prompts for the remaining datasets include emotion, star rating, product category, toxicity, and news topic.Source: A Prompts of Our Method · p. 11

B Baselines

  • The baseline prompt appendix lists templates for PromptEOL, PonTE, and PonTE+Echo.Source: B Baselines · p. 11

C Results of Multi-layer Intervention

The multi-layer intervention results are reported in Table 8; the accompanying text highlights single-layer intervention as a strong, lower-overhead option.

C Results of Multi-layer Intervention

D Similarity Measurement Before and After Steering

Before steering, value vectors under PromptEOL and PonTE remain highly similar, indicating that PonTE retains general semantic information. Similarity decreases after steering, consistent with embeddings becoming more condition-focused.

D Similarity Measurement Before and After Steering

E Results on Other Languages

On Amazon-R, the method improves performance in German and Spanish, demonstrating cross-lingual generalization.

E Results on Other Languages

F Results on Larger LLMs

The method improves performance on larger LLMs, although larger models do not always perform better on conditional text embeddings. In a token-decoding case study, its outputs are instruction-relevant and more concentrated than PonTE’s.

G Case Study

  • Unlike PonTE’s decoded tokens, which include instruction-irrelevant items, the method’s tokens are instruction-relevant and have a more concentrated probability distribution.Source: G Case Study · pp. 11–12, Table 11 · p. 13

F Results on Larger LLMs

Loading 2609.32684v1…