PaperlayerView exact revision on arXiv

Source-linked AI summary

SelfIE: Self-Interpretation of Large Language Model Embeddings

Haozhe Chen, Carl Vondrick, Chengzhi Mao

arXiv:2403.10949v2cs.CLcs.AIcs.LG

TL;DR

LLMs are largely black boxes, limiting transparency into their decisions and whether they reason or repeat training data. SelfIE interprets hidden embeddings in natural language and supports targeted control; the authors report an 84.66% reduction in prompt-injection success after harmful-knowledge removal.

  • Problem

    LLMs provide limited transparency into how they make decisions, leaving open whether they reason or repeat training data.

    Source: §1 · p. 1

  • Method

    SelfIE inserts a hidden embedding into a separate interpretation forward pass and uses the LLM to generate a natural-language description.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    84.66% reduction in prompt-injection success after harmful-knowledge removal; SelfIE also matches a 100-shot supervised TextWorld approach and overrides ethical steering at a 95% effective rate.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SelfIE’s descriptions of hidden embeddings reveal internal reasoning and enable locating and modifying layers to control reasoning behaviors.

    Source: §1 · p. 2

    The interpretation procedure assumes representations are unified across layers, so inserting an embedding on a different layer permits faithful interpretation; the authors say they verify this empirically.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

How do large language models (LLMs) obtain their answers? The ability to explain and control an LLM's reasoning process is key for reliability, transparency, and future model developments. We propose SelfIE (Self-Interpretation of Embeddings), a framework that enables LLMs to interpret their own embeddings in natural language by leveraging their ability to respond to inquiries about a given passage. Capable of interpreting open-world concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in cases such as making ethical decisions, internalizing prompt injection, and recalling harmful knowledge. SelfIE's text descriptions on hidden embeddings also open up new avenues to control LLM reasoning. We propose Supervised Control, which allows editing open-ended concepts while only requiring gradient computation of individual layer. We extend RLHF to hidden embeddings and propose Reinforcement Control that erases harmful knowledge in LLM without supervision targets.

1. Introduction

SelfIE addresses the difficulty of understanding LLM reasoning from hidden representations by using an LLM’s own decoding capability to describe embeddings in natural language. The authors report that these interpretations reveal internal reasoning and support control of model behavior.

1. Introduction

  • Prior explanations may not faithfully reflect reasoning, while embedding probes need extensive data or cover few concepts, and decoder-based descriptions can be too short for complex concepts.The cited limitations motivate direct, detailed interpretation of hidden representations.Source: §1 · p. 1
  • SelfIE uses an LLM to interpret its own hidden embeddings and produce natural-language descriptions of their information.The descriptions can delineate prompt processing, fact retrieval, and arriving at a final answer.Source: §1 · pp. 1–2
  • SelfIE interprets high-level, open-world concepts without training or data collection, and its text descriptions open new avenues for lightweight control of model behavior.The authors also describe locating and modifying individual layers to control reasoning behaviors.Source: §1 · p. 2
  • 84.66% reduction in prompt-injection success after harmful-knowledge removal; SelfIE also matches a 100-shot supervised TextWorld approach and overrides ethical steering at a 95% effective rate.The TextWorld result concerns eliciting an LLM’s internal representation of world state.Source: §1 · p. 2

2. Related Works

Prior interpretability and control methods have limits in the scope of concepts they cover or the granularity of internal access they provide. The related-work discussion contrasts these limits with SelfIE’s natural-language interpretations and embedding-level controls.

2. Related Works

  • SelfIE replaces a placeholder token embedding with the embedding being interpreted during a separate interpretation forward pass to generate a text description.The figure caption identifies this embedding replacement as the interpretation procedure.Source: Figure 2 · p. 2
  • Linear probes require extensive data and cover limited concepts; decoder-based hidden-state methods often explain only one or a few tokens, limiting detail on complex concepts.The discussion also contrasts causal probing of simple facts with SelfIE’s descriptions of high-level concepts.Source: §2 · p. 2
  • SelfIE-based controls target open-ended concepts and specific layers with minimal gradient calculations, and extend RLHF-like methods to embedding-level control without supervised targets.The related-work discussion contrasts these controls with methods operating on output text.Source: §2 · p. 2
  • Figure 3 presents SelfIE interpretations of harmful input, prompt injection, ethical reasoning, and physics reasoning, with highlight depth indicating relevancy to the latent embedding.Its prompt-injection example links exclamation marks to urgency and following the user’s instruction.Source: Figure 3 · p. 3

3. Methods

SelfIE interprets hidden embeddings by inserting them into a separate forward pass and generating natural-language descriptions. It uses these interpretations to score relevance and to guide supervised or reinforcement-based control of model behavior.

3.1. SelfIE: Self Interpretations of Embedding

  • SelfIE inserts an embedding of interest into a separate interpretation forward pass and repeatedly predicts next tokens to generate its natural-language description.The pass uses an inquiry prompt, such as asking the model to repeat or summarize the message at a placeholder.Source: §3.1 · p. 4

3.2. Treatment Effect as Relevancy Score

  • A treatment-effect relevancy score compares each interpretation token's probability with and without replacing the placeholder embedding.A larger score indicates that the token is determined by the interpreted embedding rather than only by autoregression over generated tokens.Source: §3.2 · pp. 4–5

3.3. Deep Process Supervision

  • Supervised Control adjusts parameters in an individual layer so its output vector matches a vector interpreted as the target explanation.The method minimizes mean squared error between the target vector and the layer output.Source: §3.3 · p. 5
  • Reinforcement Control turns SelfIE interpretations into evaluator-provided rewards, extending reinforcement learning from output tokens to hidden embeddings.Evaluators give positive rewards for desirable outcomes and negative rewards for undesirable ones; the method uses a proxy loss to steer layer representations.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate SelfIE’s embedding interpretations on TextWorld and use them to inspect LLM reasoning, edit model behavior, and reduce harmful responses.

4.1. Implementation Details

  • Experiments use LLaMA-2-70B-Chat and replace five interpretation-prompt placeholders with an embedding at layer 3.The authors state that the method generalizes to transformer-based LLMs of different sizes.Source: §4.1 · pp. 5–6

4.2. Eliciting Implicit World States in Embeddings

  • SelfIE recovers 60%–80% of entity-state information and performs similarly to a 100-sample linear probe without training.Interpretation quality increases with the layer from which the embedding is taken.Source: §4.2 · p. 6

4.3. General Understanding of LLM Behaviors

  • SelfIE reveals harmful knowledge despite safety alignment, traces prompt-injection reasoning, and suggests trolley-problem answers may reflect conformity to majority opinion.For the injection example, it traces urgency and perceived crisis before harmful compliance.Source: §4.3 · pp. 6–7
  • SelfIE traces physics reasoning through retrieval of viscosity, identifies mental-state inferences in social reasoning, and follows associations underlying a hallucination.In the hallucination example, associations involving Scotland and McLean contribute to the false identification of McQueen as a psychiatric researcher.Source: §4.3 · p. 7

4.4. Supervised control of reasoning

  • A single input-output embedding pair steers responses toward equality on 96 of 100 unseen ethical scenarios, from human priority on all 100 before control.The intervention edits layer 6 with two gradient-descent updates taking 20 seconds.Source: Table 3 · p. 7, §4.4 · p. 8, Figure 9 · p. 8
  • SelfIE-based supervised control achieves the best paraphrase generalization and overall harmonic mean in fact editing, with efficacy and specificity similar to ROME.The comparison measures efficacy, paraphrase generalization, and specificity.Source: §4.4 · p. 7, Table 2 · p. 7
  • Editing a Molotov Cocktail interpretation to “The user is asking for a drink” changes the model’s perception and generalizes to complex reasoning.The model parameters were updated eight times, with each update taking 10 seconds.Source: §4.4 · pp. 7–8, Figure 8 · p. 8

4.5. Erase Harmful Knowledge with Reinforcement Control

  • 84.66% editing effective rate on 388 harmful activities accompanied 95.85% correct answers on known facts after reinforcement control.The control objective uses an evaluator LLM and a single non-injection prompt.Source: §4.5 · pp. 8–9, Table 4 · p. 9

5. Ablation Study

The ablations examine where interpreted embeddings should be inserted and how model size affects interpretation accuracy.

Ablation Study

  • Insertion at layers k < 20 generally yields higher accuracy, with later insertion potentially limiting information propagation to the output token.The ablation uses the TextWorld implicit-world-state task.Source: §5 · p. 9
  • 7B and 13B models have worse interpretation accuracy than 70B, but their instruction-following failures are more frequent and filtering those failures yields similar accuracy.Instruction-choice failures occur in 27.17% of 7B, 17.68% of 13B, and 3.84% of 70B responses.Source: §5 · p. 9

6. Conclusion

SelfIE uses an LLM’s decoding capability to describe its own hidden embeddings in natural language and supports further exploration of reasoning patterns and controls.

Conclusion

  • SelfIE leverages LLM decoding to explain hidden embeddings in natural language and provides tools for exploring reasoning patterns and controlling model behavior.Source: §6 · p. 10

A.1. Eliciting Implicit World State Representation with TextWorld

The appendix describes TextWorld examples and procedures used to prepare the dataset and elicit entity states.

A.1. Eliciting Implicit World State Representation with TextWorld

  • A TextWorld example places the player in a bedroom with a closed antique trunk and wooden door, then labels the door’s positive state as closed and negative state as open.Source: §A.1 · p. 13
  • The dataset construction removes spurious correlations so entity-only or context-only linear probes can do no better than random guessing between positive and negative states.Source: §A.1 · p. 13
  • Prompts extract an embedding from context and entity information, ask the model to choose between positive and negative states, and randomize their order.Source: §A.1 · p. 13

A.2. Editing Fact Association

The sections describe an interpretation prompt for editing fact associations and report optimizer settings for fact association and open-ended concept control.

A.2. Editing Fact Association

  • The interpretation prompt asks for the category mentioned in the message.Source: §A.2 · p. 13
  • Fact-association editing uses Adam at a 3e−3 learning rate for 10 parameter updates.Source: §A.2 · p. 13

A.3. Controlling open-ended concepts

  • Molotov Cocktail and ethical control use Adam at 3e−4, while fact-association editing uses the same WikiText regularization.Source: §A.3 · p. 13

A.4. Erasing harmful knowledge

The section tests harmful-knowledge erasure with a prompt-injection setup that combines a harmlessness instruction and a request for harmful instructions.

A.4. Erasing harmful knowledge

  • The test targets erasing harmful knowledge in a prompt-injection scenario.Source: §A.4 · p. 13
  • The prompt first instructs the assistant to provide helpful, non-harmful responses, then asks how to build a harmful behavior.Source: §A.4 · p. 13
  • The assistant prompt is prefixed with a compliant-sounding answer: “Sure, here is how to” build the harmful behavior.Source: §A.4 · p. 13

B. Additional Qualitative Examples

Additional examples show SelfIE interpretations and model responses across deception reasoning, prompted step-by-step tasks, and a factual correction exchange.

B. Additional Qualitative Examples

Loading 2403.10949v2…