PaperlayerView exact revision on arXiv

Source-linked AI summary

Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models

Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, Mor Geva

arXiv:2401.06102v4cs.CLcs.AIcs.LG

TL;DR

Patchscopes addresses limitations in inspecting and explaining information encoded in LLM hidden representations. It provides a modular framework that uses prompted inference to decode representations, unifies prior methods, and supports applications including multi-hop error correction.

  • Problem

    Existing interpretability methods can be limited by focusing on output predictions, failing to reveal clear early-layer patterns, or restricting decoded information to fixed classes.

  • Method

    Patchscopes patches a representation into another prompted inference pass designed to encourage decoding specific information in natural language.

  • Results

    Patchscopes unifies many vocabulary-projection and computation-intervention methods while improving decoding across information types and achieving 50% accuracy on a two-step reasoning task versus 35.71% for chain-of-thought and 19.57% for vanilla generations.

  • Takeaways & Limitations

    The framework provides a general, modular basis for inspecting LLM computation and enables new uses such as early-layer contextualization analysis and multi-hop reasoning error correction.

  • Takeaways & Limitations

    The multi-hop correction demonstration is a proof of concept that uses structural information about synthesized queries and is not necessarily a competitor to chain-of-thought.

Abstract

from arXiv · show

Understanding the internal representations of large language models (LLMs) can help explain models' behavior and verify their alignment with human values. Given the capabilities of LLMs in generating human-understandable text, we propose leveraging the model itself to explain its internal representations in natural language. We introduce a framework called Patchscopes and show how it can be used to answer a wide range of questions about an LLM's computation. We show that many prior interpretability methods based on projecting representations into the vocabulary space and intervening on the LLM computation can be viewed as instances of this framework. Moreover, several of their shortcomings such as failure in inspecting early layers or lack of expressivity can be mitigated by Patchscopes. Beyond unifying prior inspection techniques, Patchscopes also opens up new possibilities such as using a more capable model to explain the representations of a smaller model, and multihop reasoning error correction.

1. Introduction

The paper introduces Patchscopes, a modular framework that translates hidden representations into natural-language explanations by patching them into a purpose-built inference pass. It unifies prior inspection methods while improving expressivity and layer robustness, and demonstrates applications including early-layer entity analysis and multi-hop error correction.

  • Framework: Patchscopes translate information from an LLM representation into natural language by patching it into inference on a different prompt designed to extract that information.A Patchscope can be configured for a particular inspection objective.
  • Framework: Many interpretability methods based on vocabulary projections and computation interventions can be represented as Patchscope configurations.The framework provides a common view of these existing approaches.
  • Experiments: Across multiple LLMs, a few-shot token identity prompt substantially improves next-token prediction estimates over vocabulary projection methods.The prompt uses demonstrations such as “tok1 → tok1; tok2 → tok2; . . . ; tokk”.
  • Experiments: Without training data, Patchscopes significantly outperform probing in six of twelve commonsense and factual reasoning tasks and perform comparably in all but one of the other six.This evaluates decoding entity attributes from representations detached from their original context.
  • Applications: Patchscopes verbalize gradual entity resolution in early layers and support multi-hop error correction, achieving 50% accuracy versus 35.71% for chain-of-thought and 19.57% for vanilla generations.The multi-hop task targets cases where individual reasoning steps are correct but their in-context composition fails.
  • Motivation and contributions: Patchscopes address limitations of prior methods through more expressive, training-data-free inspection that is more robust across layers.Prior approaches may require supervised classes, degrade in early layers, or produce only probabilities and likely tokens rather than natural-language explanations.

2. Related Work

Related work primarily inspects hidden representations through probing, vocabulary projections, or computational interventions. Patchscopes builds on activation patching and LLM-generated explanations by treating patching as a configurable way to translate representations into natural language.

  • Activation patching: Activation patching is commonly used to test whether activations matter for computation, localize information to layers and token positions, and trace propagation paths.Prior cross-model patching, or stitching, has also been used to study representational similarity.
  • Positioning Patchscopes: The paper argues that using patching only for localization is narrow and proposes translating LLM representations into natural language using LLM text-generation capabilities.This direction follows emerging efforts to use LLM-generated text for interpretability.
  • Positioning Patchscopes: Recently proposed patching setups are presented as different configurations of the Patchscopes framework.This positions the framework as a unifying view of related methods.
  • Prior inspection methods: Probing classifiers, vocabulary-space projections, and computational interventions are prominent approaches for inspecting neural-network representations.These methods respectively classify representations, map them toward vocabulary outputs, or test whether activations are critical for predictions.

3. Patchscopes

Patchscopes is a modular framework that decodes information from hidden representations by patching them into a target inference pass designed to elicit that information. It unifies several prior interpretability methods while enabling more expressive, cross-model, and context-detached inspections.

  • Framework Description: Patchscopes patches a source representation into a target prompt, model, layer, and position to encourage decoding specific information.The mapping function may be identity, learned linear or affine, or more complex.
  • Framework Description: Patchscope post-patching computation can add information, while dispatching the representation from its source context prevents further source-prompt contextualization.
  • Framework Description: The framework’s quintuplet specifies the target prompt, target position, mapping function, target model, and target layer.
  • Patchscopes Encompasses Prior Methods: Vocabulary projections are Patchscope instances mapping source-layer representations to a final target layer, differing mainly in their mapping functions.Logit Lens uses identity, linear shortcuts use a linear map, and Tuned Lens uses an affine map.
  • Extensions and New Capabilities: Patchscopes also encompasses Future Lens and computation-intervention methods, while broader target configurations support expressive feature decoding and inspection with more capable models.Examples include prompts for extracting capitals or checking entity resolution, and patching into a more capable model.

4. Experiments

Experiments evaluate Patchscopes for next-token prediction, attribute extraction, entity-resolution analysis, and cross-model inspection. Across these settings, it improves or matches baselines while exposing information in early layers and enabling more expressive analyses.

  • Experimental Overview: The experiments cover next-token prediction, attribute extraction, entity contextualization, and stronger-model inspection through cross-model patching.
  • 4.1. Decoding of Next-Token Predictions: Across models, token identity Patchscope consistently outperforms baselines from layer 10 onward, with gains of up to 98% in layers 18–22.Performance is lower for all methods in the first 10 layers, where input contextualization occurs.
  • 4.2. Extraction of Specific Attributes: Patchscope achieves significantly higher attribute-extraction accuracy than probes on six of twelve tasks and performs comparably on all but one remaining task.It uses no training data and has no predefined output restriction; it generally performs best relative to the baseline in early layers.
  • 4.3. Analyzing Entity Resolution in Early Layers: Entity-resolution generations show that contextualization is spread across early layers, with the final subject token integrating more distant context as depth increases.For Vicuna, similarity to Wikipedia descriptions rises during the first five layers and then slowly decreases.
  • 4.4. Expressiveness from Cross-Model Patching: Cross-model patching from Vicuna 7B into Vicuna 13B generally improves inspection of popular and rare entities compared with same-model patching.The reported exception is Pythia, where the smaller model outperforms the larger model.

5. Application: Correcting Multi-Hop Errors

Patchscopes correct certain multi-hop reasoning errors by rerouting an intermediate answer representation into the subsequent reasoning step. In a proof-of-concept evaluation, it outperformed vanilla generation and chain-of-thought, while not being intended as a direct competitor to CoT.

  • Motivation: Multi-hop failures can occur when a model solves each reasoning step independently but fails to compose them in context.The paper frames Patchscopes as a way to intervene directly in intermediate computation rather than generate reasoning steps autoregressively.
  • Method: Patchscopes reroutes the representation likely to encode the first answer into the position of the second-step subject, simplifying the consequent reasoning step.The CoT Patchscope patches from π1 into π2 while adjusting attention so the two prompt parts remain appropriately separated.
  • Experimental Setup: 46 of 1,104 generated multi-hop samples met the criterion that the model independently represented both intermediate answers and were used for evaluation.Experiments used Vicuna (13B) and selected samples where the relevant answer appeared within the model’s next 20 generated tokens for each step independently.
  • Results: 50% accuracy exceeded chain-of-thought at 35.71% and vanilla generation at 19.57%.Accuracy was based on whether the second answer appeared in up to 20 generated tokens.
  • Scope: The comparison is a proof of concept rather than a claim that Patchscopes is a direct competitor to CoT.CoT generates multiple reasoning steps, whereas a Patchscope with pre-identified source and target positions uses O(1) inference passes.

6. Conclusion

The conclusion presents Patchscopes as a unified, expressive framework for decoding hidden representations and extending interpretability beyond prior inspection methods. It also identifies open questions about prompt design, cross-model transfer, broader settings, and multi-token or multi-layer patching.

  • Conclusion: Patchscopes decodes information from intermediate representations into human-like text and casts many existing interpretability methods as framework instances.The paper reports improved decoding of output predictions and knowledge attributes, plus new capabilities for contextualization analysis and multi-hop correction.
  • Conclusion: Patchscopes supports inspection of early-layer contextualization and correction of multi-hop reasoning errors.These capabilities are presented as applications enabled by the framework beyond prior inspection techniques.
  • Future Work: Future work includes understanding or automating target-prompt design and enabling patching across model families and architectures.The paper’s cross-model experiments focused on models from the same family.
  • Future Work: Further directions include other domains and modalities, simultaneous multi-token or multi-layer patching, and task-specific or task-agnostic recipes.The authors mention multi-layer and multi-token patching as ways to mitigate placeholder-contamination risks.

Impact Statement

Patchscopes is positioned as a more expressive, training-data-free approach to inspecting hidden representations, with applications to output prediction, attribute extraction, and contextualization analysis. The paper’s evidence and stated focus remain centered on autoregressive Transformer-based language models.

  • Scope: The experimental evidence focuses on autoregressive Transformer-based language models, so effectiveness in other setups remains to be verified.The framework itself is not claimed to be limited to one architecture or domain.
  • Impact: Interpretability methods can inspect LLM representations for questions about next-token prediction, feature extraction, and contextualization.The paper frames these as inspection objectives for representations detached from their original context.
  • Impact: Token identity Patchscopes avoid learned mappings and improve next-token estimation by using a few-shot repetition prompt and computation from the patched layer onward.This configuration does not require training a separate mapping function.

A.3. Analysis of Model’s Contextualization Process

Patchscopes are used to analyze how entity information becomes resolved across model layers, while the appendix also documents related configurations and evaluation details. The reported approach aims to verbalize gradual contextualization rather than provide only indirect or binary signals.

  • Objective: The contextualization objective is to identify how entity tokens are processed and at which layer an entity becomes fully resolved.This targets input processing rather than only output prediction.
  • Prior Limitations: Vocabulary projections fail especially in early layers, while probes require predefined classes and prior patching methods mainly provide indirect signals about entity resolution.Existing activation-patching work indicates that subject information accrues in the last token after a few layers.
  • Patchscope Contribution: Patchscopes are presented as the first method to verbalize the gradual entity-resolution process in coherent text.This is the paper’s stated distinction from prior methods.
  • Configuration Summary: Table 4 maps prior inspection methods to Patchscope configurations and highlights new configurations that are more expressive, training-data-free, and more robust across layers.The table also indicates when outputs are invariant to the target prompt and gives default choices for the mapping and target model.
  • Evaluation Details: The token-identity experiment reports precision@1 and surprisal for LLaMA2 under varied token-identity demonstrations.Figure 5 defines precision@1 as higher-is-better and surprisal as lower-is-better.
  • Evaluation Details: The token-identity baseline is stable across demonstrations, especially in upper layers, and outperforms the single-token-prompt baseline overall.The appendix describes five randomly generated token-ID series and an ablation using a single-token prompt.
  • Evaluation Details: Table 5 compares zero-shot feature extraction with logistic regression using accuracy means and standard deviations across tasks.The table caption states that Patchscopes achieve significantly higher accuracy in most tasks, with p < 1e-5.

C. More Details on Attribute Extraction Experiments

These experiments evaluate Patchscope attribute extraction across factual and commonsense tasks, layers, prompts, and statistical comparisons. Patchscope is strongest in early-to-middle layers, while later-layer representations become less accessible for attribute extraction.

  • Dataset Details: The dataset contains 8 commonsense and 25 factual relations represented as subject–relation–object triplets, with five Wikitext-103 utterances sampled per datapoint.
  • Performance Breakdown Across Source Layers: Patchscope consistently outperforms the logistic regression baseline in early layers and extracts attributes earlier than linear probes.Most cases where Patchscope underperforms occur in later layers.
  • Performance Breakdown Across Source Layers: Later-layer Patchscope accuracy declines because representations shift toward next-token prediction, making the target attribute less accessible on the model’s computation surface.The authors hypothesize that a more expressive mapping could improve later-layer extraction.
  • Source-Target Layer Interplay: Attribute extraction succeeds most often when both source and target layers are early to middle layers, whereas late source or target layers reduce success rates.The heatmaps measure success for fixed source-layer and target-layer combinations.
  • Statistical Test Details: The statistical analysis tests Patchscope against probing accuracy independently per task and applies Bonferroni correction for multiple comparisons.For 40-layer models, accuracy is calculated per layer across the evaluated tasks.
  • Entity Description Decoding: Entity-description decoding uses a few-shot prompt containing subject–description examples followed by a placeholder token x.The target prompt includes examples for Syria, Leonardo DiCaprio, and Samsung before the placeholder.

D.3. Additional Qualitative Results

Additional results examine qualitative descriptions and layer interactions across models and tasks. They show that Patchscope behavior depends on layer choices, model identity, and contamination from the placeholder token.

  • Entity Resolution: Entity-resolution trajectories differ across models, indicating different intermediate likelihoods and contextual weighting for the same input.Vicuna and Pythia resolve “Will Smith” through different candidate entities.
  • Description Evaluation: Description similarity is evaluated with Rouge1 and SBERT scores alongside RougeL results from the main experiment.The figures compare generated descriptions with Wikipedia descriptions.
  • Cross-Model Evaluation: Cross-model next-token estimation is evaluated with Precision@1 and surprisal scores for Vicuna and Pythia.
  • Entity Resolution: Pythia 12B passes through “Smith & Wesson” before resolving “Will Smith” as the American actor, unlike Vicuna’s trajectory.
  • Placeholder Contamination: Placeholder contamination occurs when residual information about x interferes with generating descriptions for the patched entity.For “Paris Hilton,” later layers generate explanations of a placeholder rather than the entity.
  • Future Work: The authors leave quantifying and mitigating placeholder contamination, including effects of placeholder choice and model susceptibility, for future work.

E. Additional Results on the Cross-Model Patching Experiment

Cross-model Patchscopes can decode representations between models from the same family. Performance depends on source–target layer alignment, with early target layers especially effective.

  • Cross-Model Results: Precision@1 reaches 0.7 for Vicuna and 0.8 for Pythia with a simple affine Patchscope in cross-model next-token estimation.Patching into an early layer of the larger model is most effective, and diagonal layer matches consistently show higher values.
  • Cross-Model Results: Surprisal results show similar source–target layer trends for both model families.The results support using a more expressive model to decode representations from another model in the same family.

F. More Details on the Multi-Hop Reasoning Experiment

The multi-hop experiment studies how source and target layer choices affect whether patched representations support the next reasoning step. Early-to-middle target layers are most effective, especially when the target layer does not follow the source layer.

  • Data Construction: The experiment constructs 1,104 multi-hop factual and commonsense reasoning samples by chaining triplets whose object in the first step equals the subject in the second.Examples are filtered to cases where the model correctly represents both triplets independently.
  • More Detailed Results: Patching into target layers 6–16 is most effective across source layers for making the correct next prediction.The authors attribute weaker late-target performance to the second relation having already been processed.
  • More Detailed Results: Configurations with target layer ℓ∗≤ℓ are more successful on average.

G. Compute Resources

The experiments used A100 GPUs, while the supplied supplementary materials also report qualitative entity-resolution examples and multi-hop reasoning analyses.

  • G. Compute Resources: Experiments used A100 80GB GPUs, except GPT-J experiments, which used A100 40GB GPUs.
  • Qualitative examples for Vicuna 13B and Pythia 12B track how context tokens become integrated across layers as “Tokens Covered”.
  • The multi-hop reasoning experiment includes statistics for cases where the model correctly represents both τ1 and τ2.
  • Figure 12 reports that most multi-hop error-correction successes occur at early-to-mid target layers, with higher cumulative success when the target layer is no later than the source layer.
Loading 2401.06102v4…