PaperlayerView exact revision on arXiv

Source-linked AI summary

Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries

Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, Amir Globerson

arXiv:2406.12775v2cs.CL

TL;DR

LLMs can struggle with multi-hop knowledge queries, while precise accounts of their latent reasoning pathways have been limited in large pretrained models. The paper traces how the two hops are resolved across layers and introduces back-patching to examine failures. It finds a sequential pathway and reports that back-patching improves performance on previously incorrect queries.

  • Problem

    How LLMs internally compose knowledge to answer multi-hop queries remains insufficiently understood, with precise latent reasoning pathways not thoroughly investigated in large pretrained models.

    Source: §1 · p. 1

  • Method

    The authors use Patchscopes to locate hop answers in hidden representations and back-patch later-layer representations into earlier layers to test the pathway.

    Source: §8 · p. 9

  • Results

    The first hop is resolved in early layers before the second hop appears later, and back-patching significantly improves performance on previously incorrect queries.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The findings and methods open opportunities for understanding and improving latent reasoning in LLMs.

    Source: §8 · p. 9

    Back-patching is not a practical inference method because only a subset of back-patches produces the correct answer.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) can solve complex multi-step problems, but little is known about how these computations are implemented internally. Motivated by this, we study how LLMs answer multi-hop queries such as "The spouse of the performer of Imagine is". These queries require two information extraction steps: a latent one for resolving the first hop ("the performer of Imagine") into the bridge entity (John Lennon), and another for resolving the second hop ("the spouse of John Lennon") into the target entity (Yoko Ono). Understanding how the latent step is computed internally is key to understanding the overall computation. By carefully analyzing the internal computations of transformer-based LLMs, we discover that the bridge entity is resolved in the early layers of the model. Then, only after this resolution, the two-hop query is solved in the later layers. Because the second hop commences in later layers, there could be cases where these layers no longer encode the necessary knowledge for correctly predicting the answer. Motivated by this, we propose a novel "back-patching" analysis method whereby a hidden representation from a later layer is patched back to an earlier layer. We find that in up to 66% of previously incorrect cases there exists a back-patch that results in the correct generation of the answer, showing that the later layers indeed sometimes lack the needed functionality. Overall, our methods and findings open further opportunities for understanding and improving latent reasoning in transformer-based LLMs.

1 Introduction

The paper investigates how large pretrained language models internally answer multi-hop queries, where existing evidence for precise latent reasoning pathways was limited. It finds that the first hop is resolved before the second, and introduces back-patching to test whether earlier computation can help when the pathway fails.

1 Introduction

  • Precise latent reasoning pathways had been found mainly in small models trained on synthetic data, not thoroughly investigated in large pretrained models.The paper motivates studying internal mechanisms as a step toward strengthening latent reasoning in LLMs.Source: §1 · pp. 1–2
  • The first hop is resolved in early layers, while the second hop appears in the prompt’s last token only in later layers.The paper proposes that the first-hop answer propagates to the last token to support second-hop resolution.Source: §1 · p. 2
  • Up to 66% of initially incorrect cases have an optimal back-patch that produces the correct answer.Back-patching moves a hidden representation from a later layer to an earlier layer and continues the forward pass.Source: §1 · p. 2
  • Back-patching tests the proposed findings without requiring assumptions about prompt structure or knowledge of participating entities.It patches a later-layer hidden representation into the same prompt position at an earlier layer, allowing more layers to finish computation without added parameters or training.Source: §1 · p. 2
  • The authors introduce a two-hop query dataset to support systematic study of this setting.They also use Patchscopes to inspect entities encoded in hidden representations while models process the queries.Source: §1 · p. 2

2 Experimental Setup

The setup defines two-hop queries as compositions of linked facts, then constructs a Wikidata-based dataset and filters examples to isolate multi-hop reasoning. Experiments compare correct and incorrect cases across six LLaMA and Pythia models.

2.1 Two-Hop Queries

  • Two-hop queries compose two facts whose shared entity is the bridge between the source and target entities.The setup identifies the last token of the first hop as t1 and the last token of the full prompt as t2.Source: §2.1 · pp. 2–3

2.2 Dataset

  • The dataset is built from Wikidata by sampling bridge entities, composing predefined relations, and converting facts into natural-language queries with manual templates.The dataset passage reports 82,020 queries.Source: §2.2 · p. 3
  • Shortcut filtering removes cases where a model answers from entity popularity or source-target correlation rather than the full query.The authors test prompts with the first entity or first relation omitted, then retain model-specific filtered subsets.Source: §2.2 · p. 3
  • The experiments form per-model subsets of correct full-query answers and failures despite correct answers to both hops in isolation.The correct subset also verifies the first-hop answer; examples are sampled by bridge-entity type, with 100 per type for correct cases and 50 for failure cases when available.Source: §2.2 · p. 3

2.3 Models

  • The evaluated models are LLaMA 2 7B and 13B, LLaMA 3 8B and 70B, and Pythia 6.9B and 12B.The models range from 32 to 80 layers.Source: §2.3 · p. 4

3 Localizing First Hop Resolution

Patchscopes analysis locates first-hop resolution at t1, generally in early layers. The section describes how hidden representations are probed and finds that e2 can be resolved even in some incorrectly answered cases.

3.1 Interpreting Hidden Representations

  • Patchscopes maps a hidden representation to a natural-language description, extending vocabulary projections that map representations to single tokens.The procedure records a source token's representation, inserts it into a target prompt at a chosen layer, and continues generation.Source: §3.1 · p. 4

3.2 Experiment

  • The experiment records t1 representations across source layers, patches them into target layers, and checks whether generated text names bridge entity e2.A representation counts as encoding e2 when one of three sampled generations names it.Source: §3.2 · p. 4

3.3 Results

  • Across all models and settings, the reported non-trivial percentages are 41%-78%; in incorrect two-hop cases, e2 resolving generally occurs at higher layers than in correct cases.For LLaMA 2 13B, Figure 2 shows e2 first successfully decoded from t1 almost exclusively during early layers.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
  • The heat map shows an early-layer hotspot for successful e2 decoding, while a later-layer hotspot succeeds almost exclusively when patched after the final layer.The latter pattern is roughly equivalent to vocabulary projection, although e2 is encoded in hidden representations before it ranks among top projected tokens.Source: §3.3 · p. 5, Figure 3 · p. 5

4 Second Hop is Resolved at Last Position

The second hop is typically resolved at the prompt's last token after the first hop, with target entities appearing in mid-upper layers. MLP updates play a larger role than attention updates in promoting the predicted token.

4.1 Method

  • The analysis projects attention and MLP residual updates through layer normalization and the output embedding matrix, and also uses Patchscopes.The projections locate entities by their vocabulary rank.Source: §4.1 · p. 5

4.2 Experiment

  • The experiment checks whether Patchscopes decodes e3 from t2 and whether each sublayer update ranks the generated first token highest in its vocabulary projection.The table reports the percentage of cases and mean layer of first promotion.Source: §4.2 · p. 5, Table 3 · p. 6

4.3 Results

  • 71%-86% of correctly answered cases decode e3 from t2, compared with 33%-65% of incorrectly answered cases.The target entity is resolved in the mid-upper layers at t2, and the information needed for the second hop exists in those layers' parameters.Source: §4.3 · p. 6
  • MLP sublayers play a larger role than attention sublayers in promoting the predicted token, though attention updates also promote it in a non-negligible share of cases.Promotions occur in upper layers, after e3 has been resolved at the last token.Source: §4.3 · p. 6

5 Information Propagates to Last Token

The section tests whether bridge-entity information propagates from the first-hop position to the prompt’s last token, and finds evidence of this flow, concentrated mainly in middle layers. It also compares when the bridge entity appears at each position and how pathway timing differs between correct and incorrect cases.

5.1 Method

  • Attention knockout, vocabulary projections, and Patchscopes provide distinct signals of whether bridge-entity information reaches or is encoded at t2.The methods examine changes in generation, sublayer updates, and whether Patchscopes can detect e2 at the last prompt token.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 Experiment

  • The experiments test propagation by blocking attention from t1 to t2 across a seven-layer window and checking whether the model’s answer changes.The attention mask is set to −∞; layers are treated as critical when blocking them disrupts the correct answer or changes incorrect generations.Source: §5.2 · p. 6, §5.1 · p. 6

5.3 Results

  • Critical information is detected propagating from t1 to t2 in a relatively high percentage of cases, with relevant layers mostly in the middle.Table 4 reports percentages and mean layers for cases where attention knockout, projection, or Patchscopes first detected propagation.Source: §5.3 · p. 6, Table 4 · p. 7
  • The bridge entity is decoded from t2 in 25%-75% of cases, with t2 encoding it in mid-upper layers and t1 in lower layers.The distinct layer ranges are consistent with a sequential computation.Source: §5.3 · p. 7
  • In incorrect cases, entities generally resolve later while information propagates and is extracted earlier, highlighting the importance of early first-hop resolution.Figure 4 compares the first observed layers of each pathway stage between correct and incorrect LLaMA 3 8B cases.Source: §5.3 · p. 7

6 Back-patching Improves Two-Hop Performance

The section uses back-patching to test whether later-layer representations can benefit from earlier layers’ knowledge-extraction functionality. Back-patching corrects many previously incorrect cases, while success depends on the source and target layers and patched token.

6 Back-patching Improves Two-Hop Performance

  • The proposed failure mechanism is that the first hop resolves in lower layers and the second uses upper layers that may no longer contain needed knowledge.The authors describe this as a possible transformer limitation: information may be propagated or extracted before resolution is complete, or extraction may be confined to upper layers.Source: §6 · p. 7

6.1 Method

  • The experiment records a hidden representation at a source layer, patches it into the same prompt position at an earlier target layer, then continues generation.The source representation is thus fed into an earlier point in the forward computation.Source: §6.1 · p. 7

6.3 Results

  • Back-patching correctly answers 32%-66% of incorrect cases while keeping correct cases at 100% across models and token positions.Table 5 presents success rates by setting and patched token; the authors note that choosing source and target layers well could yield significant gains.Source: Table 5 · p. 8, §6.3 · p. 8

6.2 Experiment

  • Back-patching is tested at both t1 and t2, and a case counts as successful if any source-target pair generates the correct answer.The successful source and target layers are analyzed afterward.Source: §6.2 · p. 8

6.3 Results

  • For LLaMA models, back-patching t2 succeeds more often than back-patching t1, suggesting last-position stages may be a more dominant failure point.The higher t2 success rate occurs despite back-patching t2 making no assumption about prompt structure.Source: §6.3 · p. 8
  • For t1, lower source layers and lower target layers show higher success, while higher target layers show no success in the LLaMA 2 13B heat-map.Lower source layers correspond to first-hop resolution; higher target layers leave too little time to complete the pathway.Source: §6.3 · p. 8, Figure 5 · p. 8
  • For t2, success is higher with upper source layers and lower target layers, which may give the model time and access to lower-layer parametric knowledge.The Figure 5b discussion links upper source layers to second-hop resolution at the last token.Source: §6.3 · p. 8, Figure 5 · p. 8

7 Related Work

Prior work has traced factual knowledge and examined latent reasoning, but the paper situates its contribution in analyzing information flow across layers and positions for real-world two-hop queries. It also contrasts back-patching with approaches that assume knowledge of the bridge entity.

7 Related Work

  • Knowledge-tracing work has studied editing and transformer internals, while this paper extends mechanistic analysis to two-hop queries requiring multiple extractions and composition.The passage groups knowledge-editing studies [De Cao et al., 2021; Mitchell et al., 2022; Meng et al., 2022; Zhang et al., 2024] separately from interpretability work [Dai et al., 2022; Geva et al., 2022, 2023].Source: §7 · p. 9
  • Prior latent-reasoning studies found existential evidence without tracking information flow, while Wang et al. (2024) studied a similar circuit in a small synthetic-data transformer.This paper instead studies large language models with real-world knowledge and regular pretraining, using methods that account for forward-pass information flow.Source: §7 · p. 9
  • Sakarvadia et al. (2023) and Li et al. (2024a) assume the bridge entity is known; Li et al. also use knowledge editing, which may disrupt models.The authors say a potential back-patching-based method would not share these drawbacks.Source: §7 · p. 9

8 Conclusion

The conclusion emphasizes a sequential pathway: lower layers resolve the first hop before mid-upper layers answer the second. It presents back-patching as both an analysis method and a way to improve performance on previously incorrect queries.

8 Conclusion

  • The paper finds strong evidence that lower layers resolve the first hop before mid-upper layers answer the second.The conclusion describes this as a sequential latent reasoning pathway.Source: §8 · p. 9
  • Back-patching extracts a later-layer hidden representation and patches it into an earlier layer, improving performance on previously incorrect queries.The conclusion presents the method as a validation of the proposed pathway.Source: §8 · p. 9
  • The findings point to a possible transformer limitation: the second hop may have to use only knowledge encoded in upper layers.The paper states that its methods and findings open opportunities for understanding and improving latent reasoning in LLMs.Source: §8 · p. 9

Limitations

The section limits the study to a prominent two-hop reasoning pathway and notes that back-patching is not currently a practical inference method. It also qualifies the mechanistic evidence as approximate and describes additional analyses and experimental details.

Limitations

  • Mechanistic decoding methods are approximations, which the authors seek to mitigate with multiple techniques and experiments designed to provide lower-bound results.The authors state that these choices do not undermine their findings.Source: Limitations · p. 9
  • The study examines one prominent pathway, omits some components such as how relations contribute, and argues that this still supports sequential computation.Other pathways are considered likely to exist.Source: Limitations · p. 9
  • The analysis covers only two-hop queries; broader query and task settings are expected to involve similar mechanisms but are left for future work.The expectation is discussed alongside prior and concurrent work in other settings.Source: Limitations · pp. 9–10
  • Only a subset of back-patches generates correct answers, and chain-of-thought prompting would be more effective if best multi-hop performance were the sole goal.Efficiently selecting source and target layers in advance could make back-patching viable.Source: Limitations · p. 10

B Patchscopes First Decoded Layers

  • Figure 6 reports the percentage of cases at each layer where Patchscopes first successfully decodes target entities.The passage describes the figure's measure, not its values.Source: B Patchscopes First Decoded Layers · p. 12

C Patchscopes Heat-maps

The section presents Patchscopes heatmaps for entity decoding and compares the first observed layers of pathway stages in correct and incorrect cases.

C Patchscopes Heat-maps

  • Figures 7 and 8 show Patchscopes success heatmaps for decoding e2 from t1, e2 from t2, and e3 from t2 across models.The passage specifies the decoding targets and positions, but not the heatmap outcomes.Source: C Patchscopes Heat-maps · p. 12

D First Observed Layers of Pathway Stages

E Back-patching Heat-maps

The section includes back-patching heatmaps and additional figures describing Patchscopes decoding and pathway-stage layers across models.

Patchscopes First Decoded Layers

  • Figure 6 shows the percentage of cases per layer where target entities were first successfully decoded using Patchscopes.The figure description gives the measure, not numerical values.Source: Figure 6 · p. 13

Patchscopes Heat-maps

  • Figures 7 and 8 show Patchscopes success heatmaps for LLaMA and Pythia models.No specific outcomes are stated in the supplied figure descriptions.Source: Figure 7 · p. 14, Figure 8 · p. 15

First Observed Layers of Pathway Stages

  • Figure 9 compares the first layer of each pathway stage between correct and incorrect cases across models.The passage states the comparison but not its result.Source: Figure 9 · p. 16

E Back-patching Heat-maps

Loading 2406.12775v2…