PaperlayerView exact revision on arXiv

Source-linked AI summary

Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutation

Wenyu Huang, Pavlos Vougiouklis, Mirella Lapata, Jeff Z. Pan

arXiv:2505.11754v1cs.CL

TL;DR

Multi-hop QA requires language models to retrieve evidence and reason across multiple documents, a setting where causal masks may constrain cross-context interaction. The paper tests document permutations and bidirectional attention across model architectures, and finds that attention-based selection can also improve accuracy. Overall, encoder-decoder models, forward document order, and bidirectional attention show advantages in the reported MHQA experiments.

  • Problem

    MHQA requires language models to retrieve relevant documents and synthesize information across them, while causal masking may limit cross-document interaction.

    Source: §1 · p. 2

  • Method

    The paper evaluates three model families under document-order, distance, and completeness permutations, compares causal and bidirectional attention, and analyzes attention patterns.

    Source: §1 · p. 2, §3.2 · p. 3, §5 · p. 4

  • Results

    Encoder-decoder models outperform decoder-only counterparts, bidirectional attention improves MHQA performance, and peak-attention selection raises Qwen 7B accuracy from 28.6% to 33.7%.

    Source: §1 · p. 2, §8 · pp. 8–9

  • Takeaways & Limitations

    In the reported MHQA setting, model architecture, gold-document ordering, and attention patterns are associated with differences in performance.

    Source: §5 · p. 4, §8 · pp. 8–9

    Experiments focus on MuSiQue with at most 20 documents per question and prompts around or below 4k tokens.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-hop Question Answering (MHQA) adds layers of complexity to question answering, making it more challenging. When Language Models (LMs) are prompted with multiple search results, they are tasked not only with retrieving relevant information but also employing multi-hop reasoning across the information sources. Although LMs perform well on traditional question-answering tasks, the causal mask can hinder their capacity to reason across complex contexts. In this paper, we explore how LMs respond to multi-hop questions by permuting search results (retrieved documents) under various configurations. Our study reveals interesting findings as follows: 1) Encoder-decoder models, such as the ones in the Flan-T5 family, generally outperform causal decoder-only LMs in MHQA tasks, despite being significantly smaller in size; 2) altering the order of gold documents reveals distinct trends in both Flan T5 models and fine-tuned decoder-only models, with optimal performance observed when the document order aligns with the reasoning chain order; 3) enhancing causal decoder-only models with bi-directional attention by modifying the causal mask can effectively boost their end performance. In addition to the above, we conduct a thorough investigation of the distribution of LM attention weights in the context of MHQA. Our experiments reveal that attention weights tend to peak at higher values when the resulting answer is correct. We leverage this finding to heuristically improve LMs' performance on this task. Our code is publicly available at https://github.com/hwy9855/MultiHopQA-Reasoning.

1 Introduction

Multi-hop QA requires models to retrieve relevant documents and reason across them, but causal masking may constrain cross-document interaction. The paper studies model architectures, document permutations, bidirectional attention, and attention-based answer selection.

1 Introduction

  • MHQA requires synthesizing evidence across multiple documents, making both document retrieval and cross-document reasoning necessary for a correct answer.The study frames this as a more complex information-integration task than simpler question answering.Source: §1 · p. 2
  • The paper asks whether causal masking limits decoder-only models and what changes when it is replaced with a bidirectional prefix mask.Causal masks prevent tokens from seeing future context, unlike bidirectional encoding.Source: §1 · p. 2
  • Flan-T5 generally outperforms causal decoder-only models without fine-tuning, while forward-ordered gold documents suit fine-tuned models and Flan-T5.The study also reports benefits from bidirectional attention, effects of document distance, and retained correctness after removing the first-hop document.Source: §1 · p. 2
  • The study tests three model families and varies gold-document order, distance, and completeness to analyze MHQA behavior.The evaluated families are Flan-T5, Qwen2.5, and Llama 3.x.Source: §1 · p. 2
  • 33.7% versus 28.6%: Qwen 7B accuracy rises when answers are selected from document permutations with the largest peak attention score.The authors report that correct answers typically coincide with at least one document receiving a higher attention score.Source: §1 · p. 2

2 Related Works

Prior work studies retrieved-context use and causal-model limitations, but the cited context-use studies focus on one-hop QA. This paper extends the investigation to reasoning across multiple hops.

2.1 RAG and MHQA

  • Earlier RAG studies report retrieval harms and “lost in the middle,” but focus on one-hop QA rather than reasoning across distant contextual hops.The paper positions its MHQA analysis as building on those studies.Source: §2.1 · p. 2

2.2 Drawbacks of Causal Language Model

  • Causal masks prevent context tokens from seeing future context; prior work mitigates this through context repetition or bidirectional attention, which this paper examines for MHQA.The paper notes that context repetition has additional costs and explores bidirectional attention as an alternative.Source: §2.2 · p. 3

3 Preliminary

The paper formulates MHQA over a question and mixed relevant and irrelevant documents, then uses grouped attention and an Information Contribution score to inspect document use.

3.1 Multihop Question Answering

  • MHQA takes a question and a set of relevant and irrelevant documents, with information from fewer than all documents mandatory for answering.The task is prompted by concatenating the question and documents in the input context.Source: §3.1 · p. 3

3.2 Grouped Attention Weight

  • Grouped attention computes weights between token blocks, including instructions, documents, questions, and predictions, and normalizes by the size of block X.The prediction block uses individual prediction tokens because it may include tokens beyond the answer.Source: §3.2 · p. 3

3.3 Information Contribution Score

  • The Information Contribution score uses grouped attention to estimate how much information each document contributes to the MHQA answer.It is defined for a document token group and layer; the attention heads and answer tokens are specified in the score description.Source: §3.3 · p. 3

4 Experimental Setup

Experiments compare instruction-tuned encoder-decoder and decoder-only models across inference setups on MuSiQue. Fine-tuning and bidirectional attention are additional setups, with accuracy used for evaluation.

4 Experimental Setup

  • The experiments use instruction-tuned Flan-T5, Qwen2.5, and Llama 3.x models, testing answer-only, chain-of-thought, fine-tuned, and fine-tuned-plus-bidirectional setups.Qwen2.5 model sizes range from 0.5B to 14B; the Llama models are 1B, 3B, and 8B.Source: §4 · p. 3

4.1 Dataset

  • MuSiQue supplies 19,938 training queries and 2,417 development queries, with questions spanning two to four hops and contexts of up to 20 documents.The experiments use its answerable set and preserve the original training and development split.Source: §4.1 · p. 4

4.2 Metrics

  • Accuracy measures performance; exact match is used for fine-tuned and encoder-decoder models, while some CoT outputs are checked for the reference answer in their last line.For answer-only predictions, the boxed answer is evaluated; for CoT, the last boxed answer is used when available.Source: §4.2 · p. 4

5 Does Permutation Change LM’s Mind?

This section tests how document order, spacing, and missing first-hop evidence affect multi-hop question answering. Results vary by model and training: Flan-T5 and fine-tuned models show order-related advantages, while models may rely on parametric knowledge when context is incomplete.

5 Does Permutation Change LM’s Mind?

  • The experiments compare Forward, Backward, and Original orders; Forward_i inserts noise between ordered gold documents, and Remove First removes first-hop evidence.Forward follows the reasoning chain; Backward reverses it.Source: §5 · pp. 4–5, Figure 2 · p. 4

5.1 Order of Documents Matters

  • Fine-tuning improves all models and favors Forward; bidirectional attention adds gains and reduces variation, whereas non-finetuned decoder-only models show little consistent order effect.The authors report near-zero training-order correlations, indicating that the Forward benefit emerges through finetuning rather than sequential alignment in the training data.Source: §5.1 · p. 5

5.2 Distance between Documents Matters

  • Greater spacing between gold documents generally lowers performance; relevance-based ordering can shorten relevant-document gaps and improve end-to-end QA, motivating ranking metrics alongside recall.Finetuned models are more robust to increased spacing and to placing forward-ordered documents at the end of the context.Source: §5.2 · pp. 5–6

5.1 Order of Documents Matters

  • Flan-T5 models outperform similarly sized non-finetuned decoder-only models; 3B Flan-T5-xl beats all decoder-only models under 8B, with Forward best and Backward worst.Other tested encoder-decoder models did not show the same clear order trend.Source: §5.1 · p. 5, Table 2 · p. 6

5.3 Do LMs guess the answer?

  • When first-hop information is absent from model parameters, 2-hop accuracy generally drops; 3- and 4-hop drops are smaller for non-finetuned models, while 4-hop FT accuracy can rise.The authors interpret this as models lacking the ability to know when they lack necessary information.Source: §5.3 · p. 6
  • When removed first-hop information is stored parametrically, 2-hop accuracy usually drops less, but complex-question scores often rise as models rely on parametric knowledge despite context-only instructions.The authors warn that failure to locate all required context information may lead to hallucinations and that retrieval may sometimes harm performance.Source: §5.3 · pp. 6–7
  • Correct answers generally have higher peak IC, especially for the final hop; among 20 random shuffles, higher peak-IC rankings align with higher accuracy.Peak IC is presented as a signal for identifying a more effective context order.Source: Figure 4 · p. 7, Figure 5 · p. 7

6 Are LMs Aware of Context Permutations?

The attention analysis finds that correct answers tend to coincide with stronger peak attention, especially on the last-hop document. Ranking shuffled contexts by peak IC identifies better-performing context orders.

6.1 LMs Assign Higher Peak Attention when Correct

  • Correct samples consistently gave the largest attention score to the last-hop document, while incorrect samples distributed attention more evenly across hops.The analysis examined Answer Only, Finetuned, and Finetuned + Bi settings on MuSiQue development data.Source: §6.1 · p. 7, §6 · p. 7
  • Qwen2.5 7B median peak IC scores were 2.22 for correct samples and 1.72 for incorrect samples across 20 random shuffles.Higher peak IC among those shuffles corresponded to higher MHQA performance.Source: §6.1 · pp. 7–8
  • Across 20 random context shuffles, higher peak IC scores corresponded to higher MHQA performance, indicating peak IC as a key signal for identifying optimal context order; the best order almost doubled performance versus the worst.Source: §6.1 · p. 8

6.2 LMs Generally Favour the Last Document

  • The last document often received the most attention among noise documents, and the best shuffled contexts also showed a preference for placing a document last.The preference appeared in other causal decoder-only models, including Qwen2.5 0.5B, though less strongly.Source: §6.2 · p. 8, Figure 6 · p. 8

7 Discussion

The discussion recommends arranging retrieved documents according to the reader’s behavior: minimize relevant-document distance for off-the-shelf causal readers, and preserve forward order for fine-tuned readers. It also highlights bidirectional attention and evidence attribution as important considerations.

7.1 Optimizing LMs’ Use of Context

  • For off-the-shelf causal readers, minimize the distance between relevant documents; for fine-tuned readers, preserve forward order, and place more relevant documents near the context end in both cases.The discussion connects these recommendations to decomposed-question retrieval, where concatenating documents in question order can leave relevant documents far apart.Source: §7.1 · p. 8

7.2 Use Reader with Bidirectional Attention

  • Prefix masking with LoRA fine-tuning lets decoder-only readers outperform their causal setup and improves robustness to gold-document order; FlanT5 is a competitive alternative.The discussion frames this as a response to the limitations of causal-mask readers for MHQA.Source: §7.2 · p. 8

7.3 Attribution Is Important

  • Removing the first-hop document lowers performance but leaves accuracy relatively high, while last-hop attention remains unchanged.These findings underscore the importance of attributing predicted answers to evidence in the context.Source: §7.3 · p. 8

8 Conclusion

The conclusion reports that MHQA performance depends on architecture, document distance, and fine-tuning: encoder-decoder models outperform decoder-only counterparts, while bidirectional attention also helps decoder-only models. Fine-tuning favors forward document order and reduces sensitivity to document distance; higher peak attention aligns with accurate predictions.

8 Conclusion

  • Non-finetuned causal decoder-only LMs are invariant to gold-document order but are affected by the distance between gold documents.Fine-tuning instills a forward-order bias and makes models more robust to distance.Source: §8 · pp. 8–9
  • Encoder-decoder models outperform decoder-only counterparts, and incorporating bidirectional attention enhances performance in both encoder-decoder and decoder-only models.The conclusion also reports that increased peak attention aligns with accurate predictions.Source: §8 · p. 9

Limitations

The study’s main experiments use MuSiQue in its original setup, with at most 20 documents per question, and prompts around or below 4k tokens. The reported environment uses A100 GPUs, greedy decoding, and seed 42 for fine-tuning.

Limitations

  • The main experiments are limited to MuSiQue in its original setup, with at most 20 documents per question; most prompts are around or below 4k tokens.The authors describe MuSiQue as a challenging dataset and report additional results on relevant 2WikiMultihopQA subsets in Appendix H.Source: Limitations · p. 9

A Experiment Environment

  • Experiments used NVIDIA A100 80GB GPUs, greedy decoding, and seed 42 for fine-tuning.The study used Huggingface Transformers and PEFT for LoRA fine-tuning.Source: A Experiment Environment · p. 11

B Model Specification

These appendix sections specify model and prompting setups, then report attention-distribution analyses, additional encoder-decoder results, and a closed-book experiment setup.

B Model Specification

C Prompt Details

  • The prompt instructs models to answer using only provided search results, which may include irrelevant documents.For the Answer Only setup, the prompt ends with an added boxed-answer cue.Source: C Prompt Details · p. 11

E.1 Model behave differently with Backward and Forward

  • In Forward order, attention favors the first-hop document except the last; in Backward order, hop-level differences are minor, including for finetuned models.This behavior is mainly observed on 3- and 4-hop questions, where the first- and last-hop documents are farther apart.Source: §E.1 · pp. 11–12

D Detailed statistics of peak IC score

  • Figure 7 summarizes peak IC scores for correct and incorrect samples.The supplied caption identifies the compared groups but does not state their score differences.Source: Figure 7 · p. 12

E.2 How does distance change the LM’s mind?

  • With five noise documents between gold documents, attention favors the first hop except the last; with none, attention follows reasoning-hop order, declining from last hop to first.The authors interpret the longer-distance pattern as requiring more attention to the first hop to retain essential information.Source: §E.2 · p. 12

E.3 How does removing the first hop change the LM’s mind?

  • Removing the first-hop document most affects 2-hop samples, while attention to remaining documents is nearly unchanged for 3- and 4-hop questions.The authors connect these findings to evidence attribution and warn that ignoring evidence completeness can produce hallucinations.Source: §E.3 · p. 12

F Experiment on other encoder-decoder models

  • T0pp’s forward-order preference disappeared; because P3 is a Flan-dataset subset, the authors consider training data, rather than architecture, a possible source of the behavior.They suggest future work identify which Flan dataset may trigger the preference.Source: F Experiment on other encoder-decoder… · p. 12

G Closed-book Result

  • The closed-book MuSiQue development experiment provides no context, reflecting the authors’ view that MHQA is difficult without external knowledge.Table 6 reports the evaluation results, but its values are not included in the supplied passages.Source: G Closed-book Result · p. 12

H Experiment Result on Other MHQA dataset

On 2WikiMultihopQA, the paper tests transfer across two 2-hop subsets and reports document-order, attention-mask, and distance findings, alongside peak-IC answer selection results.

H Experiment Result on Other MHQA dataset

  • The transfer evaluation uses 5,234 compositional and 1,549 inference questions, all 2-hop, with 10 documents including two gold documents.The evaluated models include Flan T5, Qwen 2.5 7B, and Llama 3.1 8B variants, including finetuned variants where applicable.Source: H Experiment Result on Other MHQA… · pp. 12–13
  • 42.14 average accuracy across 10 shuffles versus 49.29 for the highest-peak-IC sample on compositional questions; inference accuracy was 14.96 versus 19.43.These comparisons use Qwen 2.5 7B with Answer Only and randomly shuffled document orders.Source: H Experiment Result on Other MHQA… · p. 13
  • On both subsets, forward-order preference persists, while bidirectional attention boosts performance and is more robust to gold-document order than causal attention.The tables report results for the compositional and inference development subsets.Source: H Experiment Result on Other MHQA… · p. 13, Table 7 · p. 14, Table 8 · p. 14
  • Finetuned models are more distance-robust, even without same-dataset finetuning; non-finetuned models lose performance as gold-document distance increases.Figure 11 presents distance results on the 2WikiMultihopQA development set.Source: H Experiment Result on Other MHQA… · p. 13, Figure 11 · p. 14
Loading 2505.11754v1…