PaperlayerView exact revision on arXiv

Source-linked AI summary

IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering

JungMin Yun, YoungBin Kim

arXiv:2608.13588v1cs.CLcs.AI

TL;DR

Multi-hop QA requires reasoning across interdependent evidence distributed across documents, but single-query compression methods do not capture these dependencies. IterCOMP adds iterative answerability assessment and targeted follow-up questions to compress contexts, achieving higher QA scores while reducing input length across three benchmarks.

  • Problem

    Existing query-focused compression methods rely mainly on single-query relevance, limiting their ability to model interdependent evidence and multi-hop reasoning across documents.

  • Method

    IterCOMP iteratively filters evidence, assesses answerability, and generates targeted follow-up questions to retain essential reasoning information in a compact prompt.

  • Results

    IterCOMP achieves the highest EM and F1 scores across all three benchmarks, including a MuSiQue F1 increase from 19.92 to 27.36 with a 7× input-length reduction.

  • Takeaways & Limitations

    IterCOMP provides a training-free, model-agnostic approach for improving multi-hop QA performance while reducing token usage.

  • Takeaways & Limitations

    Its effectiveness depends on the underlying LLM’s reasoning for answerability and missing-information detection, while iterative compression adds overhead and hyperparameter sensitivity.

Abstract

from arXiv · show

Multi-hop question answering requires complex reasoning across multiple evidence segments, which often overwhelms retrieval-augmented generation systems with lengthy and noisy contexts, thereby undermining both efficiency and accuracy. While existing prompt compression methods attempt to address this issue, they are typically designed for single-turn queries and fail to capture interdependent reasoning steps. We propose IterCOMP, a unified, training-free prompt compression framework that incorporates multi-hop reasoning within an iterative compression loop. IterCOMP decomposes documents into evidence segments, evaluates question answerability, and generates targeted follow-up questions to iteratively integrate essential evidence, producing a compact, reasoning-oriented prompt. Experiments on MusiQue, 2WikiMultiHopQA, and HotpotQA demonstrate that IterCOMP achieves substantial improvements in Exact Match and F1 scores while reducing the token budget, outperforming existing baselines and exhibiting robustness as reasoning complexity increases.

1 Introduction

Multi-hop QA requires integrating evidence across multiple documents, but long retrieved contexts increase inference latency, computational overhead, and API costs while single-query compression struggles with complex multi-hop scenarios. IterCOMP addresses this gap through iterative prompt compression that explicitly incorporates multi-hop reasoning and assesses whether retained evidence is sufficient.

  • Motivation: Multi-hop QA requires integrating evidence from multiple documents through implicit intermediate clues absent from the initial question.The task derives answers by reasoning over multiple pieces of evidence.
  • Motivation: Longer retrieved contexts increase inference latency, computational overhead, and operational costs.Input length scales linearly with the number of retrieved documents, increasing latency and computation; commercial API inputs also become more expensive.
  • Compression challenge: Effective compression must selectively retain query-relevant information while removing irrelevant or distracting content.Simply reducing input length is insufficient for effective compression.
  • Compression challenge: Single-query compression primarily uses surface-level query-document relevance, limiting effectiveness in complex multi-hop QA.Existing query-focused methods condense prompts around a given query but predominantly rely on a single-query paradigm.
  • IterCOMP: IterCOMP explicitly incorporates multi-hop reasoning into an iterative prompt-compression loop.The framework filters relevant evidence segments and uses an LLM to assess whether they are sufficient to answer the question.

2 Related Work

Prior work compresses prompts through soft representations or hard extraction/abstraction, with query-aware methods refining content using query-conditioned importance or similarity. However, their one-to-one relevance paradigm is insufficient for multi-hop QA, motivating reasoning over linked evidence and follow-up questioning.

  • Prompt Compression: Soft prompt compression encodes prompts into continuous vectors, while AutoCompressor segments, compresses, and concatenates soft representations and GIST generates prefix-style soft prompts.These methods support parameter-efficient adaptation while preserving the h…
  • Prompt Compression: Hard prompt compression shortens prompts through extractive or abstractive methods, removing redundant or uninformative content to improve efficiency and generated-output quality.The passage distinguishes query-agnostic compression from query-aware approaches.
  • Query-Aware Compression: Query-aware methods estimate importance with query-conditioned perplexity, jointly analyze selected and new content, or compare queries with sentence embeddings.The cited methods include LongLLMLingua, COMPACT, and RECOMP; the passage also introduces R2C as encoding queries with context chunks.
  • Limitations for Multi-Hop QA: Existing query-aware methods typically match one query against individual documents or sentences, overlooking the inter-document dependencies and linked evidence required by complex multi-hop QA.Multi-hop queries contain subcomponents that must be jointly resolved across distributed evidence.
  • Follow-Up Questioning: LLM-generated follow-up questions can improve document-generation coherence and informativeness, support deeper conversational-search exploration, and help decompose complex queries through self-ask.Self-ask involves generating and answering sub-questions.

3 Preliminary Analysis

The preliminary analyses test whether LLMs can judge evidence sufficiency and identify missing information for multi-hop questions. Results indicate that answerability judgment becomes harder as hop length increases, while classification errors can either prematurely terminate or unnecessarily extend compression.

  • Research Questions: IterCOMP relies on LLMs to determine whether evidence is answerable and, when it is not, identify the additional information needed for the correct answer.These assumptions motivate the two preliminary research questions on answerability judgment and missing-information identification.
  • Experimental Setup: The analyses sample 400 MuSiQue QA pairs for each hop length from two to four reasoning steps and evaluate LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct, GPT-3.5-Turbo, and GPT-4o.GPT-4o generates sub-question and sub-answer pairs from the dataset’s decomposition annotations.
  • Answerability Judgment: Answerability is tested with Full evidence containing all required sub-answers and Partial evidence containing sub-answers only through an intermediate hop.Full cases are labeled answerable, whereas Partial cases are labeled unanswerable.
  • Missing Information Identification: Missing-information identification provides partial evidence and evaluates generated follow-up questions against reference sub-questions using BLEU and Relatedness score.BLEU measures n-gram overlap, while Relatedness score uses GPT-4o binary classification to assess topical or conceptual alignment.
  • Answerability Judgment: 92% to 78.5%: Mistral-7B accuracy declines from 2-hop to 4-hop in the Full setting.The decline suggests that increasing hop length raises the volume of evidence and the complexity of its interconnections, making answerability judgment more than surface-level binary classification.
  • Answerability Judgment: Misclassifying an unanswerable case as answerable can terminate compression prematurely, whereas misclassifying an answerable case as unanswerable triggers unnecessary iterations.The two error types respectively risk insufficient reasoning clues and avoidable additional compression steps.

4 Methodology

IterCOMP constructs a compact prompt from retrieved documents while preserving information needed for high-fidelity multi-hop reasoning. It segments evidence, combines semantic and lexical relevance, adaptively filters candidates, and iteratively accumulates evidence until the question is answerable or the hop limit is reached.

  • Task Definition: Prompt compression synthesizes an information-sufficient evidence set whose token length is much smaller than the retrieved documents while preserving essential reasoning information.The compressed prompt is passed to a reader model to generate the output.
  • Evidence Segmentation: Documents are decomposed into sentence-level evidence segments, preserving local semantic context while enabling effective compression.Each document d_i yields an evidence-segment set E_i.
  • Relevant Evidence Filtering: Relevant evidence filtering combines semantic alignment with lexical overlap to capture contextual relevance and precise keyword cues in multi-hop settings.Semantic similarity uses text-encoder embeddings, while lexical similarity weights overlapping tokens by contextual importance.
  • Relevant Evidence Filtering: Percentile-based filtering retains segments above a global relevance threshold and reapplies this adaptive cutoff at each reasoning step to suppress noise propagation.The retained candidate set forms the basis for the compressed prompt and guides attention throughout the reasoning chain.
  • Iterative Evidence Accumulation: An LLM judges answerability at each iteration, stopping when accumulated evidence suffices; otherwise, it identifies missing information, generates a follow-up question, and refilters evidence.The cycle continues until sufficiency is reached or a predefined maximum hop limit is encountered.

5 Experiments

Experiments on three multi-hop QA dev sets show that IterCOMP improves QA performance while compressing prompts, with gains maintained across reasoning complexity and component ablations. The framework also adapts iteration depth and evidence size to hop length and reduces API inference costs.

  • Experimental Setup: IterCOMP is evaluated zero-shot on MuSiQue, 2WikiMultiHopQA, and HotpotQA dev sets, using LLaMA-3-8B as the reader model.MuSiQue uses the answerable musique_ans_v1.0_dev subset.
  • Main Results: IterCOMP achieves the highest EM and F1 across all benchmark datasets while outperforming hard and extractive compression baselines.Compared methods include LLMLingua, LongLLMLingua, LLMLingua-2, RECOMP, Selective-Context, and R2C.
  • Main Results: 27.36 F1 on MuSiQue exceeds R2C by 4.92 points in a training-free setting.The iterative design retains relevant information to address missing evidence under strict input-length constraints.
  • Main Results: On HotpotQA, IterCOMP attains 51.79 F1 and closes 53.2% of the 15.32-point Raw Documents–Oracle gap.Oracle represents an upper bound based on ideal document selection.
  • Reasoning Complexity: As MuSiQue hop length rises from 2-hop to 4-hop, F1 drops from 30.19 to 20.72, while iterations increase from 1.98 to 3.08 and compressed prompts expand from 228 to 403 tokens.These changes indicate adaptive refinement and evidence retention for longer reasoning chains.
  • Ablation and Efficiency: Removing Iterative Refinement causes the largest ablation drop, while no-compression yields 19.92 F1; API compression reduces costs by 75.5%–79.4%.The filtering threshold peaks at 27.36 F1 when k = 90, but performance drops at k = 95.

6 Conclusion

IterCOMP is a unified, training-free prompt compression framework that embeds multi-step reasoning into iterative compression for multi-hop question answering. It improves QA performance while reducing token usage and supports broad deployment, including with commercial black-box LLMs.

  • IterCOMP integrates answerability judgment and targeted follow-up question generation into iterative refinement, retaining only evidence essential for complex reasoning.The process progressively constructs a concise, information-rich prompt.
  • Experiments on three multi-hop QA benchmarks show that IterCOMP enhances QA performance while reducing token usage.
  • Its training-free, model-agnostic design enables integration with commercial black-box LLMs where fine-tuning is infeasible.The framework addresses performance degradation in long contexts and the economic burden of large-scale inference.

Limitations · A Prompts

IterCOMP’s limitations include dependence on the underlying LLM’s reasoning abilities and extra overhead from iterative compression. The paper also provides prompt examples for sub-QA generation and answerability judgments with missing-information identification.

  • Limitations: IterCOMP’s effectiveness depends on the underlying LLM’s reasoning capability for answerability judgment and missing-information detection.This dependence can introduce reasoning errors into the compression process.
  • Limitations: Errors in LLM judgments may propagate through premature termination or accumulation of irrelevant evidence.These failure modes can cause the iterative process to stop too early or retain unnecessary context.
  • Limitations: The iterative design incurs additional compression overhead compared with single-pass methods.The added cost arises from repeatedly processing and integrating evidence during compression.
  • Limitations: Lightweight controllers or parallel filtering are suggested as ways to alleviate iterative compression overhead.These approaches are presented as potential refinements rather than evaluated solutions.
  • A Prompts: The prompt appendix includes generated examples for sub-QA pair generation.These examples are presented in the top portion of Table 7.
  • A Prompts: The prompt appendix also includes examples for answerability judgment with missing-information identification.These examples are presented in the bottom portion of Table 7.

B Implementation Details · B.1 Datasets

The experiments use three standard multi-hop QA benchmarks: MuSiQue, 2WikiMultiHopQA, and HotpotQA. These datasets test multi-step reasoning, evidence integration, and transparent reasoning paths across varying complexities.

  • B.1 Datasets: MuSiQue combines multiple single-hop queries into questions requiring 2 to 4 reasoning hops.Its construction is intended to prevent answers from being derived through superficial clues.
  • B.1 Datasets: 2WikiMultiHopQA is a Wikipedia-based dataset with questions requiring up to five reasoning steps.Each question includes an explicit reasoning path and supporting evidence.
  • B.1 Datasets: HotpotQA requires retrieving and synthesizing evidence from multiple documents.Its explainable questions encourage transparent and interpretable reasoning paths.
  • B.1 Datasets: The benchmark suite comprises MuSiQue, 2WikiMultiHopQA, and HotpotQA.These are described as three widely used standard datasets for the experiments.
  • B.1 Datasets: For each dataset, the reported splits are train, dev, and test.The dataset summaries specify the number of instances in each split.
  • B.1 Datasets: The benchmark statistics are presented in Table 8.The table reports statistics for the multi-hop QA datasets used in the experiments.

B.2 Baselines

The baselines comprise extractive, token-level, and question-aware prompt compression methods implemented with officially released codebases and publicly available models. They include RECOMP, Selective-Context, LLMLingua variants, LongLLMLingua, and R2C, with Table 9 additionally reporting abstractive RECOMP and CompAct.

  • All experiments use officially released codebases and publicly available models to ensure reproducibility.
  • RECOMP selects relevant sentences with a dual-encoder model, while Selective-Context removes low-information lexical units using a compact language model.
  • Table 9 reports full results on three multi-hop QA benchmarks, including abstractive RECOMP and CompAct.
  • LLMLingua removes low-perplexity tokens under a predefined compression ratio, and LLMLingua-2 adds a refined budget controller and data distillation.
  • LongLLMLingua uses question-aware coarse-to-fine compression, whereas R2C retains important chunks and sentences based on FiD cross-attention scores.

B.3 Additional Experiments Results

The additional experiments compare IterCOMP with abstractive compression methods while highlighting fairness issues caused by inconsistent compression-length control. RECOMP and CompAct are therefore excluded from the main comparison, with their complete results reported separately in Table 9.

  • RECOMP synthesizes summaries across retrieved documents, while CompAct jointly analyzes selected content and newly introduced segments for context-aware compression.
  • Fair comparison is challenging because IterCOMP and most baselines lack strict token-level constraints, while RECOMP and CompAct deviate substantially more from the target token budget.
  • RECOMP and CompAct use free-form generation without explicit length control and are excluded from the main comparison because they cannot reliably regulate compressed-prompt length.
  • Their full results are reported in Table 9, with the comparison explicitly noting fairness limitations from inconsistent compression budgets.
Loading 2608.13588v1…