PaperlayerView exact revision on arXiv

Source-linked AI summary

Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression

Zhengpei Hu, Kai Li, Dapeng Fu, Xuechao Zou, Yuanhao Tang, Yue Li, Tengfei Cao, Jianqiang Huang

arXiv:2608.04569v1cs.CLcs.LG

TL;DR

Hard prompt compressors can retain answer text while deleting the dependent evidence needed to interpret it, producing referential dangling. The paper formalizes and measures this failure, then tests fixed-budget restoration, finding that reinserting omitted support improves accuracy by 29–34 points on affected examples.

  • Problem

    Hard compressors independently select fragments, but important retained text may depend on omitted evidence, leaving the extent of referential dangling insufficiently characterized.

  • Method

    The paper formalizes answer-path dangling, measures it across compressors and QA datasets, and evaluates fixed-budget support reselection and automatic restoration.

  • Results

    29–34 percentage points: fixed-budget reselection improves Qwen3-8B accuracy on dangling examples across three multi-hop QA datasets.

  • Takeaways & Limitations

    Hard compression should preserve both relevance and the dependencies required to interpret retained evidence.

  • Takeaways & Limitations

    Transfer beyond Beaver and annotated-support QA remains unresolved, with restoration gains on three additional compressors not statistically significant.

Abstract

from arXiv · show

Hard prompt compression reduces long-context inference cost by independently scoring tokens, sentences, or chunks and retaining the highest-scoring units under a budget. We identify a structural failure in this procedure: independent selection can split dependent evidence pairs, retaining one member while deleting the other. When retained text contains an answer but deleted text defines the entity needed to interpret it, we call the result referential dangling. At a compression ratio of 0.30, Beaver, which ranks coherent chunks using Qwen3-0.6B embeddings, leaves the answer path incomplete in 34-54% of bridge examples across three multi-hop question answering datasets. On a shared HotpotQA bridge set, all six hard compressors we test exhibit dangling at rates up to 60%, and every document in LongBench-v2 Single-Document QA contains at least one dangling reference. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing nonsupporting paragraphs to maintain the token budget improves accuracy by 29-34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Stronger answer models do not absorb the loss: on MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs. Finally, we train a compact classifier to rank omitted sentences by whether they are needed to interpret retained text and reinsert the top-ranked candidates without support annotations at inference. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while changing the compression ratio only from 0.30 to 0.31. Hard compressors should optimize both relevance and referential completeness.

1 Introduction

Long-context inference is costly, motivating hard prompt compression, but independent fragment selection can remove dependencies needed to interpret retained evidence. The paper names this failure referential dangling and shows it is widespread across compressors and datasets.

  • Motivation: Expanded context windows enable document-scale reasoning but remain expensive because self-attention scales quadratically and models underuse content away from prompt boundaries.Prompt compression prunes inputs to reduce this cost while preserving task-relevant information.
  • Illustrative mechanism: A missing bridge can leave the answer string and related facts present while deleting the statement that connects the queried entity to that answer.The Tim DuBois example shows why relevance alone does not guarantee a usable compressed context.
  • Compression procedure: Hard compressors independently score tokens, sentences, or chunks and retain the highest-scoring fragments within a token budget.Scoring signals include relevance, perplexity, embedding similarity, information content, classifier scores, syntactic salience, and attention-derived importance.
  • Failure mode: Referential dangling occurs when compression retains a relevant fragment but omits a dependency required to interpret it, making the fragment unusable.Thus, retaining an answer string does not ensure that the remaining context supports the inference needed to connect it to the query subject.
  • Empirical diagnosis: 34 to 54% of bridge examples have incomplete answer paths under Beaver at compression ratio 0.30 across HotpotQA, 2WikiMultiHopQA, and MuSiQue.A human audit finds 95% precision.
  • Empirical diagnosis: 32 to 60% dangling rates occur across all six tested compressors on a shared bridge set, while all 80 LongBench-v2 Single-Document QA documents contain dangling references.These findings extend the diagnosis beyond Beaver and bridge examples.

2 Related Work

Prior work compresses prompts through extractive, task-aware, soft or latent, and hidden-state approaches. Dependency-preserving extractive summarization constrains selection, whereas hard prompt compressors select salient units under a budget without that constraint.

  • Prompt and context compression: Prompt compression reduces long-context inference cost by shortening inputs before they reach the target model.It complements model-side efficiency methods such as elastic subnetworks and efficient encoder–decoder or state-space architectures.
  • Prompt and context compression: Task-aware selectors use reinforcement-learning rewards or key-information-density objectives, while soft approaches use gist tokens, autoencoders, semantic source coding, or learned special-token representations.These approaches represent distinct alternatives to extractive source-text selection.
  • Prompt and context compression: Hidden-state and KV-cache compression methods avoid the studied failure because they do not produce a reduced text sequence.They compress hidden activations or KV caches rather than source text.
  • Dependency-preserving selection: Extractive summarization preserves dependencies by adding anaphoricity constraints that require pronoun-dependent text to be selected with the pronouns.Hard prompt compressors inherit the selection problem but score units by salience or query relevance under a budget without this constraint.

3 Problem Formulation

The formulation defines referential dangling as retaining a task-relevant sentence while omitting every inclusion-minimal sufficient support set needed to interpret or use it. It then shows that additive hard-compression objectives do not guarantee referential completeness, even when a complete selection fits the same budget.

  • Problem setup: The downstream language model receives only the query and compressed context, while extractive compression emits at most τ tokens from an original context exceeding τ tokens.The definition applies to extractive hard compressors and distinguishes query-agnostic compressors from query-dependent ones.
  • Referential dangling: Referential dangling occurs when a retained task-relevant sentence has no fully retained sufficient support set among its inclusion-minimal alternatives.Support is defined jointly with the retained sentence and query, using only explicit information in the original context.
  • Additive fragment selection: Additive selection combines independently assigned fragment utilities under a global modular knapsack objective, with no interaction term enforcing complete support.The formulation covers nonoverlapping token, sentence, and fixed-chunk selectors whose final selection solves the stated objective.
  • No guarantee of complete support: Proposition 1 constructs three equal-cost fragments where utilities satisfy u(x) > u(z) > u(d) ≥0, causing selection of {x, z} and omission of x’s sole support d.The same budget also permits {x, d}, which is complete, so additive optimization alone provides no guarantee of complete support.

4 Empirical Diagnosis of Referential Dangling

Referential dangling is widespread under hard prompt compression, increasing as compression becomes more aggressive and recurring across datasets, document types, and compressor designs. The affected examples vary substantially by scoring method, indicating a broad diagnostic failure rather than a single selector-specific artifact.

  • Prevalence: At r = 0.30, dangling rates range from 34.2% to 54.2% across HotpotQA, 2WikiMultiHopQA, and MuSiQue.A manual audit found the omitted paragraph was required in 38 of 40 flagged examples, yielding 95% precision.
  • Long documents: 30.5% is the mean per-document dangling rate across 80 LongBench-v2 Single-Document QA documents, and every document contains at least one dangling reference.The separate first-mention diagnostic counts cases where a later reference survives but its entity’s first-mention sentence is omitted.
  • Variation across measured conditions: 53.5% is the dangling rate at r = 0.20 on the broader HotpotQA sample, compared with 6.3% at r = 0.70.Groups with larger annotated hop counts also have larger dangling rates, although dataset construction prevents isolating reasoning depth.
  • Results across scoring signals: 59.8% is the highest dangling rate among six compressors on the shared HotpotQA bridge set, versus 32.1% for Beaver.The six methods use embedding similarity, syntactic structure, self-information, token classification, perplexity, or attention across multiple output granularities.
  • Results across scoring signals: 47.8% is PartPrompt’s dangling rate on the shared set, while Beaver and LongLLMLingua occupy opposite ends despite both using the query.Thus, query access alone does not remove the diagnostic event, which also occurs under hierarchical syntactic selection.
  • Overlap of affected examples: 95% of examples dangle under at least one compressor, but only 5.4% dangle under all six; mean pairwise Jaccard similarity is 0.38.The affected sets therefore vary across scoring methods rather than reflecting one fixed subset of failures.

5 Fixed-Budget Content Reselection

Fixed-budget reselection restores omitted supporting paragraphs while removing nonsupporting content, substantially improving accuracy on dangling examples. Full-support contexts also outperform base compressed contexts across downstream models, confirming the cost of incomplete evidence.

  • Protocol limitation: The reselection protocol depends on annotated supporting paragraphs and is applied across all three multi-hop QA datasets used in the prevalence analysis.Because Base and Reselected remove different nonsupporting paragraphs, the comparison measures the joint change in retained content.
  • Comparison protocol: Fixed-budget reselection reinserts the omitted supporting paragraph and removes at least as many tokens from lowest-scoring nonsupporting paragraphs.Base is the original compressed context; Full support retains both annotated supporting paragraphs as a complete-evidence reference.
  • Accuracy on dangling examples: 28.8–34.3 accuracy points: Reselected improves over Base with p < 10−4 on all three datasets.Reselected contexts are shorter than Base on average, so the gains cannot be explained by a larger token budget.
  • Results across downstream models: 8.0–9.0 accuracy points: Full support improves over Base across four downstream models, with p < 0.01 for all models.The comparisons remain significant after Holm correction at α=0.05; Qwen3-4B and Qwen3-8B gain 8.5 and 8.0 points, respectively.

6 Automatic Context Restoration

The section evaluates targeted sentence restoration as an inference-time diagnostic using a candidate generator and classifier, finding modest accuracy gains at nearly unchanged compression. Gains depend on candidate source and do not yet reliably transfer across compressors.

  • Restoration pipeline and training: A candidate generator collects potentially supporting omitted sentences, which a bert-base-uncased classifier ranks using retained sentence, candidate sentence, and question.Training pairs come from HotpotQA and use shared entities between retained and omitted annotated supporting sentences.
  • Downstream accuracy: 4.7 points: Qwen3-8B accuracy improves while compression changes from 0.30 to 0.31.The improvement is statistically significant (p=0.022).
  • Candidate sources: 4.7 points on HotpotQA versus 0.5 points on 2WikiMultiHopQA for first-mention candidates; the hybrid gains are 4.5 and 5.5 points, respectively.Including the annotated supporting sentence diagnostically raises the gain to 8.0 points (p=0.008) while adding 0.4 sentences on average.
  • Matched addition control: 2.0 points: random sentence addition improves accuracy, compared with 4.7 points for targeted restoration.The direct contrast is suggestive but not significant at this sample size (p=0.15).
  • Transfer across compressors: Smaller, non-significant gains on three additional compressor outputs leave transfer unresolved.The configuration was calibrated on Beaver and HotpotQA, while compressor output granularity varies.

7 Conclusion

The paper identifies referential dangling as a structural failure of independent hard compression: task-relevant text can remain while interpretive support is removed. The failure is widespread across datasets, compressors, and long-document diagnostics.

  • Core finding: Referential dangling occurs when independent hard compression retains task-relevant text but removes support required to interpret it.The failure concerns incomplete evidence paths created by independently selecting compression units.
  • Core finding: At r = 0.30, paragraph-level dangling occurs in 34% to 54% of bridge examples with a retained answer paragraph under Beaver across three multihop QA datasets.This quantifies the prevalence of dangling when the answer paragraph is retained but its interpretive support is removed.
  • Core finding: The event recurs across six compressors on HotpotQA, while a LongBench-v2 Single-Document QA diagnostic flags every evaluated document.These findings indicate that dangling is not limited to Beaver or to a single dataset.

A Robustness to Experimental Choices

The dangling-reference diagnosis remains stable when varying the Beaver embedding model, overlap threshold, and compression ratio. Across these checks, the finding is not tied to a single experimental configuration.

  • Overall robustness: The appendix varies the embedding model, overlap threshold, and compression ratio, and reports that the diagnostic remains stable across all variations.Table 7 summarizes the three robustness checks.
  • Embedding scorer: At most 0.9 percentage points: replacing Qwen3-0.6B with GPT-2 changes the HotpotQA dangling rate by no more than this amount at ratio 0.30.This comparison uses n=300 HotpotQA examples and measures shifts relative to the released Qwen3-0.6B embedding setup.
  • Overlap threshold: 0.3–0.7: sweeping the content-word-overlap threshold keeps dangling substantial for both chunk-level and token-level compressors across the full range.The unified cross-compressor metric uses the sentence-level Keepθ predicate, while paragraph-level prevalence uses a separate paragraph instantiation.
  • Overlap threshold: Up to 0.6: LLMLingua-2 dangles at least as much as Beaver at every threshold through this value, including the setting most lenient to token fragments.This comparison is reported across the threshold sweep for chunk-level and token-level compressors.

B Manual Audit and Qualitative Examples

The manual audit illustrates referential dangling through real examples where the answer remains but its interpretive bridge is deleted. Among 38 confirmed cases, the omitted paragraph most often contained the dataset answer string, followed by an entity definition or another dependency.

  • Representative Cases: Real examples from each dataset show the “answer present but bridge deleted” mechanism in original contexts and compressed outputs.The examples are drawn from the experimental data and each panel displays the original context alongside the compressed output.
  • Manual Audit: 26 of 38 confirmed cases omitted a paragraph containing the dataset answer string.The manual audit found 38 confirmed dangling cases overall.
  • Manual Audit: 10 of 38 confirmed cases omitted the entity definition, while 2 omitted another dependency.These categories account for the remaining confirmed cases after those containing the dataset answer string.
  • Manual Audit: 2 false positives arose when the answer paragraph was independently sufficient or the omitted paragraph supplied only a generic modifier.These were not genuine referential-dangling failures.

C Long-Document Dangling by Subdomain · D Protocol for Fixed-Budget Content Reselection · E Compressor Configurations

At ratio 0.30, referential dangling affects retained sentences consistently across all LongBench-v2 subdomains and every document has at least one dangling reference. The reselection protocol restores omitted supporting paragraphs under a fixed budget, while six compressors are evaluated under standardized configurations.

  • C Long-Document Dangling by Subdomain: 25.1% to 36.5%: Per-kept-sentence dangling rates span all seven LongBench-v2 subdomains, and every document contains at least one dangling reference.Academic text has the highest rate and combines dense terminology, frequent cross-section references, and the longest retained-sentence counts.
  • D Protocol for Fixed-Budget Content Reselection: Base, Reselected, and Full support compare original compression, fixed-budget reinsertion of one supporting paragraph, and retention of both supporting paragraphs.Reselected removes nonsupporting paragraphs whose combined length is at least as large as the reinserted paragraph.
  • D Protocol for Fixed-Budget Content Reselection: 1.5%, 1.6%, and 9.6%: Reselected contexts are shorter than Base on average for HotpotQA, 2WikiMultiHopQA, and MuSiQue, respectively.Omitted supporting paragraphs are ranked by isolated compressor salience, and the highest-scoring paragraph is reinserted.
  • D Protocol for Fixed-Budget Content Reselection: p < 10^-4: Base versus Reselected is significant on the dangling subsets of all three datasets under an exact two-sided McNemar test.Base versus Full support reaches p < 0.01 for all four downstream LLMs on the 200-example HotpotQA set.
  • E Compressor Configurations: 184 HotpotQA bridge examples: All six compressors keep 30% of tokens at compression ratio 0.30 using standardized evaluation conditions.Official implementations are used where available, and training-free methods require no adaptation.
  • E Compressor Configurations: Beaver selects top-k 64-token pages using query-aware cosine similarity from Qwen/Qwen3-0.6B embeddings, whereas LLMLingua-2 and Selective-Context are query-agnostic token scorers.LLMLingua-2 uses a trained token classifier; Selective-Context uses token self-information under Llama-2-7b-hf.
  • E Compressor Configurations: PartPrompt uses syntactic salience, LongLLMLingua uses query-conditioned perplexity, and DAC uses attention-derived token importance with α=0.8 and compress_ratio 0.70.PartPrompt selects parse-tree nodes with dynamic programming, while DAC retains 30% of tokens using Qwen/Qwen3-0.6B attention scores.

F Implementation Details for Automatic Context Restoration

The automatic restoration pipeline uses Beaver’s sentence-level outputs and a classifier that ranks omitted candidates for reinsertion. The selected K=3 configuration adds about 1.8 sentences per example and yields the largest observed accuracy gain in the reported sweep.

  • Evaluation setup: The evaluation uses Beaver at compression ratio 0.30, while restored contexts average ratio 0.31 and the classifier is fixed at K=3.The automatic restoration evaluation uses 300 HotpotQA examples, except for Mistral-7B, which uses 200.
  • Output granularity: Beaver is used because its coherent multi-sentence blocks support controlled restoration at sentence boundaries while preserving interpretability.Token-level compressors can produce fragmented text, making complete-sentence insertion harder to compare with the original selection objective.
  • Classifier architecture and training: The classifier is bert-base-uncased with a binary head over a retained sentence, candidate sentence, and question, using a 256-token maximum sequence length.It is fine-tuned for three epochs with AdamW at 2 × 10^-5, batch size 32, 10% warmup, and fp16.
  • Training pair construction: Positive pairs are retained–omitted supporting-sentence pairs sharing an entity, while hard negatives share an entity without satisfying the positive rule.Positive pairs comprise 1.9% of candidate pairs; negative downsampling produces 7,565 rebalanced pairs with 40% positives.
  • Number of restored sentences (K): +4.7 points is the largest observed gain in the Qwen3-8B sweep, achieved with K=3 restored candidates.K=2 gives +3.7 points and adding all candidates gives +4.3 points; K=3 adds about 1.8 sentences per example.

G Automatic Context Restoration Results and Controls · H Transfer Across Compressor Outputs

The paper evaluates automatic context restoration through downstream accuracy, outcome comparisons, and a matched-addition control, then tests whether one fixed restoration configuration transfers across compressor outputs. These analyses separate restoration effectiveness from sentence-count effects and compressor-specific tuning.

  • G Automatic Context Restoration Results and Controls: Table 10 reports downstream accuracies and significance tests for automatic restoration.Figure 7 compares restoration gains with gains from retaining both supporting paragraphs.
  • G Automatic Context Restoration Results and Controls: Among 130 Beaver-incorrect examples, corrected and still-incorrect cases receive similar numbers of restored sentences.Thus, restoration size alone does not distinguish the outcomes.
  • G Automatic Context Restoration Results and Controls: Table 11 compares examples corrected by automatic restoration with those that remain incorrect among Beaver’s 130 incorrect cases.The comparison directly examines outcome differences rather than restoration volume alone.
  • G Automatic Context Restoration Results and Controls: Random insertion and targeted restoration add the same number of sentences per example in the matched addition control.This controls for the amount of added text when comparing the two procedures.
  • H Transfer Across Compressor Outputs: Table 13 applies the same restoration configuration to every compressor output with Qwen3-8B on HotpotQA.The experiment tests transfer across compressor outputs rather than separately optimized configurations.
  • H Transfer Across Compressor Outputs: The classifier and candidate generator are calibrated on Beaver’s retained sentence distribution and HotpotQA first-mention structure.This calibration fixes the restoration setup before evaluating transfer to other compressor outputs.
Loading 2608.04569v1…