PaperlayerView exact revision on arXiv

Source-linked AI summary

MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression

Ke Wan, Yifan Wang, Liheng Lai, Chen Chen

arXiv:2609.27380v2cs.CL

TL;DR

Sequential likelihood-based compression can make evidence scores depend on context order, so the same retrieved contexts may preserve different evidence under different permutations. MORSE combines a reverse-likelihood evidence-first anchor with compression-aware permutation selection. The paper reports improved evidence retention over reverse ordering and generally over compute-matched random search.

  • Problem

    Sequential scoring accounts for cross-context redundancy, but makes evidence scores dependent on the order of retrieved contexts.

    Source: §1 · p. 2

  • Method

    MORSE uses reverse query evidence for an evidence-first anchor and evaluates alternative permutations with the compressor before selecting among their compressed outputs.

    Source: §1 · p. 2, §5.2 · p. 5

  • Results

    MORSE-5 improves SF-R over RANDOMSEARCH-5 in all 12 1P settings and all 12 ISD settings, and over REVERSE across all reported settings.

    Source: §6.2 · p. 6

  • Takeaways & Limitations

    Context ordering can materially change what a sequential compressor preserves, making evidence retention a consequential outcome of ordering.

    Source: §7 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation often relies on multiple retrieved contexts that contain substantial redundancy, motivating context compression to preserve useful information under limited input budgets. Likelihood-based compressors can account for cross-context redundancy through sequential scoring, but this makes evidence scores dependent on context order. We show that permuting the same contexts under an unchanged compressor can substantially change which supporting evidence survives compression. We attribute this sensitivity to information preemption: earlier, partially relevant contexts can absorb credit for shared information, reducing the incremental scores of later, stronger evidence and increasing its risk of removal. Controlled pair-swap interventions provide direct empirical support for this mechanism by showing that placing stronger evidence before overlapping, partially relevant contexts can improve its survival. Based on this insight, we introduce MORSE, a compression-aware method for evidence-preserving context ordering. MORSE uses reverse query likelihood to construct an evidence-first anchor and to evaluate compressed candidate outputs, enabling compression-aware selection among alternative permutations. Across multi-hop Question Answering (QA) benchmarks, compression procedures, budgets, and scoring models, MORSE improves evidence retention over reverse ordering and generally outperforms matched random search, with downstream QA gains. Our code is available at https://github.com/tbn5pj/MORSE_code

1 INTRODUCTION

In sequential multi-context compression, context order changes importance scores and can substantially alter which evidence survives. MORSE uses reverse query evidence to establish an evidence-first anchor and select among compressor-evaluated permutations.

1 INTRODUCTION

  • Permuting identical retrieved contexts substantially changes evidence retention, and a controlled pair swap raises SF-R from 0.5 to 1.0.The gold sentence's score rises from 0.45 to 9.04 and its rank improves from 32nd to 2nd, preserving it at 4×, 8×, and 16× compression.Source: Figure 1 · p. 2, §1 · p. 2
  • Earlier partially relevant contexts can absorb credit for shared information, lowering later stronger evidence scores and increasing its risk of deletion.The paper calls this order-dependent credit-assignment failure information preemption.Source: §1 · p. 2
  • MORSE orders contexts using reverse query evidence, then evaluates the anchor and alternative permutations with the actual compressor and target budget.It scores the resulting compressed outputs and selects the candidate estimated to retain the strongest query evidence.Source: §1 · pp. 2–3
  • The work identifies ordering sensitivity, supports preemption with controlled pair swaps, and combines an evidence-first anchor with compression-aware permutation selection.The authors treat evidence retention as the primary outcome because it directly isolates ordering's effect on compression.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies likelihood-based compression, ordering for downstream long-context use, and redundancy-aware retrieval. MORSE instead studies how ordering a fixed retrieved set changes evidence attribution during sequential compression.

2 RELATED WORK

  • Unlike methods that design unit-level scoring or deletion rules, MORSE treats context ordering as a structural variable in sequential likelihood-based compression.Permutations alter how shared evidence is credited and consequently which information survives compression.Source: §2 · p. 3
  • Where prior long-context ordering work targets information presentation to downstream models, MORSE studies ordering within compression itself.Here, permutation changes sequential likelihood histories and evidence credit assignment.Source: §2 · p. 3
  • Unlike redundancy-aware retrieval methods that choose which contexts to retrieve or retain, MORSE keeps the retrieved set fixed and orders it before sequential compression.Its focus is order-dependent evidence attribution among already selected contexts.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation treats context permutation as the decision variable in sequential multi-context compression, where importance scores depend on the preceding history.

3 PROBLEM FORMULATION

  • The formulation represents retrieved contexts as ordered compression units and remains unit-agnostic, although experiments use complete sentences.Each context is a sequence of units.Source: §3 · p. 3
  • A unit's sequential score is its query-conditioned code-length difference, evaluated relative to its history.The history comprises preceding contexts and earlier units in the current context.Source: §3 · p. 4
  • The compressor removes low-scoring units under a token budget, and different context permutations can yield different compressed outputs.The paper names this score the sequential chain query-mutual-information (QMI) score.Source: §3 · p. 4

4 ANALYSIS OF SEQUENTIAL MULTI-CONTEXT COMPRESSION

The analysis explains how sequential scoring discounts redundancy but makes evidence credit dependent on order. It motivates evidence-first ordering while noting that standalone evidence scores cannot capture the full interactions of sequential compression.

4.1 ORDER-DEPENDENT CREDIT ASSIGNMENT

  • Sequential scoring discounts later units that add no query-dependent information beyond earlier units, avoiding repeated full credit for overlap.This benefit comes from conditioning each unit's score on the preceding history.Source: §4.1 · p. 4
  • Information preemption describes how an earlier partial context can take credit for shared information, lowering a stronger evidence carrier's score.The decomposition is an idealized mechanism illustration, and suppression affects retention only if it changes a budget-constrained deletion decision.Source: §4.1 · p. 4

4.2 EVIDENCE-FIRST ORDERING AND ITS LIMITS

  • Reverse query evidence ranks texts by how strongly they support the query and motivates an evidence-first anchor.Standalone scores cannot capture sequential redundancy, complementarity, preemption, or budget effects, so candidate orderings are also evaluated after compression.Source: §4.2 · pp. 4–5

5 PROPOSED METHOD: MORSE

MORSE anchors ordering in standalone query evidence, then evaluates candidate permutations through the compressor and selects the output with the strongest retained query evidence. This combines an evidence-first ordering with compression-aware exploration.

5.1 EVIDENCE-FIRST ANCHOR

  • MORSE sorts contexts by reverse query likelihood to form an evidence-first anchor that prioritizes stronger standalone query evidence.For a fixed query, the reverse query-evidence score induces the same ordering as log pθ(q | ci).Source: §5.1 · p. 5

5.2 COMPRESSION-AWARE GLOBAL EXPLORATION

  • Because standalone evidence misses sequential interactions, MORSE combines the anchor with up to K −1 unique global random permutations.The alternatives are sampled without replacement and exclude the anchor.Source: §5.2 · p. 5
  • MORSE runs each candidate through the same sequential compressor under budget B, then restores retained units to original document order.Canonicalization makes selection compare retained content rather than downstream presentation order.Source: §5.2 · p. 5
  • It scores each canonically reordered compressed output with reverse query evidence and selects the candidate with the highest retained-evidence score.The anchor remains in the candidate set, and the default search budget is K = 5.Source: §5.2 · pp. 5–6

6 EXPERIMENTS

Across the evaluated settings, MORSE-5 improves evidence retention over both Reverse and compute-matched random search, though downstream QA gains are less uniform. Experiments also examine search budget, selection objective, generalization, ordering effects, and preemption interventions.

6.1 EXPERIMENTAL SETUP

  • The evaluation uses HotpotQA, 2WikiMultiHopQA, and TRUE-20, comparing 1P and ISD compression at 4×, 8×, and 16× with SF-R as the primary retention metric.Answer F1 and EM measure downstream QA; the protocols separate compression order from presentation order.Source: §6.1 · p. 6

6.2 NUMERICAL RESULT ANALYSIS

  • MORSE-5 improves SF-R over compute-matched RANDOMSEARCH-5 in all 12 1P and all 12 ISD settings, and also improves on Reverse across settings.Downstream gains are broadly consistent under 1P but vary more under ISD; higher SF-R does not always mean higher F1 or EM under milder compression.Source: §6.2 · p. 6, Table 1 · p. 7

6.3 SEARCH-BUDGET ABLATION

  • Increasing K improves evidence retention with diminishing returns; K = 5 is the practical balance between search quality and latency.Larger K can provide incremental improvements when additional test-time computation is available.Source: §6.3 · p. 6, Table 2 · p. 8

6.4 SELECTION-OBJECTIVE ABLATION

  • The reverse selection objective JB selects higher-SF-R candidates and correlates more strongly with SF-R than either forward alternative.The ablation averages selected SF-R and within-example Spearman correlation over the 12 1P settings.Source: §6.4 · p. 7, Table 3 · p. 8

6.5 GENERALIZATION

  • MORSE beats Reverse in all four scorer–dataset combinations and averages better than compute-matched RandomSearch-5 in each setting, but the joint OLMo–MuSiQue shift is small and statistically inconclusive.For MuSiQue, retention is measured with Supporting-Paragraph Recall rather than Supporting-Fact Recall.Source: §6.5 · p. 8, Table 4 · p. 9

6.6 COMPRESSION ORDER VERSUS PRESENTATION ORDER

  • Compression-only Reverse ordering improved SF-R/F1/EM by 9.58/5.38/4.77 points on average; presentation-only changes averaged 0.05 F1 and 0.17 EM.Reverse improved SF-R in all 12 settings, while presentation-only effects were negligible and inconsistent.Source: §6.6 · pp. 8–9

6.7 CONTROLLED PAIR-SWAP INTERVENTION

  • In 399 eligible pairs, placing evidence before its preemptor rescued more supporting units than it harmed at 4×, 8×, and 16× compression.Rescues versus harms were 170 vs. 16, 165 vs. 8, and 149 vs. 7, respectively; stronger measured suppression was associated with higher rescue probability.Source: §6.7 · p. 9, Figure 2 · p. 9

7 CONCLUSION

Context permutation can substantially change which supporting evidence survives query-conditioned sequential compression. MORSE combines an evidence-first anchor with compression-aware global exploration to select the candidate retaining the most query evidence.

7 CONCLUSION

  • The paper identifies order-dependent credit assignment and information preemption as mechanisms behind evidence-retention changes, and supports them with controlled pair-swap interventions.MORSE uses a reverse-likelihood evidence-first anchor and compression-aware global exploration to select retained query evidence.Source: §7 · p. 10

A.1 FULL EXPERIMENTAL SETUP

The experiments test compression and ordering choices on multi-hop QA benchmarks under native and enlarged context settings. They compare evidence retention and downstream QA while varying compressor, budget, and context order.

Datasets

  • HotpotQA and 2WikiMultiHopQA are evaluated in native settings and TRUE-20 stress tests with distractors appended to create 20-context collections.Native evaluation uses 500 HotpotQA examples and 100 2WikiMultiHopQA examples; TRUE-20 uses 100-example subsets for each dataset.Source: §A.1 · p. 13

Baselines

  • MORSE is compared with independent scoring, five static ordering baselines, and RANDOMSEARCH-5, which uses five random permutations without a Reverse anchor.REVERSE is MORSE’s evidence-first anchor; RANDOMSEARCH-5 matches its K = 5 search budget.Source: §A.1 · p. 13

Compression Setting

  • The primary 1P compressor scores sentences once per ordering, while ISD repeatedly recomputes scores and deletes the lowest-scoring sentence until the budget is met.Both use Qwen2.5-0.5B-Instruct, exact token-budget accounting, and 4×, 8×, and 16× compression ratios.Source: §A.1 · p. 13

Evaluation Metrics

  • Evaluation measures supporting-fact recall and downstream Answer F1 and Exact Match, with results averaged across five seeds for stochastic K = 5 search methods.For ISD, all methods use the same post-compression Reverse query-likelihood reranking policy before downstream inference.Source: §A.1 · pp. 13–14

Evaluation Design

  • Table 6 separates compression order from downstream presentation order, with SF-R reflecting compression order rather than presentation.Canonical presentation restores original document order; Reverse presentation follows reverse query-likelihood ranking.Source: Table 6 · p. 14

A.2 COMPRESSION ORDER VERSUS PRESENTATION ORDER

The decomposition indicates that ordering during compression matters more than downstream presentation order in these experiments. Changing compression order from Original to Reverse substantially improves downstream utility, whereas changing presentation alone yields smaller and less consistent changes.

A.2 COMPRESSION ORDER VERSUS PRESENTATION ORDER

  • Across 12 settings, switching compression order from Original to Reverse substantially improves downstream utility; changing presentation alone yields smaller, less consistent changes.Table 6 varies compression and presentation order independently; SF-R depends only on compression order.Source: §A.2 · p. 14
  • The decomposition supports the interpretation that ordering primarily changes which information survives sequential compression, rather than merely repositioning retained evidence.The experiment diagnoses Reverse ordering and does not decompose the full MORSE search procedure.Source: §A.2 · p. 14

A.3 DETAILED BASELINE DEFINITIONS

The baselines vary whether scoring is sequential, how contexts are ordered, and whether candidate outputs are selected through compression-aware search. These comparisons hold the underlying compressor and budget consistent where specified.

Baseline Comparison Design

  • Static baselines differ only in supplied context order, while RANDOMSEARCH-5 adds compute-matched selection among permutations using MORSE’s JB objective.RANDOM-5 reports the mean across five fixed random permutations; ORIGINAL retains dataset or retrieval order without reranking.Source: §A.3 · pp. 14–15

Independent

  • INDEPENDENT scores units without conditioning on previously processed contexts, making its scores invariant to context permutation.It represents non-sequential compression for comparison with order-sensitive methods.Source: §A.3 · p. 15

RandomSearch-5

  • RANDOMSEARCH-5 selects the highest-JB compressed output from five unique random permutations, while MORSE replaces one shared random candidate with the Reverse anchor.The matched comparison uses the same compressor and objective, isolating the effect of the informed anchor in finite-budget search.Source: §A.3 · p. 15

Static Ordering Baselines

  • Static ordering baselines preserve the sequential compressor but vary context order: ORIGINAL, RANDOM-5, LENGTH, FORWARD, and REVERSE.REVERSE ranks contexts by reverse query likelihood and is MORSE’s evidence-first anchor; FORWARD uses forward conditional likelihood, while LENGTH sorts by increasing context length.Source: §A.3 · p. 15

A.4 DETAILED RESULTS CORRESPONDING TO TABLE 1

Across the detailed experiments, MORSE’s clearest strength is evidence retention: it leads SF-R across the reported A.4 settings and improves on Reverse across the tested generalization and scaling conditions. Its advantages over matched random search are less uniform in transferred and larger-context settings, and downstream QA does not always track evidence retention.

A.4 DETAILED RESULTS CORRESPONDING TO TABLE 1

  • MORSE achieves the highest SF-R in all 24 A.4 dataset–setting–procedure–ratio combinations and improves over RANDOMSEARCH-5 in all 12 comparisons for each compression procedure.Downstream F1 and EM are more heterogeneous, particularly under ISD.Source: §A.4 · p. 16, Table 7 · p. 16

A.5 PREEMPTION CORRECTION ANALYSIS

  • MORSE-5 corrects more initially harmful preemptor–evidence relations than RANDOMSEARCH-5 by 9.2, 9.5, and 11.9 points at 4×, 8×, and 16× compression, respectively.Its final harmful-order rate falls from 45.9% in the original order to 35.8%, versus 46.4% under RANDOMSEARCH-5; the comparison linking corrections to pair-swap effects is descriptive, not causal.Source: §A.5 · pp. 19–20, Table 15 · p. 19, Table 16 · p. 19

A.6 SELECTION-OBJECTIVE ABLATION

  • Reverse JB increases selected SF-R over Forward by 5.73 points in the MORSE pool and 5.26 points in the random pool, and has stronger within-example association with SF-R.The stronger association persists when the Reverse candidate is excluded, and reverse JB has higher oracle top-1 agreement than either forward alternative in both candidate pools.Source: §A.6 · pp. 20–21, Table 17 · p. 20, Table 18 · p. 21

A.7 CROSS-MODEL AND CROSS-DATASET GENERALIZATION

  • In transferred scorer–dataset conditions, MORSE improves retention over Reverse at every tested ratio, while its advantage over RANDOMSEARCH-5 weakens under the joint OLMo–MuSiQue shift.The equal-ratio difference over RANDOMSEARCH-5 is +0.72 points there, with a 95% confidence interval of [−0.83, +2.34].Source: §A.7 · pp. 22–23, Table 19 · p. 22, Table 20 · p. 23

A.8 CONTEXT-COUNT SCALING

  • Across 15 context-count scaling settings, MORSE-5 outperforms Reverse, but its comparison with RANDOMSEARCH-5 depends on compression ratio and context count.At 4× compression with 50 contexts, RANDOMSEARCH-5 is significantly higher; ten other comparisons have confidence intervals overlapping zero. Improved evidence retention does not guarantee better generated answers in every setting.Source: §A.8 · p. 24, Table 24 · p. 25, Table 25 · p. 25

Related papers

Loading 2609.27380v2…