Source-linked AI summary
When Learned Context Planning Fails to Beat Strong Retrieval: A Controlled Study of Planning, Routing, and Reranking for Long-Context QA
Yingrui Li, Han Chen
TL;DR
The paper asks whether learned evidence planning improves long-context multiple-choice QA when compared with strong retrieval, routing, budgeted-selection, and reranking controls. It evaluates planner-guided evidence selection against those controls and finds that planning does not robustly beat them under the studied conditions. The paper’s bounded conclusion is that learned planning is a weak relevance signal rather than a replacement for strong retrieval.
Problem
It is unclear whether learned evidence planning remains useful after comparison with strong retrieval and reranking controls, since weaker comparisons can make a planner look stronger than BM25.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Planner gains should be checked against strong retrieval, routing, budgeted-selection, and reranking controls before being attributed to planning.
Source: §1 · p. 1The conclusions are restricted to LongBench-v2 MCQ, Qwen2.5-7B-Instruct, one planner recipe, evaluated retrieval and reranking methods, and a partly transductive all-503 diagnostic.
Source: Limitations · p. 4
Abstract
from arXiv · showhide
Learned context planning selects evidence atoms before an answer model reasons over them. We test whether this learned selection improves long-context multiple-choice QA after strong retrieval, routing, budgeted-selector, and reranking controls. Our primary diagnostic uses all 503 LongBench-v2 MCQ questions with Qwen2.5-7B-Instruct. The planner is SFT-trained on outcome-selected traces from 140 training and 28 development questions; because the 503-question analysis includes those questions, it is partly transductive. At an 18k-character budget, anchored hybrid retrieval reaches 36.18% accuracy and BM25 reaches 35.98%, while the best direct planner-guided method reaches 34.19%. On the untouched 152-question test split, anchored hybrid remains higher (42.11% versus 36.84%). Leakage-safe routers cannot convert a large oracle gap. Under tight budgets, the best planner is ahead by only 0.40 points at 6k and loses at 9k; planner-guided reranking has a +1.79-point estimate at 6k with a paired interval crossing zero and ties the control at 9k. Packing-order and score-flatness analyses did not identify a stable mechanism. Under this setup, learned planning is a weak relevance signal rather than a replacement for strong retrieval.
1 Introduction
The paper asks whether learned evidence selection holds up against strong retrieval and reranking controls. It finds that, in this setting, planning does not robustly outperform those controls.
1 Introduction
- RAG ranks and packs retrieved chunks, while learned context planning selects evidence atoms for the answer model.Here, the planner outputs atom IDs rather than multi-step reasoning traces.Source: §1 · p. 1
- A planner may beat prompted planning or a direct baseline yet still lose to BM25, and an oracle routing bound need not yield a deployable router.Source: §1 · p. 1
- The study tests whether learned evidence planning survives strong retrieval and reranking controls, finding no robust win over those controls in this setting.Planning changes the evidence set, helps some examples, and yields a small 6k-character rerank signal.Source: §1 · p. 1
- The study finds that direct planner-guided construction loses to BM25 and anchored hybrid retrieval, while oracle complementarity fails to yield a leakage-safe router and planner-rerank signal is statistically unsupported.Source: §1 · p. 1
2 Experimental Setup
The experiments compare planner-selected evidence with retrieval, budgeted-selection, and reranking controls on LongBench-v2 MCQ. The setup uses outcome-selected traces to train a planner and reports both overlapping all-question diagnostics and a disjoint test partition.
2 Experimental Setup
- The 503 LongBench-v2 MCQ questions are deterministically split into 301 training, 50 development, and 152 test questions; the primary all-question diagnostic includes 168 questions used to construct planner SFT train/development records and is partly transductive.The separate 152-question test partition is untouched.Source: §2 · pp. 1–2
- Evidence atoms are character-bounded chunks packed under 18k, 9k, or 6k budgets, with BM25, dense, hybrid, anchored-hybrid, budgeted, MMR, and cross-encoder controls.Multi-document contexts use 2,000-character atoms with 200-character overlap; other contexts use 8,000-character atoms with 1,000-character overlap.Source: §2 · p. 2
- Outcome-selected traces yield 140 SFT training rows and 28 development rows, with zero test-ID overlap; an audit could not reproduce an earlier reported count of 262.For each question, sampled plans are deduplicated and executed; one correct plan is retained, preferring fewer atoms, shorter context, and earlier sample order.Source: §2 · p. 2
- The planner is a Qwen2.5-7B-Instruct LoRA adapter trained for three epochs, and training ends at global step 24 without development-checkpoint selection.Source: §2 · p. 2
- At inference, the planner sees only the question, choices, and retrieved atoms; planner-selected atoms precede BM25, hybrid, or reranked fill.An earlier-adapter sensitivity check is also reported.Source: §2 · p. 2
- Oracle routing uses gold correctness post hoc, while leakage-safe routers use inference-time features; the study reports accuracy and paired statistics alongside released analysis artifacts.Raw contexts and model weights are not redistributed, historical GPU inference is not claimed bitwise reproducible, and latency is descriptive only.Source: §2 · p. 2
3 Results
Across full-budget, routing, and tighter-budget comparisons, planner-guided methods do not robustly outperform strong retrieval or non-planner controls. Mechanism audits identify no stable planning advantage.
3.1 Direct Planning Loses to Retrieval
- At 18k, anchored hybrid (36.18%) and BM25 (35.98%) beat the best direct planner-guided method (34.19%).On the untouched 152-question subset, anchored hybrid reaches 42.11% versus 36.84% for planner-guided anchored hybrid; across domains it leads in one, ties in one, and trails in four.Source: §3.1 · pp. 2–3, Table 1 · p. 3, Table 2 · p. 3
3.2 Oracle Routing Does Not Become a Router
- Oracle routing reaches 46.52% for hybrid anchor plus planner+hybrid and 56.26% across all methods, but leakage-safe routers fail to improve over the best single method.The oracle figures are post-hoc upper bounds, not deployable policies.Source: Table 3 · p. 3, §3.2 · p. 3
3.3 Budget and Rerank Controls Erase Most Gains
- At 6k, the best planner gains only +0.40 points over the best same-budget selector (p = 0.930), then loses by 1.19 points at 9k.Planner-rerank estimates +1.79 points at 6k, but its paired interval crosses zero (p = 0.478); at 9k it ties the best non-planner reranker. The earlier adapter's 34.79% 6k score also lacks paired support and does not beat the 9k control.Source: Table 4 · p. 3, §3.3 · p. 3
3.4 Selection, Order, and Score Mechanisms
- Packing-order changes show no systematic advantage; planner–reranker Jaccard@10 is lower for wins than losses at both budgets, and none of 92 tests survives correction.These rank and context-membership analyses are not evidence-recall measurements because the study lacks gold atom-level evidence labels.Source: §3.4 · pp. 3–4
4 Discussion and Conclusion
After strong retrieval, routing, budgeted-selection, and reranking controls, most apparent planning gains disappear. The authors conclude that learned planning is a weak relevance feature rather than a replacement for retrieval.
4 Discussion and Conclusion
- Compared only with prompted, heuristic, and direct context baselines, learned planning can look useful; this study recommends checking it against stronger controls.Source: §4 · p. 4
- +1.79 points at 6k is the planner-rerank estimate, with 68 wins and 59 losses among 503 questions.Confirming a +1.5-point paired effect would require roughly 8,800 examples, while a +2.0-point effect would require roughly 4,900.Source: §4 · p. 4
- After strong retrieval, routing, budgeted-selection, and reranking controls, most apparent gains disappear, though planning sometimes changes the evidence set helpfully.Source: §4 · p. 4
Limitations
The conclusions are bounded to one LongBench-v2 multiple-choice QA setup and have limited power for small paired effects on the disjoint test set.
Limitations
- The study covers LongBench-v2 MCQ, Qwen2.5-7B-Instruct, one outcome-supervised planner recipe, one overlapping character-window atomization scheme, and evaluated lexical/dense retrieval with a MiniLM cross-encoder.Source: Limitations · p. 4
- The all-503 diagnostic is partly transductive for 168 questions represented in SFT data, while the disjoint test has only 152 questions and limited power for small paired effects.Source: Limitations · p. 4
- The study does not evaluate open-ended QA, other model scales, alternative supervision sources, or real-world deployment, and it has no external replication.Source: Limitations · p. 4
- A larger or differently mined trace set could change the outcome; historical runs also lacked strict locked-run enforcement and immutable model-revision pinning.Latency is descriptive rather than a portable efficiency claim.Source: Limitations · p. 4
- Without gold atom-level evidence labels, mechanism analyses use atom IDs, ranks, scores, and displacement rather than evidence recall.Source: Limitations · p. 4