PaperlayerView exact revision on arXiv

Source-linked AI summary

Glance: The Generator Is Its Own Reranker

Adilet Metinov, Chingiz Arziev

arXiv:2610.40454v1cs.CLcs.IR

TL;DR

RAG systems often separate reranking from answer generation, requiring another served model and processing candidate passages before the generator reads them. Glance gives the generator passage-specific probes over a shared prefill, and reports competitive retrieval rankings and improved downstream QA in the evaluated settings.

  • Problem

    RAG systems commonly use separate models for evidence selection and answer generation, so candidate passages are processed before the generator reads selected context.

    Source: §1 · p. 1

  • Method

    Glance equips the generator with a lightweight decision adapter and pointer head that scores passage-specific probes branching from a shared candidate prefill.

    Source: §1 · p. 1

  • Results

    In evaluated settings, Glance achieves competitive retrieval quality and improves downstream answers over cross-encoder-selected context under the same generator.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Adapters let one resident backbone support ranking and general decisions, and probe-only adaptation can let answer generation reuse the original passage cache.

    Source: §1 · p. 2, §6 · p. 10

    Evaluation is a proof of concept covering hundreds of queries across two BEIR datasets and one QA dataset, using one hybrid 9B backbone; latencies are from an unoptimised eager path.

    Source: §5 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance, which equips the generator with a lightweight decision adapter and pointer head to score candidate passages from a shared prefill. Short relevance probes branch from the cached context, producing one score per passage without separately prefilling each query-passage pair. A single resident 9B backbone supports generation, general decisions and reranking, with approximately 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, Glance achieves nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. On 300 HotpotQA questions, its selected context improves answer F1 by 0.045 over cross-encoder-selected context under the same base generator (paired bootstrap 95% CI [0.014, 0.076]). In a scaling study, reranking quality keeps rising up to 200 candidates (43k tokens) scored from one shared prefill, where it exceeds the cross-encoder by 0.058 nDCG@10, and a probe-only activated adapter lets generation continue from the original passage cache. These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.

1 Introduction

Glance lets the generator score candidate passages from a shared prefill, rather than relying on a separate reranker model. The paper presents retrieval, downstream QA, and unified-serving evaluations of this approach.

1 Introduction

  • The approach targets RAG systems that separately serve reranking and answer models, where candidate passages are processed before the generator reads selected context.The introduction asks whether the generator can judge candidates using computation shared across passages.Source: §1 · p. 1
  • Glance uses a lightweight decision adapter and pointer head to score passage-specific probes branching from a shared candidate cache before the same backbone generates an answer.An activated variant prefills with base weights and can continue generation from the original cache.Source: §1 · p. 1
  • The paper evaluates retrieval on two BEIR datasets, HotpotQA under the same generator, and scaling to 200 candidates per shared prefill.These evaluations use reranking fine-tuned on MS MARCO.Source: §1 · p. 1
  • Glance shares one resident backbone across ranking and general decisions, with adapters switched per request and an activated variant that continues from the reranking cache.The paper reports outputs identical to each model loaded alone.Source: §1 · p. 2
  • Glance removes the additional reranker backbone while achieving competitive retrieval quality and improved downstream answers in the evaluated settings, though a dedicated cross-encoder remains faster per query.The introduction frames these as results in the evaluated settings.Source: §1 · p. 2

2 Related Work

Prior reranking methods differ in how they process passages and produce scores, while related work also studies shared-context caching and generator–retrieval coupling. Glance is distinguished by trained passage-specific probes over a shared multi-passage context.

2 Related Work

  • Cross-encoders score query–passage pairs independently, while LLM rerankers use pointwise logits or listwise generation; Glance instead produces separate trained probe scores for all passages from one shared context.In-context reranking uses attention patterns of a frozen LLM, and Self-RAG judges each passage in its own pass.Source: §2 · p. 2
  • TurboRAG [12] and CacheBlend fuse passage KV caches for generation, while HyperRAG [1] caches document-side KV for decoder rerankers; Glance applies shared-prefix branching to RAG reranking.The paper builds on Kev, which packs typed questions over a shared state.Source: §2 · p. 2
  • REPLUG trains a retriever from LM likelihood and Self-RAG lets the generator critique passages; long-context models can still miss evidence placed mid-input.The cited limitation motivates reranking even with large context windows.Source: §2 · p. 2

3 Method

Glance scores candidate passages through branches from a shared state, using a decision adapter and pointer head on a frozen backbone. Its two answer pipelines differ in whether generation reuses the candidate cache or re-prefills selected passages.

3.1 Overview and the two answer pipelines

  • Glance’s main pipeline scores probes, selects top passages, then re-prefills them with the adapter off; Glance-Activated instead continues the base-weight candidate cache, which retains all k passages.The activated variant uses the base weights for passage prefill and the adapter for probe scoring.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 Decision mode

  • In decision mode, a pointer head compares the hidden state at <decide> with each option’s hidden state and returns a softmax over options; the backbone is frozen and only the head and rank-16 LoRA are trained.Binary yes/no questions use the options false and true.Source: §3.2 · p. 3

3.3 Multi-probe packing

  • In packed_all, the shared state contains all k passages and each branch asks whether its passage supports the query; packed_q and listwise instead place the query in the state.Branches do not attend to each other, and packed_all probes are not statistically independent because each sees all passages.Source: §3.3 · p. 3

3.4 Hybrid backbones

  • For the hybrid backbone, DeltaNet’s recurrent state summarizes the ordered prefix, so selected passages are re-prefilled in the main pipeline, while Glance-Activated continues the full candidate cache.The main pipeline also re-prefills because its state cache was computed with the adapter on.Source: §3.4 · p. 4

3.5 One resident backbone

  • The adapter remains unmerged: reranking uses it, while answer generation disables it on the same backbone; each additional decision skill costs approximately 0.1 GiB.Inference memory also depends on candidate count and cache implementation.Source: §3.5 · p. 4

3.6 Training

  • Training uses 4,000 MS MARCO queries with one positive and 7–19 hard negatives per record, packed in the packed_all format with one binary label per probe.One epoch took 39 minutes on one NVIDIA B200; no BEIR or HotpotQA data was used.Source: §3.6 · p. 4

4 Experiments

The experiments compare zero-shot and MS MARCO-trained decision adapters with retrieval baselines on two BEIR datasets. After training, packed_all exceeds the cross-encoder on both datasets, while zero-shot results show its single-pass trade-offs.

4.1 Setup

  • The setup uses the 9B MiMo-V2.6-Distill-Qwen backbone, BM25 top-20 candidates, and nDCG@10 on SciFact and NFCorpus; trained evaluations exclude BEIR and HotpotQA training data.The reported test sets contain 300 SciFact queries and 323 NFCorpus queries; BEIR gaps below about 0.02 nDCG@10 are indicative because differences are not significance-tested.Source: §4.1 · pp. 4–5

4.2 Single pass vs. k passes (zero-shot)

  • Without retrieval training, SciFact packed_all scores 0.715 versus 0.729 for 20 separate calls and 0.721 for the cross-encoder, while running 3.6× faster than sequential calls in this shared-GPU experiment.On NFCorpus it scores 0.341, above the cross-encoder’s 0.326 and separate calls’ 0.323; the cross-encoder remains far faster per query.Source: §4.2 · p. 5

4.3 Training for reranking

  • 0.020 on SciFact and 0.019 on NFCorpus nDCG@10 above the cross-encoder: trained packed_all is best on both datasets in Table 3.The differences were not significance-tested, and the trained-adapter per-passage baseline was not rerun.Source: §4.3 · p. 5

4.4 How far does one pass scale?

Glance’s trained adapter improves reranking as the candidate set grows, and on HotpotQA its selected context improves answer F1 under the same base generator. Scaling remains constrained by per-branch cache copies, while the evaluation also tests shared-backbone serving and adapter-on answering.

4.5 End to end: the same weights rerank and answer

  • With the zero-shot adapter, Glance’s top-3 yields HotpotQA F1 0.718 versus 0.683 for cross-encoder-selected context, despite retrieving fewer gold paragraphs.The adapter-off gap is not significant; with the adapter on for both contexts, Glance’s context leads by +0.055 F1 (95% CI [+0.018, +0.093]).Source: §4.5 · pp. 6–7

4.4 How far does one pass scale?

  • At k=200, trained-long reaches 0.717 nDCG@10 versus 0.659 for the cross-encoder on the same 100 SciFact queries.This uses 43.4k context tokens on Kev’s chunked path and takes 15.8 seconds per query; at k=100 it reaches 0.697, 0.060 above the cross-encoder.Source: §4.4 · p. 5, Table 4 · p. 7
  • Trained adapters keep improving as k grows, while the zero-shot adapter degrades beyond k=20; per-branch cache copies raise memory to 56 GiB at k=100.The zero-shot single call runs out of memory at k=200 under a 64 GiB cap, while chunking permits larger candidate sets.Source: §4.4 · pp. 5–6, Figure 2 · p. 7

4.5 End to end: the same weights rerank and answer

  • Adapter switching leaves weights unchanged: across 20 alternations, adapter-on probabilities and adapter-off greedy answers were bit-identical to their standalone modes.GPU memory remained 16.77 GiB, including 0.08 GiB for LoRA.Source: §4.5 · p. 7
  • On general prompts, adapter-on and adapter-off outputs were identical or nearly so, but broader evaluation is needed to determine whether one adapted model can replace toggling.On 1,024 Wikipedia tokens, NLL rose from 1.556 to 1.564; on HotpotQA, the decision LoRA shortened answers.Source: §4.5 · pp. 7–8
  • On 300 HotpotQA questions, trained Glance’s top-3 raises answer F1 from 0.684 to 0.729 (+0.045; 95% CI [+0.014, +0.076]) under the same base generator.The comparison isolates selection of different evidence from adapter effects on answer style.Source: Table 5 · p. 8, §4.5 · p. 8

4.6 Three modes on one backbone

Three request-specific modes share one resident 9B backbone, with separate adapters and heads for base chat, general decisions and reranking. Specializing the reranking adapter produces moderate forgetting on general decisions, which the multi-adapter setup sidesteps.

4.6 Three modes on one backbone

  • One resident backbone serves base chat, general decisions and reranking by switching the corresponding adapter and pointer head per request.Table 6 reports 16.85 GiB resident memory, unchanged at 16.86 GiB after the loop.Source: §4.6 · p. 8, Table 6 · p. 9
  • The general adapter scored 184/231 (79.7%) on JevBench, while the reranking adapter scored 175/231 (75.8%), a net loss of 9 items, 7 in the hard tier.The paper attributes moderate forgetting of general decisions to specializing an adapter for reranking; using the adapter trained for each request sidesteps it.Source: §4.6 · pp. 8–9
  • Each additional decision skill costs about 0.1 GiB.The general and reranking adapters are distinct modes, with probabilities differing by up to 0.81 on the same inputs.Source: §4.6 · p. 8

4.7 Activated adapter: one read for reranking and answering

Glance-Activated applies the adapter only to passage-specific probe branches, leaving the state cache identical to the base model’s. Matched reranking quality and cache-continuation results show the potential of reusing that cache, with the answer still seeing all candidates.

4.7 Activated adapter: one read for reranking and answering

  • The adapter is off for the state pass and on only for probe branches, and the resulting state cache equals the base cache bitwise in the 0.8B Qwen3.5 check.The patched check found 48 matching state tensors with max |∆| = 0, while branch outputs still differed from the base.Source: §4.7 · p. 9
  • 0.740 / 0.351 nDCG@10 on SciFact / NFCorpus for the activated adapter compares with 0.744 / 0.351 for the matched all-token control.Both used 150 training steps and 1,200 MS MARCO records; the authors report similar reranking scores.Source: §4.7 · p. 9
  • Without probe tokens, the activated model’s greedy chat output is identical to the base model’s by construction.The all-token control changed the wording on one of two test prompts.Source: §4.7 · p. 9
  • Continuing the cache yielded EM/F1 0.61 / 0.750 versus 0.59 / 0.754 for re-prefilling the top-3 passages, with 138 ms versus 181 ms median answer latency.This single-seed result used 100 HotpotQA distractor questions; the continued cache retains all 10 candidates, so the reranking scores do not select the answer context.Source: §4.7 · p. 9

4.8 Order sensitivity

Order sensitivity is small on average but can matter for individual passage scores. A prototype vLLM path serves chat and decision requests together, with measured agreement and throughput relative to Kev’s sequential fast path.

4.8 Order sensitivity

  • 0.678 nDCG@10 in BM25 order ranged to 0.693 across four random orders; averaging five orders gave 0.689 on 100 SciFact queries.The top-ranked passage matched BM25 order in 91% of cases, though individual passage changes reached a maximum of 0.36 and order effects were not negligible.Source: §4.8 · pp. 9–10

4.9 Serving chat and decisions in one vLLM engine

  • 0.995 per-query Spearman rank correlation with Kev and 1.54 requests/s versus 0.86 for Kev’s sequential fast path were measured in the vLLM serving prototype.Across 400 probes, 4 decisions flipped; production use requires a connector exporting readout positions and the head inside the engine.Source: §4.9 · p. 10

5 Limitations

The paper characterizes Glance as a proof of concept, with evaluation and runtime results constrained in scale and setting. Several findings are single-seed or prototype results, and the approach retains practical limits in scaling, memory and context selection.

5 Limitations

  • Evaluation covers hundreds of queries across two BEIR datasets and one QA dataset, using one hybrid 9B backbone family.The paper describes this as a proof of concept.Source: §5 · p. 10
  • Latencies come from an unoptimised eager path and partly shared, different GPUs, while a dedicated cross-encoder remains faster per query.Speed-up ratios depend on load, and BEIR differences are not significance-tested.Source: §5 · p. 10
  • In packed_all, scores depend somewhat on neighbouring passages and their order; zero-shot quality degrades beyond k=20 and collapses at k=200.Averaging over orders is offered as a simple remedy, while batching across queries currently lowers throughput.Source: §5 · p. 10
  • Activated-adapter and cache-reuse results use one seed and 100 questions, and cache reuse leaves all passages in the answer context.The bitwise cache check used a 0.8B model, memory grows with k × context length, and the vLLM path is a prototype.Source: §5 · p. 10

6 Conclusion

Glance uses the generator to rerank passages through probes that share a candidate prefill, with lightweight adapters supporting ranking and general decisions on one resident backbone. In the evaluated settings, its rankings are competitive with a dedicated cross-encoder and improve downstream QA under the same generator; probe-only adaptation also allows generation to reuse the original passage cache.

6 Conclusion

  • Glance uses passage-specific probes over one shared candidate prefill, allowing the generator to serve as its reranker.Lightweight adapters support ranking and general decisions on a single resident backbone.Source: §6 · p. 10
  • In the evaluated settings, Glance produces rankings competitive with a dedicated cross-encoder and improves downstream QA under the same generator.The conclusion identifies broader benchmark and backbone-family evaluation as open questions.Source: §6 · p. 10
  • Probe-only adaptation allows answer generation to reuse the original passage cache.Source: §6 · p. 10
  • Latency trade-offs against small cross-encoders remain an open question for optimized serving.The conclusion calls for further study of how optimized serving changes that trade-off.Source: §6 · pp. 10–11

Availability

The authors release Glance adapters and code under Apache-2.0, including reranker, activated-reranker, and general decision-adapter repositories.

Availability

  • The reranker, activated reranker, and general decision adapter are released under Apache-2.0.The repositories use the working name GlanceRAG.Source: Availability · p. 11
  • Code is available in the code/ folders of the two reranker repositories.Source: Availability · p. 11
  • The 39-minute MS MARCO checkpoint is listed under checkpoints/msmarco-short in the reranker repository.The reranker repository also identifies a trained-long version at its root.Source: Availability · p. 11

Related papers

Loading 2610.40454v1…