PaperlayerView exact revision on arXiv

Source-linked AI summary

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Devrim Çavuşoğlu, Emre Akbaş

arXiv:2608.29899v1cs.CLcs.AIcs.IR

TL;DR

Long-document retrieval faces token-level compute limits and a lack of aligned long-document contrastive data. REIGN trains a bi-encoder over cached chunk embeddings from a frozen Guidance Network and releases a synthetic benchmark. Across Wikipedia, LoCo, and patent retrieval, it matches dense long-context systems with smaller parameter budgets, with strongest scope in document-to-document retrieval.

  • Problem

    Long-document embedding is costly, while existing benchmarks lack semantically aligned long-document pairs and realistic hard negatives for contrastive learning.

  • Method

    REIGN contrastively trains a bi-encoder over contextual chunk embeddings produced and cached by a frozen Guidance Network, rather than over raw tokens.

  • Results

    REIGN matches dense long-context retrievers across Wikipedia, LoCo, and patent retrieval at smaller parameter budgets.

  • Takeaways & Limitations

    REIGN’s encoder is principally a document-to-document, cross-chunk retriever whose efficiency comes from separating token-level encoding from document-level reasoning.

  • Takeaways & Limitations

    REIGN adds little or slightly hurts on LoCo with stronger GTE guidance, and short-passage performance drops by 5–7 nDCG@10.

Abstract

from arXiv · show

Dense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs, primarily for document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets in each regime. A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.

1 Introduction

Long-document retrieval faces token-level compute limits and insufficient long-document contrastive data. REIGN addresses both with cached chunk embeddings and a synthetic benchmark, performing competitively across three retrieval regimes.

  • Long-document embedding remains constrained by memory and compute walls, despite efficient-attention, memory-augmented, and extended-context approaches.
  • Existing benchmarks lack semantically aligned long-document pairs and realistic hard negatives for contrastive learning at scale.
  • REIGN is a contrastively trained bi-encoder that aggregates cached chunk embeddings from a frozen Guidance Network instead of processing raw document tokens.
  • REIGN evaluates long-document retrieval on a synthetic Wikipedia benchmark, the out-of-distribution LoCo suite, and a real-world patent case study.
  • A 55M-parameter REIGN+GTE-small beats tested native long-context dense baselines on GoodWiki-Long and LoCo, while 357M REIGN+GTE-large is statistically indistinguishable from larger patent baselines.

2 Related Work

REIGN builds on hierarchical document encoders and long-context Transformers but changes the representation unit: a frozen, cached chunk embedding feeds a lightweight bi-encoder for first-stage retrieval.

  • Representation granularity: REIGN treats frozen Guidance Network chunk embeddings, rather than subword tokens, as the atomic input to its trainable encoder.
  • Hierarchical encoders: Hierarchical encoders split documents into chunks, encode them independently, and aggregate their representations.
  • Closest precedents: Unlike SMITH and PARADE, REIGN freezes and caches the chunk encoder, trains a contrastive bi-encoder, and targets substantially longer documents for first-stage retrieval.
  • Long-context modelling: Long-context alternatives modify attention or sequence backbones, whereas REIGN retains a standard Transformer over chunk embeddings.
  • Efficiency-oriented training: Caching removes GN forward passes from the training loop, addressing computational cost along an axis not addressed by parameter-efficient tuning alone.

3 Dataset

The released GOODWIKI-LONG-SYNTHETIC benchmark supplies semantically aligned long-document pairs and graded distractors in a standard information-retrieval layout.

  • Dataset motivation: Existing long-form corpora lack aligned document pairs and structural supervision for long-context contrastive learning.
  • Dataset statistics: Table 1 reports query-disjoint splits, tiktoken-based length statistics, and scores where 2 denotes an LLM-rephrased positive and 1 a topical distractor.
  • Dataset construction: GOODWIKI-LONG is created by rephrasing Wikipedia articles exceeding 16,000 characters, producing semantically aligned positive pairs.
  • Relevance structure: Semantically overlapping distractors create graded relevance, with positives receiving full weight and distractors reduced target weight during training.
  • Release: The benchmark is publicly released in the canonical BEIR/MTEB tri-config layout with training and evaluation code.

4 Method

REIGN chunks documents, encodes chunks once with a frozen GN, and trains a small cross-chunk Transformer with contrastive supervision. Cached embeddings make long-context training and inference substantially cheaper while preserving retrieval performance.

  • Training pipeline: Each document view is chunk-encoded by a frozen GN, refined by REIGN, and trained with rephrased positives, graded distractors, and in-batch negatives.
  • Chunk encoding: The GN maps token windows to contextual chunk embeddings, while REIGN operates entirely at the coarser embedding level.
  • Efficiency: Cached GN outputs reduce the document-level computation from O(M^2d) to O(N^2d), enabling lightweight models to process inputs of 10^5+ tokens.
  • Document aggregation: The cross-chunk encoder uses no positional encoding and average-pools chunk outputs, making document representations invariant to chunk order.
  • Operating regime: REIGN begins at N ≥2; single-chunk inputs use the GN because REIGN trails it by 5–7 nDCG@10 and adds an encoder pass.
  • Measured efficiency: With cached embeddings, REIGN answers queries in 0.40–0.52 ms, versus 19.8–118.2 ms when running the GN per query.
  • Measured efficiency: Peak GPU memory is 0.24–1.73 GB for REIGN, compared with 4.8–18.9 GB for native long-context dense baselines.

5 Experiments

Across in-distribution, out-of-distribution, and patent retrieval, REIGN delivers competitive dense retrieval with smaller parameter budgets, while its gains depend on benchmark and GN scale.

  • Evaluation caveat: GoodWiki-Long positives retain substantial lexical overlap, favoring bag-of-words retrieval and limiting how broadly its dense-retrieval ranking reflects semantic matching.The benchmark uses rephrased query distillations as positives.
  • GOODWIKI-LONG: REIGN+GTE-small tops dense GoodWiki-Long nDCG@10 at 55M parameters, edging BGE-M3 by 0.10 with 10× fewer parameters.It also beats Jina-v3 and Stella-1.5B by 3.4–3.7 points and exceeds GTE-large chunked by 1.4.
  • GOODWIKI-LONG: Chunked mean-pooling improves nDCG@10 over truncation by 1.7–2.7 points, while REIGN adds another 0.7–2.2 points across guidance networks.The largest cross-chunk lift occurs with the smallest guidance network.
  • LoCo: LoCo macro-average nDCG@10 reaches 70.77 with REIGN+GTE-large, within 0.65 of 20×-larger E5-Mistral.REIGN+GTE-small reaches 68.92, a +0.72 lift over its matched chunked GN.
  • LoCo: 71.38 is the highest observed LoCo macro-average, achieved by bare GTE-large chunked at 20× smaller scale than E5-Mistral.REIGN helps most at the small-GN end, while stronger chunked GNs can already saturate dominant query-to-passage subtasks.
  • DAPFAM: DAPFAM mean nDCG@100 is 33.10 for REIGN+GTE-large, statistically indistinguishable from Jina-v3 and Stella-1.5B using 1.6–4.3× fewer parameters.The lift over GTE-large chunked is +0.67, but that margin is not statistically significant.

6 Conclusion

REIGN enables document-level representation learning from cached chunk embeddings, reducing long-document costs while transferring from synthetic Wikipedia training to patent retrieval. Across the reported regimes, it achieves strong parameter-efficient retrieval, though LoCo performance is task-conditional.

  • REIGN trains a lightweight cross-chunk encoder over precomputed chunk embeddings from a frozen Guidance Network.The framework operates on document-level embedding sequences rather than raw tokens.
  • Cached Guidance Network embeddings reduce per-document training and inference cost by orders of magnitude while supporting inputs of 105+ tokens.The approach moves expensive token-level processing outside the training loop.
  • REIGN trains on the released GOODWIKI-LONG-SYNTHETIC benchmark and transfers zero-shot from synthetic Wikipedia to IN-IPC patent retrieval.
  • The 55M REIGN+GTE-small outperforms LoCo-paper zero-shot baselines, while REIGN+GTE-large reaches 70.77 macro nDCG@10 within 0.65 points of E5-Mistral using 20× fewer parameters.On DAPFAM, REIGN has the best mean overall at 33.10 nDCG@100 and is statistically indistinguishable from larger dense baselines using 1.6–4.3× fewer parameters.
  • LoCo results are task-conditional: REIGN helps multi-chunk aggregation, but bare GTE-large chunked mean-pooling is the Pareto-best configuration overall.The conclusion cautions that in-distribution sweet-spot claims require out-of-distribution verification.

Limitations

REIGN is primarily suited to long-document, document-to-document retrieval requiring cross-chunk evidence aggregation. Its effectiveness is bounded by task characteristics, the frozen Guidance Network, and deployment trade-offs such as chunk size and memory.

  • REIGN’s added value scales with the amount of cross-chunk reasoning required, while stronger Guidance Networks can already saturate some LoCo subtasks.On LoCo, REIGN adds −0.38 macro on GTE-BASE and −0.61 on GTE-LARGE.
  • REIGN underperforms truncated GTE-small by 5–7 nDCG@10 points on short-text retrieval benchmarks.With a single chunk, the cross-chunk signal REIGN exploits disappears.
  • Lexically driven legal subtasks expose a weakness: courtlistener_HTML reaches 26.15 nDCG@10, courtlistener_Plain 24.89, and legal_case_reports 32.69.The pooled chunk embeddings discard surface-level lexical contrast relied on by these corpora.
  • REIGN inherits biases and compression errors from its frozen Guidance Network, and changing the GN requires full REIGN retraining.The cached chunk-embedding artifact is GN-specific, and new-domain deployments require out-of-distribution verification.
  • Smaller chunks increase memory usage: a REIGN-SMALL-L2 model with GTE-large uses approximately 1.8, 3.6, 4.5, and 8.4 GB at chunk sizes 512, 256, 128, and 64.
  • REIGN is unsuitable for tasks requiring token-level fidelity, including syntactic parsing, numerical reasoning, and fine-grained entity matching.Lossy compression at the Guidance Network boundary discards information the downstream task cannot reconstruct.

Ethics Statement

The released datasets use public or openly licensed sources and synthetic generation, with no human subjects or personally identifying information reported.

  • GOODWIKI-LONG-SYNTHETIC derives from English Wikipedia under CC BY-SA, preserves licensing and attribution, and marks GPT-4o-mini-generated documents as synthetic.
  • DAPFAM is built from public USPTO patent records, and neither corpus contains private or personally identifying information.No human subjects were involved.
  • REIGN underperforms truncated GTE-small by 5.1 nDCG@10 on ARGUANA and 6.8 nDCG@10 on FIQA-2018.These benchmarks contain short queries paired with comparatively short documents.

B Encoder-Capacity Ablation: In-Distribution Sweep

The in-distribution encoder-capacity sweep shows an inverted-U pattern: SMALL-L2 often performs best, but the out-of-distribution comparison selects BASE-L3 as the more reliable default.

  • In-Distribution Sweep: The 4×3 GOODWIKI-LONG sweep evaluates REIGN configurations against GTE Guidance Networks at top-k = 10, with the ∆ column exposing an inverted-U pattern.
  • In-Distribution Sweep: TINY-L1 falls below the bare-GN baseline on all three Guidance Networks, while LARGE-L4 falls below baseline on GTE-BASE and GTE-LARGE.
  • In-Distribution Sweep: SMALL-L2 reaches the per-GN maximum on GTE-SMALL and GTE-LARGE, with BASE-L3 essentially tied on GTE-BASE.
  • Out-of-Distribution Verification: The SMALL-L2 in-distribution optimum is treated as an overfitting artefact, so BASE-L3 becomes the paper-default encoder.The result shows that in-distribution capacity ablations alone are insufficient for cross-domain transfer.
  • Out-of-Distribution Verification: BASE-L3 wins 11 of 12 out-of-distribution cells, with average margins of +0.71 nDCG@100 on DAPFAM and +0.59 nDCG@10 on LoCo.The remaining cell is a −0.20 near-tie.

D Positional-Encoding Ablation

The ablation supports removing chunk positional encodings: learned positions are near parity, sinusoidal positions are worse, and no-PE pooling is order-invariant.

  • The ablation arms use InfoNCE, batch 48, 20 epochs, learning rate 10−5, and best-validation selection.These settings differ from the released cosine-objective recipe, so the arms sit below the released operating point.
  • No positional encoding yields an order-invariant pooled document representation, with maximum deviation 3×10−8 under chunk reordering.Learned and sinusoidal positional arms are order-sensitive.
  • Sinusoidal encodings perform worse than the no-PE design across all three benchmarks.The ablation compares GOODWIKI-LONG, LoCo, and DAPFAM under a controlled protocol.
  • Learned absolute positions remain near parity with no positional encoding on all three benchmarks.The comparison uses BASE-L3 on GTE-SMALL with 512-token chunks and InfoNCE training.

E Theoretical Analysis: Information Loss and Recovery

REIGN separates irreversible information loss caused by the frozen Guidance Network from the trainable loss introduced by refurbishment. The analysis characterizes when the trainable gap can vanish and reports substantial efficiency gains from cached GN embeddings.

  • Information-loss decomposition: Theorem 1 decomposes REIGN’s information loss into an intrinsic GN gap and a trainable refurbishment gap, both non-negative.The intrinsic term is fixed by the frozen GN, while the trainable term is the only component affected by the REIGN encoder.
  • Information-loss decomposition: The intrinsic gap equals H(T | E), vanishing exactly when the GN embedding uniquely identifies token sequences almost surely.REIGN cannot recover information discarded by the frozen GN.
  • Recovery conditions: The trainable gap equals I(X; E | Rθ), vanishing exactly when Rθ is sufficient for X given E, including when the refurbishment map is injective on E’s support.This condition identifies the target of training without requiring a particular objective.
  • Contrastive recovery: InfoNCE supplies a mutual-information lower bound at every parameter setting, but certifies at most log B mutual information.The bound tightens as the loss decreases, while the released cosine recipe has no analogous likelihood-ratio guarantee.
  • Efficiency: REIGN’s analytic setup uses O(16nd^2 + 2n^2d) token-level encoder cost and a lightweight 1024→384 refurbishment over k=16 chunk embeddings.Training FLOPs additionally account for forward and backward propagation and contrastive-objective overhead.
  • Efficiency: 49–229× faster query serving is achieved by cached REIGN GN embeddings, with a one-time cache-build cost of 8.9–53.3 ms per document.The measured latency is 0.40–0.52 ms, and uncached REIGN performs the same GN work as the chunked baseline.

H Training-Objective Ablation

The training-objective ablation compares cosine-based training with InfoNCE under matched settings across GOODWIKI-LONG, LoCo, and DAPFAM. Results highlight sensitivity to negative-pool size and the instability of a small-batch InfoNCE configuration.

  • MTEB transfer: The cosine arm records 50.94/33.62 nDCG@10 on ARGUANA/FIQA-2018 in short-text MTEB retrieval.These are the best-of-arm MTEB measurements reported for the 20-epoch cosine configuration.
  • Ablation design: Table 12 evaluates cosine and InfoNCE objectives on GOODWIKI-LONG nDCG@10, LoCo macro nDCG@10, and DAPFAM nDCG@100 under matched conditions.The ablation uses BASE-L3 on GTE-SMALL with chunk/stride 512, generally for 20 epochs and seed 42.
  • Negative-pool sensitivity: At τ=0.07 and batch size 18, InfoNCE validation nDCG@10 peaks at 0.73 and declines to 0.20 by epoch 19.The batch-18 result is therefore an epoch-1 snapshot rather than a stable 20-epoch outcome.
  • Negative-pool sensitivity: Batch size 48 provides a 2,304-pair contrastive pool, compared with 324 pairs at batch size 18, and removes the reported collapse instability.InfoNCE contrasts each anchor with shifted in-batch positives, making the available negative pool depend directly on batch size.
  • Patent-task protocol: DAPFAM training uses binary relevance, false-negative masking, four provided negatives per sample, and additional in-batch negatives.Using all 20 provided negatives exceeds 24 GB of memory at FullText sequence length.
  • Patent-task protocol: DAPFAM evaluates nDCG@100 over the full 45,336-document FullText corpus, while validation nDCG@10 is only an in-batch proxy.Self-matches are removed before scoring, and the same evaluation protocol is applied across systems.
Loading 2608.29899v1…