PaperlayerView exact revision on arXiv

Source-linked AI summary

Contextual Embeddings Beyond the Gold Chunk

Sedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall, Jonas Golde, Louis Milliken, Ekaterina Glazkova, Denis Bykov, Bo Wang

arXiv:2610.38854v1cs.LGcs.CLcs.IR

TL;DR

Contextual embedding research typically assumes that one chunk contains all relevant information, although answers may require multiple chunks and supporting evidence. This paper trains contextual retrievers using graded chunk relevance and introduces benchmarks for answer and evidence retrieval; its 9B model sets new state of the art on the textual benchmarks and ConTEB.

  • Problem

    Prior contextual embedding training and evaluation assume one relevant chunk, despite queries that require complementary chunks to answer or verify a result.

    Source: §1 · p. 2

  • Method

    The paper distills graded chunk-level relevance from a query-aware context compression model and introduces benchmarks that label answer and evidence chunks.

    Source: §6 · p. 16

  • Results

    The 9B model sets a new state of the art on context-bench, Q2D-Chunk, and ConTEB, and both models are state-of-the-art dense retrievers on ViDoRe-V3.

    Source: §6 · p. 16

  • Takeaways & Limitations

    Evidence retrieval remains challenging on both benchmarks, and Q2D-Chunk is available as a public leaderboard.

    Source: §6 · p. 16

    Q2D-Chunk queries are not user-authored: they begin with assistant-generated search queries and are rewritten into natural-language questions.

    Source: Ethics and Privacy Statement · p. 17

Abstract

from arXiv · show

Contextual embeddings represent chunks of a document with respect to the document's global context. These models are typically trained and evaluated under the assumption that a relevant document contains only a single relevant chunk. In this work, we argue that in real-world use cases a retriever should not only find answers spread over several chunks but also surface the evidence needed to verify them. To address this, we introduce pplx-embed-v2-context, a family of contextual embedding models trained by distilling graded chunk-level relevance from a query-aware context compression model. This removes the need for gold-chunk labels and makes the training targets independent of the chunking. We utilize a vision-language backbone for training and release pplx-embed-v2-context in 9B and 0.6B sizes with compatible embedding spaces. On the evaluation side, we introduce two benchmarks that label both the chunks that answer a query and the evidence needed to verify the answer. Context-bench is a controlled diagnostic, and Q2D-Chunk is built from real-world search queries. To encourage further research, we also publish Q2D-Chunk as a public leaderboard. Our 9B model sets a new state of the art on both benchmarks and ConTEB. We find that the visual retrieval capabilities of our 9B model transfer from its vision-language base model and propose to extend chunking to image inputs. We demonstrate that our models achieve state-of-the-art performance among dense retrievers on visual ViDoRe-V3 and that performance improves with granular image chunking.

1. Introduction

Contextual embeddings retain document-wide context for each chunk, but prior training and evaluation assume a single relevant chunk. This work replaces gold-chunk supervision and introduces benchmarks that assess answer and supporting-evidence retrieval.

  • Gold-chunk supervision treats complementary chunks as negatives, costs a language-model annotation per pair, and ties labels to the original chunking.Document-level loss identifies relevant documents but not the relevant chunks.Source: §1 · p. 2
  • Single-gold-passage benchmarks miss evidence needed to verify answers and do not test retrieval among near-identical documents.Such evidence can identify the entity, date, version, definition, or condition to which an answer applies.Source: §1 · p. 2
  • The proposed training approach replaces gold-chunk labels with chunk-level relevance scores distilled from a query-aware context compression model.Source: §1 · p. 2
  • Context-bench and Q2D-Chunk label both answer chunks and evidence chunks, and Q2D-Chunk is published as a public leaderboard.Source: §1 · p. 2
  • The released 9B and 0.6B models share compatible embedding spaces, and the 9B model achieves state-of-the-art results across textual retrieval benchmarks.The benchmarks named include ConTEB, context-bench, and Q2D-Chunk.Source: §1 · p. 2

2. Chunk-level relevance distillation for contextual embeddings

The models learn contextual chunk relevance by distilling graded targets from a query-aware context compressor, rather than relying on gold-chunk labels. The section describes the 9B training objective, the 0.6B distillation approach, visual chunking, and the training setup.

2. Chunk-level relevance distillation for contextual embeddings

  • The 9B model is trained with query-aware context-compression relevance targets for chunks and a document-level contrastive loss.The context compressor is used only during training, so retrieval adds no teacher latency or index storage; the model also supports single-chunk document embeddings.Source: §2 · p. 3, §2.2 · p. 5

2.1. Background: Query-aware context compression

  • The teacher assigns token-level keep probabilities from query-document pairs, using sentence labels for answer-bearing, supporting, and irrelevant content.Its token-level graded scores can be aggregated for any chunking strategy and computed online more cheaply than large-language-model annotation.Source: §2.1 · p. 3

2.2. Training objective of the 9B model

  • Chunk targets use the mean relevance of each chunk’s k highest-scoring tokens, then fit a softmax target distribution with forward KL divergence.The teacher assigns zero probability to chunks in in-batch negative documents; the loss combines within-document relevance matching with document-level contrastive learning, weighted 1 and 2, respectively.Source: §2.2 · pp. 3–4, Figure 1 · p. 5

2.3. Distillation to the 0.6B scale

  • The authors report better results at 0.6B scale by distilling their 9B contextual model at the representation level, and the 0.6B and 9B models share an embedding space.Documents indexed with the 9B model can be queried with the 0.6B model.Source: §2.3 · p. 5

2.4. Visual document retrieval

  • The vision-language models process images through their vision tower, and the 9B model’s visual retrieval capability transfers from its base model despite text-only training.Image pages are divided into up to 16 horizontal-band chunks; the passage reports that chunking improves visual document retrieval.Source: §2.4 · p. 6

2.5. Training setup and datasets

  • The 9B model uses 2048-dimensional int8 embeddings, while the 0.6B model is trained against the 9B model’s 1024-dimensional MRL prefix.Training uses roughly 57M pairs for the 9B model across about 440 datasets and over 50 languages; visual data is mixed into 0.6B training, and context-compression supervision is disabled where within-document relevance is not meaningful.Source: §2.5 · p. 6

3. Evaluating contextual embeddings beyond the gold chunk

The section introduces complementary benchmarks for evaluating contextual retrieval beyond a single gold passage, including whether models retrieve evidence needed to interpret or verify answers. It describes their construction and the metrics used to assess answer, evidence, and document retrieval, coverage, and reading cost.

3. Evaluating contextual embeddings beyond the gold chunk

  • Context-bench and Q2D-Chunk address the single-passage evaluation gap with complementary benchmarks labeling answer and evidence chunks, including cases spanning multiple pages.Context-bench is controlled; Q2D-Chunk uses real-world search queries.Source: §3 · pp. 6–7

3.1. Context-bench: a controlled diagnostic via synthetic injection

  • Context-bench uses long documents and sentence chunks to isolate document disambiguation, answer retrieval, and evidence retrieval, and remains private to limit contamination and overfitting.Source: §3.1 · pp. 7–8

3.2. Q2D-Chunk: chunk retrieval on real search traffic

  • Q2D-Chunk draws long queries from production search logs, rewrites them as self-contained questions, and supplies validated paraphrased, indirect, and scenario variants for some queries.PII-containing queries are dropped before query text reaches a language model.Source: §3.2 · p. 8
  • Candidate pages include search results and retrieved likely-answer pages; chunk-level answer and evidence labels use query-specific rubrics, with independent labelers and adjudication for disagreements.The rubric covers answer groups and evidence for identification, conditions, and interpretation.Source: §3.2 · pp. 8–9
  • 6,069 base queries form the final Q2D-Chunk benchmark; within a page, a median of four answer chunks form three runs spanning 45% of the page.The benchmark has 11,719 queries including reworded variants, across 10 languages and 26 top-level domains.Source: §3.2 · pp. 9–10

3.3. Metrics

  • Answer@K counts recovered answer groups, while evidence recall@K measures recovered evidence groups after accepted answer chunks are removed from the ranking.Q2D-Chunk reports corpus-wide and within-document evidence retrieval; context-bench reports the within-document variant.Source: §3.3 · p. 10
  • On Q2D-Chunk, α-nDCG@10 measures aspect coverage and tokens to full coverage measures reading cost; Document@K evaluates gold-document retrieval or answer-page retrieval.Tokens to full coverage counts ranked tokens until every answer and evidence group is covered; queries never fully covered count as unreached.Source: §3.3 · pp. 10–11

4. Results

Across answer-and-evidence, document, and visual retrieval evaluations, the 9B contextual model leads on most reported comparisons, while results show that gains from chunking vary by task and storage setting.

4.1. Answer and evidence retrieval

  • 14.4 points at Answer@10 over voyage-context-4, with the 9B model also strongest for evidence retrieval on context-bench.In the single-chunk setting, its Document@10 is 29.8 points lower than when documents are ranked by their best chunk.Source: §4.1 · p. 11
  • 11–17 points at Answer@10 across four query sets over pplx-embed-context-v1 and 2–3 points over voyage-context-4; the 9B model leads except on evidence recall.Evidence recall is led by Nemotron-3-Embed-8B; the 9B model also reaches full coverage with the fewest tokens.Source: §4.1 · p. 12, Table 3 · p. 12
  • Across the ten most frequent domains, which cover 89% of queries, the 9B model leads in every domain; the 0.6B model stays within 3.0 points of voyage-context-4.The 0.6B model leads voyage-context-4 in health, business, and science.Source: Table 4 · p. 12, §4.1 · p. 13
  • 1.3 points above pplx-embed-context-v1 and 3.0 above voyage-context-4 on ConTEB average nDCG@10, the 9B model leads five of eight tasks.Unlike pplx-embed-context-v1, it had not seen ConTEB data; both non-contextual models outperform contextual models on COVID-QA.Source: §4.1 · p. 13, Table 5 · p. 13

4.2. Document retrieval

  • In document retrieval, the 9B model with 128-token chunks is outperformed only by voyage-context-4, while the 0.6B model is strongest in both settings among sub-billion models.With one chunk, the 9B model is also outperformed by voyage-4-large and Nemotron-3-Embed-8B.Source: §4.2 · pp. 13–14, Table 6 · p. 14

4.3. Retrieval quality and storage costs

  • Smaller chunks improve document retrieval until chunk size exceeds typical document length, and finer chunking with 1024 dimensions usually improves performance at the same index size.The analysis varies chunk granularity and embedding dimensions against storage; the models use int8-quantized embeddings.Source: §4.3 · p. 14, Figure 4 · p. 15

4.4. Visual document retrieval

  • On ViDoRe-V3, performance scales with index size: the 9B model with two chunks and 1024-dimensional embeddings beats Nemotron-3.5-8B with a 4× smaller index.On MIRACL-Vision, chunking has little effect; the models’ top-ranked visual chunks also agree with a decision model 72.1% for 9B and 69.8% for 0.6B, versus 25.1% for the random control.Source: §4.4 · pp. 14–16, Figure 5 · p. 15

5. Ablations

The ablation finds that chunk-level distillation is necessary for chunk retrieval, while the contrastive term improves results across all three evaluated benchmarks.

5. Ablations

  • Without distillation, Q2D-Chunk drops 15.2 points on average and ConTEB drops 14.6 points; the contrastive term improves all three benchmarks.Distillation alone preserves most document-level quality through the document-level term in its batch-wide KL.Source: §5 · p. 16, Table 7 · p. 16

6. Conclusion

The paper presents contextual embedding models trained from graded chunk relevance and introduces benchmarks that assess both answer and evidence retrieval. The 9B model sets state-of-the-art results on the named text and visual benchmarks, with visual retrieval gains from more granular image chunking.

6. Conclusion

  • The models distill graded chunk-level relevance from a query-aware context compression model instead of using gold-chunk labels.The benchmarks label answer and evidence chunks; Q2D-Chunk is published as a public leaderboard.Source: §6 · p. 16
  • The 9B model sets a new state of the art on context-bench, Q2D-Chunk, and ConTEB, while both models are state-of-the-art dense visual retrievers on ViDoRe-V3.Visual retrieval performance improves when pages are embedded as several chunks; evidence retrieval remains challenging on both new benchmarks.Source: §6 · p. 16

Ethics and Privacy Statement

Q2D-Chunk uses production search traffic and publicly accessible web pages, with privacy filters and model-assigned relevance and evidence judgments.

Ethics and Privacy Statement

  • Q2D-Chunk labels are assigned by LLM labelers and an LLM adjudicator, not human annotators, and should be read as model-assigned judgments.The hosted leaderboard is private and shares only aggregate results externally.Source: Ethics and Privacy Statement · p. 17

A. Training hyperparameters

This block records the 9B model’s training setup, describes visual-page chunking, and presents benchmark query variants, examples, and domain coverage.

A. Training hyperparameters

  • Table 8 lists the 9B model’s hyperparameters for single-stage Matryoshka and int8 quantization-aware training.Source: Table 8 · p. 20

B. Visual document chunking

  • Visual pages are divided into horizontal bands; a learned SEP separates adjacent band spans, with MRoPE coordinates placed at band interfaces.The soft visual token sequence is passed through the text tower.Source: B. Visual document chunking · p. 20, Figure 6 · p. 20

C. Context dependency types in context-bench

D. Q2D-Chunk query variants

  • Q2D-Chunk rewords queries as paraphrases, indirect descriptions, and first-person scenarios; the variants respectively reduce keyword overlap, replace named entities, or add a forum-post framing.Source: D. Q2D-Chunk query variants · p. 22

E. A Q2D-Chunk example

  • The example’s answer chunks give forecasts, but the evidence chunk identifies the ADB’s April 2026 outlook; Kazakhstan has no chunk because the page gives no Kazakhstan-only figure.Source: Figure 8 · p. 23

F. Q2D-Chunk domains and languages

G. Retrieval quality across domains and chunk-size sensitivity

The model leads MTEB query-to-chunk retrieval, while its document-retrieval performance is comparable to the strongest baselines; the block also describes dataset construction and evaluation configurations.

G. Retrieval quality across domains and chunk-size sensitivity

  • MTEB-Q2C labels relevant chunks within relevant documents using exact answer spans when available, or LLM judgments otherwise; MTEB-Q2D evaluates relevant-document retrieval.Source: G. Retrieval quality across domains… · p. 23
  • The accompanying figures chart Q2D-Chunk query counts by top-level domain and language, alongside Context-bench coverage by subject domain and dependency type.Source: Figure 10 · p. 24, Figure 11 · p. 24, Figure 9 · p. 24
  • The model achieves the highest average MTEB chunk-retrieval score, with clearest gains in legal, technical, and health domains; document retrieval is comparable to the strongest dense and contextual baselines.pplx-embed-context-v1 is markedly weaker on both suites; the authors say this is consistent with its benchmark-specific gold-chunk training supervision transferring poorly.Source: G. Retrieval quality across domains… · p. 25, Table 10 · p. 25, Table 11 · p. 25
  • Figure 12b tracks how 9B MTEB-Q2D retrieval quality changes with indexing chunk size; Figure 12a plots MTEB-Q2C mean nDCG@10 against storage per vector.Figure 12a distinguishes float32 and int8 storage and labels embedding dimensions.Source: Figure 12 · p. 25, G. Retrieval quality across domains… · p. 25

H. Document-level retrieval on ViDoRe-V3

I. Quantitative analysis of top chunk localization in ViDoRe-V3

The study tests whether the top-ranked visual chunk localizes answer-relevant information, finding that model-selected chunks are judged relevant more often than donor-control chunks.

  • For both models, the evaluation selects a top chunk from eight chunks per page across 1,200 English query–page pairs and asks a judge whether it directly helps answer.The pairs comprise 150 examples from each of eight public ViDoRe-V3 datasets.Source: §I · p. 26
  • 72.08% (9B) and 69.83% (0.6B) of model-selected chunks were judged directly relevant, versus 25.08% for the shared donor control.Figure 13 reports proportions of highlighted regions judged relevant; its pointwise bootstrap-interval overlap is not a paired significance test.Source: Figure 13 · p. 27, §I · p. 27

J. Ablations on visual document retrieval

The ablations indicate that contextual image chunking matters, while the learned separator’s effect varies by benchmark; examples show selected chunks across ViDoRe-V3 datasets.

J. Ablations on visual document retrieval

  • Independent chunk passes degrade as chunk count rises; contextual information matters, while learned separators help on ViDoRe-V3 but barely change MIRACL-Vision.The authors conclude naive chunking alone does not improve visual document retrieval and that separator benefits may depend on the setting.Source: Figure 14 · p. 27, J. Ablations on visual document… · p. 27

K. Examples of top chunks in ViDoRe-V3

  • Examples group model-selected chunks by ViDoRe-V3 dataset and include queries about FDA use of real-world data and cauliflower-versus-coast fractal dimensions.The examples are the first query–document pair for each dataset, without manual selection.Source: K. Examples of top chunks in ViDoRe-V3 · pp. 28–29

L. Results on BRIGHT

On re-chunked BRIGHT, the 9B model outperforms voyage-4-large, while GPT-4 query reformulations benefit the 0.6B model but not the 9B model.

L. Results on BRIGHT

  • The models improve over v1 baselines across all seven domains, which the authors attribute to chunk-level distillation.BRIGHT was re-chunked to a target size of 256 tokens while preserving original gold boundaries; documents without negative passages and their queries were removed.Source: L. Results on BRIGHT · p. 30
  • GPT-4 query reformulations yield no average improvement for the 9B model, which scores 0.2 points lower, while the 0.6B model gains 5.2 points.BRIGHT queries can require several chunks from several documents.Source: L. Results on BRIGHT · p. 30
  • The 9B model beats voyage-4-large by 6.0 points and trails voyage-context-4 by 0.8 points on average across BRIGHT domains.The 0.6B model trails voyage-4-large by 1.5 points.Source: Table 14 · p. 30, L. Results on BRIGHT · p. 30
Loading 2610.38854v1…