PaperlayerView exact revision on arXiv

Source-linked AI summary

What Transfers from a VLM Teacher? Comparing Supervision Signals for Visual Document Retrieval

Saba Sturua, Han Xiao

arXiv:2610.09177v1cs.IR

TL;DR

Visual document retrievers train against unlabelled pages as negatives even though labels may omit relevant pages. This paper compares teacher signals on labelled pages and mined candidates, and finds candidate-level teacher signals outperform teacher-free selection rules on the primary suites. Under the rating prompt and temperature studied, the signal reaching the student is close to a relevant/irrelevant split.

  • Problem

    Visual document retrieval commonly trains against unlabelled mined pages as negatives, although labels can omit relevant pages and recent teacher-distillation methods focus on the labelled positive.

    Source: §1 · pp. 1–2

  • Method

    The study compares positive-level attention and description signals with teacher-judged negatives and score distillation, using the same student, data, optimizer, and evaluation setup.

    Source: §1 · p. 2

  • Results

    Teacher judgement beats every tested teacher-free rule: 7.4 against 3.3 on ViDoRe v2 and 4.5 against 2.8 on v3.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Annotators judge about two of the top four mined candidates relevant, and under the studied rating prompt and temperature the student receives nearly a relevant/irrelevant split.

    Source: §1 · p. 1, §7 · p. 7, §9 · p. 10

    The study uses one student, one corpus, one late-interaction architecture, and one teacher, so a larger student or single-vector retriever could reorder the signals.

    Source: §9 · p. 9

Abstract

from arXiv · show

Visual document retrievers are trained contrastively: each query is matched to one page labelled relevant - the positive - and pushed away from negatives, pages presumed irrelevant. Recent methods distil a vision-language model (VLM) teacher into the retriever by enriching that positive, transferring the teacher's attention over it or a description of it. We ask whether the teacher is better spent on the other side, judging the candidates the retriever mines as negatives, which the label says nothing about. With student, data, optimizer and evaluation fixed, teacher-judged hard negatives and score distillation raise ViDoRe v2 nDCG@5 from 55.2 to 62.6 and 63.0; description alignment, as adapted here, gains 2.6 points and attention grounding nothing measurable. Against teacher-free rules that select four candidates from the same mined pool at identical training compute, the best of which is the positive-aware threshold current systems use, the teacher's judgement adds 4.1 points on v2 and 1.7 on v3. This is consistent with how incomplete the labels are. Annotators judge about two of a query's four top-ranked mined candidates relevant, none of them labelled, so training pushes the retriever away from relevant pages treated as negatives. What reaches the student is coarse: under a greedily decoded 0-100 rating prompt, 82% of the teacher's ratings come back at one end of the scale or the other, and a relevant/irrelevant partition keeps most of the distillation gain. A ten-annotator audit places the teacher within the range of variation among human annotators, and finds it reliable where a query has a single determinate answer. We release the code, the teacher's 3.3M judgements and page descriptions, the mined pools, the human audit and the trained adapters at https://github.com/elastic/vdr-teacher-signals.

1 INTRODUCTION

Visual document retrieval preserves page layout by representing pages as images, but its contrastive training treats one labelled page as positive and unlabelled pages as negatives. This study compares teacher supervision on the labelled page with teacher judgements of mined candidates, where incomplete labels can leave relevant pages among the negatives.

1 INTRODUCTION

  • Visual document retrieval encodes pages as images to preserve layout, then trains contrastively against one labelled positive and unlabelled negatives.Layout can carry information through table structure or labels inside figures.Source: §1 · p. 1
  • About two of each query’s four top-ranked mined candidates are judged relevant by annotators, yet none is labelled, exposing false negatives in contrastive training.The single labelled page can leave other pages that answer the query treated as irrelevant.Source: §1 · p. 1
  • Earlier VLM distillation methods enrich the labelled page, while this study compares them with teacher signals on mined candidates under a controlled setup.The comparison fixes the student, corpus, and teacher across signals except attention, which uses the smaller teacher recommended by its authors.Source: §1 · p. 2, Figure 1 · p. 2
  • Candidate judging improves ViDoRe v2 by 7.4–7.8 nDCG@5 against 2.6 for description alignment and approximately zero for attention grounding.Against teacher-free controls at identical training compute, teacher judgement adds +4.1 on v2 and +1.7 on v3.Source: §1 · p. 2

2 RELATED WORK

Prior work has varied visual document retriever backbones and deployment constraints more than supervision, while text retrieval and multimodal embedding have explored teacher-scored mined candidates. This paper compares those candidate-level signals with positive-level VLM distillation in visual document retrieval.

Visual document retrieval

  • Visual document retrieval largely retains one labelled page per query and mined or in-batch negatives, while backbone and deployment constraints have varied.Current mined-negative filters include positive-aware similarity thresholds; some systems also distil reranker scores over candidates.Source: §2 · p. 3

Distillation into retrievers

  • The two recent visual-document VLM distillation methods described here both target the positive: AGREE transfers attention and ReAlign transfers a generated description.Text retrieval distils cross-encoder scores through margin objectives or soft candidate-list scores [Hofstätter et al., 2020].Source: §2 · p. 3

Structural problems with contrastive training data

  • Mined negatives can include unlabelled positives, and prior responses include filtering, easier sampling, objective debiasing, and LLM-based relabelling or selection.In visual document retrieval, DSE uses an answer-string filter and DocReRank generates hard negative queries.Source: §2 · p. 3

Examining LLM judgements for contrastive training

  • Prior LLM-judge studies recommend graded labels and assess their agreement or ranking value, while this work studies the shape of labels used as training targets.Zhuang et al. (2024) find fine-grained relevance levels improve zero-shot rankers over yes/no prompting; UMBRELA reports agreement with human judges.Source: §2 · p. 3

3 EXPERIMENTAL SETUP

The experiments compare four teacher signals using a fixed student, corpus, and training setup. Candidate-level signals judge mined pages, while positive-level signals operate on the labelled page; retrieval is evaluated across four benchmarks.

Experimental setup

  • The student is ColQwen2.5-3B with language-tower LoRA adapters and a frozen vision tower, trained from the backbone rather than a released ColQwen checkpoint.The teacher is Qwen2.5-VL-72B-Instruct for relevance scores and region descriptions; attention uses Qwen2.5-VL-7B-Instruct.Source: §3 · p. 3
  • The training data contain 118,195 query-page pairs over 38,866 pages, and the frozen candidate pool holds 100 mined pages per query, of which the teacher scores 28.The candidate miner is jina-embeddings-v4.Source: §3 · p. 4

The four signals

  • Each arm optimises in-batch InfoNCE plus an auxiliary loss where applicable, with N and S acting on candidates and A and D on the labelled page.The InfoNCE base uses in-batch MaxSim scores, rather than ColPali’s pairwise loss on its hardest in-batch negative.Source: §3 · p. 4
  • N adds the four highest-ranked scored candidates rated irrelevant by the teacher to the InfoNCE denominator, filling any shortfall with the lowest-scored remaining candidates.The irrelevance cutoff is a score below 0.5.Source: §3 · p. 4
  • S distils teacher scores with a KL loss over four scored candidates, sampled one per quartile of the scored pool.The labelled page appears among those four for 12.1% of queries and enters only this term, not the InfoNCE denominator.Source: §3 · p. 4
  • A matches query-conditioned teacher attention over image tokens pooled to the student patch grid using a cosine loss.The attention teacher is the 7B model selected by AGREE’s authors.Source: §3 · pp. 3–4
  • D uses teacher-generated grounded descriptions of the positive page and a KL loss between query-conditioned and description-conditioned distributions over in-batch documents.The student encodes a description through its query tower.Source: §3 · p. 4

Protocol and evaluation

  • The primary recipe is three epochs at batch size 32, and evaluation reports nDCG@5 across four benchmarks, treating ViDoRe v2 and v3 as primary suites.Most arms use three seeds and some use two; D is reported at its best swept temperature, making its figure optimistic.Source: §3 · p. 4

4 WHERE DOES THE TEACHER HELP?

Teacher-judged hard negatives and score distillation outperform positive-level signals in the reported retrieval comparison. Candidate-level gains are positive across the evaluated suites, while the positive-level methods add little under this adaptation.

4 WHERE DOES THE TEACHER HELP?

  • On ViDoRe v2, hard negatives reach 62.6 nDCG@5 from a 55.2 baseline and score distillation reaches 63.0, while description alignment reaches 57.8 and attention grounding changes by about zero.Hard negatives and score distillation gain 4.5 and 5.3 on v3, respectively.Source: §4 · p. 5
  • N and S add four mined candidates per query at 4.8× the step cost, so the initial comparison does not isolate the teacher from the extra candidates.Section 5 separates the teacher from the candidates for N, and section 7 does so only partly for S.Source: §4 · p. 5
  • N, S, and D are positive on all twelve out-of-domain datasets, while A changes sign across them; the four-signal ranking is unchanged at one epoch.The result describes stability across datasets and budgets in the reported runs.Source: §4 · p. 5
  • Attention maps are dominated by page structure, and the description KL at ReAlign’s τd=0.01 is nearly one-hot, with gradients coming from a few examples.At matched seeds, adding attention to hard negatives yields +7.1 on v2 against +7.5 for hard negatives alone; the same description combination was not run.Source: §4 · p. 5
  • AGREE reports +2.95 on ViDoRe v2 with this study’s backbone, data, teacher, readout, and weight, but using its own recipe, which was not run here.That recipe uses batch 128, learning rate 10−4, a hardest-negative pairwise loss, and single runs.Source: §4 · p. 5

5 IS IT THE TEACHER, OR THE EXTRA CANDIDATES?

The section tests whether gains come from adding candidates or from the teacher choosing them. With candidate pools and compute held fixed, teacher-judged negatives outperform the best teacher-free rule on both evaluation suites.

5 IS IT THE TEACHER, OR THE EXTRA CANDIDATES?

  • Teacher-free controls append four candidates at matched exposure and compute, using fixed rank windows or a positive-aware TopK-PercPos threshold.TopK-PercPos retains candidates whose miner score falls below α times the positive’s score.Source: §5 · pp. 5–6
  • The four highest-ranked non-gold candidates are the weakest control, scoring −0.2 on v2 and −2.1 on v3; the teacher rates 62% relevant.The v2 result is no better than baseline, while the v3 result is below it.Source: §5 · p. 6
  • Four candidates from ranks 4–20 roughly recover baseline results, while ranks 21–100 gain +2.4 on v2 and +2.0 on v3.The false-negative rate also fell across deeper windows, but rank depth and window width changed together, so their effects cannot be separated.Source: §5 · p. 6
  • The adaptive positive-aware threshold is the strongest control, gaining +3.3 on v2 versus +2.4 for the best fixed window, but still trails the teacher.The adaptive rule admitted 45% teacher-rated-relevant negatives, versus 36% for the best window.Source: §5 · p. 6
  • +4.1 nDCG@5 on ViDoRe v2 and +1.7 on v3: teacher-selected negatives beat the best teacher-free rule at fixed candidate pool and compute.N leads every teacher-free rule at every seed on both suites.Source: §5 · p. 6

6 HOW INCOMPLETE ARE THE TRAINING LABELS?

The section finds substantial label incompleteness in mined candidates: queries written against one page can have other valid answer pages, and annotators judge many top-ranked candidates relevant despite none being labelled.

6 HOW INCOMPLETE ARE THE TRAINING LABELS?

  • The training queries were written or generated against single pages; 2,620 of 2,736 recurring query strings appeared against multiple pages.One query string appeared against 70 pages, and 6,377 rows repeated a string.Source: §6 · p. 7
  • Across scored pools, the teacher marked a median of 10 non-gold pages relevant per query, and rated the top non-gold candidate relevant for 70.1% of queries.Top-candidate rates varied by source, from 43% for Infographic-VQA to 83% for TAT-DQA.Source: §6 · p. 7
  • On audited queries with a complete top four, annotators judged 2.04 of four mined candidates relevant on average, while dataset labels marked none relevant.The cluster-bootstrapped interval was [1.46, 2.61].Source: §6 · p. 7
  • At least three of the top four candidates were teacher-rated relevant for 55.7% of queries, and 20% of N’s negatives remained teacher-rated relevant after back-fill.For 45% of those queries, the 28 scored candidates contained fewer than four teacher-rated irrelevant pages.Source: §6 · p. 7
  • Many rejected candidates answer the query as posed; the issue is that multiple pages can answer it, not that the query is necessarily vague.The section distinguishes whether a query has one determinate answer from whether an individual page answers it.Source: §6 · p. 7

7 WHAT REACHES THE STUDENT?

The section examines the information in the teacher’s score targets and finds that the rating readout is often near-binary. Most of the score-distillation gain remains when targets are reduced to a relevant/irrelevant partition.

7 WHAT REACHES THE STUDENT?

  • The greedily decoded 0–100 rating placed 51.7% of the teacher’s scores at 0 and 29.9% at 1.The same model’s yes/no log-probability readout returned continuous scores, indicating this concentration is a property of the rating readout.Source: §7 · p. 7
  • For 42.6% of queries, S’s four-candidate target was constant, making its softmax uniform and providing no candidate-level relevance ordering.Those rows comprised 27.0% all-zero, 15.1% all-one, and 0.5% another constant value.Source: §7 · p. 7
  • Masking constant-target rows beat S by 2.7 points on v2 and 1.2 on v3, though this post-hoc result is exploratory.The effective-weight change from masking did not account for the observed v2 difference.Source: §7 · p. 8
  • Replacing graded scores with the relevant/irrelevant partition retained 76% of S’s ViDoRe v2 gain at the two shared seeds.The partition retained nearly all v1 gain and all v3 gain, but its v2 shortfall may partly reflect more rows becoming uniform; the post-hoc comparison used only two shared seeds.Source: §7 · p. 8
  • N excludes teacher-rated relevant pages from negatives, while S raises their target; S’s softmax cannot distinguish all-irrelevant from all-relevant candidate sets.The two methods therefore use the same relevant/irrelevant partition in different ways.Source: §7 · p. 8

8 WHEN CAN THE TEACHER BE TRUSTED?

The audit finds that the teacher’s protected candidates are usually judged relevant by annotators, and its disagreement lies within the range of human-to-human variation. Reliability differs sharply between queries with one determinate answer and broad queries.

8 WHEN CAN THE TEACHER BE TRUSTED?

  • Annotators judged 71% of teacher-protected candidates relevant, while 14% of audited candidates the teacher rejected were judged relevant.The latter are teacher misses among the audited top-four candidates, separate from teacher-rated relevant pages added through back-fill.Source: §8 · p. 8
  • Teacher–human κ was 0.54 versus a mean human–human κ of 0.41, but the paper treats these different estimators descriptively.Human–human pairwise κ ranged from −0.13 to 0.87; the authors conclude teacher disagreements are not of a different order from human disagreements.Source: §8 · p. 8
  • Protected candidates were 83% relevant on specific queries and 42% on broad queries, with non-overlapping intervals.The specific/broad classification was made before any grades were read.Source: §8 · pp. 8–9
  • Protection precision rose from 71% at a ≥0.5 cut to 85% at 1.0, but a higher cut also releases more pages as eligible negatives.The study did not retrain N at other cuts, and S never thresholds.Source: §8 · p. 9
  • Of queries where the teacher hedged, 69% were broad, versus 22% where it did not; the authors say this is consistent with the near-binary readout.This was a post-hoc analysis with Fisher exact p=0.005.Source: §8 · p. 9

9 IMPLICATIONS AND SCOPE

On this corpus, judging mined candidates beat the teacher-free rules tested, but the signals’ scope and effect sizes remain bounded by the study’s setup. The authors recommend checking what negatives teach and note that the advantage may shrink with better-labelled corpora.

9 IMPLICATIONS AND SCOPE

  • A judge over mined candidates beat every teacher-free rule tested on both primary suites, while top-ranked candidates without a judge were the worst negatives tested.The top-ranked candidates were no better than the baseline on v2 and below it on v3.Source: §9 · pp. 9–10
  • N and S require one 46 GPU-hour annotation pass and 4.8× the training step, the most expensive single run among the four signals.The authors leave whether enriching the positive adds to mined negatives untested.Source: §9 · p. 9
  • The study covers one student, corpus, late-interaction architecture, and teacher; a larger student or single-vector retriever could reorder the signals.Mined negatives may also matter less at larger batch sizes.Source: §9 · p. 9
  • Two to three seeds preserve the sign and ordering of mean gains, but not precise effect sizes, and the human audit covers only top-ranked candidates.The conclusion expects the judge advantage to shrink on better-labelled corpora.Source: §9 · p. 10

USE OF AI

AI assistance covered implementation, evaluation, analysis, and drafting; the teacher scores and region descriptions and the annotation-interface OCR transcripts were method-related uses. The authors state that human judgments were not AI-produced and that they reviewed and take responsibility for the work.

USE OF AI

  • An AI coding assistant supported implementation and testing of the four signals and evaluation harness, analysis scripts, control design, result interpretation, and drafting and revising the paper.It also reviewed drafts and proposed the restructuring used in the current version.Source: USE OF AI · p. 10
  • The VLM generated teacher relevance scores and region descriptions, while machine-generated OCR supplied page transcripts in the annotation interface.Source: USE OF AI · p. 10
  • No AI produced the 400 human judgments or the specific/broad query classifications, and the human-study analysis was pre-specified and frozen before judgments were read.The one post-hoc analysis is labelled as such.Source: USE OF AI · p. 10
  • The authors reviewed AI-assisted text, code, and analysis and take responsibility for the paper, including its reported numbers.They state that every reported number comes from evaluation result files rather than being written by a model.Source: USE OF AI · p. 10

REPRODUCIBILITY

The release provides code, judgments, descriptions, mined pools, study materials, and selected trained adapters. It supports checking reported arms at one seed without retraining and regenerating the paper’s tables from included run outputs.

REPRODUCIBILITY

  • The release includes teacher relevance scores for 3.3M query-candidate pairs, region descriptions for 118,195 training rows, frozen mined pools, the human-study protocol and 400 judgments, and code.The repository regenerates every table from the included run outputs.Source: REPRODUCIBILITY · p. 10
  • Five trained adapters are released at one seed each: the baseline, N, S, TopK-PercPos at α=0.95, and the S-masked arm.The authors release these five rather than every trained arm to keep the release usable.Source: REPRODUCIBILITY · p. 10
  • The released judgments and score targets can be reproduced or reused with another student without access to a 72B teacher.The released adapters let readers check reported numbers for those arms at one seed without retraining.Source: REPRODUCIBILITY · p. 10
  • Appendices document prompts and parsing, the evaluation harness, run consistency, individual runs, costs, annotation procedures, and implementation defects.Appendix A records three implementation defects that produced plausible results before being caught.Source: REPRODUCIBILITY · p. 10

ETHICS

The audit used unpaid volunteers to judge pages from a publicly available training corpus, with consent for use and pseudonymised release of judgments. It collected no personal data or demographics, and the released judgments contain no free text.

ETHICS

  • Ten unpaid volunteers judged pages from an existing, publicly released training corpus; one was an author, five were colleagues, and four were personal acquaintances.The task took about twenty-five minutes, and all consented to use and pseudonymised release of their judgments.Source: ETHICS · p. 10
  • Annotators saw neither the teacher’s score nor which page carried the dataset label, and dropping any one annotator kept relevant pages per query within 1.93–2.24.Source: ETHICS · p. 10
  • No personal or demographic data was collected; released judgments use pseudonymous identifiers, and contain no free text.The machine transcript shown beside each page served only as a search aid.Source: ETHICS · pp. 10–11
  • The report retains the full annotator-behavior distribution, including two annotators at the extremes of the relevant-rate range.Source: ETHICS · p. 11

A THREE IMPLEMENTATION PITFALLS IN LATE-INTERACTION TRAINING

The section documents implementation and evaluation checks that affect interpretation of training signals and comparisons, alongside scoring readouts, attention-map properties, cost measurements, and human-audit design. It also examines teacher filtering’s reliability and limits across datasets, including remaining contamination and inconclusive targeted-filtering results.

A THREE IMPLEMENTATION PITFALLS IN LATE-INTERACTION TRAINING

  • At T=1, S’s target retained 97.3% of uniform entropy, versus 59.1% at T=0.02, changing S from sign-flipping to indistinguishable from N.A separate precision check found 43.6% of the output-projection weights unchanged after 3,693 steps; fp32 master weights reduced that fraction to 0.00% within 20 steps.Source: A THREE IMPLEMENTATION PITFALLS IN… · p. 14, Table 5 · p. 15

H HYPERPARAMETER SWEEPS FOR ATTENTION AND DESCRIPTIONS

B TEACHER RELEVANCE SCORING

  • On 500 queries and 9,577 pairs, the logprob readout’s tie-aware AUC was 0.880 versus 0.825 for the rating, so the paper makes no claim that rating is the better discriminator.The rating was retained because N and S use score magnitudes, and evaluating the alternative would require re-annotating 3.31M pairs and retraining; its effect on their margin over A and D remains untested.Source: B TEACHER RELEVANCE SCORING · p. 15

C WHICH TEST SETS ARE IN-DOMAIN

  • Eight of ten ViDoRe v1 test sets have training counterparts, whereas v2 and v3 share no source family with training; paired differences remain comparable within the reimplemented harness.The harness scored tomoro-colqwen3-embed-4b at 64.50 versus 64.69 on the v2 leaderboard, a 0.19-point difference that only approximately bounds its offset.Source: C WHICH TEST SETS ARE IN-DOMAIN · p. 16, D HARNESS CALIBRATION · p. 16

G WHAT AN ATTENTION MAP CONTAINS

  • Across shared pages, attention maps had mean cosine 0.689 versus 0.330 for different pages, and corrected variance attribution was 65% page and 35% query.The query-specific portion peaks on query-relevant text, but the whole-map cosine loss is computed mostly over the page-driven portion; dividing by query-agnostic maps raised the naive query-specific share from 21% to 29%.Source: G WHAT AN ATTENTION MAP CONTAINS · p. 17

F DETERMINISM

  • At one epoch, the InfoNCE-only baseline scored 49.0, 51.7, and 56.7 on ViDoRe v2 across seeds, a 7.7-point spread with nothing changed but the seed.The authors therefore compare seed-matched runs: query-level bootstrap intervals describe query sampling conditional on checkpoints, not training randomness.Source: F DETERMINISM · p. 17, I EVERY SEED OF THE MAIN ARMS · p. 19, Table 7 · p. 19

J COST, IN DETAIL

  • Description generation cost approximately 110 GPU-hours, while four appended candidate pages raised three-epoch training cost by 91 GPU-hours over the 23.9-GPU-hour baseline.Candidate mining and scoring for N and S took about 9 and 37 GPU-hours, respectively; attention extraction took 24 GPU-hours with the 7B teacher, and its estimated 72B cost was 130–280 GPU-hours.Source: J COST, IN DETAIL · pp. 21–22, Table 12 · p. 21

K THE HUMAN VALIDATION STUDY

  • The human audit sampled 50 queries for ten annotators, with every page judged twice by different people and teacher scores and positive labels concealed.The four-point UMBRELA rubric distinguished partial answers from merely on-topic pages, and nine of 60 annotator views containing a “cannot judge” grade were dropped.Source: K THE HUMAN VALIDATION STUDY · p. 22
  • Leave-one-annotator-out analysis keeps precision of protection at 68–73%, miss rate at 11–17%, and relevant-pages-per-query at 1.93–2.24, with specific/broad intervals disjoint in all ten cases.No annotator is excluded; human–human κ is 0.41, and agreement with the teacher is nearly flat across all ten annotators.Source: K THE HUMAN VALIDATION STUDY · p. 23, Table 13 · p. 23

L CONTAMINATION BY STRATUM

  • On severe queries, the filter back-fills with pages it rated relevant for 45% of queries; pooled across strata, 20.0% of trained negatives remain teacher-rated relevant.Filtering is nearly a no-op on the clean stratum, while severe-stratum analyses are confounded because that is also where filtering is least complete.Source: Table 14 · p. 23, L CONTAMINATION BY STRATUM · p. 23

M WHERE THE FILTER IS APPLIED

  • At a fixed budget of 15,813 treated queries, stratum targeting is inconclusive: severe helps and moderate does not, but random treatment matches severe on v2 and alone is consistently positive on v3.Every mean effect is under 1.5 points at two seeds, and the random draw’s v2 seeds span 2.4 points; removing severe-arm back-fill costs little despite back-fill firing on 45% of severe queries.Source: Table 14 · p. 23, M WHERE THE FILTER IS APPLIED · p. 24, Table 15 · p. 24

N PER-DATASET BREAKDOWN

  • N and S improve nDCG@5 on all 12 out-of-domain datasets; D is positive on all with smaller margins, while A changes sign and never by more than about two points.The gains vary about fourfold across the twelve datasets, but their sign does not change for N and S.Source: Table 16 · p. 25

O WHAT THE TEACHER’S FILTER REMOVES

  • The filter rejects the rank-1 candidate, which the teacher rates relevant for 70% of queries, shifting selected negatives from median rank 2 to 5 and varying depth by query.22.9% of N’s accepted negatives come from beyond rank 10. Of 115,698 queries, 40% have at least three top-four candidates rated ≥0.9; contamination is highest in form-reading sources, where hard-negative mining is ill-posed.Source: Table 17 · p. 25, O WHAT THE TEACHER’S FILTER REMOVES · pp. 25–26

P EXTENDED LIMITATIONS

The conclusions are bounded by a single student, corpus, and architecture, alongside limited tuning and audit coverage. Several signal combinations and the value of continuous teacher targets remain unresolved.

P EXTENDED LIMITATIONS

  • The study covers one student family, corpus, and late-interaction architecture; other retrievers could change the signal ordering.The descriptions method was ported to MaxSim scoring, and the attention arm used a 7B teacher while the other arms used a 72B teacher.Source: P EXTENDED LIMITATIONS · p. 26
  • The three-point hyperparameter screens and two-to-three three-epoch runs support consistent signs and gain ordering, but not precise effect sizes.Descriptions are reported at a swept value, and the authors note that an untested axis could change whether a signal helps.Source: P EXTENDED LIMITATIONS · pp. 26–27
  • The human audit covers only the four top-ranked candidates and does not test whether better judgements would train a better retriever.Negatives below the top four are unaudited; the audit finds teacher-human agreement comparable to human-human agreement.Source: P EXTENDED LIMITATIONS · p. 27
  • Because training never uses a continuous teacher target, the study cannot establish whether genuinely graded supervision would do better.The reported rating readout is close to binary, while a yes/no logprob readout of the same teacher is not.Source: P EXTENDED LIMITATIONS · p. 27
  • Compute-matched controls support N's margin and S's gain, but the study lacks an informative teacher-free target for S.S's flat-target control loses 3.1 points on v2; a control using the retriever's own scores over the same candidates remains future work.Source: P EXTENDED LIMITATIONS · p. 27
  • The study tests few signal combinations and page-level retrieval only, leaving complementarity and possible grounding benefits unresolved.Hard negatives plus attention leaves the v2 result near hard negatives alone; attention plus descriptions was run for one epoch and flips sign across seeds on v2.Source: P EXTENDED LIMITATIONS · p. 27
Loading 2610.09177v1…