Source-linked AI summary
Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines
Siddharth Vohra, Runmin Jiang, Xiaomo Li, Min Xu
TL;DR
Grounded language-model pipelines need to preserve a selected object through retrieval, but benchmark recall may evaluate a different dataset-linked object. This paper audits that identity handoff across HybridQA selector records and introduces ROP to make targets, denominators, and outcomes explicit. Ranked retrieval sometimes omits the selected object, and its presence is associated with higher reader exact match.
Problem
A single top-k recall score cannot distinguish poor selection, a broken identity handoff, and failure to use returned evidence when the checked object and responsible component are unspecified.
Method
The paper audits selected-object return and dataset-trace recall across frozen HybridQA pipelines and releases ROP, an executable profile for targets, returned fields, cutoffs, membership rules, and expected populations.
Results
On aligned records, handoff rules omit the selected object between 14 and 389 times at cutoff five, while the two target checks disagree on 106 of 1,792 original-question rankings.
Takeaways & Limitations
Selected-object return and dataset-trace recall identify different components, while ROP makes the chosen target and denominator explicit and replayable.
Takeaways & Limitations
The deletion result applies only to a deliberately selected 64-item cohort, and selector stability, semantic validity, and average answer benefit require separate evaluations.
Abstract
from arXiv · showhide
Grounded language-model pipelines can be divided into three stages: selecting an object, retrieving passages for it, and using that evidence to answer. If the selected object must reach the reader, losing it breaks the handoff. Benchmark recall checks the dataset-linked object, which can differ. We audit 600 HybridQA questions across three selector families. On 1,463 resolvable records where the selected object matches the dataset-traced passage, exact key lookup and exact title matching return the object every time. With every ranked rule given the same decoded selected title, body-only BM25 omits it on 389 records (26.6%) at cutoff five, while hybrid retrieval with reranking omits it on 14 (1.0%). The two identities differ on 329 of 1,792 resolvable records. With original-question rankings, their top-five checks disagree on 106 records (5.9%). Frozen reader comparisons associate the aligned object's presence with 28.6 to 31.0 points higher exact match. In a deliberately selected 64-item cohort, removing that passage sharply lowers exact match, while removing a similar-length comparison passage does not reproduce the drop. We release the Returned-Object Profile (ROP), an executable record of the target, returned-ID field, cutoff, membership rule, and complete expected population, with data and an offline replay.
1 Introduction
The paper frames retrieval as an identity handoff: a selected object must reach the reader, while dataset-trace recall may check a different object. On aligned HybridQA records, deterministic handoffs preserve the object, but ranked retrieval can omit it.
- Motivation: A grounded pipeline can lose a selected object when retrieval fails to return it, even if selection itself was correct.Selection, identity handoff, and reader use are distinct failure points requiring different fixes.
- Motivation: Dataset-trace recall checks G, while selected-object return checks whether the recorded selection U reaches the reader.When U and G differ, the metric must state whether it measures benchmark coverage or delivery of the selected object.
- Evaluation: On 1,463 aligned records, exact ID lookup and exact decoded-title equality return every selected object, while ranked rules omit some at cutoff five.Both ranked rules receive the same decoded selected title, isolating the handoff rule's effect on membership.
- Evaluation: U and G differ on 329 resolvable records, and original-question rankings assign different top-five labels to 106 of 1,792 records.The two membership tests therefore describe different targets within the same ranking.
- Evaluation: 389/1,463 selected objects are missing with body-only BM25, compared with 14/1,463 for the hybrid reranker.The omission counts are measured with the selected ID held fixed across handoff rules.
- Reader analysis: The aligned object's presence is associated with 28.6 to 31.0 points higher exact match, while removing it lowers exact match in a selected 64-item cohort.Removing a similar-length nontraced passage does not reproduce the loss, although the recorded-context comparison is associative.
2 Measuring an Identity Handoff
ROP defines an identity-handoff audit around the target, returned IDs, cutoff, resolver, membership rule, and expected population. It separates membership failures from unverifiable or incomplete runs so retrieval and accounting problems remain distinguishable.
- Operational definition: Selected-object return succeeds when a returned ID within cutoff k resolves to U or a declared acceptable alternative.Dataset-trace recall uses the same ranking and membership rule but targets G instead.
- Operational definition: ROP records whether a selected-return requirement applies and isolates whether the declared target appears in returned IDs.The same fields can describe interfaces without assigning a hard pass or failure.
- Declaration: The declaration records five evaluation decisions, including the target, returned field, cutoff, resolver, and membership rule.These fields make the boundary explicit for developers evaluating identity-preserving lookup or ranked evidence construction.
- Population accounting: ROP lists every expected record, distinguishes missing or malformed observations, and reports run status separately from record verdicts.A surviving-row pass fraction can remain 1.0 even when an expected row disappears.
- Verdicts: A usable returned list that omits the target is a membership failure, whereas a missing or malformed list is UNVERIFIABLE.Missing expected records make the run incomplete; duplicate or unexpected IDs invalidate its observed population.
3 HybridQA Identity-Handoff Audit
The audit fixes the selected object and evaluates whether different handoff rules return it to the reader. It also separates selected-object return from dataset-trace recall when the two identities differ.
- Handoff rules: Exact key lookup and exact decoded-title equality returned the selected object on all 1,463 aligned records.These deterministic baselines separate identity resolution from ranked retrieval.
- Reading the misses: The final reranker’s 14 misses split into six near-cutoff cases and eight absent from both released top-20 lists.Changing the cutoff could change the near-cutoff handoff result, whereas the stored rankings cannot distinguish the cause of the eight deeper misses.
- Evaluation target: Selected and dataset-traced identities differed on 329 of 1,792 resolvable records, and original-question BM25 produced different top-five labels on 106 records, or 5.9%.The same frozen rankings were scored against U and G; query construction changed the magnitude and balance of disagreement.
4 From Returned Evidence to Answers
The reader analysis follows the shared object from retrieval into answering. Its associative comparisons and preselected deletion study both examine whether the aligned passage is present in the reader’s context.
- Reader comparisons: When the shared object was present, exact match was 28.6 to 31.0 points higher across three frozen reader-prompt comparisons.The paired contexts also differed in passage identities, ranks, lengths, distractors, and input length, so the comparison is associative.
- Deletion study: The deletion study preselected 64 U = G items that were exact with the shared passage and inexact without it.The fixed cohort was chosen from earlier outputs before the fresh calls.
- Deletion study: Removing the aligned passage lowered exact match by 77.6 points, while sham-drop exceeded trace-drop by 81.4 points.The intact-versus-sham interval included zero, and the sham passage was selected by rendered length without containing the normalized reference answer.
- Deletion study: Majority exact was 63 of 64 for intact, 57 of 59 for sham-drop, and 9 of 64 for trace-drop.Answer-token F1 showed corresponding gaps of 48.5 and 52.1 points.
5 Related Work
Related work covers retrieval and generation evaluation, changing retrieval inputs, identity-bearing grounding systems, and provenance or contract-based pipeline validation. This audit adds an explicit identity handoff boundary before the reader.
- Grounding and retrieval evaluation: RAG, RAGChecker, MIRAGE, CF-RAG, and AgenticRAGTracer study retrieved evidence and its relationship to generation or answer tokens.The audit follows a named object across one component boundary before the reader sees it.
- Changing retrieval inputs: ReAct, IRCoT, STEP, Iter-RetGen, query rewriting, HyDE, and RaFe vary the inputs used for later retrieval.The paper reports query construction alongside the profile because it determines how directly retrieval is evaluated.
- Identity-bearing systems: Entity linking, graph and entity retrieval, and unified data-to-text interfaces pass identities or structured knowledge into retrievable evidence.HybridQA supplies table-cell links that support the paper’s identity-handoff audit.
- Provenance and contracts: PROV, OpenTelemetry conventions, Design by Contract, OCL, and typed validation systems provide precedents for recording and checking pipeline state.The paper applies this contract-oriented perspective to whether a declared target appears in returned IDs.
6 Discussion
The discussion separates identity handoff from benchmark coverage and reader behavior, showing that retrieval rules, query construction, and complete-run accounting determine what can be concluded. ROP makes the selected target and evaluation boundary explicit.
- Pipeline diagnosis: A complete pipeline evaluation assigns separate repair targets to selector fit, selected-return misses, evidence quality, and reader behavior.A wrong answer after both identity checks pass leaves evidence quality and reader behavior to inspect.
- Identity and evaluation boundaries: Selected-object return isolates whether the recorded choice reaches ranked reader evidence, while dataset-trace recall measures coverage of the benchmark-linked object.These targets can describe different components when the selected and dataset-linked identities differ.
- Pipeline diagnosis: ROP can reserve one reader slot for the selected object and fill remaining slots with ranked retrieval, while relevance and answer quality remain separate evaluations.Lookup preserves identity; ranking adds neighboring passages or chunks.
- Identity and evaluation boundaries: A handoff evaluation should report the target, query construction, returned-ID field, and cutoff because these choices change which identity is tested.Target disagreement rises from 106 records with the original question to 213 with the decoded title.
- Complete-run auditing: ROP marks incomplete runs as unverifiable by accounting for every expected record, preventing surviving-row pass fractions from masking missing, duplicate, or malformed observations.A two-row run with one missing row can otherwise retain a pass fraction of 1.0.
7 Conclusion
The conclusion finds that a selected object can be lost between selection and reader evidence, and that benchmark coverage and selected-object delivery are distinct checks. ROP makes this boundary explicit and replayable.
- Conclusion: On 1,463 aligned HybridQA records, exact key dereference and exact decoded-title equality return every selected object, whereas ranked rules omit 14 to 389 at cutoff five.With the selected ID fixed, the handoff rule determines whether that identity appears in the evidence list.
- Conclusion: 106 of 1,792 resolvable records, or 5.9%, receive different top-five labels when selected-object return and dataset-trace recall are checked on identical original-question BM25 rankings.The two checks therefore describe different targets within the same ranking.
- Conclusion: ROP binds the target, returned field, cutoff, resolver, and expected population, assigns record-level verdicts and a complete-run status, and supports offline replay.It makes the chosen rule and denominator explicit.
Limitations
The study is bounded by its corpus, recorded selector outcomes, unannotated identity mismatches, and deliberately selected deletion cohort. ROP v1 also relies on supplied hash-bound records and external controls for provenance.
- Scope and audit limits: ROP v1 checks endpoint membership and byte consistency in supplied, hash-bound records but does not establish when declarations were created or who produced logs.Those provenance checks require external controls.
- Scope and audit limits: The empirical study covers one English Wikipedia table-text corpus and records one selector outcome per model-question pair, so it does not estimate selector stability.Repeated-sample variation is outside this design.
- Scope and audit limits: The 329 exact-ID mismatches lack semantic annotations, and the deletion result applies only to a deliberately selected 64-item cohort.Selector quality, evidence validity for both identities, and average answer benefit require separate evaluations.
Ethical Considerations
ROP makes handoff evaluation auditable by fixing the target, population, resolver, membership rule, and run-status logic. The study uses public HybridQA data, reports no new personal-data collection, and separates diagnostic outcomes from ethical claims about generated answers.
- The study uses public, Wikipedia-derived HybridQA data and collects no new personal data or human annotations.
- Generated answers are not treated as verified facts or evaluated for social bias, while invalid outputs and item-level traces remain available for inspection.
- ROP declarations fix the target interface, policy, producer, revision, mode, and expected population before evaluation.
- ROP v1 verifies file integrity with SHA-256 but does not authenticate producers or independently establish event timing.
- The evaluator resolves identities, applies the declared cutoff and membership rule, and records verdicts for observed and missing records.
- Hard-mode runs require complete expected populations, valid files and hashes, and successful return checks for every expected record.
B Reproducibility Details
The reproducibility protocol fixes cohort selection, prompts, answer conditions, retrieval settings, and hashes before outcomes are collected. Released files preserve selection metadata, frozen inputs, and execution links for offline reconstruction.
- The study records cohort-selection salt, source hash, item-selection hashes, and final cohort hash for reproducibility.
- A hash-locked 120-question protocol-development set is excluded from later development cohorts, whose settings are fixed before model outcomes.
- The frozen selector instruction requires exactly one explicitly shown table-linked Wikipedia page and returns only the required JSON object.
- The selected link is stored as an exact /wiki/... key, while retrieval_fact is a concise search name for the same entity.
- Synthetic hard-mode fixtures exercise all seven record verdicts and five run statuses, and released hash-bound inputs connect analyses to code and outputs.
B.2 Retrieval results across models and cutoffs
Retrieval performance depends on query construction, stack, and cutoff: decoded selected-ID queries generally outperform original-question queries, while reranking improves selected-object return. The audit preserves complete rankings and answer-matching criteria for reproducibility.
- Exact title equality returns every object across the corpus, whereas title-only BM25 omits 7,667 objects at cutoff five.
- Across 1,792 resolvable selections, the final reranker returns U on 1,775 records and G on 1,497, with 278 differing labels.
- The final reranker increases selected-object return by 22.5 points over body-only BM25 in the selected-name stress test.
- Decoded selected-ID queries outperform original-question queries across retrieval stacks at both reported cutoffs.
- Answer comparisons include only complete matched-five rows with identical protocol fields, normalized exact-match scores, and question-cluster assignments.
B.4 Open-weight Qwen check
The open-weight check tests whether the selected key survives retrieval when Qwen3-32B-AWQ supplies the upstream selection and retrieval fact. Its results use fixed decoding settings and explicit handling of an unresolvable selected link.
- The open-weight check uses Qwen3-32B-AWQ with fixed reasoning, sampling, token-cap, penalty, and seed settings.
- One parser-valid selected link falls outside the displayed candidates and remains explicit in the audit.
- The open-weight table distinguishes final-hybrid retrieval using Qwen’s fact from body-BM25 retrieval using the decoded selected ID.
- At cutoff five, the joint outcomes for 334 resolvable records are (P1H1, P1H0, P0H1, P0H0) = (232, 98, 0, 4).