PaperlayerView exact revision on arXiv

Source-linked AI summary

Navigation-Informed Embeddings: Dense-Retriever Adaptation from Agent Search Traces

Shrey Shah, Levent Ozgur

arXiv:2608.15956v1cs.AI

TL;DR

Dense retrievers often need labeled query-document pairs to adapt as agentic retrieval workflows and distributions change. NIE learns from retained navigation traces, improving target support Recall@20 from 72.2 to 78.0 overall.

  • Problem

    Adapting dense retrievers to evolving specialized workflows typically requires costly relevance judgments or curated query-document pairs, despite agent traces providing behavioral evidence.

  • Method

    NIE adapts dense retrievers from retained agent traces by treating stopping documents as soft positives and path documents as ordered hard comparisons.

  • Results

    Support Recall@20 improves from 72.2 to 78.0 overall, while NIE-Path raises long-path recall from 46.7 to 55.4.

  • Takeaways & Limitations

    NIE makes retriever adaptation trace-driven rather than annotation-driven when suitable agent traces are already retained.

  • Takeaways & Limitations

    NIE relies on retained traces preserving useful coverage under gradual distribution drift; systematic coverage gaps require broader retraining or workflow intervention.

Abstract

from arXiv · show

Agentic retrieval workflows produce query, retrieval, and stopping traces as a byproduct of answering questions. We study how these traces can adapt a deployed dense retriever to changing workflow distributions without new relevance labels, synthetic queries, or LLM judgments. We introduce Navigation-Informed Embeddings (NIE), a family of trace-derived objectives. NIE-Stop turns the stopping document into a soft positive; NIE-Path additionally uses preceding path documents as hard comparisons and imposes ordinal constraints with geometric decay. A BGE encoder adapted from retained source trajectories improves support Recall@20 on an independent target benchmark from 72.2 to 78.0 overall. NIE-Stop reaches 76.9 overall and 52.3 on long paths; NIE-Path raises long-path performance to 55.4, compared with 46.7 for the unadapted encoder. A shuffled-order control under the full path objective loses 3.2 points. Without public-benchmark training, the same adapter also improves nDCG@10 by 1.9 points on standard BEIR HotpotQA. NIE therefore provides a lightweight adaptation channel for settings where trajectories are already retained, with zero incremental labeling cost.

1. Introduction

Navigation-Informed Embeddings (NIE) adapts dense retrievers from retained agent search traces, using stopping events and ordered search paths as supervision without new labels or generation calls. It addresses evolving deployed workflows by preserving the structural information in multi-document navigation.

  • Motivation: Agent navigation traces encode stopping events and relevance gradients that independent document judgments would discard.Early documents are often broad or partially relevant, while later documents more directly address the information need.
  • Framework: NIE adapts dense retrievers from observed agent queries, ordered retrieved documents, and stopping points while reserving human labels for evaluation.The framework applies when agentic search trajectories are already retained.
  • Supervision: NIE-Stop uses stopping documents as contrastive positives, whereas NIE-Path adds nonterminal documents, chronology, hard comparisons, adjacent ordering loss, and geometric decay.Terminal documents provide noisy positive signal, while later path documents tend to be more relevant as agents refine searches.
  • Contribution: The proposed adaptation loop updates the encoder from newly retained agent search traces with zero incremental labeling cost.NIE-Stop avoids new labeling or generation calls, and evaluation uses a held-out target shift with source trajectories for training.

2. Related Work

NIE extends implicit-feedback learning from static result interactions to agent trajectories, using retrieved-but-insufficient documents and stopping behavior to adapt reusable dense encoders. Unlike synthetic-supervision approaches, it derives ordering-based training signals directly from observed navigation paths.

  • Learning from Implicit Feedback: Prior implicit-feedback methods address click, session, and document-order signals, while systematic presentation and position bias motivates counterfactual and propensity-weighted learning.These methods primarily model user interactions with result lists rather than agent retrieval trajectories.
  • Agent Trajectory Feedback for Dense Adaptation: NIE applies implicit feedback at the agent-trajectory level rather than to user interactions with static result lists.Earlier documents were explicitly retrieved into the agent’s context but were insufficient for stopping.
  • Agent Trajectory Feedback for Dense Adaptation: Trajectory supervision maps searches into terminal positives, path-local contrasts, and adjacent order constraints for dense representation learning.These signals form a soft relevance gradient along search paths.
  • LLM-Generated Synthetic Supervision for Retrieval: Unlike LLM-generated synthetic supervision, NIE extracts training signals from observed agent navigation behavior and induced path ordering.Synthetic approaches generate queries for documents and train retrievers on the resulting pairs or triples.
  • Dense Passage Retrieval: NIE is positioned against dense-retrieval and multi-hop methods spanning dual encoders, hard negatives, distillation, late interaction, iterative retrieval, and reasoning-chain following.The related methods include DPR, ColBERT, MDR, Baleen, and PathRetriever.

3. Method

NIE derives weak supervision directly from agent navigation traces, treating the stopping document as a soft positive and earlier path documents as ordered, lower-confidence comparisons rather than absolute negatives. NIE-Stop uses terminal-document contrastive training, while NIE-Path adds hard path comparisons and geometric ordering constraints, with human judgments reserved for evaluation.

  • Trace-derived supervision: NIE trains from observed queries, ordered retrieval paths, stopping documents, and random corpus negatives without explicit relevance labels.Human, LLM-as-judge, and answer-derived labels are reserved for evaluation and auditing.
  • Trace-derived supervision: The terminal document d_T serves as a soft positive because the observed stopping event indicates it was more useful than earlier retrieved documents.NIE does not require a calibrated stopping score.
  • Trace-derived supervision: Earlier path documents provide lower-ranked, challenging comparisons, while the path encodes a preference ordering rather than absolute relevance labels.Earlier documents may remain genuinely relevant, especially in multi-hop questions, so the ordering is interpreted as a behavioral preference within the episode.
  • Limitations: Trajectory-derived preferences depend on the generating agent’s behavior, motivating held-out human evaluation and behavioral-noise analysis.Missed evidence, narrow retrieval patterns, or premature stopping can be reflected in the supervision.
  • Objectives: NIE-Stop excludes nonterminal path documents, whereas NIE-Path adds earlier path documents as hard comparisons and, for long paths, an ordering loss.The ordering loss compares adjacent steps and uses geometric decay to scale the required margin.
  • Objectives: The full path objective combines contrastive and ordering losses, assigning weight 1 to the terminal document and geometrically decreasing weights to predecessors.Short and moderate paths use the augmented contrastive term, while long paths contribute both terms; λ=1 in all experiments.

4. Experimental Setup

The experiments test trajectory-based dense-retriever adaptation under source-to-target workflow shift, using retained source traces for training and independent target evidence for evaluation. They compare BGE-based NIE variants with reference encoders and assess transfer on HotpotQA.

  • Source navigation traces: 2,197 query-level training examples are extracted from 2,484 source training queries using ordered retrieved documents and the agent’s stopping document.Trajectories are stratified into short, moderate, and long paths based on the number of documents viewed.
  • Target workflow benchmark: The target benchmark uses independently annotated supporting evidence to measure retrieval quality after a shifted workflow distribution, without target queries, documents, or labels during NIE training or model selection.Source and target corpora were collected independently, with disjoint queries and document identifiers.
  • Training and evaluation protocol: NIE treats terminal documents as soft positives and earlier path documents as contrastive, order-aware training signals, while target path information is used only for evaluation stratification.All retrievers are evaluated on the same path-length buckets, and target terminal documents are excluded from training supervision.
  • Evaluation labels: The primary metric is support Recall@20 averaged across independently annotated target support sets and reported by observed path length.Target trajectories do not define relevance; curated answer/evidence records provide the ground-truth support sets.
  • Dense adaptation protocol: The protocol centers on BGE-base and compares its NIE-Stop and NIE-Path adaptations with off-the-shelf Contriever and E5-base reference encoders.NIE-Stop excludes nonterminal path documents; NIE-Path adds them as hard comparisons and incorporates path order.
  • External transfer: The already-trained adapter is additionally evaluated on HotpotQA using a global corpus of titled context paragraphs and identifier-based gold-passage matching.This evaluation measures transfer on independently constructed multi-hop data beyond the primary workflow benchmark.

5. Results · 5.1. Main Results · 5.2. Matched Trace-Adaptation Baselines

NIE improves target-domain support Recall@20 through trajectory-derived adaptation without target labels, with the largest gains on moderate- and long-path queries. Stop-based adaptation reaches 76.9 overall, while the full path objective reaches 78.0 and outperforms matched trace-derived controls.

  • 5.1. Main Results: Figure 1 reports target-domain support Recall@20 across short-, moderate-, and long-path query types.The main comparison is organized by query path length.
  • 5.1. Main Results: Across 542 target queries, support Recall@20 rises from 72.2 to 78.0, a 5.8-point gain over BGE-base.NIE gains 1.5, 7.7, and 8.7 points on short-, moderate-, and long-path queries, respectively.
  • 5.1. Main Results: NIE-Stop reaches 76.9 overall and improves long-path recall by 5.6 points using only stopping documents.NIE-Path reaches 78.0 overall and adds 3.1 points on long paths through nonterminal comparisons, adjacent ordering, and geometric decay.
  • 5.1. Main Results: The reported gains come from trajectory-derived adaptation using separate source trajectories and evaluation data, without target-label fine-tuning.The target benchmark and source trajectories are separate, establishing a no-target-label setting.
  • 5.1. Main Results: The largest improvements occur on moderate- and long-path queries, where support Recall@20 measures annotated evidence coverage across each query.This metric penalizes suppressing required multi-document evidence rather than rewarding retrieval of only one relevant document.
  • 5.2. Matched Trace-Adaptation Baselines: All matched baselines use the same BGE initializer, source-query budget, optimizer schedule, and validation split, without human or LLM judgments.The controls test generic adaptation against stopping- and path-structured objectives.
  • 5.2. Matched Trace-Adaptation Baselines: 51.9 long-path and 75.9 overall Recall@20 are achieved by retained tool-query pairs, while NIE-Stop reaches 52.3 and 76.9.Adding nonterminal comparisons without ordering reaches 52.8 and 77.1; NIE-Path improves over this no-order control by 2.6 long-path points and 0.9 overall points.
  • 5.2. Matched Trace-Adaptation Baselines: Figure 3 compares objective-component ablations, trajectory-order controls, and negative-source ablations on long-path queries with 4+ documents viewed.It reports that NIE-Path adds 3.1 support Recall@20 beyond terminal-only adaptation.

5.3. Ablation Study · 5.4. Effect of Path Length

The ablations show that terminal positives, path-document comparisons, adjacent ordering, and geometric weighting each contribute to long-path support Recall@20, with full NIE reaching 55.4. Improvements persist across path lengths, although Recall@20 declines as navigation paths become longer for both baseline and adapted model.

  • 5.3. Ablation Study: 55.4 long-path support Recall@20 is achieved by full NIE with geometric weighting, versus 52.8 for the matched no-order control.The full objective adds 2.6 points over the matched no-order objective.
  • 5.3. Ablation Study: 5.6 points is the long-path gain from BGE to NIE-Stop using terminal positives without nonterminal path documents.The estimated 95% CI is [2.1, 9.0].
  • 5.3. Ablation Study: 54.1 long-path support Recall@20 results from adding adjacent ordering with an unweighted margin to the matched no-order setup.The matched no-order control reaches 52.8 before adjacent ordering is added.
  • 5.3. Ablation Study: 3.2 points is the observed-order advantage over shuffled order under the full path objective.The 95% CI for this advantage is [0.9, 5.6].
  • 5.3. Ablation Study: 55.4 ± 0.4 is NIE-Path long-path support Recall@20 across three training seeds, compared with 52.8 ± 0.5 for the matched no-order control.Values are reported as mean ± standard deviation.
  • 5.4. Effect of Path Length: NIE provides consistent improvements across path lengths, with gains persisting across increasingly long searches.Figure 4 reports long-path support Recall@20 by trajectory length.
  • 5.4. Effect of Path Length: Support Recall@20 decreases as paths lengthen for both the baseline and NIE model, showing that very extended navigation sequences remain challenging.Performance varies with navigation-path length in the test set.

5.5. Robustness to Behavioral Noise · 5.6. Transfer to HotpotQA · 5.7. Standard Public-Benchmark Transfer

NIE remains effective despite noisy stopping behavior and transfers beyond the primary target benchmark to pooled and standard BEIR HotpotQA. Across evaluations, NIE-Stop supplies the larger gain, while NIE-Path adds further retrieval improvements.

  • 5.5. Robustness to Behavioral Noise: 26% of sampled terminal documents were wrong, yet NIE-Stop improved long-path support Recall@20 by 5.6 points and NIE-Path by 8.7 points over BGE.The audit used a three-of-four aggregation rule, and NIE did not filter sessions using audit judgments.
  • 5.5. Robustness to Behavioral Noise: 47.1 was the random-terminal control’s long-path support Recall@20, near BGE’s 46.7, indicating stopping documents carried signal beyond path membership.The control supports the behavioral-noise analysis without using audit judgments for filtering.
  • 5.6. Transfer to HotpotQA: 3.4 points was NIE’s HotpotQA gain for retrieving both supporting passages, while its gain for retrieving either passage was 3.2 points.The pooled experiment measured Recall@20 against two independently annotated gold supporting passages from the global corpus.
  • 5.6. Transfer to HotpotQA: 1.6 points was NIE-Path’s additional gain over NIE-Stop on the stricter HotpotQA “both” metric.NIE-Stop improved over BGE-base, and the full path objective added multi-document retrieval accuracy.
  • 5.7. Standard Public-Benchmark Transfer: The same frozen checkpoints were evaluated on standard BEIR HotpotQA using its released corpus, test queries, qrels, and evaluation script.This public-benchmark evaluation used a ranking-sensitive metric and was separate from the pooled Recall@20 protocol.
  • 5.7. Standard Public-Benchmark Transfer: 1.9 nDCG@10 points was NIE’s total gain over BGE-base on standard BEIR HotpotQA, with NIE-Path adding 0.6 points beyond NIE-Stop.NIE-Stop first improved nDCG@10 by 1.3 points; the paired query-bootstrap interval for the total gain was [0.8, 3.0] points.
  • 5.7. Standard Public-Benchmark Transfer: 1.5 Recall@100 points was NIE’s total gain over the initializer, following NIE-Stop’s 0.9-point improvement and NIE-Path’s further 0.6 points.These results mirrored the decomposition in which stopping-document adaptation contributed the larger increment.

5.8. Cost Analysis

NIE uses retained agent navigation traces as supervision, avoiding new relevance-labeling campaigns when trajectories already exist. Its reported adaptation is an offline source-to-target encoder update, while repeated trace-window updates are proposed but not evaluated online.

  • Data collection: NIE treats retained navigation traces as a supervision byproduct, shifting incremental costs toward storage, filtering, and training rather than new annotation campaigns.If trajectories are generated solely for training, their LLM inference cost becomes part of data generation.
  • Labeling: NIE requires zero incremental labeling cost when suitable trajectories already exist, using terminal documents as soft positives and earlier path documents as lower-ranked items or hard contrasts.Human labels remain useful for evaluation and auditing, but are not required to construct the training objective.
  • Training: The reported adapter trains for three epochs on 2,197 trajectory-derived examples without new LLM inference or relevance annotation.Its incremental inputs are retained traces and embedding-model optimization.
  • Distribution-shift adaptation: The experiment performs one offline encoder update across disjoint source and target datasets, while successive trace-window updates are described without a multi-round online evaluation.The reported results establish the source-to-target update that the repeated process would reuse.

6. Analysis

Across 555 filtered source trajectories, NIE shifts query-document geometry toward observed navigation order: correlations improve, terminal documents rank higher, and adjacent violations decline. Long paths add signal because earlier documents can provide partial or contextual evidence before terminal resolution.

  • Query-document geometry: 555 trajectories with at least four retained path documents were evaluated using Pearson correlation between document position and query similarity.The analysis covers every trajectory passing the diagnostic filters, not selected examples.
  • Query-document geometry: 60.9% of paths improve correlation after training, with mean correlation rising from 0.031 to 0.105 and terminal-document rank improving by 0.28 positions.The mean correlation shift is 0.074.
  • Query-document geometry: 4.8 points: the adjacent-pair violation rate falls from 49.5% to 44.7% after training.These diagnostics indicate movement toward the observed navigation order; human-labeled target evaluation determines retrieval impact.
  • Long-path patterns: Long-path queries often accumulate partial evidence, with earlier documents supplying background or missing pieces before the terminal document resolves the remaining fact.This pattern explains why path order can add signal beyond terminal positives.
  • Long-path patterns: Earlier path documents may also provide context needed to interpret the answer, motivating soft ordering supervision rather than treating nonterminal documents as irrelevant.Agents can progressively refine retrieval from broad results toward increasingly specific query aspects.

7. Limitations

NIE depends on retained, policy-compliant search traces and sufficient query volume, while its path-order prior can be wrong when terminal documents are incorrect or earlier documents contain essential evidence. Results on pooled HotpotQA are secondary diagnostics because duplicate identifiers affect absolute scores.

  • NIE requires retained agent search traces, compliant data practices, and—when generated for training—an inference-cost budget.The studied setting assumes trajectories are already available from the retrieval workflow.
  • 26% of audited terminal documents are wrong, limiting the reliability of the path-order prior despite improved supporting-evidence coverage.Later documents tend to be more useful, but the prior is explicitly soft because agents can stop incorrectly and early documents may contain necessary evidence.
  • The adjacent ordering loss accommodates useful earlier documents by imposing local rather than all-pairs preferences.Shuffled-order and reversed-order controls test whether chronology is more informative than arbitrary or inverted ordering.
  • Low-traffic and cold-start settings may lack enough query volume to expose meaningful navigation patterns and may need synthetic data or selective annotation.These complementary supervision sources can bootstrap learning when implicit supervision is insufficient.
  • Pooled HotpotQA is a secondary diagnostic because duplicate identifiers make absolute scores sensitive to duplicate competition and tie handling.The primary claim remains the source-to-target adaptation result on the workflow benchmark.

8. Conclusion

Navigation-Informed Embeddings adapts dense retrievers from retained agent search traces, with path-aware objectives improving support Recall@20, especially on long searches. The approach provides lightweight, annotation-free updates as workflow distributions evolve.

  • 8. Conclusion: NIE adapts dense retrievers using objectives derived from agent search traces.NIE-Stop uses the observed stopping document, while NIE-Path adds nonterminal comparisons and ordinal constraints for extended searches.
  • 8. Conclusion: 55.4 long-path support Recall@20 follows the complete path objective, versus 52.3 for terminal-only adaptation and 46.7 before adaptation.Together, the objectives improve target support Recall@20 from 72.2 to 78.0 overall.
  • 8. Conclusion: NIE makes retriever adaptation trace-driven rather than annotation-driven, with zero incremental labeling cost when suitable traces are retained.Each update requires only lightweight encoder training, and the construction can be applied to subsequent trace windows as workflow distributions evolve.
  • 8. Conclusion: Retained agent trajectories provide a practical supervision channel for low-cost dense-retriever adaptation, while ordered path structure adds signal on the hardest queries.The reported experiment evaluates one offline update, with later updates possible as workflow distributions change.
Loading 2608.15956v1…