Source-linked AI summary
Learning to Route in Visual Space via Multi-Step Embedding Retrieval
Tianyu Chen, Mingyuan Zhou, Jiaxing Wu
TL;DR
Visual agentic search is bottlenecked by single-step retrieval when clues are hard to express in text or needed intermediate images are not returned. VHop provides a controllable benchmark, and VHop-Router learns multi-step retrieval in visual embedding space. The approach improves retrieval and agentic-search performance while reducing token use, without changing the agent.
Problem
Single-step retrievers limit visual agentic search when clues are difficult to express in text or needed intermediate evidence is not among the returned results.
Source: §1 · p. 1Method
VHop supplies controllable, verified multi-hop retrieval data, while VHop-Router trains an embedding model to retrieve linked image chains autoregressively in one tool call.
Source: §1 · p. 2Results
VHop-Router improves retrieval-only and agentic-search performance while maintaining efficiency; agentic-search gains are 52.7% versus 3.7% from upgrading the agent.
Source: §1 · pp. 2–3Takeaways & Limitations
The retrieval tool can handle multi-hop search while leaving LLM agents untouched and preserving their native capabilities.
Source: §1 · pp. 2–3Scaling data generation remains challenging because it relies on explicitly defined relationships, and the policy struggles beyond hop counts seen during training.
Source: §7 · p. 11
Abstract
from arXiv · showhide
LLM agents rely on retrieval tools to access external knowledge, yet visual agentic search remains severely bottlenecked by standard single-step retrievers. In current pipelines, the agent must issue text queries for every intermediate step, struggling when visual clues are difficult to describe or when the retriever fails to surface necessary intermediate evidence within its top results. We hypothesize that offloading multi-step navigation across the entire embedding space directly to the retrieval tool resolves this performance bottleneck. To study this systematically, we introduce VHOP, a flexible data generation framework and benchmark with five core difficulty levels testing both visual matching and search planning. Using this framework, we develop VHOP-Router, an end-to-end training pipeline---combining supervised fine-tuning, online imitation learning, and reinforcement learning---that transforms a standard embedding model into an autoregressive multi-step retriever. Operating directly in the visual latent space, VHOP-Router retrieves linked image chains in a single tool call without requiring the agent to formulate intermediate text queries. Experiments show VHOP-Router boosts retrieval performance from under 5\% to 76.3\%. In agentic search, it improves task success rates by 52.7\% and reduces the average token length by 61\% from 1886 to 728, whereas upgrading the agent yields only a 3.7\% gain. Compared to a strong baseline where the agent retrieves the top 50 results per step, VHOP-Router maintains superior performance while reducing in-context images by $23\times$ and cutting the cumulative API payload by $35\times$. The models also generalize robustly to unseen difficulty levels and realistic test sets. Ultimately, VHOP and VHOP-Router provide an efficient and effective solution for visual agentic search that leaves native LLM capabilities entirely intact.
1. Introduction
Visual agentic search is limited by single-step retrieval: visual clues may be hard to express in text, and needed intermediate evidence may not appear among returned results. The paper introduces VHop and VHop-Router to move multi-step search into visual embedding space, reporting larger task-success gains from upgrading the retrieval tool than from upgrading the agent.
1. Introduction
- 52.7% increase from replacing Qwen3 with VHop-Router exceeds the 3.7% gain from upgrading the agent model.The comparison holds the other component fixed; Figure 1 summarizes task performance and upgrade gains.Source: Figure 1 · p. 1, §1 · p. 2
- VHop provides controllable visual multi-hop retrieval data with configurable hop counts, distractors, and difficulty levels, plus structured verification of ground-truth answers.Verifiers check that precise instructions yield valid labels and include all valid answers.Source: §1 · p. 2
- VHop-Router transforms a single-step embedding model into an autoregressive multi-step retriever trained with supervised fine-tuning, online imitation learning, and reinforcement learning.It returns linked image chains in one tool call without intermediate agent text queries.Source: §1 · p. 2
- The approach leaves LLM agents untouched, preserving their native capabilities while shifting multi-hop search to the retrieval tool.The framework is compatible with any LLM-based agent.Source: §1 · pp. 2–3
- Experiments report performance gains in retrieval-only and agentic search with improved efficiency, and generalization to unseen tasks and realistic test sets.The introduction states that generalization spans varying difficulty levels and hop counts.Source: §1 · p. 3
2. Related Work
VHop-Router differs from prior visual search and QA work by learning multi-step visual retrieval, including recovery from dead ends, rather than relying on text reasoning or searching for a named object.
2. Related Work
- VHop controls retrieval paths across images, including hop counts and dead ends, and provides verified answers, unlike benchmarks focused on scene reasoning or text corpora.Related work contrasts its visual matching and recovery from dead ends with multi-step text search, LLM-guided retrieval, object search, and visual-evidence QA.Source: §2 · p. 3
3. VHop: Controllable Multi-Hop Visual Search in Latent Space
VHop formalizes multi-hop visual search as retrieving an image sequence in which each intermediate image supplies the clue for the next step. Its controllable difficulty levels and verified data construction support evaluation of both visual matching and search planning.
3. VHop: Controllable Multi-Hop Visual Search in Latent Space
- A solver retrieves an image sequence from a starting image and text instruction, using each intermediate image to determine the next search and treating the final image as the answer.Precise prompts specify color and spatial constraints without naming objects; vague prompts omit those constraints and step-by-step instructions.Source: §3 · pp. 3–4
- The overview figure contrasts an L4 search with dead ends and backtracking against a complete L1 chain and image comparisons for L2, L3, and L5.It also places VHop-Router’s training stages beneath the examples.Source: Figure 3 · p. 4
- VHop’s five difficulty levels add constraints or distractors while retaining earlier requirements; L4 introduces dead ends that require exploration and recovery.Hop count, corpus size, and distractor count are configurable, and matching can extend to logos and printed text.Source: §3 · p. 4
- VHop constructs image collections symbolically, then renders and checks them for consistency in object identities, colors, and spatial relations.An independent verifier checks for exactly one valid sequence under precise instructions and records all valid answers for vague prompts.Source: §3 · p. 4
4. VHop-Router: Training Framework
VHop-Router turns an embedding model into an autoregressive visual retriever trained to follow image chains, recover from dead ends, and stop at an answer. Its training and evaluation use separate generated worlds, and its three-stage pipeline progresses from supervised retrieval to imitation and verifiable-reward learning.
4. VHop-Router: Training Framework
- VHop-Router trains an embedding model to retrieve linked image chains autoregressively, returning them in one tool call without intermediate agent text queries.Its three stages are supervised fine-tuning (SFT), online imitation learning (Online IL), and reinforcement learning with verifiable rewards (RLVR).Source: §4 · pp. 4–5
4.1. Experimental Setup
- Training uses L4 data, while evaluation uses disjoint worlds and tests transfer across L1–L5 without further fine-tuning.The evaluation includes three L4 worlds with 100 queries each and approximately 100 three-hop queries at each other level.Source: §4.1 · p. 5
- Baselines include retriever-only methods and agentic search, with success rate and per-query token, image-input, and API-call counts measuring performance and efficiency.Agentic comparisons hold Gemini 3.5 Flash fixed when replacing its retriever with VHop-Router.Source: §4.1 · pp. 5–6
4.2. Autoregressive Retrieval Policy
- At each decision, the policy encodes the instruction and active image stack, scores candidate images, and uses learned gates to select, backtrack, or stop.Image candidates are scored by cosine similarity, with selection from the top-K candidates; the active stack retains the query image.Source: §4.2 · p. 6
4.3. Training Pipeline
- The training stages address sequential retrieval in order: ranking a valid continuation, recovering from mistakes, and optimizing final-answer success.The stages are SFT, Online IL, and RLVR, respectively.Source: §4 · pp. 4–5, §4.3 · p. 6
4.3.1. Supervised Fine-Tuning
- SFT aligns the encoded instruction, query image, and gold trajectory prefix with the next gold image using an InfoNCE objective.Training first updates the state encoder with the action encoder frozen, then optimizes both encoders.Source: §4.3.1 · p. 7
4.3.2. Online Imitation Learning for Recovery and Stopping
- Online IL collects fresh policy rollouts and uses oracle actions on visited states to train recovery and stopping behavior.The action encoder is frozen while the state encoder and gates are updated using states from the current iteration.Source: §4.3.2 · p. 7
4.3.3. Outcome-Based Reinforcement Learning
- RLVR trains the state encoder and gates from sampled rollouts, rewarding a rollout only when the policy stops with a valid answer.The action encoder stays fixed, and the method uses a PPO-style clipped surrogate with DAPO-style asymmetric clipping.Source: §4.3.3 · p. 7
5. Results
Across five difficulty levels, VHop-Router improves retrieval and agentic search over baseline retrievers. It also outperforms wider single-step retrieval, transfers to shorter queries and added distractors, and shows gains on a realistic test set.
5.1. Significant Improvements Across Difficulty Levels
- 76.3% standalone success on L4 versus 3.7% for the strongest standard single-step retriever; with Gemini 3.5 Flash fixed, agentic success rises from 32.7% to 85.4% on L4 and from 28.0% to 84.0% on L5.Across all four training configurations, VHop-Router improves retrieval and agentic search across L1–L5.Source: Table 4 · p. 8, §5.1 · p. 8
5.2. Retriever Upgrades Yield 14x the Gain of Agent Upgrades
- 52.7 percentage points on L4 and 56.0 on L5 are the gains from upgrading the tool, compared with 3.7 and 6.0 points from upgrading the agent.The comparisons keep the other component fixed in each case.Source: Table 5 · p. 8, §5.2 · p. 8
5.3. Can Retrieving More Images Match a Learned Multi-Step Retriever?
- 85.4% success with VHop-Router exceeds both single-step retrievers at k=50, while requiring 23× fewer in-context images and a 35× smaller cumulative API payload.Flash with VHop-Router averages 2.1 calls, 12 returned images, and 33 cumulative image inputs; Qwen3 at k=50 averages 275 in-context images and 1,163 cumulative image inputs.Source: Figure 4 · p. 9, §5.3 · p. 9, Table 6 · p. 9
5.4. Generalization on One- and Two-Hop Queries
- 99.7% success on one-hop L4 queries and 90.0% on two-hop L4 queries show that the mixed-hop policy handles both without further fine-tuning.Source: §5.4 · p. 9
5.5. Robustness to Additional Distractor Objects
- 52.2% success with an added unseen distractor remains 24.8 percentage points above the strongest single-step agentic baseline at 27.4%.The added third foreground object increases retrieval difficulty without changing the answers; standard hop images contain two foreground objects.Source: §5.5 · p. 9, Table 7 · p. 10, Figure 5 · p. 10
5.6. Generalization to Realistic Queries and Settings
- 36.0% of realistic queries return the correct answer with precise three-hop training and 34.0% with precise mixed-hop training, versus 22.0% for GE2 and 26.0% for Qwen3.The test set comprises 50 questions and 386 images, with no additional training for these evaluations.Source: Table 8 · p. 10, §5.6 · p. 10
6. Analysis
A modest length penalty shortens RLVR rollouts while preserving L4 accuracy, whereas a stronger penalty degrades performance. Trajectory examples illustrate the distinct roles of SFT, Online IL, RLVR, and the agent.
6.1. Controlling Rollout Length in RLVR
- λ=0.02 shortens RLVR rollouts while achieving L4 accuracy comparable to the unpenalized objective; λ=0.05 shortens them further but degrades performance.Increasing λ consistently shortens rollouts; the tested terminal reward is r_λ = r − λ|ρ|.Source: §6.1 · p. 11, Figure 7 · p. 11
6.2. Trajectory Pattern Analysis
- SFT improves first-hop visual matching, Online IL learns recovery and stopping, and RLVR refines decisions through final-answer rewards to recover from an initial mistake and complete the chain.The agent can select an answer from a returned chain or re-query when the answer is missing.Source: §6.2 · p. 11
7. Conclusion and Limitations
The paper concludes that VHop and VHop-Router shift multi-hop visual search to an autoregressive retriever, boosting retrieval and agentic-search success while reducing context and API payloads. It identifies scaling to more objects and generalizing beyond training hop counts as challenges.
7. Conclusion and Limitations
- VHop-Router boosts retrieval and agentic-search success while reducing LLM context sizes and API payloads, preserving native LLM capabilities.Scaling VHop to more objects remains challenging because it relies on explicitly defined relationships, and the trained policy struggles beyond hop counts seen during training.Source: §7 · p. 11
Reproducibility statement
- The stdlib-only generator and verifier are deterministic from stored seeds and θ; the paper points to appendices for implementation and training details.Appendix B covers rendering checks, data splits, and prompts; Appendix D gives training procedures and hyperparameters.Source: Reproducibility statement · p. 11
AI use statement
- The authors report using AI tools for manuscript and figure work, reference checking, benchmark-image rendering, and retrieval-baseline captions.They state that the authors take responsibility for the final content and experimental claims.Source: AI use statement · pp. 11–12
A. Additional Related Work
- VHOP studies image-linked visual search, where intermediate images supply clues for the next retrieval and the final answer is itself an image.Unlike object-search tasks, the starting image may not show the target, so intermediate images must guide what to retrieve next.Source: A. Additional Related Work · p. 14
- VHOP makes task structure controllable across separate images, with explicit dependencies and alternative branches for evaluation.Its guarantees are conditional on faithful rendering; additional retrieval bounds also depend on solver restrictions and sampling assumptions.Source: A. Additional Related Work · p. 14
- The related-work discussion distinguishes retrieval policies that learn selection, backtracking, and stopping on image sequences from language-guided search and multi-step text retrieval.The policy can also serve as a tool for an agent that revises queries or selects an answer from returned images.Source: A. Additional Related Work · pp. 14–15
B. Benchmark Generation and Verification
VHop separates symbolic task construction and verification from image rendering, enabling controlled visual multi-hop retrieval tasks. Its worlds vary matching rules and distractors, while prompts, data splits, and rendering conditions define how tasks are evaluated.
B. Benchmark Generation and Verification
- VHop constructs object links and question constraints symbolically, verifies the search problem over the corpus, then renders images to vary visual appearance and difficulty.The separation provides exact task labels, conditional on faithful rendering.Source: B. Benchmark Generation and Verification · p. 15, §B.3 · p. 17
B.1. Difficulty Levels and World Construction
- L1–L5 cumulatively add requirements, while the generator controls hop count, matching rules, relations, colors, families, distractors, and forks.Distractors include constraint-violating decoys, dead ends, look-alike instances, and extra portraits; the corpus is available from the start.Source: Table 9 · p. 15, §B.1 · p. 15, Figure 8 · p. 16
B.2. Prompt Variants and Data Separation
- Precise prompts specify matching rules, relations, and filters without naming intermediate values; vague prompts may allow multiple verifier-enumerated endpoints.Generation verifies satisfying trajectories over pooled images and rejects families that fail checks; one- and two-hop questions are verified suffixes of three-hop chains.Source: §B.2 · pp. 15–16
B.3. Rendering Conditions
- Training and evaluation use separate worlds and appearance and mark pools; rendering prompts specify objects, attributes, and spatial arrangements.An image audit checks identity, instance distinctions, colors, marks, and relations against the manifest; the verifier alone does not certify rendering fidelity or rule out every captioning shortcut.Source: §B.2 · p. 16, §B.3 · p. 17
C. Symbolic Guarantees
The symbolic guarantees establish when VHop queries have unique trajectories and bound certain retrieval strategies under explicit assumptions. These statements concern symbolic worlds or restricted retrieval settings, not unrestricted search or pixel-level fidelity.
C. Symbolic Guarantees
- Under fresh link keys, unique continuations, unsatisfiable fork alternatives, and a unique eligible terminal portrait, a precise query has exactly one trajectory in the pooled corpus.The verifier checks these conditions across the corpus, including decoys and appearance twins.Source: C. Symbolic Guarantees · p. 17
- Reusing appearance templates preserves symbolic uniqueness when link keys remain fresh and Theorem 1’s conditions hold.For identity links, distinct instances must still be distinguishable in rendered images.Source: C. Symbolic Guarantees · pp. 17–18
- Under conditional symmetry, a pairwise scorer’s top-1 answer accuracy is at most 1/(m + 1) when at least m + 1 portraits share query-specified properties and a scene hop precedes the answer.The scorer is restricted to the query and candidate image, without intermediate evidence, corpus features, or role-revealing metadata.Source: C. Symbolic Guarantees · p. 18
- With at most H image retrievals for a complete H-hop chain, the success probability is bounded by the product of inverse branch counts under the stated local-information and fork assumptions.The argument does not require unconditional independence between forks; the proposition does not bound larger-budget recovery or finding the answer anywhere in a returned set.Source: C. Symbolic Guarantees · p. 18
D.1. Data, Models, and Evaluation
The paper trains and evaluates visual multi-hop retrievers using verified data, staged learning, and outcome-based reinforcement learning. The appendices detail training procedures and one- and two-hop evaluation settings.
D.1. Data, Models, and Evaluation
- Training and evaluation use L4 data with Qwen3-VL-Embedding-2B; SFT and Online IL use gold trajectories, while RLVR uses separate practice worlds and a final-answer verifier.The seed world contains 978 questions and 4,492 corpus images; the precise three-hop run produces checkpoint rloo10w10_200.Source: §D.1 · p. 18
D.2. Supervised Fine-Tuning
- SFT first trains the state encoder on gold prefixes with full-corpus InfoNCE, then adds hop-level hard negatives and updates the action encoder.Later stages freeze the action encoder and its corpus index.Source: §D.2 · p. 19
D.3. Online Imitation Learning
- Online IL collects rollouts with oracle labels at every visited state, mixing oracle actions, policy actions, and forced dead-end excursions.Training uses only states collected in the current iteration; records retain the stack, candidate mask, parent score, and oracle action.Source: §D.3 · p. 19
- Online IL averages per-state losses over collected data, while selection gradients update the state encoder and gate gradients update the gates and state encoder.The action encoder and corpus index remain frozen; evaluation depth floors constrain stopping separately from the learned stop decision.Source: §D.3 · p. 20
D.4. Outcome-Based RL Details
- RLVR samples complete rollouts, scores them with a terminal answer verifier, computes leave-one-out advantages, then updates the state encoder and gates using clipped branch terms and KL penalties.Groups with identical rewards have zero advantage and provide no policy-gradient signal.Source: §D.4 · pp. 21, 23
- The RL surrogate omits complementary gate factors, making it generally biased relative to the full policy gradient; behavior and reference KL penalties constrain policy updates.The 𝜆=0.02 penalty run also changes gradient microbatching and the stack limit, a difference to consider alongside the reward change.Source: §D.4 · pp. 21–23
E.1. One- and Two-Hop Queries
- Tables 11 and 12 report one- and two-hop success rates under the stated corpora, checkpoints, and scoring conventions.Their captions specify L4 test worlds and development-world evaluation for other levels; the supplied passages do not report the table values.Source: Table 11 · p. 24, Table 12 · p. 25
E.2. Agent Model and Retrieval Tool
On the 300-query L4 test set, replacing Qwen3 with VHop-Router improves Flash-agent success more than upgrading Flash, while using fewer images than a wide-retrieval baseline. The section also reports decision-penalty and distractor-scene evaluations.
E.2. Agent Model and Retrieval Tool
- 52.7 percentage points: replacing Qwen3 with mixed-hop VHop-Router raises Flash-agent L4 success from 32.7% to 85.4%, versus 3.7 points from upgrading Flash to Pro.With the mixed-hop tool, Flash and Pro reach 85.4% and 84.7%, respectively.Source: §E.2 · p. 26, Table 13 · p. 26
E.3. Number of Images Returned per Call
- 85.4%: precise mixed-hop VHop-Router exceeds GE2’s 81.3% L4 success at k=50 while returning fewer images.The image-occurrence measure sent to the model includes repeated conversation history and is distinct from retrieved-image count, latency, and monetary cost.Source: §E.3 · p. 26, Table 14 · p. 27
E.4. Decision Penalty
- At iteration 200, the λ=0 and λ=0.02 runs reach 48.0% and 47.7% on L4; vague-prompt mean accuracy rises from 26.0% to 32.3%, while L4 vague accuracy falls from 20.0% to 18.0%.Mean stop rate increases and mean backtracks decrease; each setting uses one run, and the λ=0.02 run also changes gradient-memory settings.Source: §E.4 · p. 28, Figure 12 · p. 28, Table 15 · p. 29
F.1. Three-Object Scenes
- With an added distractor per hop scene, precise three-hop and mixed-hop policies fall from 48.0% to 30.4% and from 76.3% to 52.2%, respectively.The results show sensitivity to competing visual content but do not distinguish object-recognition failures from search-control failures.Source: §F.1 · p. 29
F.2. Four-Hop Extrapolation
These evaluations test transfer beyond the main L4 setting, including longer searches, human-authored queries, and logo or text matching. Results are weak on four-hop and logo/text tasks, while realistic-query results support only a limited transfer claim.
F.2. Four-Hop Extrapolation
- At most 8.2%: all four policies remain at or below this accuracy on four-hop queries, and removing the depth floor does not resolve the failure.The test also increases corpus size and distractor layers, so it tests longer, harder searches rather than isolating hop count; four-hop training recovery is not established.Source: §F.2 · p. 30, Table 16 · p. 30
F.3. Realistic Queries and Settings
- On 50 human-authored queries using rendered images, precise VHop-Router tools reach 36.0% and 34.0% agent set success, versus 22.0% and 26.0% for GE2 and Qwen3.Answer accuracies are 20.0%, 20.0%, 12.0%, and 8.0%, respectively; the results support limited transfer, not reliable performance across arbitrary photo collections.Source: §F.3 · pp. 30–31, Table 17 · p. 30
F.4. Extensions to Logo and Text Matching
- With precise mixed-hop training, standalone VHop-Router reaches 5.3% on L6 and 2.1% on L7; Flash with the same tool reaches 10.6% and 5.3%.Training used L4 identity links, so logo and printed-text matching rules were absent from post-training; the results do not isolate perception, rule application, or search-control errors.Source: §F.4 · pp. 31–32, Table 18 · p. 32
G. Trajectory Case Studies
- The trajectory case study compares precise three-hop L4 checkpoints on a query whose gold chain is 00076→00079→00352.The solver must find Q to the right of yellow object A, then A to the left of green object B, and finally B alone.Source: §G.1 · p. 33, G. Trajectory Case Studies · p. 33
A RL recovers from a wrong branch
The trajectory examples contrast learned recovery with fixed retrieval, and show how an agent can retry a chain policy or choose an answer already in its output. On a vague query with two valid endpoints, RLVR stops at an acceptable alternative while Online IL exhausts its budget.
A RL recovers from a wrong branch
- After Online IL and RLVR make the same incorrect first selection, RLVR backtracks and retrieves the three gold images, while Online IL stays on the wrong branch.The figure records RLVR backtracking with pbt=0.636 and stopping with pstop=0.999; Online IL stops on a wrong answer.Source: Figure 13 · p. 33
- Untrained retrieval misses the first hop, while SFT retrieves it before leaving the gold chain; both show five-step trajectories without learned recovery.Flash with GE2 retrieves the first gold hop but not the second hop or answer, whereas Flash with Qwen3 retrieves none of the three gold images.Source: Figure 14 · p. 34, Figure 15 · p. 34, Figure 16 · p. 35
G.2. Revising the Query and Selecting an Answer
- An agent can retry the policy with revised text when a chain lacks the answer, or select the answer directly when it is already in the returned chain.On one example, a retry retrieves a valid chain and stops correctly; on another, Flash selects the correct third image from a chain that continued searching.Source: §G.2 · p. 36, Figure 18 · p. 37, Figure 19 · p. 37
G.3. A Vague Query with Two Acceptable Answers
- With two verified endpoints under a vague instruction, RLVR backtracks once and stops at the alternative valid answer, while Online IL exhausts its 16-decision budget.The two chains share the first hop; the vague question omits the relation distinguishing them, and either endpoint is accepted.Source: §G.3 · pp. 37–38, Figure 20 · p. 38