Source-linked AI summary
Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents
Feiyu Gavin Zhu, Xiaoyu Zhu, Jiqi Yang, Rui Yang, Arnab Kumar Mondal, Yancheng Wang, Xinke Deng, Jean Oh, Reid Simmons, Joerg Liebelt, Xiang Kong, Zhongyu Jiang
TL;DR
Small multimodal agents can incur substantial costs generating lengthy freeform reasoning before each action, even when that reasoning provides little useful guidance. SSR selects among reusable natural-language reasoning candidates and scores them in parallel; across seven multimodal search benchmarks, it achieves competitive task performance with lower reasoning and total inference latency.
Problem
Small multimodal agents generate sequential freeform reasoning before each action, which can be lengthy, costly, and provide little useful action guidance.
Source: §1 · p. 1Method
SSR selects context-conditioned reusable natural-language reasoning candidates and scores them in parallel instead of generating freeform reasoning.
Source: §1 · p. 1, §3.1 · p. 3Results
Takeaways & Limitations
Across GRPO, GSPO, SAPO, and SFT, SSR maintains competitive performance while reducing reasoning and model inference latency.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Multimodal agents commonly generate free-form reasoning before each action. For small models, limited model capacity can result in lengthy reasoning that provides little useful guidance for action generation while incurring substantial inference cost. To address this challenge, we introduce Selection-based Structured Reasoning (SSR), a framework that reformulates reasoning as selection instead of open-ended generation. SSR represents recurring high-level reasoning as pre-specified, reusable natural-language candidates. At each turn, the model selects from these reasoning candidates based on their likelihoods given the current context, without requiring an auxiliary task head. Using pre-specified reasoning traces enables parallel scoring, where teacher-forced prefilling computes token likelihoods concurrently within and across candidates using a shared context KV cache. We evaluate SSR on seven multimodal search benchmarks using 2B and 4B models. Across multiple reinforcement learning objectives and supervised fine-tuning, SSR delivers significant efficiency gains without sacrificing task performance. SSR achieves an average success rate competitive with leading search agents of the same scale, while reducing per-turn reasoning latency by over 90% and total per-question model inference latency by 28-54%. Project page: https://zfy0314.github.io/ssr-webpage/.
1 INTRODUCTION
Small multimodal agents incur sequential reasoning costs that may not improve action guidance, even though high-level search rationales often recur across questions. SSR selects reusable reasoning candidates and scores them in parallel, achieving competitive task performance with lower reasoning and inference latency.
Motivation
- Small models may spend costly sequential reasoning tokens on lengthy deliberations that provide little useful guidance for action generation.The authors note that multimodal search often reuses high-level rationales even as entities, queries, and observations vary.Source: §1 · p. 1
Approach
- SSR replaces freeform reasoning with context-conditioned selection among reusable natural-language candidates, then generates instance-specific actions from the selected rationale.The action generation supplies details such as a question-specific search query without generating freeform reasoning traces.Source: §1 · pp. 1–2
Parallel scoring
- Teacher-forced prefill scores candidate tokens in parallel, and candidates share the context KV cache, avoiding autoregressive reasoning generation.The language model scores candidates by length-normalized likelihood without an additional classification head.Source: §1 · p. 2
Results
- 61.37% average success rate for 4B SSR with GRPO matches leading trained 4B agents at 61.25%, while reasoning latency per turn falls by over 90% and per-question model latency by 28–54%.These efficiency gains hold across four training objectives and two model sizes, with comparable or higher success rates.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK
Prior work improves multimodal search through agent training, efficiency controls, and reusable reasoning or skills. SSR takes a distinct approach: selecting natural-language reasoning candidates from the acting model’s context-conditioned likelihoods.
2.1 MULTIMODAL SEARCH AGENTS
- Multimodal search agents have expanded tool use and training approaches, while SSR explores inference efficiency as an orthogonal direction.Prior systems included image operations, code execution, and web search, alongside work on training mixtures, gradient routing, and credit assignment.Source: §2.1 · pp. 2–3
2.2 EFFICIENT REASONING FOR MLLM AGENTS
- Efficiency methods control deliberation or tool use, whereas SSR reduces the reasoning-span cost within each interaction round by scoring complete candidates in parallel.HyperEyes, for example, reduces interaction rounds by dispatching concurrent grounded queries and discouraging superfluous tool calls.Source: §2.2 · p. 3
2.3 REUSABLE REASONING AND AGENT SKILLS
- Unlike fixed trajectory strategies or executable skills, SSR selects natural-language guidance each turn from updated history, leaving queries and tool arguments flexible.It uses the acting model’s likelihoods without an auxiliary knowledge-management or retrieval system.Source: §2.3 · p. 3
3 SELECTION-BASED STRUCTURED REASONING
SSR replaces freeform reasoning with context-conditioned selection from reusable natural-language candidates. It scores candidates in parallel and trains selection and action generation under supervised or reinforcement-learning objectives.
3.1 OVERVIEW
- SSR replaces freeform reasoning in ReAct agents with selection among reusable reasoning candidates conditioned on the interaction history.The history includes the initial request and prior reasoning, actions, and tool observations; the selected rationale guides context-specific action generation.Source: §3.1 · p. 3
3.2 PARALLEL REASONING DECODING
- Teacher-forced prefill scores candidate tokens in parallel, while different candidates are scored concurrently using the same history KV cache.The selected candidate is inserted as reasoning text before generating an instance-specific action.Source: §3.1 · p. 3, §3.2 · p. 4
- SSR’s parallel-depth comparison removes the freeform trace-length factor L through token-level parallelism.The analysis assumes fixed model layers, token dimensions, and vocabulary size.Source: §3.2 · p. 5
3.3 SSR TRAINING
- Selection and action generation share model parameters and can be optimized jointly under supervised fine-tuning or reinforcement learning.The framework supports SFT and RL objectives without requiring a separate model component.Source: §3.3 · p. 5
- SFT combines categorical reasoning-candidate selection loss with token-level action loss, weighted by λSFT.The inputs include the context, ground-truth reasoning candidate, and ground-truth tool call or answer.Source: §3.3 · p. 5
- SSR can train directly with GRPO-style RL, replacing token-level importance ratios for generated reasoning with a categorical ratio for selection.The action tokens retain their usual importance ratios; GSPO and SAPO variants are also supported.Source: §3.3 · p. 5
4 EXPERIMENTS
Across seven multimodal search benchmarks, SSR attains competitive success rates while substantially reducing reasoning and model inference latency. Experiments compare training setups, reasoning methods, and library configurations.
4.1 SETUP
- Experiments use 2B and 4B Qwen3-VL models, seven evaluation benchmarks, and a shared six-candidate reasoning library; latency profiling samples 100 questions per benchmark.The tool harness includes reverse-image search, text search, and image cropping; profiling excludes environment-side processing.Source: §4.1 · p. 6
4.2 SSR ACHIEVES COMPETITIVE TASK PERFORMANCE
- 4B SSR reaches 61.37% average success rate versus 61.25% for the best 4B SOTA model, while 2B SSR reaches 51.26%, comparable to a zero-shot 8B model.The table reports success rates on multimodal search benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7
4.3 COMPARISON BETWEEN SSR AND FREEFORM REASONING
- SSR cuts per-turn reasoning latency by over 90% and mean per-question model latency by 28–54% while maintaining comparable success rates across training objectives.The comparison covers GRPO, GSPO, SAPO, and SFT.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4 INFERENCE LATENCY AND REASONING THROUGHPUT
- SSR has the highest success rate among the efficient-reasoning baselines, with approximately 7× or greater reasoning speedup and around 2.5s mean model latency, about 30% below the fastest baseline.At the 95th percentile, reasoning time is 0.071s versus 0.798–1.892s for baselines; question-level latency is 4.6s versus 6.6s for the fastest baseline.Source: §4.4 · p. 8, Table 3 · p. 8
4.5 COMPARISON BETWEEN DIFFERENT REASONING METHODS
- Selecting candidate indices instead of full reasoning text lowers average success by 3.9% across all seven benchmarks, while autoregressive selection requires SFT to conform to the library.Without SFT, a prompt and RL format reward do not enforce use of library candidates; full-text selection scores candidate semantics.Source: §4.5 · pp. 8–9, Table 4 · p. 8
4.6 REASONING LIBRARY SIZE AND SELECTION DISTRIBUTION
- Across seven benchmarks, success rate rises monotonically from 37.24% to 61.37% as the library grows from one to six candidates; moving from one generic candidate to two gains 17.27 percentage points.The same checkpoint selects reasoning differently by benchmark: SimpleVQA favors answering from the image, while HR-MMSearch favors examining image detail; generated action types match the selected candidate in 99.9% of turns.Source: §4.6 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9
5 CONCLUSION
The conclusion presents SSR as selection among reusable reasoning candidates, with parallel scoring supporting competitive performance and lower inference latency across multimodal search benchmarks.
5 CONCLUSION
- Across seven multimodal search benchmarks, SSR achieves competitive task performance while reducing per-turn reasoning latency by over 90% and per-question model latency by 28–54%.The conclusion attributes the efficiency to parallel scoring of pre-specified candidates rather than token-by-token reasoning generation.Source: §5 · p. 9
- SSR selects reusable natural-language candidates according to their likelihood under the current context instead of generating open-ended reasoning.Source: §5 · p. 9
- The results suggest that small search agents may use a compact reusable reasoning space rather than reason over the full language space at every step.Source: §5 · p. 9
AI USE STATEMENT
The authors report using generative AI tools for manuscript assistance and parts of the experimental pipeline, while retaining responsibility for the final work.
AI USE STATEMENT
- Generative AI tools assisted with drafting, revision, mathematical-claim checks, derivation checks, and reference locating and formatting.Source: AI USE STATEMENT · p. 10
- For SFT experiments, Gemini 3.1 Pro generated reasoning traces and tool-call trajectories and mapped retained traces to reasoning-candidate labels.Source: AI USE STATEMENT · p. 10
- Qwen models summarized retrieved content and judged open-ended answers; the authors take responsibility for the final text, claims, results, and other AI-assisted content.Source: AI USE STATEMENT · p. 10
A PSEUDOCODE FOR SSR INFERENCE AND GRPO TRAINING
SSR inference scores reusable reasoning candidates conditioned on the interaction history, selects one, and then generates an action. Grouped GRPO trains the selector and action generator while approximating selector gradients with detached competitor scores.
A.1 INFERENCE
- Candidates are teacher-forced against the running history and scored by length-normalized likelihood; the selected reasoning is appended before autoregressive action generation.The library can be built by grouping recurring information needs from successful traces into reusable candidates that omit question-specific entities and answers.Source: §A.1 · p. 14
A.2 GROUPED GRPO WITH DETACHED COMPETITORS
- Grouped GRPO collects multiple trajectories per question, normalizes rewards within each group, and updates selector and generated-action terms while applying KL only to generated action positions.Only the selected candidate’s score receives gradients; detached competitor scores avoid backward computation through competing candidates.Source: §A.2 · p. 14, §B.1 · p. 16
B.1 EXACT AND APPROXIMATE GRADIENTS
- The exact selector gradient requires forward and backward computation for all candidates; detached competitors avoid their backward passes, but their scores can become stale after an update.The selected probability is exact at the rollout parameters on the first gradient update.Source: §B.1 · pp. 16–17
B.2 SSR ROLLOUTS ARE ON-POLICY IN GRPO
- SSR rollout collection is on-policy for its composite behavior: one categorical probability records reasoning selection, while token-level probabilities record subsequent autoregressive actions.Reusing a collected batch after parameter updates makes the current policy differ from the behavior policy, as in standard GRPO.Source: §B.2 · p. 17
B.3 GSPO AND SAPO WITH REUSED ROLLOUTS
GSPO and SAPO adapt their update rules to SSR’s reasoning selections and generated actions. The cost analysis distinguishes computational operations, cache memory, and latency, and gives a hardware-specific comparison of candidate scoring with freeform decoding.
B.3 GSPO AND SAPO WITH REUSED ROLLOUTS
- GSPO applies a clipped surrogate to a length-normalized sequence ratio, whereas SAPO applies a smooth surrogate to each ratio with slope set by the advantage sign.Both normalize by trajectory length and average across trajectories; inserted reasoning tokens, prompts, observations, and padding are excluded from the length.Source: §B.3 · p. 17
C DETAILED INFERENCE COST
- The latency comparison models freeform generation as L/vdec plus overhead and SSR candidate scoring as Ktot/vscore plus overhead; SSR is faster when its total is lower.The FLOP analysis assumes a fixed Transformer and a shared history already stored in the KV cache.Source: C DETAILED INFERENCE COST · p. 18
- On an NVIDIA RTX PRO 6000 GPU, generating 100 freeform reasoning tokens takes around the same time as processing a 15,200-token library.The analysis distinguishes computational depth from FLOPs, KV-cache memory, and inference latency, and estimates shared-prefix cache storage as mKV(H + Ktot) bytes.Source: C DETAILED INFERENCE COST · p. 18
D.1 TRAINING DATASETS
Training draws on a 3,859-example multimodal-search mixture that combines factual visual questions, fine-grained image inspection, and high-resolution visual search.
D.1 TRAINING DATASETS
- The 3,859-example training mixture is drawn from TAPO/SenseNova-MARS, with subset sizes and accuracy rewards reported in Table 5.The supplied description does not specify the table’s cell values.Source: §D.1 · p. 18, Table 5 · p. 19
- FVQA-train mixes search-required and search-free factual image questions, while DeepEyes-4K covers fine-grained perception, chart understanding, and challenging reasoning.FVQA-train is intended to train the agent to identify when image or text retrieval is needed; DeepEyes-4K includes multiple-choice and non-multiple-choice subsets.Source: §D.1 · p. 19
- VisualProbe-train focuses on small targets in high-resolution images with distractors, encouraging repeated inspection and exploration of image regions.Source: §D.1 · p. 19
D.2 EVALUATION BENCHMARKS
The evaluation uses seven multimodal search benchmarks spanning retrieval difficulty, temporal freshness, and image resolution. The paper also describes a fixed six-candidate reasoning library used throughout training and evaluation.
D.2 EVALUATION BENCHMARKS
- The seven benchmarks collectively span retrieval difficulty, temporal freshness, and image-resolution requirements.The evaluation uses benchmark versions distributed in the SenseNova collection.Source: §D.2 · p. 19
- HR-MMSearch has 305 knowledge-intensive questions paired with 4K images across eight domains, with approximately 60% classified as hard and requiring at least three tool invocations.MMSearch uses a 171-question system-level split and includes image-grounded, up-to-date factual questions.Source: §D.2 · p. 19
- FVQA-test contains 1,800 samples split evenly across automatically constructed, human-corrected InfoSeek, and expert-annotated subsets.InfoSeek targets fine-grained factual attributes grounded in Wikidata; the evaluation samples 2,000 instances from its test split.Source: §D.2 · p. 19
- SimpleVQA covers 1,013 English samples, LiveVQA aggregates 3,602 image-question pairs emphasizing recent content, and MAT-Search tests composite multi-hop queries and tool use.Source: §D.2 · pp. 19–20
E REASONING CANDIDATE LIBRARY
- The default reasoning library contains six manually written short natural-language candidates, and remains fixed during training and evaluation.Each candidate describes how to make progress from the current state.Source: E REASONING CANDIDATE LIBRARY · p. 20, Table 6 · p. 20
F TRAINING DETAILS
The training setup specifies an RL recipe and interaction limits, a four-action interface, and accuracy-plus-format rewards.
F TRAINING DETAILS
- RL training uses VeRL with batch size 64, learning rate 1 × 10−5, and eight rollouts per GRPO group, without an SFT warm-up.The setup adapts TAPO’s multimodal-search configuration.Source: F TRAINING DETAILS · p. 20
- Training trajectories allow at most 10 interaction rounds, 8,192 generated tokens per round, and 32,768 tokens total; exceeding any limit counts as incorrect.Training text retrieval uses a local Wikipedia index, while evaluation uses Serp API web search.Source: F TRAINING DETAILS · p. 20
- The agent can use image search, query-based text search, image zoom, or a terminal answer, with SSR selecting the thinking block before the tool call.The tool call follows the same JSON schema as the base protocol.Source: F TRAINING DETAILS · p. 20
- RL combines accuracy and format rewards; non-multiple-choice answers receive model-based judgments, while multiple-choice answers use exact match.SSR receives full format reward only when it uses the expected tool for the selected reasoning, otherwise it receives partial format reward for a correct tool call.Source: F TRAINING DETAILS · p. 20
G REASONING COMPARISON: NO THINKING, FREEFORM, AND SSR
Under GRPO, SSR has the highest unweighted mean success rate among the three reasoning conditions and outperforms no thinking on all seven benchmarks. Freeform reasoning improves over no thinking on only two benchmarks.
G REASONING COMPARISON: NO THINKING, FREEFORM, AND SSR
- 61.37% mean success rate for SSR exceeds 58.91% for no thinking and 58.65% for freeform reasoning across seven benchmarks.These are unweighted benchmark means; SSR’s gains are 2.47 and 2.72 percentage points, respectively.Source: G REASONING COMPARISON: NO THINKING,… · p. 21, Figure 5 · p. 21
- SSR beats no thinking on all seven benchmarks, with gains from 0.99 to 7.34 percentage points; freeform reasoning does so on only two.SSR also exceeds freeform reasoning on five benchmarks, while HR-MMSearch and MAT-Search favor freeform reasoning.Source: G REASONING COMPARISON: NO THINKING,… · p. 21