PaperlayerView exact revision on arXiv

Source-linked AI summary

Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search

Tianming Liang, Qirui Du, Jian-Fang Hu, Haichao Jiang, Zicheng Lin, Wei-Shi Zheng

arXiv:2602.04454v1cs.CV

TL;DR

Existing MLLM-based segmentation systems are limited by frozen knowledge when queries require current or domain-specific information. Seg-ReSearch interleaves reasoning with external search and trains this behavior using hierarchical rewards, then evaluates it on OK-VOS and existing reasoning-segmentation benchmarks, where it substantially improves over prior approaches. Its broader deployment may introduce internet-bias and privacy risks.

  • Problem

    Existing segmentation systems cannot continuously acquire external information, limiting their handling of dynamic queries beyond MLLMs’ frozen knowledge.

  • Method

    Seg-ReSearch interleaves task decomposition, reasoning, and external search, trained with hierarchical rewards balancing outcome feedback and step-wise supervision.

  • Results

    Seg-ReSearch substantially improves over existing approaches, including state-of-the-art models and baselines equipped with the same search tools.

  • Takeaways & Limitations

    The framework supports segmentation of dynamic, open-world queries requiring information beyond the frozen knowledge of MLLMs.

  • Takeaways & Limitations

    The approach may amplify internet bias and raise privacy concerns when accessing the external world.

Abstract

from arXiv · show

Segmentation based on language has been a popular topic in computer vision. While recent advances in multimodal large language models (MLLMs) have endowed segmentation systems with reasoning capabilities, these efforts remain confined by the frozen internal knowledge of MLLMs, which limits their potential for real-world scenarios that involve up-to-date information or domain-specific concepts. In this work, we propose \textbf{Seg-ReSearch}, a novel segmentation paradigm that overcomes the knowledge bottleneck of existing approaches. By enabling interleaved reasoning and external search, Seg-ReSearch empowers segmentation systems to handle dynamic, open-world queries that extend beyond the frozen knowledge of MLLMs. To effectively train this capability, we introduce a hierarchical reward design that harmonizes initial guidance with progressive incentives, mitigating the dilemma between sparse outcome signals and rigid step-wise supervision. For evaluation, we construct OK-VOS, a challenging benchmark that explicitly requires outside knowledge for video object segmentation. Experiments on OK-VOS and two existing reasoning segmentation benchmarks demonstrate that our Seg-ReSearch improves state-of-the-art approaches by a substantial margin. Code and data will be released at https://github.com/iSEE-Laboratory/Seg-ReSearch.

1. Introduction

Existing segmentation systems can reason about visual content but remain constrained by frozen MLLM knowledge when queries require new or domain-specific information. Seg-ReSearch addresses this gap through interleaved external search, hierarchical rewards, and the OK-VOS benchmark.

  • Motivation: Current segmentation models struggle with queries involving up-to-date or domain-specific concepts beyond the frozen knowledge of MLLMs.They lack the ability to continuously acquire necessary information from external sources.
  • Framework: Seg-ReSearch iteratively decomposes tasks, reasons, and interacts with search engines until identifying the target objects.This interleaved reasoning-and-search paradigm is designed for broader open-world queries.
  • Reward Design: The hierarchical reward mechanism balances sparse outcome rewards with rigid step-wise supervision.Expert actions guide the first step, while format-based rewards and a tapering bonus support valid exploration without infinite search loops.
  • Benchmark: OK-VOS is a human-annotated benchmark designed to require external knowledge for video object segmentation.Its queries contain up-to-date information or new concepts beyond existing MLLMs’ internal knowledge and may require single- or multi-hop searches.
  • Results: Seg-ReSearch significantly outperforms state-of-the-art reasoning segmentation models and baselines using the same search tools.The reported evaluation covers OK-VOS and establishes a substantial performance advantage.

2. Related Work

Language-guided segmentation has progressed from explicit visual references toward reasoning-based instructions, while related agentic methods remain limited in their reward designs and application domains. Seg-ReSearch combines iterative search with segmentation to address these limitations.

  • Language-guided Segmentation: Referring segmentation and referring video object segmentation let users specify image or video objects through natural language.Recent reasoning-oriented approaches also train MLLMs to generate positional prompts for external segmentation models.
  • Reward Designs in Agentic RL: Agentic reinforcement-learning methods commonly use either sparse outcome rewards or rigid process rewards.The former supervise the whole trajectory, whereas the latter supervise each action step.
  • Search Agents: Retrieval-augmented generation retrieves passages for user queries, but static retrieval can suffer from noisy information.Recent systems therefore train language models to perform iterative search during reasoning.
  • Search Agents: Most iterative-search agents focus on general question answering rather than fine-grained visual tasks such as segmentation.Seg-ReSearch extends interleaved reasoning and external search to segmentation through multi-turn search interactions during dynamic MCoT.

3. Seg-ReSearch

Seg-ReSearch combines interleaved reasoning, external search, and hierarchical rewards for language-guided segmentation beyond MLLMs’ internal knowledge. Its pipeline iteratively searches, selects a keyframe, localizes the target, and generates masks.

  • Overview: Seg-ReSearch iteratively decomposes queries, reasons, and interacts with search engines to identify objects beyond MLLMs’ internal knowledge.The framework supports image and video inputs and dynamically updates reasoning with retrieved information.
  • Segmentation Pipeline: The video pipeline selects a target-containing keyframe, predicts a bounding box and point, then passes them to a frozen mask generator for mask prediction and propagation.Static image segmentation bypasses keyframe selection.
  • Hierarchical Reward Designs: The hierarchical reward balances sparse outcome rewards against rigid step-wise supervision through process and outcome components.The process reward includes initial guidance and tapering process rewards, while the outcome reward captures keyframe and localization quality.
  • Hierarchical Reward Designs: Initial guidance rewards a first search query semantically matching an expert trajectory while allowing any valid entry point rather than exact imitation.Similarity is computed with a light Sentence Transformer.
  • Hierarchical Reward Designs: TPR grants format-based bonuses for valid actions while tapering cumulative rewards toward 1 to support exploration without infinite search loops.p is the base reward, k counts valid-format actions, and M sets the upper bound.
  • Reward and Optimization: The outcome reward combines binary localization checks with a continuous frame-quality reward, and GRPO optimizes grouped outputs using their relative rewards.Riou, Rl1, and Rpoint evaluate localization, while Rframe prioritizes prominent, less-occluded target frames.

4. OK-VOS: A VOS Benchmark Requiring Outside Knowledge

OK-VOS is a video object segmentation benchmark designed for queries requiring outside knowledge rather than evidence already present in the visual input. It contains expert-reviewed samples spanning one-hop, multi-hop, and relational identification.

  • Motivation: Existing language-guided benchmarks provide necessary identification evidence, while reasoning benchmarks generally test only basic common sense.OK-VOS targets up-to-date information and long-tail knowledge beyond those simplified settings.
  • Dataset Construction: OK-VOS contains 1,000 test samples across 150 videos and 500 objects, annotated and reviewed by five human experts.Queries answerable without external web search were discarded or refined, and visual shortcuts were explicitly mitigated.
  • Query Categories: The benchmark categorizes queries as one-hop, multi-hop, or relational according to the number and structure of retrieval and reasoning steps required.Relational queries define a target through its visual relationship to a knowledge-based anchor target.

5. Experiments

Experiments evaluate Seg-ReSearch on external-knowledge and conventional reasoning segmentation benchmarks, then analyze reward designs, training behavior, and search settings. The results show strong benchmark performance and benefits from hierarchical rewards, multi-turn search, retrieval, and browsing choices.

  • SOTA Comparisons: 50.0 overall J &F makes Seg-ReSearch-8B substantially stronger than OK-VOS competitors, while the 4B model outperforms the search-augmented baseline by nearly 10%.VideoLISA-3.8B reaches 19.3 overall J &F, UniPixel-7B reaches 34.2, and Qwen3-VL-8B*+Search improves its base model by only 1.8%.
  • SOTA Comparisons: Seg-ReSearch-8B establishes new state-of-the-art results on ReasonSeg and ReasonVOS, improving +8.3 J &F over the same-MLLM OneThinker-8B.These results support robustness in general reasoning segmentation.
  • Analysis of Reward Designs: Hierarchical rewards achieve the best performance by balancing outcome feedback with exploration, unlike sparse outcome rewards or rigid step-wise imitation.Outcome rewards provide insufficient guidance for complex multi-turn reasoning, while imitation constrains exploration.
  • Analysis of Reward Designs: LPR causes reward hacking and degrades J &F to 41.1, whereas BPR and w/o PR leave search usage nearly unchanged during training.The comparison uses Qwen3-VL-4B as the base model and indicates that fixed or sparse process signals do not reliably motivate multi-step search.
  • Ablation on Search Settings: Increasing maximum search turns from 1 to 5 yields +7.4%, while increasing from 5 to 10 adds only 0.4%, indicating diminishing returns.The default setting therefore benefits from multi-turn reasoning without requiring the largest action budget.
  • Ablation on Search Settings: Retrieving three textual entries adds +2.1%, image retrieval adds 0.8%, Google Search beats DuckDuckGo by 2.9 J &F, and full-page browsing adds +4.4 J &F.The adopted default retrieves 3 textual and 1 visual entry; the browsing result suggests further gains from more comprehensive retrieval.

6. Conclusion

Seg-ReSearch addresses the knowledge bottleneck in segmentation by interleaving reasoning with external search and training this capability with hierarchical rewards. It also introduces OK-VOS, and experiments report substantial improvements over existing approaches.

  • Conclusion: Seg-ReSearch handles dynamic, open-world queries beyond frozen MLLM knowledge by interleaving reasoning with external search.The framework uses hierarchical rewards to balance outcome feedback and step-wise supervision.
  • Conclusion: OK-VOS is a challenging video object segmentation benchmark designed to require external knowledge.The benchmark complements the proposed framework’s evaluation of knowledge-intensive segmentation.
  • Conclusion: Extensive experiments demonstrate substantial improvements over existing approaches.

Impact Statement

Seg-ReSearch is presented as a step toward visual recognition and segmentation systems that access the external world. The technique is positioned for more intuitive visual interaction and potential use in robotic and educational settings.

  • Seg-ReSearch enables models to access the external world for visual recognition and segmentation.
  • The approach aims to support more intuitive interaction with visual content and insights beyond pixel-level perception.
  • The technique may extend open-world and dynamic capabilities in robotic and educational assistant scenarios.

A. More Implementation Details

Seg-ReSearch uses a structured video-target identification workflow combining visual analysis, optional web search, and explicit localization outputs. Its implementation defines tool interactions, response formats, training settings, and baseline comparisons.

  • Training Setup: Training uses retrieved-token masking, a global batch size of 16, a mini-batch size of 8, and a maximum response length of 6,144 tokens.The maximum response comprises 4,096 generation tokens and 2,048 environmental-feedback tokens.
  • Search Policy: When external knowledge is needed, the model plans a search and invokes text_search or image_search with a query.Search results are returned as environmental information for subsequent reasoning.
  • Target Identification Workflow: The system analyzes video frames and outputs a selected keyframe, bounding box, and point for the target.The workflow first compares candidate objects and selects the clearest frame, then predicts 2D localization coordinates.
  • Search Policy: The assistant must inspect video frames before searching and avoid searching for information available directly from the visual input.Search is reserved for external information needed to identify the target, not ordinary frame analysis.
  • Baselines: Experiments compare Seg-ReSearch with specialist video segmentation, MLLM-based video segmentation, and MLLM-based image segmentation methods.The listed baselines include SAMWISE, ReferDINO, ReferEverything, VideoLISA, GLUS, RGA3, UniPixel, OneThinker, LISA, and RSVP.

B. Training Dynamics of Seg-ReSearch

Training rewards improve as Seg-ReSearch-8B learns to follow the required interaction format and optimize task accuracy. Process rewards rise early while wrong response length approaches zero, alongside increasing outcome rewards.

  • Training Dynamics: Outcome rewards Riou, Rl1, Rpoint, and Rframe show a steady upward trend during Seg-ReSearch-8B training.These rewards track localization and frame-selection outcomes described by the training procedure.
  • Training Dynamics: The process reward rises rapidly early in training while wrong response length declines to near zero.This indicates that the model quickly learns the required interaction format.
  • Training Dynamics: The hierarchical reward design first guides strict format adherence before optimizing task accuracy.The training dynamics support a progression from interaction-format compliance to outcome improvement.

C. Comparison with SFT Baselines

On OK-VOS, supervised fine-tuning on the same 100 samples is insufficient, whereas Seg-ReSearch achieves larger improvements with the same limited data. The comparison attributes the difference to training paradigm rather than simply additional data.

  • Comparison with SFT: 9.8% and 12.4% performance improvements over the 4B and 8B baselines are achieved with only 100 samples.These improvements are reported for Seg-ReSearch on the OK-VOS comparison.
  • Comparison with SFT: SFT degrades GLUS-7B by 13.1% in overall J &F when trained on the same 100 samples.The passage presents this as evidence that SFT can be data-hungry and prone to overfitting.
  • Comparison with SFT: SFT on UniPixel-7B improves performance by 2.8% but still lags far behind Seg-ReSearch.The comparison contrasts a slight SFT gain with the larger improvements reported for the proposed method.
  • Comparison with SFT: The experiment tests whether gains arise from the 100 samples or from the training paradigm by applying public SFT procedures to existing SOTA models.Table 10 reports the resulting comparison on OK-VOS.

D. More Qualitative Analysis

A qualitative case contrasts visual-only reasoning with Seg-ReSearch’s use of external information. Seg-ReSearch identifies the target by combining text search, image verification, and frame-level localization.

  • Qualitative Comparison: Seg-ReSearch retrieves band information through text search and verifies the singer’s identity with image search.The retrieved image is used to compare the candidate identity against the video frames.
  • Qualitative Comparison: The qualitative example reports successful target identification for Seg-ReSearch after combining external identity evidence with visual comparison.The baseline instead hallucinates the target identity from the scene’s prominent visual cues.
  • Qualitative Comparison: Qwen3-VL-8B*+Search incorrectly selects the prominent pink-haired female singer as the target through visual reasoning alone.Its reasoning treats visible appearance and singing activity as evidence that she is the lead singer.
  • Qualitative Comparison: Seg-ReSearch selects frame 2 and predicts a bounding box and point for the identified target.The displayed answer gives bbox_2d [120, 78, 607, 998] and point_2d [275, 675].
Loading 2602.04454v1…