Source-linked AI summary
Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, Jiaya Jia
TL;DR
Reasoning segmentation methods trained on simple supervised descriptions can lose out-of-domain generalization and lack explicit reasoning processes. Seg-Zero decouples MLLM reasoning from segmentation and trains the reasoning model exclusively with GRPO and format-and-accuracy rewards, without explicit reasoning data. Seg-Zero-7B achieves 57.5 on ReasonSeg zero-shot, surpassing LISA-7B by 18%.
Problem
Existing reasoning-segmentation methods rely on supervised fine-tuning with simple descriptions and show degraded out-of-distribution performance.
Method
Seg-Zero uses a decoupled reasoning and segmentation architecture, training the reasoning model with pure GRPO reinforcement learning and format-and-accuracy rewards without explicit reasoning data.
Results
57.5 on ReasonSeg is achieved by Seg-Zero-7B in zero-shot evaluation, surpassing LISA-7B by 18%.
Takeaways & Limitations
Seg-Zero exhibits emergent test-time reasoning and strong generalization performance compared with models of the same scale.
Takeaways & Limitations
Seg-Zero currently focuses only on single-object reasoning segmentation.
Abstract
from arXiv · showhide
Traditional methods for reasoning segmentation rely on supervised fine-tuning with categorical labels and simple descriptions, limiting its out-of-domain generalization and lacking explicit reasoning processes. To address these limitations, we propose Seg-Zero, a novel framework that demonstrates remarkable generalizability and derives explicit chain-of-thought reasoning through cognitive reinforcement. Seg-Zero introduces a decoupled architecture consisting of a reasoning model and a segmentation model. The reasoning model interprets user intentions, generates explicit reasoning chains, and produces positional prompts, which are subsequently used by the segmentation model to generate precious pixel-level masks. We design a sophisticated reward mechanism that integrates both format and accuracy rewards to effectively guide optimization directions. Trained exclusively via reinforcement learning with GRPO and without explicit reasoning data, Seg-Zero achieves robust zero-shot generalization and exhibits emergent test-time reasoning capabilities. Experiments show that Seg-Zero-7B achieves a zero-shot performance of 57.5 on the ReasonSeg benchmark, surpassing the prior LISA-7B by 18\%. This significant improvement highlights Seg-Zero's ability to generalize across domains while presenting an explicit reasoning process.
1. Introduction
Reasoning segmentation targets complex queries that require logical reasoning and cross-domain knowledge, but supervised methods trained on simple descriptions generalize poorly out of domain. Seg-Zero addresses this with a decoupled reasoning-and-segmentation architecture trained using reinforcement learning and structured rewards.
- Reasoning segmentation interprets implicit queries to produce pixel-wise masks for nuanced targets requiring logical reasoning and cross-domain knowledge.
- Existing methods typically use supervised fine-tuning on categorical or basic factual descriptions, achieving alignment on specific datasets but degrading on out-of-distribution samples.
- Seg-Zero decouples reasoning from segmentation: an MLLM generates reasoning chains, bounding boxes, and points, while a segmentation model converts these prompts into masks.
- Training fine-tunes only the reasoning model with pure GRPO reinforcement learning, without explicitly annotated reasoning data, while keeping the segmentation model frozen.
- Format rewards constrain reasoning and output structure, whereas accuracy rewards use IoU and L1 distance to guide optimization.
- 57.5 on ReasonSeg is achieved by Seg-Zero-7B in zero-shot evaluation, surpassing LISA-7B by 18%.The model uses only 9,000 training samples derived from RefCOCOg.
2. Related Works
Prior work spans semantic, referring-expression, and MLLM-based segmentation, while newer multimodal reasoning studies focus mainly on high-level tasks. Seg-Zero targets the remaining gap of fine-grained pixel-level reasoning.
- Recent multimodal reasoning works address high-level mathematical or counting tasks but do not consider fine-grained pixel-level image understanding.
- Seg-Zero is designed to enhance pixel-level reasoning through reinforcement learning.
- Semantic segmentation predicts masks for explicit classes, whereas referring expression segmentation identifies target objects from short text queries.
- LISA and related methods bridge MLLMs with segmentation models through special semantic tokens or tokens such as <SEG>.
- Token-based approaches require extensive data to fine-tune both the MLLM and segmentation decoder and may compromise pixel-level capabilities.
3. Method
Seg-Zero separates reasoning-based localization from mask generation: an MLLM reasons over an image and object description to produce spatial prompts, and a segmentation model uses them to generate a fine-grained mask. Reinforcement learning trains the reasoning model from zero with structured prompts and rewards.
- 3.1. Pipeline Formulation: The method section defines segmentation as producing a binary mask M for the region corresponding to an image label T.
- 3.1. Pipeline Formulation: The reasoning model generates a reasoning process plus a bounding box and two points that localize the target object.
- 3.2. Seg-Zero Model: The segmentation model receives the bounding box and points as prompts and produces a precise, fine-grained mask.
- 3.2. Seg-Zero Model: The user prompt instructs the model to analyze and compare image objects, generate reasoning first, and then output a predefined final-answer format.
- 3.3. Reward Functions: Thinking-format rewards require reasoning inside <think> tags and the final answer inside <answer> tags.
- 3.3. Reward Functions: Segmentation-format rewards evaluate whether bbox and point coordinates satisfy soft or strict output constraints.
- 3.4. Training Details: Training data uses original masks to derive bounding boxes and centers of two largest inscribed circles, and GRPO starts from a pretrained model without reasoning data.
4. Experiment
Experiments evaluate Seg-Zero against supervised fine-tuning, ablate prompt and reward choices, and test scaling and zero-shot generalization. Reinforcement learning with chain-of-thought rewards improves in-domain and out-of-domain segmentation while preserving visual QA ability.
- SFT vs. RL: RL + CoT thinking reward achieves the best performance on both in-domain and OOD segmentation tasks.CoT further improves performance over RL without CoT, indicating that the reasoning process benefits OOD samples.
- SFT vs. RL: SFT suffers catastrophic forgetting of visual QA ability, while RL preserves the model’s general Visual QA capability.This comparison extends the evaluation beyond segmentation tasks.
- Ablation Study: Combining bbox and point prompts produces the best performance, while using only point prompts performs worst.The result supports the prompt design’s role in pixel-level localization accuracy.
- Ablation Study: Strict format rewards significantly improve OOD ReasonSeg performance, although they progress more slowly initially and later produce longer responses.Soft accuracy rewards slightly improve ReasonSeg but significantly underperform hard rewards on RefCOCOg.
- Ablation Study: Performance on both in-domain and OOD data improves as reasoning-model scale increases from 2B to 7B parameters.Completion length also tends to increase with model size and during later training after format rewards saturate.
- Comparison with Other Methods: Qualitative results show that reasoning helps distinguish contextually appropriate objects and positional relations among multiple same-class objects.Examples include selecting a recreational vehicle rather than a truck for a road-trip query and identifying a conductor at the front of a stage.
5. Conclusion
Seg-Zero integrates chain-of-thought reasoning into segmentation and uses format and accuracy rewards under exclusive reinforcement learning. The paper reports emergent reasoning without supervised reasoning data and introduces additional localization evaluation.
- Seg-Zero integrates the CoT reasoning process into segmentation tasks through a reward mechanism combining format and accuracy constraints.
- Exclusive reinforcement learning enables Seg-Zero to develop emergent reasoning capabilities without supervised reasoning data.
- Table 11 reports bbox IoU and Point Accuracy as additional localization metrics.
A. Rewards Changes During Training
During training, format rewards converge rapidly while accuracy rewards increase gradually. The changing reward balance corresponds to an initially shorter response length.
- Format rewards converge to 1 within a few steps, while accuracy rewards gradually increase during training.
- The early dominance of format rewards decreases response length during the initial training steps.
B. More Visualization Examples
Figure 9 provides additional visualization examples.
- Figure 9 provides more examples of the model’s visual outputs.
C. Additional Analysis of RefCOCO(+/g)
RefCOCO(+/g) mask annotations contain substantial size and edge inaccuracies, which lower IoU-based evaluation. Additional localization metrics indicate approximately 90% success in target-object localization.
- In 100 randomly selected samples, 5% inaccurately represent object sizes and almost 70% have imprecise edge handling.
- Annotation errors produce lower IoU scores despite the model’s reported high precision.
- Prior methods fine-tune their segmentation decoders to adapt to imprecise annotations, whereas Seg-Zero keeps its segmentation model frozen.
- Approximately 90% of predictions successfully localize the target object under the added bbox accuracy and point accuracy metrics.
- Figure 8 shows mean reward changes across a training batch.
D. Limitations and Future Works
Seg-Zero currently focuses on single-object reasoning segmentation, while future work considers multi-object and instance-level reasoning segmentation.
- Future work considers extending Seg-Zero to multi-object and instance-level reasoning segmentation.