Source-linked AI summary
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
Yuting Yang, Haichao Jiang, Tianming Liang, Quan Zhang, Jian-Fang Hu
TL;DR
Existing referring segmentation assumes precise queries, although real-world instructions can be ambiguous and lead models to guess user preferences. IC-Seg uses multi-turn intent clarification, Hi-GRPO, and the Ambi-RVOS benchmark; it reports superior ambiguous-query performance while retaining state-of-the-art results on existing reasoning benchmarks.
Problem
Existing referring segmentation methods assume user queries contain enough precise details to uniquely distinguish the intended target, despite ambiguity being common in real-world instructions.
Method
IC-Seg performs multi-turn interactive intent clarification before segmentation, while Hi-GRPO supplies hierarchical supervision at trajectory, turn, and step levels.
Results
IC-Seg achieves superior performance on Ambi-RVOS, including nearly 20% higher overall J &F, and preserves state-of-the-art performance on existing image and video reasoning segmentation benchmarks.
Takeaways & Limitations
The framework resolves ambiguous references through interaction while maintaining strong grounding and segmentation capabilities across existing reasoning benchmarks.
Takeaways & Limitations
IC-Seg primarily supports text-based interaction, leaving pointing gestures and audio-visual cues for future exploration.
Abstract
from arXiv · showhide
Referring segmentation aims to segment the target objects in images or videos based on the textual query. Despite remarkable progress over the past years, existing works always assume that the user-provided queries are already precise and clear. However, this assumption is impractical. In real-world scenarios, it is unrealistic to expect all users to thoroughly review their visual content and carefully ensure their queries are unique and unambiguous. When encountering such cases, existing segmentation models tend to arbitrarily guess the user preferences, often resulting in undesired outcomes. To address this limitation, we propose IC-Seg, a novel agentic framework that proactively clarifies user intent through multi-turn conversation before segmentation. To effectively incentivize this capability, we further introduce Hi-GRPO, a new hierarchical optimization strategy that injects dense and informative supervision signals at the trajectory, turn, and step levels. This strategy encourages efficient intent clarification, effectively eliminating redundant interactions and improving overall dialogue quality. For evaluation, we establish Ambi-RVOS, a referring video object segmentation benchmark with ambiguous user queries. Extensive experiments demonstrate that IC-Seg not only outperforms existing methods by a large margin in resolving ambiguous queries, but also maintains state-of-the-art performance on standard reasoning segmentation benchmarks. Code and data will be released at https://github.com/iSEE-Laboratory/IC-Seg.
1 Introduction
Existing referring segmentation methods assume queries are precise, but real-world instructions are often ambiguous. IC-Seg addresses this gap through proactive multi-turn clarification, supported by Hi-GRPO and evaluated on Ambi-RVOS and existing reasoning benchmarks.
- Real-world referring queries may omit details needed to uniquely identify the intended target, making the assumption of precise instructions impractical.
- IC-Seg proactively interacts with users to confirm intent and preferences before segmenting the desired target.
- Nearly 20% improvements in overall J &F distinguish IC-Seg on ambiguous queries, while it preserves state-of-the-art performance on existing image and video reasoning benchmarks.
- Hi-GRPO uses hierarchical supervision signals to foster effective multi-turn clarification capabilities.
- Ambi-RVOS is a benchmark designed to reflect ambiguity in human instructions for referring video object segmentation.
2 Related Work
Related work spans language-guided segmentation, multi-turn visual interaction, and reinforcement learning for language models. These areas provide foundations for visual grounding, dialogue, and interactive policy optimization.
- Language-guided Segmentation: Language-guided segmentation identifies and segments image or video targets from natural-language descriptions, with recent methods incorporating MLLMs for complex reasoning.
- Multi-turn Visual Interaction: Multi-turn visual interaction combines visual understanding, dialogue-history tracking, and context-aware responses for conversations about images or videos.
- Reinforcement Learning with LLMs: Reinforcement learning with LLMs uses verifiable rewards and GRPO-style relative advantages to improve reasoning without relying on a learned value model.
3 IC-Seg: Referring Segmentation with Multi-turn Intent Clarification
IC-Seg resolves ambiguous referring-segmentation queries through multi-turn clarification before localization and segmentation. Hi-GRPO supervises final accuracy, dialogue quality, and fine-grained reasoning across hierarchical levels.
- Clarification-then-Segmentation Pipeline: IC-Seg uses a trainable policy MLLM to clarify ambiguous queries through multi-turn dialogue before localizing and segmenting the confirmed target.A User Simulator supports simulated human-in-the-loop interaction during training.
- Clarification-then-Segmentation Pipeline: The segmentation stage selects a prominent video keyframe, predicts a bounding box and interior point, and feeds them to frozen SAM2 for whole-video mask generation.
- Hierarchical Optimization: Hi-GRPO provides hierarchical supervision at trajectory, turn, and step levels instead of relying only on sparse final-outcome rewards.The hierarchy targets localization accuracy, inquiry quality, and token-level reasoning quality.
- Trajectory-Level Optimization: Trajectory-level rewards supervise keyframe localization using IoU, box-center distance, point accuracy, and keyframe quality terms.The reward combines R_iou, R_box, R_point, and R_keyframe.
- Turn-Level Optimization: Turn-level rewards combine entropy reduction with interaction efficiency to discourage premature termination and redundant, ineffective inquiries.Questions count as effective when they reduce the candidate set, encouraging concise and discriminative clarification.
- Step-Level Optimization: Step-level optimization uses expert diagnostic feedback from the User Simulator to provide dense token-quality signals and align the policy with privileged evaluations.The resulting probability ratio is incorporated into the final objective.
4 Ambi-RVOS: An RVOS Benchmark with Ambiguous User Queries
Ambi-RVOS is a referring video object segmentation benchmark designed to test intent clarification when queries match multiple candidate objects. It constructs ambiguity from visually similar scenes and controlled query simplification.
- Benchmark Design: Ambi-RVOS contains ambiguous queries where multiple candidate objects match the expression but only one individual is the intended ground truth.
- Benchmark Construction: The benchmark curates videos from ReVOS and MeViS, emphasizing scenes with high densities of visually similar objects.
- Ambiguity Construction: Ambiguous queries are created either by preserving source queries that already match multiple objects or by progressively removing identifying attributes.
- Benchmark Splits: The 1,500-sample test set is divided into Simple, Medium, and Difficult levels according to the initial number of candidate objects.Simple has 400 samples with two candidates, Medium has 600 with three to five, and Difficult has 500 samples.
5 Experiments
Experiments evaluate IC-Seg on ambiguous and standard reasoning segmentation benchmarks, then examine Hi-GRPO components, privileged guidance, efficiency, simulator reliability, and qualitative clarification behavior.
- Main Results: 55.1 J &F establishes IC-Seg-8B as the best-performing method on Ambi-RVOS, while IC-Seg-4B improves 21.6 J &F over Qwen3-VL-4B*.Traditional specialist RVOS models and reasoning MLLMs perform substantially worse on ambiguous queries.
- Main Results: +4.1 Test gIoU over SAM3-Agent-7B and +6.0 J &F over OneThinker-8B show IC-Seg-8B remains competitive on standard reasoning benchmarks.The comparison includes ReasonSeg and ReasonVOS, with OneThinker-8B using the same Qwen3-VL-8B base model.
- Ablation Study: 52.0 J &F is reached after adding trajectory-, turn-, and step-level supervision, compared with 46.4 J &F for trajectory-level optimization alone.Combining turn-level rewards raises the score to 48.9 J &F before step-level supervision produces the final result.
- Ablation Study: Expert process-oriented guidance outperforms empty guidance and ground-truth answers, while direct privileged-distribution distillation remains insufficient for long-horizon interaction.The guidance diagnoses reasoning flaws rather than only supplying the final target.
- Efficiency Analysis: IC-Seg achieves its best accuracy with 2.5 clarification turns on average, and step-level removal increases step time by only approximately 13% over the no-step-level variant.Removing step- or turn-level design reduces accuracy and increases interaction turns; the extra training cost mainly comes from simulator calls and one additional forward pass.
- User Simulator Analysis: Performance improves from 45.3 to 52.0 as the training simulator scales from 4B to 32B, while simulator-based evaluation produces results similar to human-user evaluation.The analyses vary training and inference simulators separately and compare simulator evaluation with five human users on 150 test samples.
- Qualitative Results: In a representative ambiguous case, IC-Seg excludes implausible candidates through feedback and verifies the white car instead of repeating an unsupported guess.The trajectory-only GRPO model repeatedly focuses on the orange bicycle after negative feedback, whereas IC-Seg uses the interaction to resolve the reference.
6 Conclusion
The paper studies intent clarification in language-guided segmentation through IC-Seg, trains this behavior with Hi-GRPO, and introduces Ambi-RVOS for evaluation. Experiments show effective ambiguity resolution while preserving strong grounding and segmentation capabilities.
- IC-Seg resolves ambiguous references through user interaction before segmentation.
- Hi-GRPO combines trajectory-, turn-, and step-level optimization to provide dense supervision for clarification behavior.
- Ambi-RVOS is a referring video object segmentation benchmark containing ambiguous user queries.
- Experiments on Ambi-RVOS and existing reasoning segmentation benchmarks demonstrate effective ambiguity resolution with strong grounding and segmentation capabilities.
A More Implementation Details
The implementation trains Hi-GRPO with sampled multi-turn trajectories, hierarchical rewards, and expert guidance for token-level credit assignment. The setup uses bounded interaction and multimodal video or image inputs with specified resizing procedures.
- Implementation setup: IC-Seg limits both training and inference to a maximum of five query turns.
- Implementation setup: For video segmentation, six frames are uniformly sampled and resized to 448 × 448, while the selected keyframe is resized to 864 × 864.
- Implementation setup: The implementation includes expert guidance that can refine spatial localization and keyframe selection for ambiguous video targets.
- Hi-GRPO training: Hi-GRPO samples on-policy multi-turn trajectories and computes sequence-level advantages from trajectory and turn rewards.
- Hi-GRPO training: Expert guidance provides diagnostic information for refining token-level credit assignment before the GRPO update.
- Hi-GRPO training: The training procedure uses a dataset of multimodal inputs, grouped trajectory sampling, reward weighting, mixing, and clipped policy updates.
C Training Dynamics of IC-Seg
IC-Seg’s training dynamics show improving localization and process rewards alongside stabilized query counts. Additional supervised fine-tuning on Ambi-RVOS samples does not improve baseline robustness, whereas IC-Seg gains consistently across difficulty levels.
- Training dynamics: Localization rewards RIoU, Rbox, Rpoint, and Rframe steadily increase during IC-Seg-8B training.These rewards indicate improving final grounding accuracy and keyframe selection.
- Training dynamics: The process reward rises consistently, indicating more effective clarification behavior during training.
- Training dynamics: Query counts increase early and then stabilize around a moderate range as training progresses.
- Comparison with SFT baselines: Fine-tuning representative baselines on the 120 Ambi-RVOS training samples does not improve robustness to ambiguous references.
- Comparison with SFT baselines: GLUS-7B drops by 9.6 overall J &F and UniPixel-7B decreases by 2.8 points after SFT.
- Comparison with SFT baselines: IC-Seg obtains large and consistent gains over its Qwen3-VL* initialization across all difficulty levels.
E Sensitivity Analysis of λ.
The sensitivity analysis examines λ, which balances holistic sequence-level advantage against dense step-level supervision. λ = 0.5 performs best, while lower or higher values provide less effective guidance.
- λ sensitivity: λ = 0.5 achieves the best performance in the step-level reward sensitivity analysis.
- λ sensitivity: λ = 0.3 gives less weight to step-level supervision and does not fully highlight key reasoning steps in long dialogues.
- λ sensitivity: λ = 0.7 gives more weight to expert guidance, which tends to encourage the model to over-rely on that guidance.
F Details of Efficiency Analysis
IC-Seg’s hierarchical supervision reduces redundant interaction while preserving segmentation quality, with modest additional training cost.
- Removing turn-level or token-level optimization increases inference turns while segmentation performance drops.The reported inverse trend indicates that multi-level supervision helps prevent redundant questioning loops and supports discriminative questions.
- 295s to 333.5s: an extra forward pass raises training step time by approximately 13%.The extra pass bypasses time-consuming text generation.
- The non-interactive Qwen3-VL-4B baseline is faster at inference but obtains the worst performance.It directly identifies and segments the target without resolving the user’s intent.
- Table 12 reports average inference turns, per-sample processing time, and average per-step training time.“Turns” measures average query turns, “Time” measures seconds per sample, and “Training cost” measures seconds per training step.
G More Qualitative Analysis
Qualitative analyses contrast arbitrary selection and repetitive questioning with IC-Seg’s targeted clarification process for resolving ambiguous video targets.
- Qualitative comparison: Qwen3-VL-4B* directly selects the object it considers most suitable when the query is ambiguous.Its example selects the central figure and produces an incorrect localization.
- Qualitative comparison: Qwen3-VL-4B*+GRPO asks clarifying questions but falls into repetitive questioning loops and reaches the maximum number of turns.The example repeats position and motion questions without resolving the target.
- Qualitative comparison: IC-Seg asks targeted position and action questions to identify the intended person instead of relying on an initial visual judgment.In the example, it confirms the right-side person and then asks whether that person is spinning, jumping, or walking.
- Prompt design: The interaction protocol requires analyzing candidates, extracting distinguishing features, and asking a question that isolates them.Supported attributes include color, direction, action, relative position, and shape.
- Prompt design: After the target is uniquely identified, the system selects a clear frame and outputs a bounding box with an interior point.The prompt separates frame selection from high-resolution target localization.
- Prompt design: The user simulator tracks a red-contoured target across indexed frames and answers questions about its appearance, motion, and state changes.Its rules require focusing only on the contoured object and requesting clarification when an answer is ambiguous.
J Limitations and Future Work
IC-Seg’s current interaction is primarily text-based, while the paper identifies multimodal interaction as a direction for more natural ambiguity resolution and human-AI collaboration.
- IC-Seg primarily supports text-based interaction, limiting how users can convey intent in real-world scenarios.The paper suggests pointing gestures and audio-visual cues as possible extensions.
- Multimodal interaction could make clarification more intuitive and efficient by incorporating pointing gestures or audio-visual cues.This is presented as future research rather than a demonstrated capability of the current system.
- The paper connects asking rather than guessing with safer and more reliable interactive AI in assistive robotics and autonomous navigation.It also describes reduced pressure on users to provide precise initial prompts.