Source-linked AI summary
DRAgent: Discriminative Reasoning Agent for Referring Expression Segmentation
Yujie Qi, Luyan Zhang
TL;DR
MLLM-based RES methods can suffer from errors when generating localization coordinates as discrete text tokens. DRAgent uses detector-generated candidates and two-stage MLLM discrimination, then segments the selected box; it achieves competitive results across RefCOCO, RefCOCO+, and RefCOCOg, with strongest reported performance on RefCOCOg.
Problem
One-pass MLLM coordinate generation can lose spatial continuity and cause localization bias, alignment errors, and confusion with distractors in complex scenes.
Method
DRAgent uses an open-vocabulary detector to create candidates, applies MLLM screening and instance-wise verification, and feeds the selected box to SAM for segmentation.
Results
DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg, reaching 76.78% on RefCOCOg val(U) and 77.54% on test(U).
Takeaways & Limitations
The results support using the MLLM as a discriminative target selector over detector-generated candidates rather than as a coordinate generator.
Abstract
from arXiv · showhide
Referring Expression Segmentation (RES) aims to generate a pixel-level mask for the object specified by a language expression. Recent methods based on multimodal large language models (MLLMs) often rely on one-pass coordinate prediction for visual localization, which serializes continuous spatial locations as discrete text tokens and may lead to localization bias and alignment errors. To address these issues, we propose DRAgent, an MLLM-driven discriminative reasoning (DR) framework for RES. Instead of requiring the MLLM to generate localization coordinates, DRAgent first constructs a detector-generated candidate space and then uses the MLLM as a visual-semantic target discriminator. Specifically, the MLLM performs reliable target selection among potential distractors through a two-stage DR mechanism, which first screens high-recall candidates and then performs instance-wise verification. The selected target box is subsequently used as a spatial prompt for a foundation segmentation model to produce the final pixel-level mask. Furthermore, we construct a self-consistency-filtered reasoning-chain data pipeline for LoRA-based fine-tuning, providing more reliable supervision for enhancing the MLLM's discriminative reasoning capability. Experiments demonstrate that DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg.
I. INTRODUCTION
RES requires pixel-level masks for language-specified objects, but MLLM-based one-pass coordinate generation can introduce localization and alignment errors. DRAgent addresses this by selecting targets from detector-generated candidates through discriminative reasoning and reports competitive benchmark performance.
- Motivation: RES generates a pixel-level mask for the object specified by a natural language expression.Precise identification supports subsequent task execution in autonomous systems.
- Motivation: MLLM-based methods use high-level semantic reasoning to generate spatial prompts for foundation segmentation models such as SAM.These prompts guide downstream mask generation.
- Limitations: Autoregressively serialized coordinate tokens can lose spatial continuity and increase coarse or biased localization.In dense or visually similar scenes, direct one-pass localization can confuse the referent with distractors and propagate incorrect prompts to segmentation.
- Approach: DRAgent reformulates localization as discriminative target selection, using screening and instance-wise verification over detector-generated candidates instead of one-pass coordinate generation.The selected bounding box is used as a spatial prompt for pixel-level segmentation.
- Approach: DRAgent introduces a two-stage DR mechanism and a self-consistency-filtered reasoning-chain pipeline for LoRA-based supervised fine-tuning.The two components improve target discrimination from inference and training perspectives, respectively.
- Results: DRAgent achieves competitive performance on RefCOCO, RefCOCO+, and RefCOCOg benchmarks.The contribution is evaluated across all three benchmarks.
A. MLLMs for RES
RES research progressed from task-specific alignment architectures to MLLM-based systems coupled with foundation segmentation models. DRAgent instead assigns the MLLM a discriminative target-selection role over detector-generated candidates.
- Earlier RES Methods: Early RES methods use language-aware vision transformers, relation-aware cross-modal modeling, or sequential polygon generation.These architectures still struggle with fine-grained expression-region alignment in complex scenes.
- MLLM-Based RES: Recent MLLM-based RES frameworks improve open-vocabulary semantic understanding and cross-modal modeling, often connecting MLLMs with SAM or mask decoders.Examples include LISA, GSVA, and OpenWorldSAM.
- Localization Generators: Visual localization generator methods produce boxes or points as prompts and may iteratively refine masks through external segmentation tools.SAM-Veteran and SegAgent exemplify multi-round or interactive localization prompting.
- DRAgent: DRAgent uses an open-vocabulary detector to generate candidates while the MLLM performs target discrimination through screening and instance-wise visual verification.This decouples candidate generation from target discrimination and avoids direct coordinate decoding.
III. METHOD
DRAgent decomposes RES into candidate detection, discriminative target selection, and pixel-level segmentation. Its MLLM selects the referred candidate through two-stage reasoning, with self-consistency-filtered data supporting fine-tuning.
- Overview: DRAgent divides RES into candidate object detection, discriminative target selection, and pixel-level segmentation.The MLLM handles target selection, while the final selected target is converted into a mask.
- Training: DRAgent further enhances discriminative reasoning through self-consistency-filtered reasoning-chain data for LoRA-based fine-tuning.The data pipeline is part of the framework’s training design.
- Candidate Object Detection: Candidate detection constructs a high-recall discrete decision space from the query and an open-vocabulary detector.The query category is extracted before candidate objects are detected.
- Discriminative Target Selection: The MLLM selects the candidate best matching the query through screening followed by instance-wise visual verification.This replaces direct coordinate generation with discriminative selection.
- Pixel-Level Segmentation: SAM receives the selected target as a spatial prompt and generates the corresponding pixel-level mask.The segmentation stage maps the final target decision back to pixel space.
B. Two-Stage Target Selection
DRAgent stabilizes target selection by cascading candidate screening with instance-wise visual verification. It replaces direct coordinate generation with relative visual-semantic evidence from constrained answer-token logits.
- Two-stage discriminative reasoning: The MLLM first screens candidates, then verifies each retained instance to reduce distractor confusion in complex scenes.This cascaded design narrows the search space before finer visual-semantic comparison.
- Two-stage discriminative reasoning: Candidate verification asks whether each red-boxed region tightly encloses the object described by the query.
- Verification scoring: The verification score compares the relative strength of positive and negative answer-token sets instead of relying on direct generative decoding.The score is intended as a more stable discriminative signal when answer-token probabilities are ambiguous.
- Verification scoring: Positive and negative token sets contain case variants of “Yes” and “No,” respectively, while their average logits define candidate-specific evidence.
- Target selection: The selected candidate is the one with the highest verification score, which ranks visual-semantic matching rather than estimating calibrated probability.
C. Data Pipeline and Training Strategy
DRAgent constructs training supervision by generating structured reasoning, descriptions, and decisions, then filtering chains through closed-loop self-verification. This targets reasoning that is visually grounded rather than merely linguistically plausible.
- Data pipeline: The pipeline filters reasoning chains that are linguistically plausible but not grounded in correct visual target discrimination.
- Stage 1: Self-consistency generation: Stage 1 renders either the referred target or a hard negative and elicits a structured reasoning chain, visual description, and Yes/No answer.
- Stage 2: Self-verification: Stage 2 reconstructs a verification query from the generated description and feeds it back to the MLLM for closed-loop verification.
A. Experimental Settings
The experiments compare DRAgent with non-MLLM, implicit MLLM, and explicit localization-generator RES methods on standard benchmarks. DRAgent is competitive overall and reaches its strongest reported results on RefCOCOg.
- Experimental settings: DRAgent uses Grounding DINO for candidate detection, Qwen3-VL-8B-Instruct for target discrimination, and SAM for pixel-level segmentation.LoRA is applied to the visual and language layers during fine-tuning.
- Compared methods: The comparison covers non-MLLM methods, implicit MLLM fusion methods, and MLLMs used as explicit visual localization generators.
- Main results: DRAgent remains competitive on RefCOCO and RefCOCO+ but does not consistently surpass the strongest methods there.
- Evaluation: Table I reports cIoU (%) and highlights the best and second-best results across RES benchmarks.
C. Ablation Study
The ablation study shows that both inference-time two-stage discriminative reasoning and training with filtered reasoning chains improve target selection. Their combination yields the strongest RefCOCOg result.
- 75.42% average cIoU follows the introduction of two-stage DR, versus 66.84% for the single-stage baseline on RefCOCOg.The mechanism narrows candidates before instance-wise verification, reducing irrelevant-candidate interference during visual-semantic matching.
- 72.51% average cIoU from SFT without CoT supervision is lower than the two-stage DR variant, indicating limited guidance from final-answer supervision alone.Final-answer supervision provides limited information about comparing candidates and making discriminative decisions.
- 77.16% average cIoU is achieved by full DRAgent with self-consistency-filtered reasoning-chain data, the best result among the ablations.Filtering retains reasoning chains whose decisions align with visual evidence, producing more reliable supervision.
D. Analysis
The analysis examines candidate-space and overlap-compensation settings, alongside qualitative segmentation examples. It identifies settings that balance target recall with distractor suppression and reports more accurate DRAgent masks in challenging cases.
- Candidate count: Average cIoU increases as the initially selected candidate count K grows from 2 to 4, improving target recall and enriching the verification decision space.
- Qualitative results: Qualitative examples compare referring text, input images, baseline masks, DRAgent masks, and ground truth across challenging RES scenarios.The caption states that DRAgent achieves more accurate segmentation consistent with ground truth.
- Candidate count: K = 4 achieves the best performance, whereas larger candidate sets slightly reduce average cIoU by adding irrelevant or visually confusable candidates.
- Overlap compensation: γ = 60% provides the best trade-off between candidate recall and noise suppression in overlap compensation.Increasing γ from 40% to 60% improves average cIoU, but further increases reduce performance by retaining fewer relevant candidates.