PaperlayerView exact revision on arXiv

Source-linked AI summary

SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation

Linnan Zhao, Xu Liu, Lingling Li, Licheng Jiao, Fang Liu, Wenping Ma

arXiv:2609.08867v1cs.CV

TL;DR

Reasoning segmentation must identify the referent of an indirect description and locate it precisely, while existing interfaces often couple those signals. SeGDeP separates semantic and geometric prompts and trains their executable interface before policy optimization. It reports results across RefCOCO-family and ReasonSeg benchmarks, with small and crowded targets remaining a limitation.

  • Problem

    Existing MLLM-segmenter interfaces use an implicit trigger or coupled context for semantic identification and spatial grounding, despite these requiring different evidence and supervision.

    Source: Introduction · p. 1

  • Method

    SeGDeP converts resolved MLLM states into separate semantic prompts and geometric box predictions, which jointly condition a mask decoder.

    Source: Introduction · p. 1, Problem Formulation and Overview · p. 2

  • Results

    SeGDeP-4B reports 82.7 average cIoU across eight RefCOCO-family splits and 66.0/59.6 gIoU on ReasonSeg val/test.

    Source: Introduction · p. 1

  • Takeaways & Limitations

    Gradient diagnostics and prompt interventions support complementary semantic and geometric roles, while small, crowded targets remain a residual localization challenge.

    Source: Conclusion · p. 7

    The current single-query head predicts one referred mask and cannot emit or rank independent instance hypotheses for genuinely multi-target instructions.

    Source: D Limitations and Future Work · p. 12

Abstract

from arXiv · show

Reasoning segmentation converts an implicit linguistic conclusion into a precise mask, requiring both semantic identification and spatial grounding. Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context, although they receive different supervision and fail differently. This coupling obscures whether a failure arises from target interpretation or from localization. We present SeGDeP, an explicit what-where interface. A semantic prompt branch and an independent geometric projection path transform resolved MLLM states into semantic features and a DETR-predicted box, which jointly condition a SAM 3 mask decoder. Training first aligns this executable interface, then uses group reward-decoupled policy optimization (GDPO) to balance format, box-IoU, and mask-IoU feedback. SeGDeP-4B reaches 82.7 average cIoU over eight RefCOCO-family splits and 66.0/59.6 gIoU on ReasonSeg val/test while adapting only 0.38% of Qwen3-VL parameters through LoRA. Controlled stage-wise ablations, gradient diagnostics, and prompt interventions further show that the two paths develop complementary semantic and geometric specialization rather than duplicating the same evidence.

Introduction

Reasoning segmentation must resolve what an indirect description refers to and where that referent lies, but existing interfaces couple these decisions. SeGDeP separates semantic and geometric prompts, and its reported results span RefCOCO-family and ReasonSeg benchmarks.

  • Reasoning segmentation must identify objects from indirect attributes, functions, or relations and ground them as pixel masks.Source: Introduction · p. 1
  • Trigger and coupled-context interfaces leave semantic identity and spatial support in one implicit or coupled representation.Source: Introduction · p. 1, Figure 1 · p. 1
  • In a shared-context model, 37–46% of samples produce opposing semantic and geometric gradients.Source: Introduction · p. 1
  • SeGDeP separates role-typed semantic prompts from an independent geometric projection, then jointly uses them to condition localization and mask decoding.Source: Introduction · pp. 1–2
  • 82.7 average cIoU across eight RefCOCO-family splits and 66.0/59.6 gIoU on ReasonSeg val/test are reported for SeGDeP-4B.Source: Introduction · p. 1

Related Work

Prior work addresses referring segmentation, reasoning interfaces, and policy optimization, but many reasoning-segmentation interfaces still couple semantic identity with spatial support. SeGDeP instead translates resolved reasoning into separate text and box prompts.

Method

SeGDeP translates resolved MLLM states into separately parameterized semantic and geometric paths, then recombines their prompts for mask prediction. Its two-stage training first aligns the executable interface and then optimizes reasoning with decoupled reward feedback.

Problem Formulation and Overview

Decoupled Prompt Connector

Two-Stage Optimization

  • Stage 1 freezes the MLLM and SAM 3 components while aligning prompt paths and localization modules to executable text and box prompts.Source: Two-Stage Optimization · p. 3
  • Stage 2 activates MLLM LoRA and updates the connector and mask decoder using component-wise standardized format, box, and mask rewards alongside supervised geometric and mask objectives.Source: Two-Stage Optimization · pp. 3–4

Experiments

SeGDeP leads the RefCOCO-series and ReasonSeg comparisons, while controlled ablations and diagnostics examine prompt-path separation, reward aggregation, transfer, and residual errors.

Setup

  • Evaluation uses official RefCOCO-series and ReasonSeg splits, with GSEval held out for zero-shot transfer and cIoU used for the RefCOCO comparison.The main model pairs Qwen3-VL-4B-Instruct with SAM 3; Stage 2 adds ReasonSeg and activates LoRA while keeping the base MLLM and image backbone frozen.Source: Setup · p. 4

Ablation Studies and Diagnostic Analysis

  • With foundation models frozen, the decoupled interface reaches 56.4 versus 53.1 gIoU after Stage 1; after Stage 2, it reaches 66.0 versus 64.3 for the coupled interface.Negative semantic–geometric gradient cosines occur in 37.0%, 46.0%, and 44.5% of samples across RefCOCO val, ReasonSeg val, and GSEval, despite a 0.95 gradient-norm ratio.Source: Ablation Studies and Diagnostic Analysis · pp. 4–5

Main Results

  • 82.7 average cIoU across eight RefCOCO splits puts SeGDeP-4B first on seven splits; it improves over LENS by 3.4 points on RefCOCO+.The average per-image gIoU is 83.2, and the largest cIoU margin is +5.3 on RefCOCO+ testB.Source: Main Results · p. 4, Table 1 · p. 5
  • On ReasonSeg, SeGDeP ranks first on all four metrics, exceeding LENS by 3.9/0.4 validation gIoU/cIoU and 2.4/0.8 test gIoU/cIoU points.The larger gIoU gains indicate improvement distributed across examples, and test gains under both aggregations extend beyond validation.Source: Main Results · p. 4, Table 2 · p. 5

Ablation Studies and Diagnostic Analysis

  • Learned text and box translation reach 63.1 and 63.8 gIoU versus 58.6 and 62.3 for direct outputs; combining them reaches 66.0.Ground-truth boxes add 4.8/7.0 cIoU, while hard-negative boxes reduce it by 83.0/81.0 points; semantics also contributes before box prediction and during mask conditioning.Source: Ablation Studies and Diagnostic Analysis · pp. 5–6, Table 3 · p. 5, Table 4 · p. 6
  • RefCOCOg examples show the semantic branch preserving attributes, actions, and relations while the predicted box selects a compact region for mask decoding.The examples include thin skis and a fork, where modest box shifts can remove foreground pixels, and larger people or dogs competing with distractors.Source: Ablation Studies and Diagnostic Analysis · p. 6, Figure 3 · p. 6

Additional Empirical Studies

  • On ReasonSeg validation, component-normalized GDPO reaches 66.0 gIoU, a 2.3-point gain over summed-reward GRPO with other Stage-2 settings fixed.Format success rises from 96.0% to 97.5%, while completion length falls from 367 to 318 tokens and low-IoU formatted outputs fall from 26.2% to 21.5%.Source: Additional Empirical Studies · p. 6
  • Without GSEval training or model selection, SeGDeP reaches 68.4 gIoU versus LENS at 67.0, while its pooled cIoU is 75.2 versus 78.3.On RefCOCOg, small targets and crowded scenes have lower box/mask IoUs than the overall 81.51/80.88; the analysis identifies localization as the dominant residual bottleneck.Source: Additional Empirical Studies · pp. 6–7

Conclusion

SeGDeP uses decoupled semantic and geometric branches to produce text and box prompts for SAM 3, with GDPO aligning reasoning to localization and mask quality.

  • The paper concludes that consistent gains across RefCOCO and ReasonSeg, supported by matched ablations, gradient diagnostics, and prompt interventions, validate the explicit what–where interface.Errors on small, crowded targets motivate stronger localization and multi-instance support.Source: Conclusion · p. 7

A Training and Implementation Details

Training first aligns the executable prompt interface, then adapts the reasoning model and mask decoder with reinforced feedback while retaining supervised interface losses.

  • Stage 1 freezes Qwen3-VL and SAM 3 image, text, and mask modules while training the prompt interface and coarse-to-fine localization path on RefCOCO, RefCOCO+, and RefCOCOg.It uses AdamW for eight epochs with global batch size 128 and cosine decay.Source: A Training and Implementation Details · p. 10, Table A1 · p. 10
  • The semantic extractor uses 64 queries and a 32-token role-typed prompt bank, while projected context enters DETR memory for coarse-box prediction and ROI-feature refinement.The geometric path is supervised by box regression and GIoU; the semantic branch uses valid-token distillation, with padding excluded.Source: A Training and Implementation Details · p. 10
  • Stage 1 excludes ReasonSeg and GSEval and applies no segmentation loss; ground-truth boxes come from masks, and the frozen SAM 3 text encoder supplies training targets only.Source: A Training and Implementation Details · p. 10
  • Stage 2 adds ReasonSeg to the RefCOCO mixture and updates the prompt interface and SAM 3 mask decoder; the serialized JSON box is context, while DETR and the ROI refiner produce the executable box.Qwen is adapted with 17M LoRA parameters, 0.38% of the 4B backbone, while base weights remain frozen.Source: A Training and Implementation Details · p. 10
  • Format, box, and mask rewards are equally weighted and normalized component-wise, while supervised box and mask losses remain active during reinforcement learning.The 0.38% adaptation figure applies only to Qwen3-VL; the prompt interface and mask decoder are also trainable.Source: A Training and Implementation Details · p. 10
  • Increasing the semantic context-slot budget from 16 to 64 raises RefCOCO val cIoU from 72.2 to 76.6, with no further gain at 128 slots.The ablation varies semantic extraction capacity while keeping the 32-token prompt bank and geometric path fixed.Source: A Training and Implementation Details · p. 10, Table A3 · p. 10

Shared-Bottleneck Gradient Diagnostic

At a deliberately shared context, semantic and geometric supervision often request opposing updates despite comparable average gradient scales. The diagnostic recurs across three datasets, but is scoped to the controlled coupled bottleneck.

Shared-Bottleneck Gradient Diagnostic

  • The diagnostic separately backpropagates semantic and geometric losses to the same upstream context, then measures their gradient compatibility.The semantic loss is masked distillation; the geometric loss contains box-regression and GIoU terms.Source: Shared-Bottleneck Gradient Diagnostic · p. 10
  • 37–46% of samples show opposing gradients across three datasets, while mean cosine stays near zero and norm balance is 0.95.The diagnostic uses the controlled coupled baseline; Table A4 reports the per-dataset values.Source: Table A4 · p. 11, Shared-Bottleneck Gradient Diagnostic · p. 11
  • Negative-gradient-rate intervals recur on RefCOCO, ReasonSeg, and held-out GSEval, but characterize this controlled bottleneck rather than all coupled connectors.The 95% intervals are 31.5–45.0%, 39.0–52.5%, and 37.5–51.5%, respectively.Source: Shared-Bottleneck Gradient Diagnostic · p. 11

B Additional GDPO Details

The GDPO details describe a soft format reward and component-wise advantage normalization over completions sampled for the same input. Normalization separates relative reward signals before aggregation.

B Additional GDPO Details

  • The format reward gives one point to each of the first four valid QA pairs, then halves credit for each later pair.A separate final-answer point checks parseability rather than target correctness, discouraging repetition without a hard reasoning-length cutoff.Source: B Additional GDPO Details · p. 11

Decoupled Advantage Estimation

  • GDPO computes each component's reward statistics over completions sampled for the same input, then detaches the aggregated advantage before policy log-probabilities.This changes credit assignment among observed trajectories without making the format or IoU evaluators trainable.Source: Decoupled Advantage Estimation · p. 11
  • Component-wise normalization makes updates approximately invariant to positive rescaling of an individual reward source, except when its group variance is zero.Raw format rewards are typically four to five times either box or mask rewards; a constant-valued component contributes no relative preference for that prompt.Source: Decoupled Advantage Estimation · p. 11

Token-Level Objective with KL Regularization

The token-level objective uses a reference-policy KL constraint and averages token contributions within each completion before averaging across the group. On ReasonSeg val, component-normalized GDPO improves visual outcomes over summed-reward GRPO while producing shorter traces.

Token-Level Objective with KL Regularization

  • The token policy-gradient term uses a non-negative k3 KL estimator that is zero when the policy matches its reference and grows as sampled-token probabilities diverge.The estimator supplies a stable local constraint.Source: Token-Level Objective with KL… · p. 11
  • Averaging valid completion tokens within each trace before averaging the group gives sampled trajectories equal mass rather than favoring longer traces.The objective combines the GDPO term with box and mask losses.Source: Token-Level Objective with KL… · p. 11
  • 2.3 points higher gIoU than summed-reward GRPO accompanies 1.5 points higher format success and fewer formally valid, low-IoU traces.Mean QA pairs decrease from 4.92 to 4.63, completion length from 367 to 318 tokens, and low-IoU traces from 26.2% to 21.5%.Source: Token-Level Objective with KL… · pp. 11–12, Table B1 · p. 12

C Additional Diagnostics and Transfer Results

The additional diagnostics test channel-specific prompt interventions, held-out transfer, and inference cost. Transfer comparisons are end-to-end because the compared model foundations are not scale matched.

C Additional Diagnostics and Transfer Results

  • Prompt interventions use the same same-category, visually similar distractor and replace only the semantic prompt or the box under test.The ground-truth-box row measures mask-decoding headroom after localization correction; the SAM 3-text row substitutes a frozen text-encoder feature.Source: C Additional Diagnostics and Transfer… · p. 12
  • SeGDeP leads LENS by 1.4 gIoU on held-out GSEval, while LENS retains a 3.1-point advantage in pooled cIoU.GSEval is excluded from training and model selection; the comparison measures end-to-end transfer, not connector effects in isolation.Source: Table C1 · p. 12, C Additional Diagnostics and Transfer… · p. 12
  • Context extraction and the connector take 2.8 ms, or 0.14% of SeGDeP's 2.0009-second single-image total.The 0.8791-second total latency difference from LENS is almost completely accounted for by unmatched MLLM and segmenter foundations.Source: Table C2 · p. 12, C Additional Diagnostics and Transfer… · p. 12
  • SeGDeP's context-plus-connector interface has 318.5M resident parameters versus 474.8M for LENS, a 32.9% reduction.These parameter counts are resident, not trainable; they are distinct from the 17M LoRA figure.Source: C Additional Diagnostics and Transfer… · p. 12
  • The latency measurements support component-level attribution under a single-image protocol, not estimates of training cost, peak memory, or batched throughput.Those quantities depend on scheduling and caching.Source: C Additional Diagnostics and Transfer… · p. 12

D Limitations and Future Work

SeGDeP’s clearest limitations concern difficult small or occluded targets and the current single-query design, which cannot provide independently ranked instance hypotheses.

D Limitations and Future Work

  • Very small, thin, or heavily occluded targets remain SeGDeP’s clearest limitation.The geometric head depends on spatial detail preserved by the frozen SAM 3 backbone.Source: D Limitations and Future Work · p. 12
  • Box IoU for 5–10% targets is 7.8 points below the overall value, and scenes with more than ten objects show a 4.8-point drop.Higher-resolution features and stronger ROI refinement may improve localization and boundary fidelity, but increase memory and latency.Source: D Limitations and Future Work · p. 12
  • The current single-query head can produce one mask with disconnected components but cannot emit or rank independent instance hypotheses.Supporting genuinely multi-target instructions requires multiple coordinated queries, instance-level supervision, and evaluation.Source: D Limitations and Future Work · p. 12

E Additional Qualitative Examples

The examples make SeGDeP’s reasoning-to-mask interface inspectable across functional, causal, role-specific, and set-level requests. They also distinguish semantic resolution from geometric support and mask quality.

E Additional Qualitative Examples

  • The trace–box–mask presentation separates semantic failure, geometric translation failure, and mask-decoding error.These are selected successful cases, not an estimate of failure frequency; quantitative slices and hard-negative interventions measure how often stages become limiting.Source: E Additional Qualitative Examples · p. 13
  • The QA trace, final answer, and displayed box expose visual evidence, resolved identity, and the spatial support available to the mask decoder.The visualizations serve as an interface audit, not proof that every generated trace sentence is causally necessary.Source: E Additional Qualitative Examples · p. 13
  • Across the examples, the tested demands range from preserving object scale and extent to role-specific identity and set membership.The cases include a tiny fixture, a complete appliance, an elongated causal object, same-category role selection, and a disconnected set.Source: E Additional Qualitative Examples · p. 13
  • The heat-source and weightlifting examples require selecting the complete stove and barbell, not merely locally relevant flames or a bar center.The long horizontal barbell support tests whether box refinement preserves the plates for mask decoding.Source: E Additional Qualitative Examples · p. 14
  • For the goalkeeper and pigs, the requested output is respectively one role-specific instance and one set-valued region.The pig mask may contain disconnected components because one query can represent the union implied by a shared state.Source: E Additional Qualitative Examples · p. 14

Related papers

Loading 2609.08867v1…