PaperlayerView exact revision on arXiv

Source-linked AI summary

ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentation

Ruicheng Zhang, Kaiwen Shen, Jiaqi Hou, Shuhan Yang, Junchao Huang, Kewei Zhang, Jun Zhou, Li Jiang, Shen Zhao

arXiv:2610.12107v1cs.CV

TL;DR

GRES must identify variable numbers of language-specified objects while preserving precise boundaries, but cascaded, single-pass methods limit adaptive perception and geometric correction. ContourVLA uses evolving contours to guide closed-loop multimodal perception and geometric actions, and outperforms the compared methods on the reported gRefCOCO gIoU and RES mIoU results.

  • Problem

    GRES requires identifying variable numbers of language-specified objects while delineating precise boundaries, demands that static feature interfaces and single-pass prediction do not adaptively support.

    Source: §1 · p. 2

  • Method

    ContourVLA uses editable contours as evolving policy states that condition multimodal perception and geometric actions in a closed loop.

    Source: §1 · pp. 2–3

  • Results

    ContourVLA achieves the highest gIoU among compared methods on all three gRefCOCO splits, by 2.7–8.7 points, and the highest mIoU on all eight RES splits.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Ablations support the benefits of re-observation and instance-level credit, highlighting the value of coupling adaptive perception with explicit geometric refinement.

    Source: §5 · p. 10

    An initially missed referent cannot be recovered through later contour updates because the rollout does not add instances or regenerate boxes.

    Source: §A.11 · pp. 29–30

Abstract

from arXiv · show

Generalized referring expression segmentation (GRES) requires dynamically balancing high-level semantics for identifying a variable number of language-specified referents with fine-grained visual evidence for precise boundary delineation. This requirement challenges existing cascaded vision-language architectures, which typically rely on static feature interfaces and single-pass mask prediction, limiting adaptive perception and geometric correction. We introduce ContourVLA, a vision-language-action policy that recasts GRES as a closed-loop visuomotor process, in which editable contours serve as explicit policy states that condition multimodal perception and are updated by geometric action chunks. Evolution-Aware Semantic Scheduling (EASS) couples contour-guided bidirectional boundary sampling with state-conditioned routing of multilevel multimodal features, adapting perception to each contour state. Following supervised initialization, Dustbin-Augmented Entropic Credit Transport GRPO (DECT-GRPO) jointly optimizes discrete grounding and continuous contour actions with instance-level credits. Its rollout rewards and credits are derived from soft prediction-target correspondences that account for false positives and missed targets. ContourVLA improves gIoU over the strongest evaluated baselines by 8.7, 2.8, and 2.7 points on gRefCOCO val, testA, and testB, respectively, and achieves the highest mIoU across all eight RefCOCO, RefCOCO+, and RefCOCOg splits.

1 INTRODUCTION

GRES requires balancing semantic identification of variable numbers of referents with precise boundary evidence, while cascaded, single-pass systems limit adaptive perception and geometric correction. ContourVLA instead uses evolving contours to condition perception and geometric actions in a closed loop.

1 INTRODUCTION

  • Static feature interfaces and single-pass mask prediction limit adaptive evidence acquisition and geometric refinement for GRES.GRES must balance high-level semantics for identifying referents with fine-grained visual evidence for precise boundaries.Source: §1 · pp. 1–2
  • ContourVLA frames GRES as a closed-loop process in which editable contours condition multimodal perception and evolve through geometric actions.The policy iteratively converts observations into action chunks that update contours, progressing from coarse localization to precise boundaries.Source: §1 · pp. 2–3
  • ContourVLA uses supervised fine-tuning to establish grounding and contour-trajectory priors, followed by joint GRPO post-training with self-rollout feedback.Its perception expert grounds referents and supplies observations, while its action expert predicts geometric action chunks.Source: §1 · p. 2
  • EASS adapts perception to contour state through spatial boundary sampling and routing of multilevel multimodal features.Its spatial and semantic components support adaptive boundary-cue acquisition and state-conditioned feature selection.Source: §1 · pp. 2–3
  • DECT-GRPO assigns instance-level credits to grounding tokens and contour actions using dustbin-augmented transport that accounts for false positives and missed targets.It couples module-specific likelihood optimization with soft prediction-target correspondences.Source: §1 · pp. 2–3

2 RELATED WORK

Prior GRES work ranges from cross-modal fusion to pretrained VLMs connected to segmentation decoders, while contour and reinforcement-learning methods address related but distinct pieces of the task. ContourVLA combines evolving contour states with adaptive multimodal perception and instance-level credit assignment.

Generalized Referring Expression Segmentation

  • Recent GRES methods connect pretrained VLMs to segmentation decoders, but fixed readouts and single-pass prediction limit adaptive evidence selection and iterative correction.Earlier approaches used cross-modal fusion between separate visual and language encoders (Yang et al., 2022; Liu et al., 2023a; Luo et al., 2025).Source: §2 · p. 3

Contour Evolution and Vision-Language-Action Policies

  • DeepSnake and related methods refine contours with local visual features, while PolyFormer predicts language-conditioned vertices; these methods do not adapt multimodal evidence to evolving GRES contours.ContourVLA instead conditions geometric action prediction on vision and language in a closed loop (Kim et al., 2024).Source: §2 · p. 3

Reinforcement Learning for Vision-Language Reasoning and Grounding

  • Seg-Zero and SAM-R1 apply GRPO to segmentation and grounding, but standard GRPO shares rollout-level advantages rather than distinguishing instance quality.MCR-GRPO estimates instance importance through marginal contribution; DECT-GRPO uses dustbin-augmented transport for instance-level credits (Han et al., 2026).Source: §2 · p. 3

3 METHOD

ContourVLA formulates GRES as a closed-loop visuomotor process: editable contours act as geometric policy states that condition perception and evolve through actions. Its training uses supervised initialization followed by DECT-GRPO post-training with instance-level credit assignment.

3.1 STATE-CONDITIONED PERCEPTION-ACTION LOOP

  • At each iteration, EASS builds contour-conditioned visual and semantic observations that GAD uses to predict geometric action chunks updating the contour state.VLIM first predicts grounding boxes to initialize contours and provides cached visual and multilevel multimodal features.Source: §3 · p. 3, §3.1 · p. 4

3.2 EVOLUTION-AWARE SEMANTIC SCHEDULING

  • EASS adapts visual evidence acquisition and multimodal feature selection to the evolving contour state throughout refinement.Its spatial and semantic axes provide the two forms of state-conditioned perception.Source: §3.2 · p. 4

3.2.1 SPATIAL VISUAL SAMPLING

  • Spatial Visual Sampling expands the receptive field along contour-vertex normals, sampling bidirectional interior and exterior visual features.Box-relative scaling adapts the receptive field to object scale.Source: §3.2.1 · pp. 4–5

3.2.2 MULTIMODAL SEMANTIC ROUTING

  • Multimodal Semantic Routing selects VLIM features using queries that encode instance characteristics, contour state, evolution stage, and decoder depth.Routing is recomputed after contour updates so the semantic abstraction can adapt during evolution.Source: §3.2.2 · p. 5

3.3 GEOMETRIC ACTION CHUNKING

  • GAD combines contour-conditioned visual evidence, contour geometry, instance context, and routed semantics to predict parallel two-dimensional vertex displacements.The displacements update the contour and trigger the next iteration.Source: §3.3 · p. 5

3.4 POLICY OPTIMIZATION WITH DECT-GRPO

  • Training first initializes VLIM with teacher-forced grounding and GAD with reference-contour positional supervision, then samples joint grounding and continuous-action rollouts for DECT-GRPO optimization.The rollout policy uses autoregressive categorical grounding outputs and bounded continuous geometric actions.Source: §3.4 · p. 6
  • Dustbin-augmented entropic transport creates globally consistent soft prediction–target correspondences, with null matches accounting for false positives and missed targets.Transport-derived instance credits modulate shared rollout feedback at grounding-token spans and contour-action positions.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

ContourVLA leads the evaluated GRES and RES benchmarks, and ablations support the contributions of EASS re-observation and DECT-GRPO instance-level credit. The experiments also compare contour evolution and latency under full versus frozen observations.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers GRES on gRefCOCO and RES on RefCOCO, RefCOCO+, and RefCOCOg; one jointly trained checkpoint is evaluated across benchmark splits.The reported metrics are gIoU and cIoU for GRES, and mIoU and cIoU for RES.Source: §4.1 · p. 7

4.2 COMPARISON WITH EXISTING METHODS

  • 83.5, 80.3, and 74.4 gIoU on gRefCOCO val, testA, and testB exceed the strongest evaluated baselines by 8.7, 2.8, and 2.7 points, respectively.ContourVLA also has the highest cIoU on val and testA; its testB cIoU of 71.8 is 0.1 point below HiMTok.Source: Table 1 · p. 8, §4.2 · p. 8
  • ContourVLA achieves the highest mIoU on all eight evaluated RES splits, with gains over strongest baselines of 1.3–1.8 points on RefCOCO, 2.8–3.2 on RefCOCO+, and 0.7 and 0.3 on RefCOCOg val and test.The same checkpoint is used across benchmarks without further benchmark-specific fine-tuning.Source: Table 1 · p. 8, §4.2 · p. 8
  • Qualitative examples show recovery of both person-and-umbrella referents, delineation of a truck's specified front portion, and an empty prediction for an absent target.The examples illustrate fewer omitted referents and less over-segmentation while preserving fine boundaries.Source: §4.2 · p. 8, Figure 4 · p. 9

4.3 ABLATION STUDIES AND ANALYSIS

  • With identical grounding predictions and initial contours, combining SVS and MSR raises gIoU from 71.1 to 77.6 and cIoU from 60.8 to 67.4 on gRefCOCO val.Freezing initial EASS readouts while contours continue to evolve reduces performance to 72.0 gIoU and 61.3 cIoU.Source: §4.3 · p. 9, Table 2 · p. 9
  • Full DECT-GRPO gains a further 3.0 gIoU and 5.1 instance F1@0.5 points over shared-advantage GRPO, and 1.3 gIoU and 2.0 F1@0.5 points over hard-assignment credit.Removing dustbins lowers all four reported metrics, particularly no-target accuracy and instance F1@0.5.Source: §4.3 · p. 9, Table 3 · p. 9
  • At the matched eight-step budget, Full EASS improves Boundary IoU by 6.07 points over frozen observations, from 59.91% to 65.98%, while adding 6.67 ms (2.94%) contour latency.Figure 5(b) visualizes Full EASS evolution from initial contours to final masks.Source: §4.3 · p. 9, Figure 5 · p. 10

5 CONCLUSION

ContourVLA treats GRES as a closed-loop perception–action process, with editable contours guiding multimodal perception and geometric actions. It achieves the highest gIoU on all three gRefCOCO splits and the highest mIoU on all eight RES splits among compared methods.

5 CONCLUSION

  • Among compared methods, ContourVLA achieves the highest gIoU on all three gRefCOCO splits, by 2.7–8.7 points, and the highest mIoU on all eight RES splits.Ablations support the benefits of re-observation and instance-level credit.Source: §5 · p. 10
  • ContourVLA recasts GRES as a closed-loop perception–action process in which editable contours guide multimodal perception and evolve through geometric actions.EASS couples contour-guided boundary sampling with multilevel semantic routing; DECT-GRPO jointly optimizes grounding and contour actions through transport-based rewards and instance-level credit.Source: §5 · p. 10

A.1 TRAINING AND EVALUATION PROTOCOL

ContourVLA training proceeds from grounding and detection pretraining to supervised contour fine-tuning and DECT-GRPO post-training. The stages progressively train referent grounding, contour control, and joint grounding-action optimization.

A.1 TRAINING AND EVALUATION PROTOCOL

  • The training pipeline has three chronological stages: grounding pretraining, supervised fine-tuning, and DECT-GRPO post-training.The visual and language backbones are initialized from MoonViT and Qwen2.5-3B-Instruct, respectively.Source: §A.1 · p. 15
  • Stage I trains VLIM on 139M LocateAnything-Data instances, with detection providing most supervision and UI, referring, OCR, layout, and pointing data contributing complementary localization signals.This stage trains grounding and detection, not contour actions.Source: §A.1 · p. 15, Figure 6 · p. 15
  • Stage II jointly fine-tunes on the training sets of gRefCOCO, RefCOCO, RefCOCO+, and RefCOCOg, using ground-truth boxes for grounding and mask-derived contour targets for GAD.Perturbed ground-truth boxes initialize the contour-action branch, and reference trajectories supervise evolution toward target boundaries.Source: §A.1 · p. 15
  • Stage III applies DECT-GRPO to the same four-dataset mixture, initialized from the Stage-II checkpoint, for joint grounding and contour-action optimization.The training-pipeline table describes the mixture as the four benchmarks' training sets and the feedback as final segmentation feedback.Source: §A.1 · p. 15, Table 4 · p. 16

III. DECT-GRPO

DECT-GRPO uses final segmentation feedback to optimize grounding and contour-action policies jointly, deriving a rollout reward and instance-level credit maps from transport correspondences. Its supplied evaluation protocols define mask-level scores, instance matching, boundary overlap, and contour latency.

III. DECT-GRPO

  • VLIM samples discrete grounding outputs that define initial contours; GAD then samples continuous geometric action trajectories, whose final contours are rasterized into masks for dustbin-augmented entropic transport.The resulting plan provides a transport-based rollout reward and instance-level credit maps for corresponding grounding tokens and contour actions.Source: §III · p. 16
  • The transport-based reward computes a shared group-relative advantage, while instance-level credit maps provide instance-specific corrections to grounding tokens and contour actions.The joint optimization uses final segmentation feedback without retaining the supervised contour losses used in Stage II.Source: §III · p. 16
  • GRES gIoU averages sample-level foreground IoU while including no-target samples: a correct empty prediction scores one, whereas a nonempty prediction on an empty target scores zero.cIoU is summed intersection area divided by summed union area over the split; correct empty predictions add zero to both sums.Source: §III · p. 16
  • Instance F1@0.5 uses one-to-one prediction-target matching at mask IoU of at least 0.5, counting unmatched predictions and targets as false positives and false negatives.For no-target expressions, predicted instances count as false positives, while correct empty predictions add no counts.Source: §III · p. 16
  • Boundary IoU compares predicted and target boundary regions after erosion-based boundary extraction, averaging expression-level union-mask scores at original image resolution.Samples with no predicted mask receive zero and remain in the average.Source: §III · p. 17
  • Contour latency is measured at action steps 2, 4, and 8 and includes contour-branch preparation, EASS readouts, GAD execution, and contour updates.The protocol excludes image encoding, grounding generation, and mask rasterization; Full EASS and frozen observations use identical grounding predictions and initial contours.Source: §III · p. 17

A.2 ADDITIONAL QUALITATIVE COMPARISONS

The appendix adds side-by-side qualitative comparisons on challenging expressions, covering plural and heterogeneous referents as well as an absent target. The examples assess set completeness, false-positive suppression, and boundary quality.

A.2 ADDITIONAL QUALITATIVE COMPARISONS

  • The side-by-side examples complement the quantitative evaluation in the main text.Source: §A.2 · p. 17
  • Figure 7 compares plural fruit referents, an absent target, and a multi-referent query combining visual attributes with spatial relations.Different mask colors distinguish referred instances, and “No Target” denotes an empty prediction.Source: §A.2 · p. 17
  • Figure 8 covers numerical and spatial constraints for multiple suitcases and a heterogeneous pot–person query.Together with Figure 7, it covers plural and multi-instance descriptions, heterogeneous referents, and the no-target case.Source: §A.2 · p. 17

A.3 OUT-OF-DISTRIBUTION VIDEO SEGMENTATION

The section qualitatively assesses the trained checkpoint on sampled frames from four out-of-distribution videos. Masks generally remain aligned with queried regions despite visual changes, but video-level temporal consistency is not measured.

A.3 OUT-OF-DISTRIBUTION VIDEO SEGMENTATION

  • The checkpoint is evaluated on sampled frames from four out-of-distribution video examples, with predictions shown in temporal order.Source: §A.3 · p. 17
  • Across object manipulation, hand–object interaction, and human motion, masks generally stay aligned with referred regions despite pose, position, appearance, and partial-occlusion changes.Source: §A.3 · p. 17
  • The assessment is qualitative and reports no video-level temporal-consistency metrics.Source: §A.3 · p. 17

A.4 STATE DESCRIPTORS AND DECODER INITIALIZATION

The implementation represents contour state with seven descriptors and initializes decoder pathways to preserve existing signals before introducing state-dependent offsets. Routing is regularized during supervised training, while its score scale has no temperature or extra logit scale.

A.4 STATE DESCRIPTORS AND DECODER INITIALIZATION

  • The seven-dimensional state descriptor combines instance geometry, point allocation, contour irregularity, evolution index, and displacement from the preceding iteration.The first three descriptors remain constant across iterations for fixed boxes and point allocations; initial-iteration displacement entries are zero.Source: §A.4 · p. 17
  • Zero-initialized projections make the initial query depend on the evolution-depth slot and preserve the GAD spatial and instance-context pathway; router regularizers apply during SFT, not DECT-GRPO.During DECT-GRPO, EASS is optimized through action likelihood and the straight-through routing path without an additional router loss.Source: §A.4 · pp. 17–18
  • Equation 6 uses neither routing temperature nor additional logit scale; its forward argmax selects one layer, while the soft-probability confidence objective remains bounded when prior weight is zero.Source: §A.4 · p. 18

A.5 COORDINATE CONVENTIONS AND ACTION-CHUNK EXECUTION

This appendix details contour-coordinate sampling and chunk execution, supervised initialization, DECT-GRPO replay and transport-based instance rewards and credits, alongside ablations of semantic routing and grounding/action objectives. It also clarifies that the closed loop refines initialized contours and notes limitations in referent recovery and fixed contour topology.

A.5 COORDINATE CONVENTIONS AND ACTION-CHUNK EXECUTION

  • EASS observes each contour once per cycle, then GAD executes T residual actions without refreshing observations before recomputing contour geometry and perception for the next cycle.K sets the number of perception-action cycles; with K = 2 and T = 4, the policy executes eight steps and re-observes every four steps, with each step moving a vertex at most 5% of box width and height.Source: §A.5 · p. 20

A.6 SUPERVISED TARGETS AND LOSS DETAILS

  • Contour actions are initialized from perturbed elliptical boxes with 64–128 vertices, while aligned ground-truth contours provide soft trajectory references rather than executable expert actions.The references may be unreachable within a chunk, so reaching the aligned target at each chunk endpoint is a supervision objective, not a hard constraint.Source: §A.6 · p. 21
  • SFT combines trajectory and region supervision with final-contour smoothness, spacing, signed-distance, and router objectives; these supervised losses initialize the policy and are not retained during DECT-GRPO.The region loss compares differentiably rasterized contours and masks within expanded ROIs, while routing regularizers are used only during SFT.Source: §A.6 · pp. 22–23

A.7 DECT-GRPO REPLAY AND TRAINING PROTOCOL

  • DECT-GRPO replay holds sampled outputs and historical contour states fixed while recomputing trainable observations and action means; grounding replay uses a branch-wise policy-gradient surrogate, not an exactly unbiased estimator of its transformed sampler.The collection sampler uses temperature and nucleus sampling, whereas replay scores re-tokenized answers under the untruncated softmax; the grounding reference disables LoRA adapters while sharing the current projector.Source: §A.7 · pp. 23–24

A.8 DERIVATION OF DUSTBIN-AUGMENTED ENTROPIC TRANSPORT

  • Entropy-regularized transport jointly matches predictions and targets with dustbin assignments, allowing unmatched instances to reduce reward and credit while distributing mass across plausible correspondences.The null-prediction capacity can account for missed targets and the null-target capacity for unmatched predictions; entropy and dustbin scores are fixed at 0.05 and 0.25, respectively.Source: §A.8 · pp. 24–25
  • For nearly interchangeable predictions and targets, transport preserves matching uncertainty, gives both predictions credit 0.6837, and yields rollout reward 0.6837.The equal credits produce zero normalized instance correction, avoiding an arbitrary preference between predictions.Source: §A.8 · p. 26
  • Dustbin transport assigns most target mass and credit to the stronger duplicate prediction, while a missed target lowers recall and the shared rollout reward.The weaker duplicate is assigned to the null target; a missed target can affect sampled outputs through the shared advantage despite having no instance-specific span.Source: §A.8 · p. 26

A.9 DECT-GRPO OPTIMIZATION DETAILS

  • DECT-GRPO normalizes instance credits separately for grounding and action branches, then modulates shared advantages without reversing their sign when 0 ≤ λc < 1.At the default λc = 0.5, the modulation ratio is [0.5, 1.5]; centering requires valid nonoverlapping grounding spans and valid action-point masks.Source: §A.9 · p. 26
  • For no-target examples, grounding advantages use αneg and the action branch is disabled; missed targets affect optimization through target coverage and shared rollout advantage.Positive-target inputs with no predicted objects receive a separately specified rollout reward.Source: §A.9 · p. 27
  • Both policy branches use clipped surrogate objectives, while reference-policy regularization applies only to grounding.The action branch has no reference-policy regularizer.Source: §A.9 · p. 27
  • Replay fixes sampled outputs and historical contours, recomputes trainable features, and detaches rewards and transport-derived credits; action gradients use score-function estimates.Losses are normalized independently, invalid or zero-advantage rollouts are omitted, and each sampled group receives one policy update.Source: §A.9 · pp. 27–28

A.10.1 STATE CONDITIONING IN SEMANTIC ROUTING

  • Full state-conditioned routing adds 1.1 gIoU, 2.3 cIoU, and 1.7 bIoU+ points over static-instance routing.The comparison retains SVS and spatial re-observation; the boundary score is evaluated on the fixed positive-target subset after eight action steps.Source: §A.10.1 · p. 28, Table 6 · p. 28

A.10.2 GROUNDING AND CONTOUR-ACTION OPTIMIZATION

  • Joint DECT-GRPO achieves the highest scores in both panels, improving end-to-end gIoU by 2.1 points and F1@0.5 by 2.6 points over grounding-PG only.Under fixed SFT grounding, it also gains 2.2 mIoU+ and 2.6 bIoU+ points over grounding-PG only, indicating refinement benefits beyond changes in grounding outputs.Source: §A.10.2 · p. 29, Table 7 · p. 29

A.11 LIMITATIONS AND SCOPE

  • The closed loop refines initialized instances but cannot recover initially missed referents, and fixed ordered-vertex contours retain their initial topology and point budget.Re-grounding or cardinality-changing actions could broaden referent recovery; topology-changing actions or adaptive vertex insertion could better handle disconnected regions, holes, or thin structures.Source: §A.11 · pp. 29–30

Related papers

Loading 2610.12107v1…