PaperlayerView exact revision on arXiv

Source-linked AI summary

Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervision

Aadarsh Sahoo, Georgia Gkioxari

arXiv:2602.13195v1cs.CV

TL;DR

Existing referring image segmentation benchmarks underrepresent functional, affordance, safety, and physical reasoning needed for intent-driven pixel grounding. The paper introduces CIS and CONVERSEG, builds a VLM-driven engine producing large-scale prompt–mask supervision, and trains CONVERSEG-NET, which achieves strong benchmark performance while remaining competitive on standard referring segmentation.

  • Problem

    Existing language-guided segmentation benchmarks emphasize categorical and spatial references, leaving functional, affordance, safety, and physical reasoning underrepresented.

  • Method

    The paper introduces CIS and CONVERSEG, then uses an automated VLM-driven generate-and-verify engine to create 106K prompt–mask pairs for training CONVERSEG-NET.

  • Results

    CONVERSEG-NET achieves strong results on CONVERSEG and remains competitive on standard referring expression benchmarks.

  • Takeaways & Limitations

    The benchmark and scalable supervision engine provide a framework for evaluating and training segmentation systems on conversational concepts beyond standard object reference.

Abstract

from arXiv · show

Conversational image segmentation grounds abstract, intent-driven concepts into pixel-accurate masks. Prior work on referring image grounding focuses on categorical and spatial queries (e.g., "left-most apple") and overlooks functional and physical reasoning (e.g., "where can I safely store the knife?"). We address this gap and introduce Conversational Image Segmentation (CIS) and ConverSeg, a benchmark spanning entities, spatial relations, intent, affordances, functions, safety, and physical reasoning. We also present ConverSeg-Net, which fuses strong segmentation priors with language understanding, and an AI-powered data engine that generates prompt-mask pairs without human supervision. We show that current language-guided segmentation models are inadequate for CIS, while ConverSeg-Net trained on our data engine achieves significant gains on ConverSeg and maintains strong performance on existing language-guided segmentation benchmarks. Project webpage: https://glab-caltech.github.io/converseg/

1. Introduction

The paper introduces Conversational Image Segmentation to ground intent-driven, functional, and physical concepts in pixel-accurate masks, addressing gaps in categorical and spatial benchmarks. It contributes the CONVERSEG benchmark, an automated supervision engine, and CONVERSEG-NET.

  • Existing referring image segmentation benchmarks primarily emphasize categorical and spatial references, while functional and physical reasoning remains underrepresented.
  • Conversational Image Segmentation grounds high-level conversational concepts into pixel-accurate masks in natural images.
  • CIS organizes queries into five families: entities, spatial and layout, relations and events, affordances and functions, and physics and safety.
  • CONVERSEG contains 1,687 human-verified image–mask pairs spanning all five concept families and broader conversational reasoning than prior benchmarks.
  • The automated VLM-driven data engine synthesizes 106K prompt–mask pairs across all five concept families through iterative generation and verification without human supervision.
  • CONVERSEG-NET trained on this synthetic data achieves strong CONVERSEG results and remains competitive on standard referring expression benchmarks.

2. Related Work

Prior language-guided segmentation work largely targets literal entities and spatial relations, whereas this paper emphasizes conversational reasoning through broader concept coverage and automated supervision. Its approach combines segmentation priors, language understanding, and diverse synthetic prompts.

  • RefCOCO benchmarks are dominated by object-centric, low-level spatial phrases and seldom test affordances, stability, or user intent.
  • Reasoning-focused segmentation systems can support complex reasoning but often rely on heavy backbones and costly multi-stage inference.
  • SAM and SAM2 provide strong promptable segmentation priors but lack native text conditioning.
  • The paper combines SAM-like priors with lightweight vision–language adapters for end-to-end conversational grounding.
  • Existing datasets focus mainly on entities and spatial relations, while CONVERSEG offers near-uniform coverage across five concept categories.
  • The data engine synthesizes prompts about affordances, layout constraints, and physical safety, then filters them through multi-stage visual verification.

3. Conversational Image Segmentation

Conversational Image Segmentation extends referring segmentation from object-centric references to high-level queries whose masks may depend on function, affordance, safety, and physical reasoning. The task predicts pixels satisfying a natural-language prompt and organizes queries into five reasoning families.

  • Given an image and prompt, CIS predicts a binary mask identifying pixels that satisfy the query.
  • Unlike RIS, CIS prompts may require functional or physical reasoning, target non-visible properties, and have context-dependent groundings.
  • The concept families are intended to reflect natural queries about surroundings and reasoning abilities beyond object-centric reference.
  • CIS includes five families: entities, spatial and layout, relations and events, affordances and functions, and physics and safety.

4. The CONVERSEG Benchmark

CONVERSEG is a benchmark designed to evaluate conversational image segmentation across balanced concept coverage, using automated candidate generation followed by human verification. Its data engine supports prompt–mask synthesis across diverse reasoning categories.

  • CONVERSEG targets conversational image segmentation and is introduced with a data collection and annotation process.
  • Existing datasets are heavily skewed toward entities and spatial relations, whereas CONVERSEG provides balanced representation across all five concepts.
  • The conversational data engine uses scene recognition, mask generation, prompt creation, and verification modules.
  • CONVERSEG combines automated candidate generation with human verification of conversational samples.
  • The benchmark contains 1,687 samples: 1,194 SAM-seeded and 493 human-annotated.

5. The Conversational Data Engine

The conversational data engine automatically constructs and verifies diverse prompt–mask pairs by combining VLM-generated descriptions, masks, conversational prompts, and alignment checks. Its multi-stage pipeline uses SAM2 and VLM verification to produce training and benchmark data without human supervision.

  • Pipeline overview: The five-stage pipeline performs scene understanding, mask generation, mask refinement, conversational prompt generation, and prompt–mask alignment verification.VLMs generate region descriptions, SAM2 produces candidate masks, and only VLM-accepted prompt–mask pairs proceed.
  • Scene understanding: Scene understanding generates 5–7 concise region descriptions specifying categories, attributes, locations, and relations for subsequent mask generation.Each description is limited to 15 words and serves as a target for localization.
  • Mask refinement: Mask refinement samples SAM2 with a dense point grid, compares candidates using IoU, and uses a VLM to select the better mask by coverage, boundary precision, and artifacts.The selected mask becomes the final mask paired with the verified description.
  • Conversational prompts: Concept-driven prompts cover reasoning types such as affordances, functions, counterfactual uses, containment, and ordinality, while trivial or mismatched pairs are pruned.Concept-specific meta-prompts guide generation of up to three prompts per concept.
  • Verification: A VLM verifies that each prompt–mask pair matches the target, excludes irrelevant content, and is reasonably described by the prompt before acceptance.All VLM components use Gemini-2.5-Flash, and the engine produces pairs without human annotation.

6. Model

CONVERSEG-NET combines frozen SAM2 image features with Qwen2.5-VL language-conditioned representations through lightweight adapters and a SAM2 mask decoder. Curriculum training progresses from literal segmentation to conversational concepts while mixing foundational data to preserve basic segmentation performance.

  • Architecture: CONVERSEG-NET fuses SAM2 image-only components with a compact Qwen-2.5-VL-3B backbone through lightweight prompt adapters.The single-pass design combines segmentation priors with vision-language conditioning.
  • Architecture: The frozen SAM2 image encoder produces a spatial image embedding, while Qwen2.5-VL jointly processes the image and prompt to produce text-token hidden states.The extracted text states have attended to image tokens through Qwen’s backbone.
  • Prompt representations: Text-token states provide sparse embeddings for fine-grained information, while the EOS state provides a dense embedding for global image-text context.Linear and MLP adapters project these representations into the decoder’s input space; Qwen is fine-tuned with LoRA.
  • Mask decoder: The SAM2 mask decoder applies bidirectional cross-attention between prompt and image embeddings before producing per-pixel foreground probabilities and the final mask.The image embedding is upsampled after two modified Transformer blocks.
  • Training curriculum: Curriculum training increases complexity from literal concepts to abstract conversational concepts because SAM2 has no prior exposure to language.The training data groups contain 440K literal, 321K referring, 48K open-vocabulary, and 106K conversational prompt–mask pairs.
  • Training curriculum: Two-phase training first pretrains on literal, referring, and open-vocabulary groups, then conversationally post-trains with balanced positives, negatives, and foundational samples.The mixing strategy is intended to maintain performance on foundational segmentation tasks while adapting to conversational concepts.

7. Experiments

Experiments show that CONVERSEG-NET improves conversational segmentation across abstract concepts while remaining competitive on referring-expression and reasoning benchmarks. Ablations indicate that curriculum design, prompt-encoder adaptation, and visual context are important for balancing and enabling performance.

  • Main Results: 72.4% gIoU from CONVERSEG-NET-7B improves over the strongest baseline, Seg-Zero at 69.2%, by +3.2% on the SAM-seeded split.The 3B full model reaches 70.8%, improving over Seg-Zero by +1.6%; the same trends hold on the human-annotated split.
  • Per-Concept Analysis: Phase-2 conversational training raises Physics & Safety performance from 41.8% to 64.2%, narrowing its gap with Entities to 9.8%.Scaling to the 7B model produces the best results across all concept families while preserving strong entity-level performance.
  • Qualitative Analysis: CONVERSEG-NET better localizes abstract concepts than LISA in qualitative examples, including affordances, physical mechanisms, examination, spill risk, and attention-related objects.LISA sometimes includes nearby distractors or misses subtle physical cues, although it can produce a more canonical mask for comfortable full-body rest.
  • Cross-Benchmark Results: 52.2% gIoU on ReasonSeg test is achieved zero-shot by the 3B model, while scaling to 7B reaches 57.0%, ahead of LISA-13B’s 51.5%.On RefCOCO, CONVERSEG-NET reaches 78.4% on val, competitive with larger-data methods such as GSVA and EVF-SAM.
  • Curriculum Learning: 74.5% on RefCOCO/+/g and 67.4% on CONVERSEG are achieved by the final mixed curriculum, balancing performance across both benchmarks.Conversational-only training gives 66.0% on CONVERSEG but 56.1% on RefCOCO/+/g, while joint training without a curriculum reaches 75.5% and 65.4%, respectively.

8. Conclusion

The paper introduces CIS and CONVERSEG for grounding high-level conversational concepts, then scales supervision with an automated data engine and trains CONVERSEG-NET to perform this grounding.

  • CIS grounds high-level concepts about affordances, physics, and function into pixel-accurate masks.
  • CONVERSEG provides 1,687 human-verified samples with balanced coverage across five concept families underrepresented in prior work.
  • An automated data engine synthesizes 106K prompt-mask pairs through iterative VLM generation and verification.
  • CONVERSEG-NET achieves state-of-the-art performance on CONVERSEG while remaining competitive on standard benchmarks.
  • A curriculum from literal to conversational concepts adapts promptable segmentation to language conditioning.

A. Qualitative Results

Qualitative results show that CONVERSEG-NET often follows conversational intent across in-distribution and shifted images, while revealing ambiguity and annotation-related evaluation limitations.

  • Qualitative predictions: CONVERSEG-NET typically produces masks that more closely match conversational intent across both CONVERSEG splits.The model does so despite using a smaller backbone than the compared LISA variants.
  • Out-of-distribution behavior: On DROID and Warehouse images, CONVERSEG-NET often localizes regions implied by prompts despite distribution shift.These qualitative results suggest prospective applications in domestic and warehouse robotics.
  • Failure cases: Ambiguous prompts can cause reflection-versus-object errors or selection of only one among several valid targets.The latter yields high precision but low recall in examples involving cylindrical vessels and upward-pointing signs.
  • Annotation quality: Noisy RefCOCO/+/g masks can assign low gIoU to semantically reasonable predictions.Incorrect regions and poor boundary alignment can artificially deflate numerical performance.

B. Conversational Data Engine Details

The conversational data engine builds training triplets by generating scene descriptions, masks, prompts, and verification decisions, while negative examples target plausible hallucinations.

  • The engine automatically constructs conversational segmentation triplets from COCO images and generates 106K training examples for CONVERSEG-NET.
  • Scene understanding produces region-level descriptions that provide the semantic backbone for later stages.Mask generation converts descriptions into boxes and SAM2 masks, with VLM-based comparison used for refinement and selection.
  • Concept-driven prompt generation converts region descriptions into prompts anchored in five concept families.Separate concept-specific meta-prompts are used for each family, including affordances and functions.
  • Prompt-mask alignment verification uses a VLM to judge whether each prompt correctly and unambiguously describes the selected mask.
  • Negative data: Concept-specific negative prompts train the model to reject contextually plausible prompts that do not correspond to valid image regions.The strategy includes absent-object neighbors and incorrect-attribute neighbors; only verified negatives enter training.

C.1. Annotation Protocols

CONVERSEG combines automated candidate generation with human verification, using simple accept-or-reject judgments to preserve semantic alignment and benchmark diversity.

  • Annotation protocol: Annotators judge whether each conversational prompt and AI-generated mask are semantically aligned, choosing Accept or Reject.They review the image, overlaid mask, prompt, and verifier suggestion, but do not edit rejected masks.
  • Verifier limitations: The VLM may accept partially satisfying masks, while humans reject examples with incomplete coverage or irrelevant regions.For example, a mask containing both a blanket and a person can be rejected for a thermal-insulation prompt.
  • Verifier reliability: When both systems reject an example, the VLM reliably identifies severe coverage errors or obvious semantic mismatches.
  • Benchmark diversity: Human verification excludes duplicate-object prompts from CONVERSEG to prevent similar prompts from dominating the benchmark.Rejected pairs remain useful for training but are excluded from benchmark construction.
  • Verifier reliability: About 70% of VLM verifier and human decisions agree across the evaluated conditions.
  • Dataset coverage: CONVERSEG represents all five concept families and includes objects, parts, surfaces, functional areas, and amorphous regions.The dataset contains diverse region types across its SAM-seeded and human-annotated splits.

D.1. Architecture

CONVERSEG-NET combines a frozen multimodal language-image backbone with SAM2 mask decoding, while its training and data engine support conversational segmentation across diverse concepts. Qualitative and quantitative comparisons show stronger intent-grounded segmentation than existing baselines on CONVERSEG.

  • Architecture: CONVERSEG-NET uses frozen Qwen2.5-VL-3B embeddings, projected into sparse and dense language prompts for the SAM2 mask decoder.Text-token hidden states form sparse language tokens, while the EOS embedding becomes a spatially broadcast dense bias.
  • Training: The model fine-tunes the SAM2 mask decoder, SAM2 prompt encoder, and language adapter through pretraining and conversational post-training.Each stage runs for 35 000 steps with AdamW, cosine scheduling, and linear warmup.
  • Benchmark: CONVERSEG covers entities, spatial and layout concepts, relations and events, affordances and functions, and physics and safety across diverse region types.The benchmark includes non-instance regions such as object parts, surfaces, and functional areas, alongside human-annotated qualitative examples.
  • Evaluation: 70.8% and 67.4% gIoU on the SAM-seeded and human-annotated CONVERSEG splits exceed SAM3’s 39.7% and 35.4%, respectively.The comparison reports gIoU on CONVERSEG and attributes the gains to conversational training and the concept-driven data engine.
  • Data Engine: The data engine uses scene understanding, mask consistency checks, mask selection, concept-specific prompt generation, and final prompt–mask verification.Additional templates generate and verify negative prompts so invalid requests correspond to empty masks.
  • Evaluation: Qualitative comparisons report that CONVERSEG-NET more reliably segments regions implied by conversational intent than LISA baselines, despite using a smaller 3B backbone.The comparisons include LISA with LLaVA-7B and Llama2-13B backbones.
Loading 2602.13195v1…