PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning

Junchi Wang, Lei Ke

arXiv:2404.08767v1cs.CVcs.LG

TL;DR

Traditional segmentation relies on explicit targets or predefined categories, while reasoning segmentation asks models to interpret human questions and identify a target. LLM-Seg separates prompt reasoning from segmentation, and the authors introduce a pipeline and dataset for the task. They report strong benchmark comparisons for LLM-Seg and describe LLM-Seg40K as useful for training and validation.

  • Problem

    Traditional segmentation requires explicit target objects or predefined categories, while reasoning segmentation requires identifying a target from a human question despite unclear target granularity.

    Source: §1 · p. 1

  • Method

    LLM-Seg uses LLaVA to select frozen SAM mask proposals, alongside a ChatGPT-4 pipeline that creates the LLM-Seg40K reasoning-segmentation dataset.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    LLM-Seg achieves state-of-the-art performance on both metrics in fully supervised evaluation and higher scores than LISA zero-shot; fine-tuning improves both methods.

    Source: §5.2.2 · p. 8

  • Takeaways & Limitations

    The pipeline automatically generates high-quality question-segmentation pairs, and LLM-Seg40K is useful for model training and validation.

    Source: §6 · p. 8

    A fixed prediction threshold makes LLM-Seg more likely to select large-area false masks, a limitation consistent with its low cIoU.

    Source: §5.1.4 · p. 7

Abstract

from arXiv · show

Understanding human instructions to identify the target objects is vital for perception systems. In recent years, the advancements of Large Language Models (LLMs) have introduced new possibilities for image segmentation. In this work, we delve into reasoning segmentation, a novel task that enables segmentation system to reason and interpret implicit user intention via large language model reasoning and then segment the corresponding target. Our work on reasoning segmentation contributes on both the methodological design and dataset labeling. For the model, we propose a new framework named LLM-Seg. LLM-Seg effectively connects the current foundational Segmentation Anything Model and the LLM by mask proposals selection. For the dataset, we propose an automatic data generation pipeline and construct a new reasoning segmentation dataset named LLM-Seg40K. Experiments demonstrate that our LLM-Seg exhibits competitive performance compared with existing methods. Furthermore, our proposed pipeline can efficiently produce high-quality reasoning segmentation datasets. The LLM-Seg40K dataset, developed through this pipeline, serves as a new benchmark for training and evaluating various reasoning segmentation approaches. Our code, models and dataset are at https://github.com/wangjunchi/LLMSeg.

1. Introduction

The introduction frames reasoning segmentation as a response to complex human instructions that existing segmentation methods handle only at the category or phrase level. It presents LLM-Seg as a two-stage approach and introduces an automated pipeline and dataset for the task.

1. Introduction

  • Open-vocabulary segmentation extends beyond predefined categories but remains limited to vocabulary- or phrase-level prompts, leaving long, complex instructions unhandled.Traditional segmentation also requires explicit target objects or predefined categories, limiting its fit for open-world applications such as home robots, self-driving, and augmented reality.Source: §1 · p. 1
  • Reasoning segmentation asks a model to identify and segment a target from a human question, requiring reasoning while leaving the target's granularity unclear.The introduction describes the task as more extensive and dynamic in application than traditional segmentation.Source: §1 · p. 1
  • LLM-Seg separates prompt understanding from segmentation, using LLaVA to select among frozen SAM mask proposals and combining vision-language and vision foundation models.The introduction favors this design because limited-data fine-tuning could harm foundation-model performance and generality, while decoupling may simplify learning and speed convergence.Source: Figure 1 · p. 1, §1 · p. 2
  • The authors propose a ChatGPT-4 data-generation pipeline and LLM-Seg40K, reusing semantic segmentation datasets to produce reasoning-segmentation questions and ground truth.They describe the pipeline as low-cost and efficient, and the dataset as intended for training and evaluation.Source: §1 · p. 2

2. Related Work

Related work describes vision-language and vision foundation models as sources of visual understanding and segmentation capabilities, while noting limits in adapting their outputs to reasoning segmentation. Prior approaches either lack complex reasoning or fine-tune segmentation components, and alternatives decouple reasoning from mask generation.

Vision Language Models

  • Current vision-language models can understand images, but their text outputs are difficult to use directly for segmentation; LLaVA uses visual instruction tuning with a linear projection layer.Other integration designs include Flamingo's gated cross-attention and perceiver resampler, and BLIP-2's Q-Former.Source: §2 · p. 2

Vision Foundation Models for Image Segmentation

  • Vision foundation models offer zero-shot adaptation, but applying them to multimodal reasoning segmentation remains limited; SAM provides promptable segmentation across image objects regardless of category.The related work also notes DINOv2's all-purpose image features and reports that its features outperform CLIP features on different vision tasks.Source: §2 · p. 2

Methods for Reasoning Segmentation

  • Referring-segmentation models using text encoders such as BERT are weak reasoning-segmentation baselines, while LISA connects LLaVA and SAM end-to-end but reportedly harms mask quality through decoder fine-tuning.The reported mask issues include holes and coarse boundaries.Source: §2 · pp. 2–3
  • Decoupled alternatives can ask a vision-language model for a phrase to pass to an open-vocabulary segmenter, or use a predicted bounding box as SAM's prompt.The passage presents both as ways to divide reasoning and segmentation between stages.Source: §2 · p. 3

3. Method

LLM-Seg separates prompt understanding from segmentation: LLaVA reasons over the image and query, while SAM supplies mask proposals that a trainable selection module scores. Its two selection heads support both best-match and multi-instance mask selection.

3.1. Model Structure

  • LLM-Seg combines pretrained LLaVA-7B, SAM, DINOv2, and a mask-selection module; only the selection module is fully trainable, while SAM and DINOv2 are frozen and LLaVA is LoRA-tuned.The final segmentation combines the selected mask proposals.Source: §3.1 · p. 3

3.2. Mask Proposals

  • SAM’s Everything Mode generates binary mask proposals from uniformly sampled point prompts, filtering low-IoU and duplicate masks; DINOv2 features are pooled within proposals to form mask embeddings.The image encoder runs once per image, and proposal generation takes about 3.5 seconds on one RTX-3090 GPU.Source: §3.2 · p. 3

3.3. Mask Selection

  • The fusion module aligns mask embeddings with the < SEG > token; an IoU head selects the most similar proposal, while an IoP head regresses per-mask IoP for threshold-based multi-mask selection.The IoU head uses KL-divergence supervision; the IoP head uses weighted L2 loss to address imbalanced IoP values.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4. Model Training

  • Training combines the IoU-head and IoP-head losses, fully updates the mask-selection module, applies LoRA to LLaVA-7B, and freezes SAM and DINOv2.The two loss weights are λiou and λiop.Source: §3.4 · p. 4

4. LLM-Seg40K

LLM-Seg40K addresses the limited scale of reasoning-segmentation data by pairing images with reasoning questions and binary masks. The pipeline reuses existing segmentation data, uses LLaVA descriptions and ChatGPT-4 question generation, and produces a 14K-image dataset.

4. LLM-Seg40K

  • LLM-Seg40K is introduced to address the limited research and small existing dataset for reasoning segmentation.The paper presents it alongside a data-generation pipeline.Source: §4 · p. 4

4.1. Dataset Definition

  • Each image is paired with multiple questions and binary masks, with questions requiring basic knowledge and reasoning to identify one or more target objects.The example contrasts asking where a headphone is with asking which device allows someone to listen without disturbing others.Source: §4.1 · pp. 4–5

4.2.1 Data Sources

  • The dataset reuses LVIS and EgoObjects segmentations; LVIS sampling balances simple and complex images, while sampled EgoObjects boxes are converted to binary masks with SAM.The LVIS sample comprises 6,000 simple and 4,000 complex images; 3,000 EgoObjects images with more than two categories are sampled.Source: §4.2.1 · p. 5

4.2.2 Prompt Engineering of GPT-4

  • LLaVA-v1.5-13B first describes each image, then text-only ChatGPT-4 generates questions from a prompt template using the description and ground-truth object categories.The template includes annotator instructions and an example; its example remains fixed while the summary and important objects are replaced.Source: §4.2.2 · p. 5, Figure 4 · p. 5

4.3. Dataset Analysis

  • LLM-Seg40K contains 14K images split into 11K training, 1K validation, and 2K test images; training averages 3.95 questions per image and includes 1,458 categories.The average training question length is 15.2 words.Source: §4.3 · p. 5

5. Experiments

The section introduces empirical results for the LLM-Seg model and the LLM-Seg40K dataset.

  • The experiments section presents empirical results for LLM-Seg and LLM-Seg40K.Source: §5 · p. 5

5.1. Evaluation of LLM-Seg

LLM-Seg is evaluated on ReasonSeg through comparisons with LISA and ablations of its model structure, mask-selection strategy, and image encoder. The reported results include stronger gIoU than LISA across the listed training settings, alongside a cIoU disadvantage and qualitative examples of mask quality and failure cases.

5.1.1 Implement Details

  • The implementation uses LLaVA-lightning-7B-v1 with DINOv2-ViT-L mask features and trains for 5,000 steps over 15 hours on two RTX-A6000 GPUs.Training mixes semantic segmentation, referring segmentation, and ReasonSeg datasets.Source: §5.1.1 · pp. 5–6

5.1.2 Quantitative Results

  • LLM-Seg's gIoU exceeds LISA's across zero-shot, 10-epoch, and 20-epoch settings (47.4 vs. 44.4; 52.3 vs. 51.0; 55.4 vs. 52.9), while zero-shot ncIoU is 48.8 vs. 48.1.The authors report cIoU about 10 points lower than LISA's and attribute it to over-segmentation on high-resolution images.Source: §5.1.2 · p. 6, Table 1 · p. 6

5.1.3 Ablation Study

  • Combining the IoU and IoP heads gives the optimal model-structure result, although the final mask selection uses IoP only.The fusion module alone does not help on either metric, but is necessary for adding the IoP head.Source: §5.1.3 · p. 6, Table 2 · p. 7
  • Thresholding predicted IoPs gives optimal performance among the tested mask-selection strategies.The strategies include selecting the top IoP mask, thresholding IoP, combining top IoU with thresholded IoP, and thresholding IoP among the top five IoU proposals.Source: §5.1.3 · pp. 6–7, Table 3 · p. 7
  • DINOv2 outperforms CLIP and SAM as the mask-embedding backbone, with gIoU gains of 3.0 and 6.1 points, respectively.The authors associate the gap with the importance of semantic information in extracted image features.Source: §5.1.3 · p. 7, Table 4 · p. 7

5.1.4 Qualitative Results

  • In qualitative examples, LLM-Seg preserves SAM's mask quality, while its fixed-threshold selection can produce large-area false positives.The paper also reports success cases where LISA identifies the correct target but produces lower segmentation quality.Source: §5.1.4 · p. 7

5.2. Benckmark on LLM-Seg40K dataset

The LLM-Seg40K validation benchmark compares segmentation methods across zero-shot and fully supervised settings. LLM-Seg leads both reported metrics when fully supervised, and its performance improves more than LISA’s after fine-tuning.

5.2. Benckmark on LLM-Seg40K dataset

  • The validation benchmark omits ncIoU because image sizes vary little in LLM-Seg40K.Table 5 compares methods on the dataset’s validation split; its labels distinguish LLM use and zero-shot versus training-split fine-tuning.Source: §5.2 · p. 7, Table 5 · p. 7

5.2.1 Implement Details

  • GRES is evaluated zero-shot with its released checkpoint, while LISA and LLM-Seg are also evaluated after training on the LLM-Seg40K training split.The fine-tuning setup uses 2,500 updates for LISA; LLM-Seg uses the same batch size and update count.Source: §5.2.1 · p. 7
  • The paper provides visual comparisons against other methods and on the LLM-Seg40K validation split, including multiple-instance examples.For LISA and LLM-Seg, the Figure 6 results are from fine-tuned models.Source: Figure 5 · p. 8, Figure 6 · p. 8

5.2.2 Analysis

  • LLM-Seg achieves state-of-the-art performance on both metrics in the fully supervised setting and scores higher than LISA in the zero-shot setting.After fine-tuning, both methods improve, but LLM-Seg improves more; the authors attribute this partly to its frozen SAM-generated masks being unaffected by training-set errors.Source: Table 5 · p. 7, §5.2.2 · p. 8

6. Conclusion

The conclusion presents LLM-Seg’s reasoning-segmentation method and the LLM-Seg40K dataset pipeline as the paper’s two contributions. It reports competitive method performance and says the pipeline generates useful question-segmentation pairs for training and validation.

6. Conclusion

  • The paper’s conclusion frames its contributions around both a reasoning-segmentation method and a dataset.It characterizes LLM-Seg as a two-stage approach that decouples reasoning from segmentation to maintain segmentation quality.Source: §6 · p. 8
  • The authors report competitive qualitative and quantitative performance for LLM-Seg against current state-of-the-art methods.Source: §6 · p. 8
  • The authors report that their pipeline automatically generates high-quality question-segmentation pairs and that LLM-Seg40K is useful for model training and validation.Source: §6 · p. 8
Loading 2404.08767v1…