PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, Chu-Song Chen

arXiv:2409.10542v3cs.AIcs.CLcs.CV

TL;DR

Pixel-level segmentation approaches for MLLMs can require architectural changes, specialized tokens, or additional losses. SAM4MLLM encodes masks as text prompts and uses MLLM queries to guide SAM; it reports competitive benchmark performance with substantially fewer training images and masks than GLaMM.

  • Problem

    Prior MLLM segmentation approaches can require architectural changes, extra structures, specialized tokens, or multiple losses, while bounding boxes alone do not precisely indicate pixel locations.

    Source: §1 · p. 2

  • Method

    SAM4MLLM encodes object masks as discrete text prompts and uses MLLM-generated prompt points to guide SAM without changing the MLLM architecture or adding losses.

    Source: §1 · pp. 2–3

  • Results

    SAM4MLLM achieves competitive performance across RES benchmarks, with comparable quality to GLaMM using nearly 100 times fewer images and 10000 times fewer masks, and better performance on RefCOCOg than other methods.

    Source: §4.3 · p. 11, §6 · p. 14

  • Takeaways & Limitations

    The approach offers a simple way to combine MLLMs’ pixel-level localization information with SAM while retaining the simplicity and generalizability of the original language models.

    Source: §1 · p. 2, §6 · p. 14

    Final mask quality is constrained by SAM’s segmentation capability, though the approach may improve as SAM progresses.

    Source: §5 · pp. 22–23

Abstract

from arXiv · show

We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models (MLLMs) for pixel-aware tasks. Our method enables MLLMs to learn pixel-level location information without requiring excessive modifications to the existing model architecture or adding specialized tokens. We introduce an inquiry-based approach that can effectively find prompt points for SAM to perform segmentation based on MLLM. It combines detailed visual information with the powerful expressive capabilities of large language models in a unified language-based manner without additional computational overhead in learning. Experimental results on pubic benchmarks demonstrate the effectiveness of our approach.

1 Introduction

MLLMs can process visual information and produce bounding boxes, but pixel-level segmentation approaches often require architectural changes, specialized tokens, or additional losses. SAM4MLLM encodes pixel-level information in language-model training and uses MLLM-generated prompts to guide SAM, addressing RES without those modifications.

1 Introduction

  • Bounding-box outputs do not provide pixel-level precision, while existing MLLM segmentation methods can require architectural changes, extra structures, specialized tokens, or multiple losses.The paper identifies added complexity as a potential obstacle to extending MLLMs to other tasks.Source: §1 · p. 2, §2 · p. 4
  • SAM4MLLM introduces pixel-level information into training data and post-processes MLLM outputs with SAM to obtain higher-precision masks.It proactively queries the MLLM to obtain effective SAM prompt points.Source: §1 · p. 2
  • SAM4MLLM teaches MLLMs pixel-level details without changing their architecture, adding new tokens, or using additional losses.It connects MLLMs to SAM for segmentation masks, and the paper reports favorable performance on RES, GRES, and ReasonSeg.Source: §1 · p. 3

2 Related Works

  • GRES extends referring expression segmentation to phrases that may describe multiple objects or refer to no object, while ReasonSeg tests reasoning with implicit descriptions.The paper evaluates SAM4MLLM on both benchmarks.Source: §2 · p. 3
  • Prior MLLM approaches address regional information through detectors, region-of-interest instructions, grounding data, or linguistic representations, but their outputs are limited to bounding boxes.Other approaches add segmentation tokens, model structures, or specialized mask outputs; SAM4MLLM instead integrates off-the-shelf SAM.Source: §2 · pp. 4–5

3 Method

SAM4MLLM represents masks as discrete prompts and trains with standard text auto-regression loss. It uses either direct prompt-point generation or proactive MLLM queries, and aligns the model with RES-related data while retaining visual-dialogue examples.

3.1 Encode Segmentation Mask into SAM Prompt

  • Masks are encoded as a bounding box and K points, each represented by x and y coordinates plus an inside-or-outside label.This converts continuous masks into discrete SAM prompts without adding tokens or changing the model structure, while retaining text auto-regression cross-entropy training.Source: §3.1 · p. 5

3.2 Prompting SAM Using MLLM

  • PPG trains an MLLM to generate a bounding box and point groups, filters sampled training prompts by SAM mask IoU, and directly generates prompts at inference.Its implementation uses two positive and one negative point per group, typically samples 64 groups, and retains one; producing multiple points can be challenging.Source: §3.2 · pp. 6–7
  • PQPP first obtains a bounding box, then queries the MLLM about uniformly sampled points inside it and uses the answers as SAM prompts.Inference typically samples a 5 × 5 grid, with low-confidence points removed; PQPP probes multiple points through query-answering.Source: §3.2 · pp. 6–7

3.3 RES Training

  • RES and gRefCOCO supply ground-truth masks, while VQAv2 provides visual-dialogue examples without masks for training.Most MLLM parameters are frozen; training adjusts the visual resampler and LoRA adapter.Source: §3.3 · p. 8
  • RES subsets include RefCOCO, RefCOCO+, and RefCOCOg; RefCOCO+ excludes location-based descriptions, while RefCOCOg uses longer, more detailed phrases.gRefCOCO also includes phrases referring to multiple objects or absent objects; the model is trained to predict “object not in the image” for absent referents.Source: §3.3 · p. 8
  • Training uses no data augmentation because flipping or cropping may change objects’ relative positions or relationships.Source: §3.3 · p. 8

4 Experiments

The experiments evaluate SAM4MLLM across RES, generalized RES, zero-shot reasoning segmentation, and VQA, alongside implementation and training-data comparisons. Results report strong segmentation performance across these settings while preserving VQA capability.

4.1 Implementation Details

  • Implementation uses Qwen-VL-7B-Chat with LoRA and EfficientViT-XL1-SAM; training uses text cross-entropy loss and runs for three epochs.Most MLLM parameters are frozen while the visual resampler and LoRA adapter are adjusted; the EfficientViT-SAM choice incurred minor accuracy loss in a pilot study.Source: §4.1 · pp. 8–9
  • For PPG, training selects one point group from 16 high-IoU groups, each containing two positive and one negative point; testing retrieves the same point pattern.The groups are sampled within the ground-truth bounding box, and the selected points are encoded as text labels.Source: §4.1 · p. 9
  • For PQPP, training samples 10 labeled points, while testing queries the MLLM on a 5×5 grid and sends points with confidence above 0.9 to SAM.Points are labeled by whether they fall inside the ground-truth mask.Source: §4.1 · p. 9

4.2 Benchmarks

  • The method uses 100K images and 82K object masks, compared with GLaMM’s 11M images and 810M masks.The authors report comparable quality to GLaMM despite using nearly 100 times fewer images and 10,000 times fewer masks.Source: §4.2 · p. 10, §4.3 · p. 11

4.3 Main Results

  • SAM4MLLM-PQPP outperforms PPG in all reported cases, while requiring a two-turn conversation instead of PPG’s single turn.PQPP predicts a bounding box and classifies points; the authors describe a cost–quality trade-off between the variants.Source: §4.3 · p. 12
  • SAM4MLLM is comparable to GLaMM on RES, reaches state-of-the-art among 7B models on most GRES splits and metrics, and outperforms LISA-13B-LLaVA1.5 on ReasonSeg with less training data.On GRES Test Set A, it lags slightly behind GSVA; the ReasonSeg evaluation is zero-shot.Source: §4.3 · pp. 10, 12
  • VQA scores indicate that the approach does not compromise the backbone’s pre-training VQA abilities, and performance is reported to improve.The authors suggest fine-tuning on more datasets may explain the increase.Source: §4.3 · p. 12
  • Qualitative examples show RES segmentation of targets described by appearance and relative position, plus multiple-instance segmentation for GRES.Examples span refCOCO, refCOCO+, and refCOCOg, including complex narrative expressions.Source: §4.3 · p. 12, Figure 3 · p. 13

5 Ablation Study

The ablations examine PQPP’s confidence threshold and point-sampling strategy. A 0.9 threshold and 5x5 grid sampling yield the reported best outcomes.

5 Ablation Study

  • The ablations focus on filtering PQPP points by confidence and sampling points within the bounding box.The authors frame these investigations as examining the method’s robustness and adaptability.Source: §5 · p. 13

Points Filtering Threshold

  • A confidence threshold of 0.9 gives the best cIoU on RefCOCOg validation; increasing or decreasing it causes a notable decline.The tested threshold range was 0.6–0.95.Source: §5 · p. 13

Points Sampling Strategy

  • A 5x5 grid sampling pattern yields the highest reported accuracy among the tested point-sampling strategies.The study compares grid-based and random sampling while varying the number of points.Source: §5 · pp. 13–14

6 Conclusion

SAM4MLLM integrates SAM with MLLMs for referring expression segmentation, encoding object masks as discrete text prompts to provide pixel-level localization without complex architectural changes or additional losses. The conclusion reports competitive performance on RES benchmarks and proposes broader visual-reasoning tasks and deeper analysis as future work.

6 Conclusion

  • SAM4MLLM encodes object masks as discrete text prompts to provide pixel-level localization without complex architectural modifications or additional loss functions.The approach integrates SAM with MLLMs for RES.Source: §6 · p. 14
  • The authors report competitive performance on RES benchmarks while maintaining the simplicity and generalizability of the original language models.Source: §6 · p. 14
  • Future work could extend the approach to broader visual reasoning tasks and analyze the interplay between language and visual foundation models more deeply.Source: §6 · p. 14

Method RefCOCOg val

On RefCOCOg validation, the listed 5x5 grid configuration has the highest reported value among the four point-sampling configurations.

Method RefCOCOg val

  • 74.5 for 5x5 grid (N=25) is the highest listed value, versus 74.4 for 6x6 grid (N=36), 73.7 for random sample (N=25), and 74.2 for random sample (N=36).Source: Method RefCOCOg val · p. 15

1 Details of point sampling strategies.

The supplementary analysis compares SAM prompt performance with a ground-truth-mask-sampled upper bound and reports that adding more point groups or points worsens accuracy. These results support the authors’ claim that MLLMs find simultaneous multi-point prediction challenging.

1 Details of point sampling strategies.

  • Around 87.8% IoU is the upper bound from multiple SAM prompts sampled using ground-truth masks, while the method’s predicted prompts achieve around 75% IoU.The authors note room to improve MLLM prompting toward the upper-bound quality.Source: Table 1 · p. 19, §1 · p. 19
  • Using more point groups (k) or more points (n1, n2) results in worse accuracy, supporting the claim that MLLMs struggle to predict multiple points simultaneously.Source: Table 2 · p. 19, §1 · p. 19

2 Details of SAM in SAM4MLLM

SAM’s decoder is fine-tuned on COCO-extended RefCOCO and GRES data, while off-the-shelf SAM is used on ReasonSeg because of bias in COCO mask annotations. The fine-tuned setup yields masks more consistent with COCO’s coarser ground-truth granularity.

2 Details of SAM in SAM4MLLM

  • Fine-tuned SAM produces masks that better match the often-coarser ground truths in COCO datasets.The comparison is between off-the-shelf and fine-tuned SAM.Source: §2 · p. 20, Table 3 · p. 20
  • SAM’s decoder is fine-tuned on COCO-extended RefCOCO and GRES datasets, following LISA and GSVA.Source: §2 · p. 20
  • Off-the-shelf SAM is used on ReasonSeg because of inherent bias in COCO mask annotations.Source: §2 · p. 20

3 Ablation regarding the dataset combination.

The ablation compares training-data combinations and reports that adding VQA data does not compromise VQA capability or affect segmentation performance when RES data are used alone.

3 Ablation regarding the dataset combination.

  • Using only RES datasets for training does not influence segmentation performance.Source: §3 · p. 20
  • Training with both RES and VQA datasets preserves the model’s VQA capabilities.Source: §3 · p. 20
  • Table 4 compares the different training data combinations.Source: Table 4 · p. 20

4 Segmentation Samples

Visual comparisons show SAM4MLLM segmenting query-specified regions on RefCOCOg and maintaining robust segmentation on ReasonSeg’s implicit descriptions.

4 Segmentation Samples

  • On RefCOCOg, SAM4MLLM better captures query intent and predicts more accurate segments than GLaMM and LISA.The comparison uses longer queries than RefCOCO and RefCOCO+, making RefCOCOg more challenging for referring expression segmentation.Source: §4 · p. 21, Figure 2 · p. 21
  • On RefCOCOg, GLaMM and LISA may segment incomplete or larger objects rather than the regions specified by the query.Source: §4 · p. 21, Figure 2 · p. 21
  • On ReasonSeg, SAM4MLLM demonstrates robust segmentation despite implicit query descriptions that omit specific object names.Source: §4 · p. 21, Figure 3 · p. 22

5 Discussion

The discussion identifies SAM’s segmentation capability as a constraint on generated-mask quality and points to alternative segmentation models or output refinement as future directions. The image-license section identifies the sources of the example images.

5 Discussion

  • Generated-mask quality is constrained by SAM’s segmentation capability.Source: §5 · pp. 22–23
  • Future work could integrate other state-of-the-art segmentation models or refine SAM outputs in challenging scenarios.Source: §5 · p. 23

6 Image License

  • Images used for illustration come from RefCOCO, RefCOCOg, and RefCOCO+, with their sources and Flickr URLs listed in Table 5.Source: §6 · p. 23, Table 5 · p. 23

Related papers

Loading 2409.10542v3…