PaperlayerView exact revision on arXiv

Source-linked AI summary

GSVA: Generalized Segmentation via Multimodal Large Language Models

Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, Gao Huang

arXiv:2312.10103v3cs.CV

TL;DR

GRES challenges RES systems by requiring multiple-target segmentation and rejection of referents absent from the image. GSVA addresses these cases with multiple [SEG] tokens and a [REJ] token, achieving strong results on GRES and related referring tasks. The paper reports state-of-the-art GRES benchmark performance and gains across classic RES and REC evaluations.

  • Problem

    GRES extends RES to multiple objects and empty targets, but existing segmentation MLLMs struggle with multiple subjects in one prompt and referents absent from the image.

  • Method

    GSVA uses multiple weight-sharing [SEG] tokens for multiple targets and predicts [REJ] tokens to reject empty targets.

  • Results

    GSVA achieves state-of-the-art performance on GRES benchmarks and strong performance across classic RES and REC tasks.

  • Takeaways & Limitations

    GSVA provides a multimodal segmentation approach that handles multiobjective segmentation and empty-target rejection in practical referring-expression scenarios.

Abstract

from arXiv · show

Generalized Referring Expression Segmentation (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. GRES poses challenges in modeling the complex spatial relationships of the instances in the image and identifying non-existing referents. Multimodal Large Language Models (MLLMs) have recently shown tremendous progress in these complicated vision-language tasks. Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient in understanding contexts with visual inputs. Among them, LISA, as a representative, adopts a special [SEG] token to prompt a segmentation mask decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt or provide descriptions incongruent with any image target. In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to address this gap. Specifically, GSVA reuses the [SEG] token to prompt the segmentation model towards supporting multiple mask references simultaneously and innovatively learns to generate a [REJ] token to reject the null targets explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue, marking a notable enhancement and setting a new record on the GRES benchmark gRefCOCO dataset. GSVA also proves effective across various classic referring segmentation and comprehension tasks.

1. Introduction

Classic RES assumes one referring expression matches one existing object, leaving multiple-target and empty-target cases unsupported. GSVA addresses these GRES challenges with multiple [SEG] tokens for multiple targets and [REJ] tokens for empty targets.

  • Motivation: RES predicts masks for interested objects but traditionally assumes each expression matches one existing image object.This leaves a gap between current RES algorithms and real-world applications involving multiple or absent referents.
  • Motivation: GRES extends RES to instructions referring to multiple subjects or descriptions with no corresponding image target.It requires models to handle complex spatial relationships and reject empty targets.
  • Background: MLLMs align language models with visual foundation models to support complex reasoning and instruction following from visual inputs.Segmentation MLLMs connect an MLLM with a segmentation model through special output tokens.
  • GSVA: GSVA predicts multiple weight-sharing [SEG] tokens, guided by entity expressions, to segment multiple requested targets.The design addresses the flexibility restriction imposed by a single constant [SEG] token.
  • GSVA: GSVA introduces [REJ] tokens to reject empty targets and reports state-of-the-art performance on GRES benchmarks.The method is also described as handling target spatial relationships and applying across models.

2. Related Works

Related work covers RES and GRES methods alongside multimodal architectures that align vision and language for segmentation and reasoning.

  • RES and GRES: RES assumes one expression matches one existing target, while GRES supports both multiple-target and empty-target scenarios.GRES is the paper's main research scope.
  • RES and GRES: Existing RES methods bridge image and language through cross-attention, cross-modal alignment, or unified segmentation decoders.DMMI addresses one-to-many target settings within this broader line of work.
  • MLLM: MLLMs align vision and language using techniques including cross-attention, prompt-tuning tokens, Q-Former modules, projections, and unified architectures.These models provide the multimodal foundation for the paper's approach.

3. Generalized Segmentation Vision Assistant

GSVA extends an MLLM–segmentation model architecture to generalized referring expression segmentation by supporting multiple referred targets and explicitly rejecting absent ones. It uses multiple [SEG] tokens for target masks and [REJ] tokens for empty targets.

  • Model Architecture: GSVA combines an MLLM for aligned vision-language reasoning with a Segmentation Foundation Model for producing target masks.The MLLM processes image and text instructions, while the SFM decodes segmentation queries into masks.
  • Model Architecture: The MLLM encodes image and tokenized text inputs, concatenates them with fixed prompt tokens, and autoregressively generates output token embeddings.A linear classifier converts output embeddings into text responses and special tokens.
  • Model Architecture: GSVA selects every generated [SEG] token as a segmentation query while discarding [REJ] tokens, enabling multiple masks and empty-target rejection.Each segmentation query corresponds to one mask, whereas rejected referents produce no segmentation query.
  • GRES: Task and Challenges: GRES removes RES’s one-target constraint, allowing expressions to identify multiple instances or no target in an image.A GRES system may segment referenced objects while producing an empty mask when a described object is absent.
  • Multiple [SEG] Tokens for Multiple Targets: A single [SEG] token can confuse segmentation MLLMs when one instruction names multiple objects, producing tattered or incorrect masks.GSVA instead associates separate [SEG] tokens with target descriptions through autoregressive decoding.
  • [REJ] Token for Empty Targets: For absent, incorrectly attributed, or incorrectly located objects, GSVA predicts [REJ] tokens so the segmentation model does not generate masks for null referents.This mechanism is intended to improve mask quality and reduce errors in identifying nonexistent objects.

4. Experiments

GSVA is evaluated on generalized and classic referring tasks, with ablations and visualizations examining its multiple-[SEG] and [REJ] designs. It achieves stronger reported performance than LISA across GRES, RES, and REC settings.

  • GRES: gRefCOCO contains 278,232 expressions across 19,994 images, including 80,022 multi-target and 32,202 empty-target expressions.Evaluation uses gIoU, cIoU, and N-acc. to measure segmentation and empty-target classification.
  • GRES: GSVA-Vicuna-7B improves over finetuned LISA by about 4% in gIoU and over 5% in N-acc. across all three evaluation splits.Larger-LLM GSVA variants exceed 70% gIoU on validation, 73% on test-A, and 65% on test-B.
  • RES: GSVA-Llama2-13B surpasses LISA by at least 2.8% cIoU on 8 RES evaluation sets, with a 5.9% margin on RefCOCOg test.This supports GSVA’s competitiveness beyond the generalized setting.
  • REC: In REC, finetuned GSVA-7B consistently achieves higher Prec@0.5 than LoRA-finetuned uLLaVA-7B, while larger LLMs add over 3% across datasets.Without finetuning, GSVA-Vicuna-7B exceeds LISA by over 7% in almost all evaluation sets.
  • Ablation Study: Removing [REJ] causes a sharp relative N-acc. drop exceeding 25% and about a 10% gIoU decline on gRefCOCO validation.Reducing multiple [SEG] tokens to one decreases gIoU by nearly 7% and N-acc. by over 15%.
  • Visualization: Qualitative examples show GSVA segments all requested targets and rejects mismatched empty targets, whereas LISA often segments only one or an incorrect instance.The visualization uses 7B models on the gRefCOCO validation set.

5. Conclusion

The paper presents GSVA as a multimodal large language model for generalized referring segmentation. Its multiple [SEG] and [REJ] tokens support multi-target segmentation and rejection of non-existing referents, with reported effectiveness across GRES, RES, and REC.

  • Conclusion: GSVA addresses generalized segmentation by introducing multiple [SEG] tokens for multiobjective masks and [REJ] tokens for empty-target rejection.The [REJ] approach provides a unified output-space alternative to methods that adjust prompts toward substitute referents.
  • Conclusion: The reported experiments cover GRES, classic RES, and REC tasks and demonstrate superior performance for GSVA.The conclusion frames these results as relevant to practical segmentation scenarios.

A.2. GSVA and Reasoning Segmentation

GSVA is evaluated on ReasonSeg with a generalized configuration to test instructions involving complex logic and additional empty targets.

  • Reasoning Segmentation: GSVA achieves competitive results to LISA on ReasonSeg under a generalized configuration.The evaluation tests instructions with complex logic.
  • Reasoning Segmentation: Qualitative ReasonSeg examples show GSVA segments the referent while rejecting an additional empty target that LISA fails to reject.This extends the demonstrated use of [REJ] beyond gRefCOCO.

A.3. Support of Various Question Types

GSVA is evaluated with diversified question forms and training data spanning segmentation, VQA, and reasoning segmentation. The results include robustness testing and an identified failure mode involving small, vague image regions.

  • Question diversity: Training uses diversified prompts, including “Please segment {objs} in this image” and “Can you segment {objs} in the image”.The training data also contains VQA and Reasoning Segmentation questions and answers.
  • Question diversity: Table 6 reports GRES results for different question types, including a substituted “Where” question alongside “What”.The pretrained model is tested for robustness across question formulations.
  • Evaluation examples: GSVA’s GRES evaluation includes generalized reasoning segmentation and examples of different question types.These examples are presented in Figures 5 and 6.
  • Failure cases: A failure case shows an empty target incorrectly predicted with a [SEG] token, producing a false positive near a vague image fragment.The model predicts a nearby fridge when no fridge is clearly present in the specified corner.

B. Implementation Details

GSVA is implemented by extending a pretrained multimodal language model and segmentation foundation model, then training with mixed task data and specified optimization settings. The paper provides configuration tables for reproducibility.

  • Reproducibility: Implementation details and hyperparameters for pretraining and finetuning are listed in Table 7.The table distinguishes training by fixed steps from training across the whole dataset by epochs.
  • Pretraining: GSVA pretraining starts from LLaVA-Vicuna-7B or another pretrained MLLM and SAM-ViT-H as the segmentation foundation model.The mask decoder, segmentation query projector, LoRA adapter weights, and vocabulary token embeddings are trainable.
  • Pretraining: Pretraining mixes Semantic Segmentation, RES, VQA, and Reasoning Segmentation data at a 9:6:3:1 sampling ratio.The ratio largely follows LISA while increasing RES weighting for gRefCOCO.
  • Optimization: All three variants are trained for 50,000 steps on 8 Tesla A100 GPUs with batch size 2 per device.The variants are Vicuna-7B, Vicuna-13B, and Llama2-13B.
  • Optimization: The learning rate is 3×10^-4 with gradient accumulation of 10, zero weight decay, gradient clipping at maximum norm 1, warmup for 100 steps, and linear decay.Training uses AdamW with stage 2 ZeRO; LoRA ranks are 8 for 7B and 64 for 13B variants.

C. GSVA on Ref-ZOM

GSVA is evaluated on Ref-ZOM as a generalized referring segmentation task covering one-to-one, one-to-many, and one-to-zero cases. The reported results show competitive performance relative to DMMI.

  • Dataset and protocol: Ref-ZOM contains 55,075 images and 74,942 annotated objects across one-to-one, one-to-many, and one-to-zero referring cases.The dataset includes 56,972 one-to-one, 21,290 one-to-many, and 11,937 one-to-zero cases.
  • Dataset and protocol: Ref-ZOM evaluation uses mIoU and oIoU for non-empty targets, while empty-target accuracy requires a strictly all-zero predicted mask.The protocol follows gRefCOCO while substituting equivalent metrics.
  • Results: GSVA and LISA are evaluated on the Ref-ZOM test split using the stated generalized referring segmentation protocol.Baselines in Table 8 are excerpted from Hu et al.
  • Results: GSVA with Vicuna-7B keeps competitive performance to DMMI.The supplied result passage reports this comparison without providing the associated metric value.

D. GSVA on Semantic Segmentation

The pretrained GSVA-Vicuna-7B and LISA-Vicuna-7B are additionally evaluated on ADE20K semantic segmentation. The evaluation tests whether they segment semantic regions rather than only specific objects.

  • Evaluation setup: GSVA-Vicuna-7B and LISA-Vicuna-7B are evaluated on the ADE20K validation dataset for vanilla semantic segmentation ability.Because pretraining covers ADE20K’s training set, evaluation focuses on segmenting semantic regions rather than specific objects.

E. More Ablation Study

The ablation studies examine the number of [SEG] tokens used during training and the design choice of sharing their weights. Increasing [SEG] tokens improves reported metrics before performance saturation, while also increasing memory use.

  • The number of targets in training: GSVA sweeps the number of training [SEG] tokens from 1 to 10 and evaluates gIoU, cIoU, and N-acc on gRefCOCO validation data.The experiment uses GSVA-Vicuna-7B and reports results in Figure 8.
  • The number of targets in training: Performance across gIoU, cIoU, and N-acc increases as more than 5 [SEG] tokens are involved.
  • The number of targets in training: Memory exceeds 24GB on RTX3090 or RTX4090 after 6 [SEG] tokens, motivating 5 as the default under performance saturation.
  • Additional ablations: The section also evaluates weight sharing for multiple [SEG] tokens and different SAM ViT backbones.The supplied captions identify these as separate ablation studies.

F. More Visualizations

The visualizations present GSVA as dialogue-based GRES interactions on gRefCOCO test sets A and B. Across examples, GSVA assigns [SEG] to matching targets and [REJ] to descriptions without corresponding image targets.

  • Test examples: In one example, GSVA segments the goose on the right, middle bird, and left referent, while rejecting the unmatched description.
  • Test examples: Other examples show multiple vases, apples, people, animals, vehicles, and glassware receiving [SEG] or [REJ] according to the responses.
  • Test examples: The displayed cases include successful segmentation of several people, sheep, giraffes, ambulances, trucks, and vases alongside rejection of absent targets.
  • Test examples: Further examples cover catcher, umpire, batter, and other queried objects, with non-existing cow, person, or pizza descriptions marked [REJ].
  • Visualization setup: Figure 9 visualizes GSVA inputs and outputs as human–chatbot dialogues using examples from gRefCOCO test sets A and B.
Loading 2312.10103v3…