PaperlayerView exact revision on arXiv

Source-linked AI summary

LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model

Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, Jiaya Jia

arXiv:2312.17240v3cs.CV

TL;DR

LISA can produce segmentation masks but cannot distinguish instances of the same category and uses relatively rigid response formats. LISA++ retains its architecture while adding instance segmentation and Segmentation in Dialogue through curated instruction-tuning data. It improves on ReasonSeg-Inst and reports improved ReasonSeg-Sem test results compared with LISA.

  • Problem

    LISA lacks the ability to distinguish same-category instances, and its relatively rigid output format limits natural, adaptable responses.

    Source: §1 · p. 2

  • Method

    LISA++ retains LISA’s architecture and reconstructs instruction-tuning data to add instance segmentation and Segmentation in Dialogue.

    Source: §1 · p. 2, §3.1 · p. 3, §4.1 · p. 5

  • Results

    LISA++ improves ReasonSeg-Inst over LISA-7b and raises ReasonSeg-Sem overall test results by 1.4% gIou and 2.6% cIoU versus LISA.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The results indicate that LISA’s mask-as-embedding paradigm can accommodate more complex scenarios without structural modifications.

    Source: §5 · p. 7

    LISA’s earlier version could not distinguish same-category instances and typically relied on a rigid response format.

    Source: §1 · p. 2

Abstract

from arXiv · show

While LISA effectively bridges the gap between segmentation and large language models to enable reasoning segmentation, it poses certain limitations: unable to distinguish different instances of the target region, and constrained by the pre-defined textual response formats. In this work, we introduce LISA++, an update to the existing LISA model, focusing on improving core functionalities while keeping the base architecture intact. The main enhancements in LISA++ include: \textbf{1) Enhanced Segmentation}: The instance segmentation ability has been added, providing a more detailed scene analysis along with the existing multi-region semantic segmentation. \textbf{2) More Natural Conversation}: Improved capability for multi-turn dialogue, with the ability to incorporate segmentation results directly into text responses, i.e., Segmentation in Dialogue (SiD). These improvements are achieved by curating the existing samples of generic segmentation datasets, aimed specifically at enhancing the segmentation and conversational skills without structural change and additional data sources. Comparative analysis with the original LISA model shows significant advancements in these areas, positioning LISA++ as a notable upgrade in visual understanding and interaction. LISA++'s adaptability and improved features highlight the versatility of the mask-as-embedding paradigm proposed by LISA, and the potential as a foundational model for diverse applications.

1. Introduction

LISA++ extends LISA’s mask-as-embedding approach while retaining its architecture, adding instance segmentation and more flexible dialogue. The paper also expands ReasonSeg to assess multi-target instance segmentation.

1. Introduction

  • LISA cannot distinguish objects of the same category and typically uses a rigid response format that limits natural, adaptable interaction.The paper contrasts detailed segmentation masks with bounding-box outputs from detection frameworks.Source: §1 · p. 2
  • LISA++ adds instance segmentation and Segmentation in Dialogue (SiD) while retaining LISA’s architecture and reconstructing instruction-tuning data from existing segmentation datasets.The added abilities support distinguishing individuals within a category and placing segmentation results directly in text responses.Source: §1 · p. 2
  • ReasonSeg is expanded with a multi-target instance-segmentation benchmark alongside its existing single-target evaluation.The expansion is intended to assess both semantic and instance segmentation.Source: §1 · p. 2

2. Background

Prior multimodal models connect language and visual inputs in different ways, but their outputs are predominantly textual. Work on dense perception adds segmentation, detection, or spatial grounding, while LISA integrates segmentation through mask-as-embedding.

Large Vision-language Models (VLMs)

  • Multimodal models such as Flamingo [1], LLaVA [15] [16], and MiniGPT-4 [40] use visual-context attention or image-text alignment, yet remain limited to text outputs.BLIP-2 [12] and mPLUG-OWL [33] encode image features for integration with textual embeddings in an LLM.Source: §2 · p. 2

VLMs with Dense Perception

  • Dense-perception work includes open-world segmentation and detection [8] [11] [42], vision-centric instruction tuning [26], user-guided detection [20] [2] [21], and region-text understanding [37] [34] [6].The passage notes limits in handling complex instructions or intricate reasoning for some approaches.Source: §2 · p. 2
  • LISA [9] differs by integrating segmentation into VLMs with an end-to-end mask-as-embedding training pipeline.The passage contrasts this with approaches based on detection, spatial boxes, or region-text pairs.Source: §2 · p. 2

3. Method

LISA++ updates LISA with reasoning instance segmentation and Segmentation in Dialogue (SiD), using instruction-tuning pairs to support both capabilities. The section also introduces ReasonSeg-Inst as an instance-level extension to ReasonSeg.

3. Method

  • The method updates LISA with instance segmentation and more natural, flexible text responses through SiD, using corresponding instruction-tuning pairs.The overview describes these changes as additions to the original LISA.Source: §3.1 · p. 2, §3 · p. 2

3.1. An Improved Baseline

  • Instance-annotated datasets are used to prompt GPT-4V to generate questions that reflect detailed instance information.The annotations and images are provided with designed prompts so GPT-4V can yield questions and answers about selected instances.Source: §3.1 · p. 3, Figure 2 · p. 3
  • SiD places segmentation results at contextually appropriate points in dialogue rather than a fixed response position, covering reasoning Q&A and captioning.The annotations used for SiD can support instance segmentation in dialogue.Source: §3.1 · p. 3
  • Task templates coordinate semantic and instance segmentation with dialogue; training also uses bipartite matching to assign ground-truth masks to predicted instance masks.Different <SEG> tokens may denote individuals within the same category, unlike LISA’s category-based use; SiD data can also be adapted for semantic or text-only dialogue.Source: §3.1 · pp. 4–5

3.2. Instance-level Evaluation for ReasonSeg

  • ReasonSeg-Inst extends the single-target ReasonSeg benchmark to assess instance discrimination alongside the semantic-only ReasonSeg-Sem task.ReasonSeg-Sem does not require distinguishing instances within the same category.Source: §3.2 · p. 5
  • ReasonSeg-Inst contains 62K training and 1.8K evaluation image-query pairs curated from COCO and ADE20K.The curation excludes images smaller than 512x512 and objects with areas under 400 square pixels.Source: §3.2 · p. 5

4. Experiments

LISA++ is evaluated on ReasonSeg instance and semantic segmentation, with implementation largely following LISA’s training configuration. Results report gains on instance segmentation and comparable or improved semantic segmentation, alongside qualitative examples of captioning, instance masks, and multi-turn dialogue.

4.1. Implementation Details

  • Training largely follows LISA’s configuration, with new instruction-tuning sets for reasoning instance segmentation and SiD using COCO images already used by LISA.No additional visual training data is introduced.Source: §4.1 · p. 5

4.2. Results

  • Against LISA-7b, LISA++-7b achieves 34.1% AP50, 22.1% AP75, and 21.5% mAP; against LISA-13b, it improves by 18.4%, 14.3%, and 12.9%, respectively.The latter comparisons are for AP50, AP75, and mAP, respectively.Source: §4.2 · p. 6
  • On ReasonSeg-Sem, LISA++ reaches 64.2% gIoU and 68.1% cIoU on validation, and improves overall test-set gIoU by 1.4% and cIoU by 2.6% versus LISA.The added instance-segmentation capability does not compromise semantic segmentation, according to the authors.Source: §4.2 · p. 6

4.3. Qualitative Results

  • Figure 4 illustrates LISA++ integrating segmentation results into textual responses for image captioning.The authors describe these responses as more natural and flexible.Source: §4.3 · p. 6, Figure 4 · p. 6
  • Figure 5’s examples assess recognition and precise segmentation of queried objects despite possible interference from other objects.Examples named in the passage include girls, white cars, and whole oranges.Source: §4.3 · p. 7, Figure 5 · p. 7
  • Figure 6 presents multi-turn instance segmentation examples in which LISA++ identifies two people and perceives color and spatial attributes.The passage gives “white” and “holding the bat” as examples.Source: §4.3 · p. 7

5. Conclusion

LISA++ adds reasoning instance segmentation and segmentation dialogue to LISA, enabling more natural, flexible conversations with segmentation outcomes. The work presents this as evidence that the mask-as-embedding paradigm can extend to more complex scenarios without structural modifications.

5. Conclusion

  • LISA++ incorporates reasoning instance segmentation and segmentation dialogue into LISA, with segmentation outcomes included in more natural and flexible conversations.Source: §5 · p. 7
  • The study demonstrates that LISA’s mask-as-embedding paradigm can accommodate more complex scenarios without structural modifications.Source: §5 · p. 7
  • The authors hope LISA++ and the paradigm will offer new perspectives and stimulate further research.Source: §5 · p. 7
Loading 2312.17240v3…