Source-linked AI summary
LISA: Reasoning Segmentation via Large Language Model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, Jiaya Jia
TL;DR
Existing perception systems rely on explicit target descriptions and struggle with implicit user intent, motivating segmentation from complex reasoning queries. The paper introduces reasoning segmentation, ReasonSeg, and LISA, which emits a <SEG> token decoded into a mask. LISA handles complex reasoning and achieves over 20% gIoU performance boost over existing works, with further gains from 239 reasoning samples.
Problem
Existing perception systems rely on explicit target indications or predefined categories and cannot actively reason about implicit user intention.
Method
The paper introduces reasoning segmentation, the ReasonSeg benchmark, and LISA, which decodes a generated <SEG> token embedding into a segmentation mask.
Results
LISA handles complex reasoning and achieves more than 20% gIoU performance boost over existing works on reasoning segmentation.
Takeaways & Limitations
LISA unlocks reasoning segmentation capabilities for multimodal LLMs, including robust zero-shot ability from reasoning-free training and further improvement from 239 reasoning samples.
Takeaways & Limitations
Polygon-sequence mask representations can introduce optimization challenges and may compromise generalization without massive data and computational resources.
Abstract
from arXiv · showhide
Although perception systems have made remarkable advancements in recent years, they still rely on explicit human instruction or pre-defined categories to identify the target objects before executing visual recognition tasks. Such systems cannot actively reason and comprehend implicit user intention. In this work, we propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction-mask data samples, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of multimodal Large Language Models (LLMs) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a <SEG> token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving complex reasoning and world knowledge. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation data samples results in further performance enhancement. Both quantitative and qualitative experiments show our method effectively unlocks new reasoning segmentation capabilities for multimodal LLMs. Code, models, and data are available at https://github.com/dvlab-research/LISA.
1. Introduction
The paper identifies a gap in perception systems that depend on explicit target descriptions and introduces reasoning segmentation with LISA to combine implicit-query reasoning and mask generation. It also establishes ReasonSeg and reports zero-shot performance, further improved with 239 reasoning samples.
- Existing perception systems require explicit target indications or predefined categories and cannot reason about implicit user instructions.
- Reasoning segmentation generates a binary mask from implicit queries involving complex reasoning or world knowledge.
- LISA adds segmentation capabilities to multimodal LLMs while retaining their language-generation abilities.It generates a <SEG> token whose hidden embedding is decoded into a segmentation mask.
- LISA handles complex reasoning, world knowledge, explanatory answers, and multiple segmentation masks in one answer.
- ReasonSeg contains over one thousand image-instruction-mask samples for evaluating reasoning segmentation.The benchmark uses annotated images and implicit queries involving complex reasoning.
- LISA shows robust zero-shot ability from reasoning-free training and improves further after fine-tuning on just 239 reasoning samples.
2. Related Work
Prior segmentation work supports semantic, instance, referring, and multimodal interaction, while multimodal LLMs primarily emphasize reasoning and text generation. LISA addresses the described gap by injecting reasoning-based segmentation capability into multimodal LLMs.
- Semantic, instance, panoptic, and referring segmentation research developed diverse pixel-, instance-, and language-guided segmentation methods.
- Existing multimodal segmentation systems unify tasks and support human interaction but do not explore the injection of reasoning capability.
- Multimodal LLMs transfer language-model reasoning to vision through visual encoders, cross-attention, and instruction-based learning.
3. Reasoning Segmentation
Reasoning segmentation requires a binary mask for an image and an implicit query whose complexity exceeds straightforward referring expressions. The ReasonSeg benchmark evaluates this task using short and long implicit instructions across 1218 annotated samples.
- Reasoning segmentation maps an input image and implicit query text to a binary segmentation mask.
- Unlike referring segmentation, its queries contain intricate expressions or longer sentences requiring complex reasoning or world knowledge.
- ReasonSeg contains 1218 image-instruction-mask samples collected from OpenImages and ScanNetv2.
- The benchmark includes both short-phrase and long-sentence instruction types.
4. Our Method
LISA connects a multimodal LLM to a segmentation decoder through a generated <SEG> token embedding, enabling end-to-end text generation and mask prediction. Its training combines semantic, referring, and VQA data, while LoRA supports efficient fine-tuning and reasoning segmentation performance improves with limited reasoning data.
- Architecture: LISA expands the LLM vocabulary with <SEG>, whose hidden embedding is decoded with visual features into a binary segmentation mask.The pipeline uses an MLP projection for the <SEG> embedding and a decoder that combines it with dense vision-backbone features.
- Training Objectives: The model is trained end-to-end with text-generation and mask losses combined in a weighted objective.The mask loss combines per-pixel BCE and DICE losses, while the text loss is autoregressive cross-entropy.
- Training Data: LISA’s training data combines semantic segmentation, referring segmentation, and visual question answering datasets.These sources convert segmentation examples into question-answer formats and retain the multimodal LLM’s VQA ability.
- Reasoning Segmentation: 239 reasoning segmentation samples further improve performance after zero-shot reasoning segmentation emerges from reasoning-free training data.The datasets used before this fine-tuning contain explicitly indicated targets rather than complex reasoning queries.
- Trainable Parameters: LoRA enables efficient fine-tuning while the vision backbone remains frozen and the decoder is fully fine-tuned.The token embeddings, language-model head, and projection layer are also trainable.
- Training Effects: LISA preserves text-generation and conversation abilities, which the authors associate with LoRA fine-tuning and inclusion of VQA data.The preservation is reported as verified in supplementary material.
5. Experiment
Experiments show that LISA handles reasoning segmentation, outperforms a decoupled two-stage approach, and remains effective across referring segmentation and design ablations.
- 5.2. Reasoning Segmentation Results: More than 20% gIoU performance boost over existing works demonstrates LISA’s ability to handle complex reasoning segmentation.The authors attribute the difference to multimodal LLM-based understanding of implicit queries.
- 5.2. Reasoning Segmentation Results: LISA significantly outperforms the vanilla two-stage method by training end-to-end and transmitting information through hidden embeddings rather than text.The comparison uses an LLM to generate text followed by a referring or open-vocabulary segmentation model.
- 5.2. Reasoning Segmentation Results: LISA-13B substantially outperforms LISA-7B, especially on long-query scenarios, indicating query understanding remains a performance bottleneck.The authors associate stronger multimodal LLMs with better results.
- 5.3. Vanilla Referring Segmentation Results: LISA achieves state-of-the-art results across refCOCO, refCOCO+, and refCOCOg referring segmentation benchmarks.The evaluation covers validation and testing sets.
- 5.4. Ablation Study: SAM performs best among evaluated vision backbones, while Mask2Former remains effective and significantly outperforms X-Decoder.The findings indicate that the framework is not limited to SAM.
- 5.4. Ablation Study: Fine-tuning SAM with LoRA performs worse than keeping it frozen, while removing SAM pre-trained weights causes performance to fall substantially.The authors suggest fine-tuning may impair SAM’s generalization ability.
- 5.4. Ablation Study: 2.2% gIoU and 2.9% cIoU gains result from GPT-3.5 instruction rephrasing during fine-tuning.The comparison is between Experiments 2 and 3 in Table 4.
- 5.4. Ablation Study: Removing semantic segmentation data causes a large performance drop, while adding more ReasonSeg samples improves both gIoU and cIoU.Adding the 200-sample ReasonSeg validation set during fine-tuning yields better performance on both metrics.
6. Conclusion
The paper introduces reasoning segmentation, the ReasonSeg benchmark, and LISA, which injects segmentation capabilities into multimodal LLMs and performs effectively on the task.
- 6. Conclusion: The paper proposes reasoning segmentation, a ReasonSeg benchmark with over one thousand samples, and LISA for multimodal LLM segmentation.The conclusion presents these as the work’s central contributions.
- 6. Conclusion: LISA performs surprisingly effectively on reasoning segmentation and connects language-model reasoning with vision-task output.The authors frame the work as a direction for combining LLMs and vision tasks.