PaperlayerView exact revision on arXiv

Source-linked AI summary

Osprey: Pixel Understanding with Visual Instruction Tuning

Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, Jianke Zhu

arXiv:2312.10032v4cs.CV

TL;DR

Existing MLLMs mainly support image- or box-level understanding, leaving pixel-level vision-language alignment and mask-based instruction data limited. Osprey introduces mask-text instruction tuning with a 724K-sample dataset and a mask-aware model, achieving superior performance across region understanding tasks. It also works with SAM masks to provide semantic captions and detailed descriptions at multiple granularities.

  • Problem

    Image- and box-level MLLMs lack fine-grained pixel-level vision-language alignment, while mask-based instruction data remains limited.

  • Method

    Osprey interleaves precise mask features from a mask-aware extractor and convolutional CLIP backbone with language instructions, trained on Osprey-724K mask-text data.

  • Results

    Osprey outperforms previous state-of-the-art methods across a wide range of region understanding tasks.

  • Takeaways & Limitations

    Osprey enables pixel-level instruction tuning and can use class-agnostic SAM masks to generate fine-grained semantics.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have recently achieved impressive general-purpose vision-language capabilities through visual instruction tuning. However, current MLLMs primarily focus on image-level or box-level understanding, falling short in achieving fine-grained vision-language alignment at pixel level. Besides, the lack of mask-based instruction data limits their advancements. In this paper, we propose Osprey, a mask-text instruction tuning approach, to extend MLLMs by incorporating fine-grained mask regions into language instruction, aiming at achieving pixel-wise visual understanding. To achieve this goal, we first meticulously curate a mask-based region-text dataset with 724K samples, and then design a vision-language model by injecting pixel-level representation into LLM. Specifically, Osprey adopts a convolutional CLIP backbone as the vision encoder and employs a mask-aware visual extractor to extract precise visual mask features from high resolution input. Experimental results demonstrate Osprey's superiority in various region understanding tasks, showcasing its new capability for pixel-level instruction tuning. In particular, Osprey can be integrated with Segment Anything Model (SAM) seamlessly to obtain multi-granularity semantics. The source code, dataset and demo can be found at https://github.com/CircleRadon/Osprey.

1. Introduction

Existing MLLMs largely provide image- or box-level understanding, leaving fine-grained region-language alignment and detailed region reasoning unresolved. Osprey addresses this gap with mask-aware pixel-level instruction tuning and reports superior performance across region understanding tasks.

  • Existing MLLMs primarily align vision and language at the image level, limiting fine-grained region classification, captioning, and reasoning.
  • Box-based region inputs can include irrelevant background, produce inexact region-text alignment, and struggle with dense-object details at low image resolution.
  • Osprey uses a mask-aware visual extractor and convolutional CLIP backbone to interleave precise mask features with language instructions.
  • Osprey-724K provides carefully annotated mask-text pairs, including object-level and part-level samples, detailed descriptions, conversations, attributes, and robustness-oriented instructions.
  • Osprey generates fine-grained semantics from class-agnostic SAM masks and demonstrates superiority across region recognition, classification, description, and reasoning tasks.

2. Related Work

Prior region-level vision-language systems locate regions before understanding them, while segmentation systems often lack rich semantics. Osprey contributes mask-text instruction data to support more comprehensive region understanding.

  • Region-level image understanding first identifies regions of interest before applying visual understanding.
  • SAM provides zero-shot object, part, and subpart segmentation from simple prompts, but vanilla SAM does not provide semantic labels.
  • Semantic segmentation extensions can predict primary categories, but primary labels alone are often insufficient for real-world applications.
  • Most region-based MLLMs use bounding boxes that may include irrelevant features, weaken region-text alignment, and operate at small image sizes.
  • Osprey addresses these gaps with a comprehensive dataset of mask-text pairs for instruction-based region learning.

3. Osprey-724K Dataset

Osprey-724K is an instruction dataset built from object- and part-level mask-text data, with additional samples designed to improve robustness and response flexibility. Its construction expands short region annotations into richer instructional supervision.

  • Osprey-724K contains around 724K multimodal dialogues covering object-level and part-level mask-text instruction data.
  • 3.1. Object-level Instructions: The dataset expands sparse captions into region instructions covering categories, types, actions, locations, colors, and statuses.
  • 3.1. Object-level Instructions: 197K unique object-level samples are generated from public datasets using GPT-4, detailed descriptions, and conversation formats.
  • 3.2. Part-level Instructions: 306K part-level samples draw on 456 part classes across 75 object categories and 55 attributes, including colors, patterns, materials, and reflectance.
  • 3.3. Robustness and Flexibility: Robustness data uses positive and negative region-category instructions, while flexibility data adds concise responses for categories, colors, types, locations, and quantities.

4. Method of Osprey

Osprey combines an image-level encoder, a pixel-level mask-aware visual extractor, and an LLM to interleave mask, spatial, image, and language representations for fine-grained region understanding. Its design uses convolutional CLIP features, mask pooling, spatial tokens, and staged supervised training to support high-resolution pixel-level instruction following.

  • Architecture: Osprey tokenizes the image, referring masks, and language, then sends interleaved mask features and language embeddings to an LLM.The architecture includes an image-level vision encoder, a pixel-level mask-aware visual extractor, and an LLM.
  • Convolutional CLIP Vision Encoder: Convolutional CLIP replaces the common ViT-based encoder to support larger input resolutions efficiently and robustly for fine-grained regions.ViT-based CLIP commonly uses 224×224 or 336×336 inputs, while global attention makes higher resolutions computationally burdensome.
  • Mask-Aware Visual Extractor: Mask pooling aggregates multi-level image features inside each region, whose projected representations are summed and adapted by an MLP into a visual mask token.For each region, features are pooled from each single-level feature map before projection and fusion.
  • Mask-Aware Visual Extractor: Osprey preserves region geometry by resizing each binary mask to 224×224, flattening and projecting it into a spatial token paired with the visual mask token.The two tokens together form the embeddings for each mask region.
  • Tokenization for LLM Model: A <region> placeholder is replaced by mask and spatial tokens, allowing region references to mix with text in a shared tokenization space.The placeholder follows region names such as “region1” or “region2” in textual inputs.
  • Tokenization for LLM Model: Image-level, region-level, and text tokens are interleaved and fed to Vicuna, with an <image> placeholder representing the image-level embedding.Vicuna is a decoder-only LLM instruction-tuned on top of LLaMA.
  • Training: Osprey trains in three supervised stages: image-text alignment, mask-text alignment, and end-to-end fine-tuning for complex pixel-level region understanding.Stage 2 trains the mask-aware extractor, while Stage 3 uses Osprey-724K together with Visual Genome and VCR data and keeps the vision encoder fixed.

5. Experiments

Osprey is evaluated on pixel-level region recognition, classification, description, reasoning, and hallucination tasks, using mask-based regions and comparisons with image-, box-, and mask-level methods. It achieves strong results across these tasks, while higher input resolution improves accuracy but increases computational cost.

  • Evaluation settings: Osprey evaluates pixel-level region recognition, classification, detailed description, reasoning, region captioning, and hallucination across multiple benchmarks.Experiments include Cityscapes, ADE20K-150, LVIS, PACO, RefCOCOs, Ferret-Bench, POPE, and RefCOCOg.
  • Open-vocabulary segmentation: 15.94% PQ, 7.24% AP, and 13.05% mIoU separate Osprey from box-level GPT4RoI on Cityscapes; against mask-level Ferret, gains are 15.07%, 2.23%, and 11.38%.On ADE20K-150, Osprey obtains 41.89% PQ, 41.24% AP, and 29.63% mIoU.
  • Referring object classification: 65.24% SS and 38.19% S-IoU on LVIS, plus 73.06% SS and 52.72% S-IoU on PACO, demonstrate strong object- and part-level classification.On PACO, Osprey surpasses Ferret by 14.38% SS and 26.76% S-IoU.
  • Description and reasoning: 72.2% and 67.8% accuracy make Osprey-Chat the best-performing model on Ferret-Bench’s Referring Description and Referring Reasoning tasks.Ferret-Bench uses box regions because mask annotations are unavailable.
  • Region captioning: 16.6% METEOR and 108.3% CIDEr on RefCOCOg surpass GLaMM by 0.4% and 3.3%, respectively.The evaluation uses short region-captioning prompts on the RefCOCOg validation set.
  • Ablation and efficiency: 800×800 input reaches 68.29% SS and 42.66% S-IoU, but raises image tokens to 2,500 and lowers inference speed, motivating the 512×512 setting.The 800×800 configuration is reported as the slowest among evaluated models.

6. Conclusion

The conclusion presents Osprey as a mask-instruction approach for fine-grained visual understanding, combining pixel-level mask references, a Mask-Aware Visual Extractor, convolutional CLIP, and the Osprey-724K dataset. The trained model achieves superior performance across region-understanding tasks and is intended to support real-world applications.

  • 6. Conclusion: Osprey incorporates pixel-level mask region references into language instructions to enhance MLLMs for fine-grained visual understanding.Its design combines a Mask-Aware Visual Extractor with a convolutional CLIP backbone.
  • 6. Conclusion: Osprey-724K contains 724K mask-based region-text pairs for fine-grained pixel-level alignment between vision and language.The model is trained on this curated dataset.
  • 6. Conclusion: Osprey demonstrates superior performance on various region-understanding tasks and sets new state-of-the-art results.The conclusion positions the dataset and model as resources for visual region understanding in real-world applications.

A.1. Additional Main Results

Additional experiments show that Osprey-724K improves a retrained GPT4RoI model, especially for part-level classification and detailed description. The results support the dataset’s effectiveness beyond the original Osprey model.

  • A.1. Additional Main Results: 20.89% SS and 15.33% improvement are observed when GPT4RoI is retrained with Osprey-724K, especially on part-level classification and detailed description.The comparison is between the original GPT4RoI model and the version retrained with Osprey-724K.

A.2. More Ablation Studies

Ablations examine mask-feature scale, vision-encoder choice, input resolution, and instruction-data composition. Multi-level features, convolutional CLIP at larger resolutions, and short-form plus positive/negative samples improve reported performance, with higher resolution incurring computational cost.

  • Mask features: Multi-level mask features significantly outperform single-level features in open-vocabulary segmentation and referring object classification.Single-level features use the vision encoder’s res4 output, while the comparison tests multi-scale features.
  • Vision encoders: CNN-based CLIP generalizes better as input resolution scales, whereas ViT-Surgery-L performs better at 224 resolution but struggles at higher resolutions.This comparison motivates using a CNN-based CLIP vision encoder in Osprey.
  • Instruction-data composition: 65.24% SS and 38.19% S-IoU on LVIS result from training with short-form prompts and positive/negative samples.Short-form prompts provide larger gains on PACO, while positive/negative samples add smaller improvements on both LVIS and PACO.

A.3. More Qualitative Results

Osprey produces accurate, fine-grained semantic predictions for challenging referring regions and outperforms prior region-level and image-level approaches qualitatively. The section also illustrates the instruction-following data and prompt-template setup used for training.

  • Qualitative Examples: Osprey accurately predicts semantics for unusual regions, including a catcher’s face mask, bottle cap, ladder step, and plate rim.These examples highlight robustness on challenging scenes.
  • Comparisons: Compared with previous region-level and image-level methods, Osprey provides more accurate region-level understanding with finer-grained details.The comparison uses the same randomly selected prompt, while image-level LLaVA-1.5 receives box-cropped regions.
  • Training Data: Osprey-724K includes object-level and part-level instruction-following data generated through GPT-4 interactions.The appendix examples cover object-level, part-level, and robustness data.
  • Training Setup: Training uses different prompt templates for different instruction-following data, with question templates randomly selected from corresponding lists.The detailed template lists are provided in Tables A12–A14.

C. Discussion on Types of Input Region

Osprey’s mask-aware extractor accepts multiple referring-region formats, while fine-grained masks provide the most precise object representation for pixel-level alignment. The appendix documents the associated instruction examples and prompt-template variants.

  • Supported Input Types: Osprey accepts point, box, and scribble inputs as free-form masks through its Mask-Aware Visual Extractor.This compatibility supports varied referring-region instructions.
  • Granularity Trade-off: Fine-grained masks represent objects more precisely than coarse point, box, or scribble inputs, supporting more accurate pixel-level semantic alignment.EfficientSAM and EdgeSAM are noted as faster, lower-cost options for acquiring masks.
  • Instruction Evaluation: The qualitative comparison evaluates prior region-level and image-level approaches using a shared prompt for detailed descriptions.The same prompt is randomly selected from Table A12.
  • Instruction Data: Osprey-724K examples cover object-level, part-level, robustness, detailed-description, conversation, short-form, attribute, and positive/negative-category instructions.These formats are illustrated across Tables A5–A14.
  • Part-level Instructions: Part-level attribute instructions use GPT-4-generated conversations based on mask-region attributes.The appendix separately documents the prompts used to generate these data.
  • Prompt Templates: Separate template lists specify detailed descriptions, brief descriptions, and positive/negative mask-region categories.These correspond to Tables A12, A13, and A14.
Loading 2312.10032v4…