Osprey: Pixel Understanding with Visual Instruction Tuning

• Existing MLLMs primarily align vision and language at the image level, limiting fine-grained region classification, captioning, and reasoning.Source: §1 · p. 1
• Box-based region inputs can include irrelevant background, produce inexact region-text alignment, and struggle with dense-object details at low image resolution.Source: §1 · p. 1
• Osprey uses a mask-aware visual extractor and convolutional CLIP backbone to interleave precise mask features with language instructions.Source: §1 · pp. 1–2
• Osprey-724K provides carefully annotated mask-text pairs, including object-level and part-level samples, detailed descriptions, conversations, attributes, and robustness-oriented instructions.Source: §1 · p. 2
• Osprey generates fine-grained semantics from class-agnostic SAM masks and demonstrates superiority across region recognition, classification, description, and reasoning tasks.Source: §1 · p. 2, Figure 1 · p. 2

• Region-level image understanding first identifies regions of interest before applying visual understanding.Source: §2 · p. 2
• SAM provides zero-shot object, part, and subpart segmentation from simple prompts, but vanilla SAM does not provide semantic labels.Source: §2 · p. 3
• Semantic segmentation extensions can predict primary categories, but primary labels alone are often insufficient for real-world applications.Source: §2 · p. 3
• Most region-based MLLMs use bounding boxes that may include irrelevant features, weaken region-text alignment, and operate at small image sizes.Source: §2 · p. 3
• Osprey addresses these gaps with a comprehensive dataset of mask-text pairs for instruction-based region learning.Source: §2 · p. 3

• Osprey-724K contains around 724K multimodal dialogues covering object-level and part-level mask-text instruction data.Source: §3 · p. 3

3.1. Object-level Instructions

• The dataset expands sparse captions into region instructions covering categories, types, actions, locations, colors, and statuses.Source: §3.1 · pp. 3–4
• 197K unique object-level samples are generated from public datasets using GPT-4, detailed descriptions, and conversation formats.Source: §3.1 · p. 4

3.2. Part-level Instructions

• 306K part-level samples draw on 456 part classes across 75 object categories and 55 attributes, including colors, patterns, materials, and reflectance.Source: §3.2 · p. 4

3.3. Robustness and Flexibility

• Robustness data uses positive and negative region-category instructions, while flexibility data adds concise responses for categories, colors, types, locations, and quantities.Source: §3.3 · p. 4

Architecture

• Osprey tokenizes the image, referring masks, and language, then sends interleaved mask features and language embeddings to an LLM.The architecture includes an image-level vision encoder, a pixel-level mask-aware visual extractor, and an LLM.Source: §4.1 · p. 4, Figure 4 · p. 5

Convolutional CLIP Vision Encoder

• Convolutional CLIP replaces the common ViT-based encoder to support larger input resolutions efficiently and robustly for fine-grained regions.ViT-based CLIP commonly uses 224×224 or 336×336 inputs, while global attention makes higher resolutions computationally burdensome.Source: §4.1.1 · pp. 4–5

Mask-Aware Visual Extractor

• Mask pooling aggregates multi-level image features inside each region, whose projected representations are summed and adapted by an MLP into a visual mask token.For each region, features are pooled from each single-level feature map before projection and fusion.Source: §4.1.2 · p. 5
• Osprey preserves region geometry by resizing each binary mask to 224×224, flattening and projecting it into a spatial token paired with the visual mask token.The two tokens together form the embeddings for each mask region.Source: §4.1.2 · p. 5

Tokenization for LLM Model

• A <region> placeholder is replaced by mask and spatial tokens, allowing region references to mix with text in a shared tokenization space.The placeholder follows region names such as “region1” or “region2” in textual inputs.Source: §4.1.3 · pp. 5–6
• Image-level, region-level, and text tokens are interleaved and fed to Vicuna, with an <image> placeholder representing the image-level embedding.Vicuna is a decoder-only LLM instruction-tuned on top of LLaMA.Source: §4.1.3 · p. 6

Training

• Osprey trains in three supervised stages: image-text alignment, mask-text alignment, and end-to-end fine-tuning for complex pixel-level region understanding.Stage 2 trains the mask-aware extractor, while Stage 3 uses Osprey-724K together with Visual Genome and VCR data and keeps the vision encoder fixed.Source: §4.2 · p. 6

Evaluation settings

• Osprey evaluates pixel-level region recognition, classification, detailed description, reasoning, region captioning, and hallucination across multiple benchmarks.Experiments include Cityscapes, ADE20K-150, LVIS, PACO, RefCOCOs, Ferret-Bench, POPE, and RefCOCOg.Source: §5.2 · p. 6, §5.2.1 · p. 7, §5.2.2 · p. 7

Open-vocabulary segmentation

• 15.94% PQ, 7.24% AP, and 13.05% mIoU separate Osprey from box-level GPT4RoI on Cityscapes; against mask-level Ferret, gains are 15.07%, 2.23%, and 11.38%.On ADE20K-150, Osprey obtains 41.89% PQ, 41.24% AP, and 29.63% mIoU.Source: §5.2.1 · p. 7

Referring object classification

• 65.24% SS and 38.19% S-IoU on LVIS, plus 73.06% SS and 52.72% S-IoU on PACO, demonstrate strong object- and part-level classification.On PACO, Osprey surpasses Ferret by 14.38% SS and 26.76% S-IoU.Source: §5.2.2 · p. 7

Description and reasoning

• 72.2% and 67.8% accuracy make Osprey-Chat the best-performing model on Ferret-Bench’s Referring Description and Referring Reasoning tasks.Ferret-Bench uses box regions because mask annotations are unavailable.Source: §5.2.3 · p. 8, Table 5 · p. 8

Region captioning

• 16.6% METEOR and 108.3% CIDEr on RefCOCOg surpass GLaMM by 0.4% and 3.3%, respectively.The evaluation uses short region-captioning prompts on the RefCOCOg validation set.Source: §5.2.5 · p. 9, Table 7 · p. 9

Ablation and efficiency

• 800×800 input reaches 68.29% SS and 42.66% S-IoU, but raises image tokens to 2,500 and lowers inference speed, motivating the 512×512 setting.The 800×800 configuration is reported as the slowest among evaluated models.Source: §5.3 · pp. 9–10, Table 9 · p. 9

6. Conclusion

• Osprey incorporates pixel-level mask region references into language instructions to enhance MLLMs for fine-grained visual understanding.Its design combines a Mask-Aware Visual Extractor with a convolutional CLIP backbone.Source: §6 · p. 10
• Osprey-724K contains 724K mask-based region-text pairs for fine-grained pixel-level alignment between vision and language.The model is trained on this curated dataset.Source: §6 · p. 10
• Osprey demonstrates superior performance on various region-understanding tasks and sets new state-of-the-art results.The conclusion positions the dataset and model as resources for visual region understanding in real-world applications.Source: §6 · p. 10

A.1. Additional Main Results

• 20.89% SS and 15.33% improvement are observed when GPT4RoI is retrained with Osprey-724K, especially on part-level classification and detailed description.The comparison is between the original GPT4RoI model and the version retrained with Osprey-724K.Source: §A.1 · p. 13, Table A1 · p. 13

Mask features

• Multi-level mask features significantly outperform single-level features in open-vocabulary segmentation and referring object classification.Single-level features use the vision encoder’s res4 output, while the comparison tests multi-scale features.Source: §A.2 · p. 13, Table A2 · p. 13

Vision encoders

• CNN-based CLIP generalizes better as input resolution scales, whereas ViT-Surgery-L performs better at 224 resolution but struggles at higher resolutions.This comparison motivates using a CNN-based CLIP vision encoder in Osprey.Source: §A.2 · p. 13, Table A3 · p. 13

Instruction-data composition

• 65.24% SS and 38.19% S-IoU on LVIS result from training with short-form prompts and positive/negative samples.Short-form prompts provide larger gains on PACO, while positive/negative samples add smaller improvements on both LVIS and PACO.Source: §A.2 · p. 13, Table A4 · p. 13

Qualitative Examples

• Osprey accurately predicts semantics for unusual regions, including a catcher’s face mask, bottle cap, ladder step, and plate rim.These examples highlight robustness on challenging scenes.Source: §A.3 · p. 13

Comparisons

• Compared with previous region-level and image-level methods, Osprey provides more accurate region-level understanding with finer-grained details.The comparison uses the same randomly selected prompt, while image-level LLaVA-1.5 receives box-cropped regions.Source: §A.3 · p. 14

Training Data

• Osprey-724K includes object-level and part-level instruction-following data generated through GPT-4 interactions.The appendix examples cover object-level, part-level, and robustness data.Source: §B.1 · p. 14

Training Setup

• Training uses different prompt templates for different instruction-following data, with question templates randomly selected from corresponding lists.The detailed template lists are provided in Tables A12–A14.Source: §B.2 · p. 14

Supported Input Types

• Osprey accepts point, box, and scribble inputs as free-form masks through its Mask-Aware Visual Extractor.This compatibility supports varied referring-region instructions.Source: C. Discussion on Types of Input Region · p. 14

Granularity Trade-off

• Fine-grained masks represent objects more precisely than coarse point, box, or scribble inputs, supporting more accurate pixel-level semantic alignment.EfficientSAM and EdgeSAM are noted as faster, lower-cost options for acquiring masks.Source: C. Discussion on Types of Input Region · p. 14

Instruction Evaluation

• The qualitative comparison evaluates prior region-level and image-level approaches using a shared prompt for detailed descriptions.The same prompt is randomly selected from Table A12.Source: Figure A2 · p. 15, Table A12 · p. 20

Instruction Data

• Osprey-724K examples cover object-level, part-level, robustness, detailed-description, conversation, short-form, attribute, and positive/negative-category instructions.These formats are illustrated across Tables A5–A14.Source: Table A5 · p. 16, Table A6 · p. 17, Table A7 · p. 17

Part-level Instructions

• Part-level attribute instructions use GPT-4-generated conversations based on mask-region attributes.The appendix separately documents the prompts used to generate these data.Source: Table A6 · p. 17, Table A11 · p. 19

Prompt Templates

• Separate template lists specify detailed descriptions, brief descriptions, and positive/negative mask-region categories.These correspond to Tables A12, A13, and A14.Source: Table A12 · p. 20, Table A13 · p. 21, Table A14 · p. 21