Source-linked AI summary
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, Ping Luo
TL;DR
Existing image-text instruction tuning lacks region-level alignment for fine-grained multimodal understanding. GPT4RoI introduces spatial instruction tuning by replacing bounding-box references with region features interleaved with language embeddings, achieving 81.6% accuracy on VCR and supporting region-level interaction and understanding.
Problem
Image-text instruction tuning lacks region-text alignment needed for fine-grained multimodal understanding.
Method
GPT4RoI replaces bounding-box references in spatial instructions with corresponding region features interleaved with language embeddings for an end-to-end LLM input sequence.
Results
81.6% accuracy on VCR placed GPT4RoI 6 points ahead of second place and near the 85.0% human-level benchmark.
Takeaways & Limitations
GPT4RoI enables users to combine language with region references for interactive, versatile region-level image understanding.
Abstract
from arXiv · showhide
Visual instruction tuning large language model(LLM) on image-text pairs has achieved general-purpose vision-language abilities. However, the lack of region-text pairs limits their advancements to fine-grained multimodal understanding. In this paper, we propose spatial instruction tuning, which introduces the reference to the region-of-interest(RoI) in the instruction. Before sending to LLM, the reference is replaced by RoI features and interleaved with language embeddings as a sequence. Our model GPT4RoI, trained on 7 region-text pair datasets, brings an unprecedented interactive and conversational experience compared to previous image-level models. (1) Interaction beyond language: Users can interact with our model by both language and drawing bounding boxes to flexibly adjust the referring granularity. (2) Versatile multimodal abilities: A variety of attribute information within each RoI can be mined by GPT4RoI, e.g., color, shape, material, action, etc. Furthermore, it can reason about multiple RoIs based on common sense. On the Visual Commonsense Reasoning(VCR) dataset, GPT4RoI achieves a remarkable accuracy of 81.6%, surpassing all existing models by a significant margin (the second place is 75.6%) and almost reaching human-level performance of 85.0%. The code and model can be found at https://github.com/jshilong/GPT4RoI.
1 Introduction
Image-level visual instruction tuning lacks region-level alignment for fine-grained understanding. GPT4RoI addresses this with spatial instruction tuning, enabling region interaction and strong performance across region-understanding benchmarks.
- Image-text alignment limits fine-grained tasks such as region captioning and reasoning, while external vision-model pipelines are non-end-to-end.
- Spatial instruction inserts region-of-interest references into language instructions and replaces them with corresponding region features before LLM processing.The region feature extractor can use RoIAlign or deformable attention.
- Region-text datasets provide bounding boxes and descriptions for training, covering detection, referring expression, captioning, and visual commonsense reasoning data.The collected data includes COCO, RefCOCO variants, Flickr30K Entities, Visual Genome, and VCR, alongside LLaVA150K.
- GPT4RoI supports language-and-region interaction and diverse capabilities including region captioning and complex region reasoning.Users can provide region references through interaction with the model.
- 81.6% accuracy on VCR placed GPT4RoI 6 points ahead of second place and near the 85.0% human-level benchmark.
2 Related Work
Prior multimodal systems either coordinate external vision models or specialize in separate region-level tasks, leaving a general-purpose region-understanding model absent. GPT4RoI uses an LLM-based generative approach to handle varied region-level questions, alongside concurrent approaches using alternative region-reference formats.
- LLM-based multimodal systems can call external vision models, but these non-end-to-end architectures are not optimal for general-purpose models.
- Region-level computer vision commonly detects potential regions before understanding them, with separate methods for recognition, captioning, scene graphs, and reasoning.
- A general-purpose region-understanding model had yet to emerge, while GPT4RoI uses generation to answer diverse region-level questions freely.
- Concurrent and follow-up MLLMs integrate region data through quantized location tokens, numeric positions, or specialized feature-extraction operations.
3 GPT4RoI
GPT4RoI converts spatial references in user instructions into region features interleaved with language embeddings, then trains the model in two stages for progressively complex region understanding.
- 3.1 Model Architecture: GPT4RoI combines a vision encoder, image projector, region feature extractor, and LLM to process spatial instructions as interleaved region-feature and text-embedding sequences.The model uses CLIP ViT-L/14 features and Vicuna for language processing.
- 3.1 Model Architecture: A multi-level image feature pyramid and scale-shuffle modules provide region representations across varying scales, with feature coordinates addressing translation invariance.The pyramid selects four CLIP vision-encoder layers and fuses them with five lightweight modules.
- 3.2 Tokenization and Embedding: The special token <region{i}> is replaced by the corresponding region feature, while explicit region labels preserve references that the LLM can use in generated responses.The architecture also inserts a global <image> feature so the LLM receives holistic visual context.
- 3.3 Spatial Instruction Tuning: Training partitions region-text data into two stages: Stage 1 aligns simple category, color, and position descriptions, while Stage 2 fine-tunes complex instruction following and multi-region reasoning.Stage 2 trains the region extractor, image projector, and LLM while keeping the vision encoder fixed.
- 3.3 Spatial Instruction Tuning: Stage 2 uses diverse region captioning and reasoning instructions, supplemented by LLaVA150K data and detector-generated region prompts to improve multi-round conversation.The resulting model performs complex single- and multiple-region captioning and reasoning tasks.
4 Demostrations
The demonstrations contrast LLaVA’s failures when users specify regions through language with GPT4RoI’s bounding-box-based recognition and reasoning across one or multiple regions.
- Single-instance dialogue: LLaVA misses the book in a center-region scene, causing incorrect answers about the boy’s activity and whether his behavior is dangerous.The model sees the umbrella and bag but not the book.
- Single-instance dialogue: GPT4RoI uses the supplied bounding box to identify that the boy is reading a magazine and infer that crossing the street while reading is dangerous.The demonstration connects visual detail recognition with commonsense reasoning.
- Multiple-instance dialogue: LLaVA incorrectly resolves language references such as “the right” and “the middle” when multiple instances are present, overlooking the intended woman.Its errors produce inaccurate descriptions of who is looking at whom.
- Multiple-instance dialogue: GPT4RoI accurately identifies region-specific actions and roles, and infers that two people are likely a couple from their observed expressions and interaction.The example identifies who should be called when ordering food and who is looking at the menu.
- Benchmark evaluation: ViP-Bench evaluates recognition, OCR, knowledge, math, relationship, and language generation dimensions for multimodal region understanding.The table caption defines the assessed dimensions and marks concurrent work with †.
5 Experiments
GPT4RoI is evaluated on comprehensive, recognition, captioning, and reasoning benchmarks for region-level understanding. It improves over prior methods, with especially strong VCR results.
- Comprehensive Region Understanding: ViP-Bench evaluates recognition, OCR, knowledge, math, relationship, and language generation for multimodal region understanding.
- Comprehensive Region Understanding: GPT4RoI surpasses comparison methods on ViP-Bench by a clear margin while using much less training data.
- Region Caption: GPT4RoI achieves better region captioning than Shikra and surpasses specialist model GRiT after fine-tuning on Visual Genome.
- Region Caption: GPT4RoI-7B and GPT4RoI-13B show comparable region captioning performance, indicating similar results across these model sizes.
- Region Reasoning: GPT4RoI improves over previous methods across VCR Q →A, QA →R, and Q →AR tasks.
- Region Reasoning: 81.6% accuracy on VCR Q →AR is over 6 points above previous methods and approaches the 85.0% human-level benchmark.
6 Conclusions
GPT4RoI uses spatial instruction tuning to execute user instructions for region-level image understanding. It combines bounding-box-derived region features with language embeddings and improves interaction and region-level task performance.
- GPT4RoI is an end-to-end vision-language model that executes user instructions for region-level image understanding.
- Spatial instruction tuning converts bounding-box references into region features and combines them with language embeddings as the LLM input sequence.
- GPT4RoI enhances user interaction through accurate region references and achieves strong performance on region-level image understanding tasks.