PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, Jifeng Dai

arXiv:2305.11175v2cs.CV

TL;DR

VisionLLM addresses the gap between LLMs’ open-ended instruction capabilities and vision tasks constrained by predefined formats. It aligns vision-centric tasks with language instructions, uses prompt-guided image tokenization and an LLM decoder, and demonstrates customizable performance across tasks, including over 60% mAP on COCO.

  • Problem

    Vision tasks remain constrained by modality differences and predefined formats, limiting their alignment with LLMs’ open-ended, instruction-driven capabilities.

  • Method

    VisionLLM uses unified language instructions, a language-guided image tokenizer, and an LLM-based open-ended decoder to manage vision-centric tasks.

  • Results

    VisionLLM demonstrates customizable performance across diverse vision-centric tasks and achieves over 60% mAP on COCO.

  • Takeaways & Limitations

    VisionLLM provides a unified, open-ended, and customizable framework for vision and vision-language tasks.

Abstract

from arXiv · show

Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60\% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The demo shall be released based on https://github.com/OpenGVLab/InternGPT. The code shall be released at https://github.com/OpenGVLab/VisionLLM.

1 Introduction

VisionLLM addresses the mismatch between LLM-style open-ended instructions and vision tasks constrained by modality and predefined formats. It proposes a unified, customizable framework and reports effectiveness across diverse task granularities, including over 60% mAP on COCO.

  • LLMs’ zero-shot, instruction-driven capabilities do not directly transfer to vision tasks because modalities and task formats differ.
  • Vision generalist models remain constrained by predefined tasks, while visual prompt tuning uses formats that differ from language instructions.
  • VisionLLM unifies vision and language tasks through language instructions, an image tokenizer, and an LLM-based open-ended decoder.
  • The framework supports customization of target objects, output formats, and task descriptions across different levels of granularity.
  • VisionLLM demonstrates generality across random object categories, output formats, and task descriptions, achieving over 60% mAP on COCO.

2 Related Work

Related work spans API-based vision-language systems, generalist sequence-generation models, and instruction-tuned multimodal models. These approaches motivate VisionLLM while leaving visual perception tasks insufficiently addressed by many language-instruction systems.

  • API-based applications connect visual APIs with language models for visual decision-making or planning.
  • Generalist models seek shared architectures and parameters for multiple tasks, often representing diverse tasks as sequence generation.
  • Instruction tuning improves language models’ zero-shot and few-shot capabilities by expressing tasks and examples through language instructions.
  • Multimodal instruction-following models mainly focus on image-to-text tasks and do not address visual perception tasks such as detection or segmentation.

3 VisionLLM

VisionLLM defines vision-centric tasks with language instructions, encodes images in a prompt-aware token representation, and uses an LLM decoder to produce open-ended outputs. Its decoder and tokenization designs unify perception and vision-language task processing.

  • 3.1 Overall Architecture: VisionLLM’s architecture combines unified language instructions, a language-guided image tokenizer, and an LLM-based open-task decoder.
  • 3.1 Overall Architecture: The end-to-end pipeline encodes an image using the task prompt, then feeds image tokens and instructions to the open-ended decoder.
  • 3.2 Unified Language Instruction: Vision-only instructions specify both the task description and desired output format, enabling flexible task customization.
  • 3.2 Unified Language Instruction: Perception outputs use tuple (C, P), discretizing class indices and coordinates into tokens; two points represent boxes, while multiple boundary points represent masks.
  • 3.3 Language-Guided Image Tokenizer: The language-guided tokenizer injects prompt features into multi-scale visual features and extracts tokens containing semantic and positional information.
  • 3.4 LLM-based Open-Ended Task Decoder: The decoder expands the LLM vocabulary with localization tokens and uses output-format-as-query decoding to generate structured task outputs.

4 Experiment

VisionLLM is evaluated as a shared-parameter generalist model across vision-centric tasks, customization settings, and component ablations. Results show strong detection performance, flexible task customization, and trade-offs from multitask learning and tokenizer design.

  • Experimental Settings: VisionLLM is evaluated on object detection, instance segmentation, visual grounding, image captioning, and visual question answering.The experiments unify vision and language outputs as vocabulary generation and investigate general modeling capacity across five vision-centric tasks.
  • Task-Level Customization: 60.2% mAP is achieved on object detection with InternImage-H, close to a detection-specific state-of-the-art model.With ResNet-50, the model reaches 44.6 mAP and outperforms Pix2Seq by 1.4 mAP using the same backbone.
  • Task-Level Customization: VisionLLM supports object-level customization from 10 to 80 classes and output-format customization by changing the number of boundary points through language instructions.These experiments are conducted on COCO using the VisionLLM-R50 variant.
  • Ablation Study: Multitask training slightly reduces performance relative to separate training on most tasks, revealing a trade-off between accuracy and generalization.The jointly trained model is slightly worse than the separately trained model except for image captioning.
  • Ablation Study: Freezing BERT hinders vision-language alignment and degrades performance on both object detection and visual grounding.BERT is not essential for object detection but is crucial for visual grounding.
  • Ablation Study: +9.9 AP is obtained when localization tokens increase from 257 to 1025, after which performance eventually reaches a saturation point.The results indicate that more localization tokens improve fine localization ability until saturation.

5 Conclusion

VisionLLM presents an LLM-based framework for open-ended, customizable vision-centric tasks and validates its generality across tasks with different granularities. The authors also discuss its potential to promote visual-language fusion while requiring low training resources.

  • Conclusion: VisionLLM combines unified language instructions, a language-guided image tokenizer, and an LLM-based task decoder for open-ended vision-centric tasks.The framework handles tasks according to language instructions and is evaluated across different task granularities.
  • Broader Impact: The work aims to promote the fusion of visual and language tasks through an open-source framework built on pre-trained vision foundation models and large language models.The authors state that using these pre-trained models requires low training resources and reduces the carbon footprint.

A Example Instructions

The appendix illustrates language instructions for customizing five vision-centric tasks: object detection, instance segmentation, visual grounding, image captioning, and visual question answering.

  • Task-Level Customization: Task-level customization uses diverse language descriptions generated from seed instructions with placeholders.The examples cover object detection, instance segmentation, visual grounding, image captioning, and visual question answering.
  • A.1 Object Detection: Object-detection instructions specify target categories, coordinate ranges, bounding-box corner offsets, and tuple output formats.The class index starts from 0, and coordinates represent top-left and bottom-right corners relative to the image center.
  • A.2 Instance Segmentation: Instance-segmentation instructions represent each mask with a class index and multiple boundary-point coordinates.Examples vary the number of boundary points, including formats with 14, 20, and 24 coordinate values.
  • A.3 Visual Grounding: Visual-grounding instructions locate objects described by expressions and return bounding-box coordinates in the same tuple style.The category set uses expression-to-class mappings, while corner offsets are measured relative to the center point.
  • A.4 Image Captioning: Image-captioning instructions ask the model to describe an input image in natural language.The examples vary between writing, describing, and generating a caption for the image.
  • A.5 Visual Question Answering: Visual-question-answering instructions provide an image and question, then request an answer based on the image.The examples differ in wording but retain the same image-conditioned question-answering structure.

B Loss Function

VisionLLM combines losses for its language-guided image tokenizer and LLM-based open-task decoder, using distinct supervision for regular words and unordered visual outputs.

  • Loss Decomposition: The total training loss is the sum of tokenizer loss Ltok and decoder loss Ldec.These terms correspond to the language-guided image tokenizer and LLM-based open-task decoder, respectively.
  • Tokenizer Loss: The image tokenizer uses category-agnostic focal classification and L1 center-point regression to extract image tokens.Each token contains an embedding and the absolute coordinates of its center point.
  • Decoder Loss: The decoder applies next-token supervision to regular word prediction and bipartite matching for unordered outputs such as bounding boxes.Both decoding cases use cross-entropy supervision after their respective output-alignment procedures.

C Training Schedule

VisionLLM uses a two-stage training schedule that progresses from object detection to unified multi-task supervision; this easy-to-hard schedule converges faster than single-stage training.

  • Schedule Overview: A two-stage training schedule is introduced to speed VisionLLM convergence.The schedule separates initial alignment from subsequent multi-task training.
  • Stage 1: Stage 1 initializes the tokenizer and decoder from Deformable DETR, BERT, and Alpaca while training primarily on object detection.Most Alpaca parameters remain frozen, with only a few LoRA parameters tunable.
  • Stage 2: Stage 2 builds on Stage 1 and freezes the visual backbone for efficiency.The supplied passage identifies this stage as continuing from the first-stage model weights.
  • Ablation Setup: The ablation table includes comparisons involving Seq2Seq, LoRA, image-token count, randomness, and large-vocabulary object detection.The table is presented as additional ablation analysis for VisionLLM.
  • Seq2Seq Ablation: Seq2Seq decoding records 44.8 AP in the reported ablation row.The supplied row reports the metric as Seq2Seq AP.
  • Multi-Task Supervision: Unified supervision combines object detection, instance segmentation, visual grounding, image captioning, and visual question answering.This multi-task training is intended to help the model understand and manipulate visual information holistically.

D More Ablation Studies

The ablations examine randomness, image-token count, prompt robustness, instruction following, decoding, and large-vocabulary recognition, showing customization benefits alongside several design trade-offs.

  • Randomness: 45.2 box AP falls to 44.6 when training randomness is introduced, while customization and open-ended capabilities improve overall.Randomness covers task descriptions, object categories, and output formats.
  • LoRA: LoRA enables convergence when randomness is used, supporting alignment between language and visual tokens.Without LoRA, the randomized model fails to converge.
  • Image Tokens: Performance improves as image tokens increase from 50 to 300, but VisionLLM uses 100 tokens to balance computation.The paper attributes the improvement to more detailed image descriptions.
  • Prompt Robustness: Eight prompts produce results clustered near 44.8 AP, with only marginal differences across task descriptions and category orders.This evaluates robustness to prompt wording and random category ordering.
  • Instruction Following: When prompts contain 40 classes, those categories retain normal performance while unmentioned categories approach zero.The result indicates dynamic adherence to the instructed category set.
  • Decoding: Seq2Seq decoding is difficult to converge for randomized open-ended detection, suggesting a need for more design or training.The alternative is compared with VisionLLM’s output-format-as-query framework.
  • Output Customization: Mask customization changes only the number of segmentation points specified in the prompt.The appendix uses this setup to vary instance-mask output formats.

E Qualitative Analysis

VisionLLM qualitatively demonstrates language-controlled customization of segmentation outputs and category sets, alongside image description and visual question answering with complex reasoning.

  • Customization of Segmentation Points: Increasing the requested number of segmentation points produces more refined object masks, demonstrating fine-grained output-format customization.The experiment modifies the number of points specified for instance segmentation.
  • Customization of Category Set: Changing the instructed category set enables VisionLLM to predict objects according to the specified categories in open-vocabulary scenarios.The results indicate that the model understands and reasons over the semantic information in the language instruction.
  • Image Description & VQA: VisionLLM generates long image descriptions and answers visual questions requiring complex reasoning.Examples of these capabilities are presented in Figure 10.
Loading 2305.11175v2…