PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Ping Luo, Yu Qiao, Jifeng Dai

arXiv:2406.08394v3cs.CV

TL;DR

Existing MLLMs are largely limited to text outputs, leaving structured and visual tasks such as detection, pose estimation, and image generation insufficiently addressed. VisionLLM v2 uses super links, task-specific decoders, curated multi-task data, and end-to-end training to unify hundreds of tasks, achieving performance comparable to specialized models while retaining broad applicability.

  • Problem

    Existing MLLMs are limited mainly to text outputs, while scaling embedding-based connections to many visual tasks requires routing and conflict management.

  • Method

    VisionLLM v2 connects a central MLLM to task-specific decoders with routing tokens and super-link queries, supported by curated multi-task data and staged joint training.

  • Results

    VisionLLM v2 unifies visual perception, understanding, and generation and achieves performance comparable to specialized models while maintaining broad applicability.

  • Takeaways & Limitations

    A shared-parameter end-to-end MLLM can support hundreds of vision-language tasks through different textual and visual prompts.

  • Takeaways & Limitations

    Training uses three relatively complex stages, and integration with downstream tools has only been preliminarily validated.

Abstract

from arXiv · show

We present VisionLLM v2, an end-to-end generalist multimodal large model (MLLM) that unifies visual perception, understanding, and generation within a single framework. Unlike traditional MLLMs limited to text output, VisionLLM v2 significantly broadens its application scope. It excels not only in conventional visual question answering (VQA) but also in open-ended, cross-domain vision tasks such as object localization, pose estimation, and image generation and editing. To this end, we propose a new information transmission mechanism termed "super link", as a medium to connect MLLM with task-specific decoders. It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios. In addition, to support the diverse range of tasks, we carefully collected and combed training data from hundreds of public vision and vision-language tasks. In this way, our model can be joint-trained end-to-end on hundreds of vision language tasks and generalize to these tasks using a set of shared parameters through different user prompts, achieving performance comparable to task-specific models. We believe VisionLLM v2 will offer a new perspective on the generalization of MLLMs.

1 Introduction

VisionLLM v2 addresses the limited output formats and scaling challenges of existing MLLMs by unifying hundreds of vision and vision-language tasks in an end-to-end generalist model. Its super-link mechanism connects the MLLM with task decoders, while curated data and joint training support broad applicability and performance comparable to specialized models.

  • Existing MLLMs primarily produce text, limiting their ability to represent structured or visual information for detection, pose estimation, and image generation.
  • Text-based tool connections restrict information transfer and cannot relay decoder feedback gradients to the MLLM, while naive embedding connections are difficult to scale across tasks.
  • Curated data from hundreds of public tasks and multi-stage joint training support diverse domains while maintaining foundational VQA capabilities and performance comparable to expert models.
  • VisionLLM v2 introduces an end-to-end generalist MLLM covering visual perception, understanding, and generation across hundreds of vision and vision-language tasks.
  • The super-link technique integrates the MLLM with task-specific decoders, enabling flexible information transmission, decoder-gradient feedback, and reduced task conflicts.
  • The model is evaluated across perception, understanding, interaction, and long-tailed domains, with performance comparable to task-specialized models on various standard benchmarks.

2 Related Work

Related work extends MLLMs through text outputs, downstream tools, unified vision interfaces, visual prompting, and diffusion-based interaction. VisionLLM v2 differs by integrating these directions into a broader vision-language generalist.

  • Conventional MLLMs have achieved strong vision-language performance, but related approaches often remain centered on instruction-following and text-based outputs.
  • Tool-augmented MLLMs transmit task information through text or decoder embeddings, but text communication hinders end-to-end optimization.
  • Unified vision models combine multiple visual tasks, while visual prompting directs models toward image regions using marks such as points, boxes, scribbles, or masks.
  • Diffusion models provide natural-language interfaces for visual generation and manipulation, including image synthesis and editing.
  • VisionLLM v2 extends the vision-generalist scope by integrating LLMs and supporting a broader range of vision-language tasks and visual prompting paradigms.

3 VisionLLM v2

VisionLLM v2 combines multimodal encoders, a central LLM, task-specific decoders, and a super link to route information across visual tasks. Routing tokens select decoders, while super-link queries provide task-specific conditions for end-to-end processing.

  • Model Design: The architecture contains image and region encoders, an LLM for multimodal modeling, task-specific decoders, and a super link for conflict-free information transmission.
  • Model Design: Text prompts, images, and visual prompts are converted into features for the LLM, with visual prompts represented using binary masks for points, boxes, scribbles, and masks.
  • Task-specific Decoders: Grounding DINO, a mask decoder, UniPose, Stable Diffusion, and InstructPix2Pix provide localization, segmentation, pose estimation, generation, and editing capabilities.
  • Super Link Technique: Routing tokens such as [DET], [POSE], [SEG], [GEN], and [EDIT] select the downstream decoder required for a task.
  • Super Link Technique: Super-link queries are appended after routing-token embeddings, processed by the LLM, projected, and sent to the selected decoder as a task condition.
  • Super Link Technique: The super link combines special tokens with learnable queries to support precise decoder invocation, efficient message transmission, and mitigation of multi-task conflicts.
  • Training Strategy: A three-stage training strategy first builds image- and region-level understanding, then adds task-specific decoders and advanced capabilities.

4 Experiments

Experiments evaluate one generalist VisionLLM v2 across multimodal dialogue, recognition, reasoning, perception, generation, and ablations. The model shows strong performance across tasks while using shared parameters and specialized decoders.

  • Evaluation Scope: The evaluation uses one generalist model with the same parameters across diverse tasks, domains, and interactive modes.The experiments cover multimodal dialogue, visual perception, generation, and multiple ablations.
  • Multimodal Dialogue: VisionLLM v2-Chat surpasses LLaVA-NeXT-7B by +9.7 and +7.0 points on MMBench-EN/CN, respectively.The model also achieves comparable performance to VisionLLM v2-Chat on the reported multimodal benchmarks.
  • Visual Commonsense Reasoning: 82.9% accuracy on VCR Q→AR exceeds ASMv2 by +3.5 points without task-specific fine-tuning.VisionLLM v2 also outperforms previous methods across all reported VCR metrics.
  • Object Detection and Instance Segmentation: 56.7 APb and 47.8 APm on COCO with Swin-T are comparable to Grounding-DINO-T and exceed prior ResNet50-based methods.On CrowdHuman, VisionLLM v2 surpasses the previous best generalist model by 0.9 AP50 points.
  • Pose Estimation: VisionLLM v2 achieves competitive pose-estimation performance with UniPose-T and sets a new state-of-the-art result on CrowdPose.The model is especially strong on AP-10K and Macaque and supports keypoint detection for multiple objects.
  • Ablation Study: Separate super-link queries for each decoder outperform shared queries, whose keypoint AP decreases during training and whose box AP is also inferior.Increasing the number of super-link queries consistently improves performance across the evaluated visual perception tasks.

5 Conclusion & Limitation

VisionLLM v2 unifies visual perception, understanding, and generation while achieving performance comparable to specialized models across broad applications. The authors identify complex three-stage training and preliminary downstream-tool integration as limitations.

  • Conclusion: VisionLLM v2 unifies visual perception, understanding, and generation within a single framework.The model is presented as a comprehensive multimodal large language model.
  • Conclusion: Experiments show performance comparable to specialized models while maintaining broad applicability.The conclusion emphasizes the model’s coverage across visual and language tasks.
  • Limitations: The three-stage training process is relatively complex, and downstream-tool integration has only been preliminarily validated.Future work is proposed to improve performance and efficiency.

A.1 More Experimental Results

VisionLLM v2 performs strongly across region captioning, visual grounding, and semantic and interactive segmentation, with results varying by task and training setting.

  • Region Captioning: VisionLLM v2-Chat improves CIDEr by +8.6 on RefCOCOg and +8.4 on Visual Genome region captioning.The generalist VisionLLM v2 also shows promising RefCOCOg performance.
  • Visual Grounding: VisionLLM v2 achieves the best RefCOCO performance among MLLMs and reaches 51.0 gIoU on ReasonSeg without further fine-tuning.The results cover referring expression comprehension and pixel-level referring expression segmentation.
  • Semantic Segmentation: After 45k fine-tuning iterations on ADE20K, VisionLLM v2 reaches 52.3 mIoU and surpasses previous methods using the same backbone.Its generalist performance is slightly inferior before fine-tuning because ADE20K is a small portion of the joint-training dataset.
  • Interactive Segmentation: VisionLLM v2 outperforms SEEM-B across all interactive-segmentation metrics but trails PSALM before region-encoder fine-tuning.Unfreezing the region encoder significantly improves performance on COCO-interactive.

A.2 Evaluation on Various Domains.

Across object discovery, captioning, detection, and multimodal evaluation, VisionLLM v2 shows competitive or strong performance across datasets and domains, with some results requiring task-specific fine-tuning.

  • Object Detection: VisionLLM v2 achieves the highest performance on 4 of 5 classical salient-object-detection benchmarks.The evaluation uses S-measure, weighted F-measure, E-measure, and mean absolute error.
  • Object Detection: VisionLLM v2 performs competitively with expert camouflaged-object detectors trained for longer schedules, including 150 epochs.The comparison concerns camouflaged object detection performance.
  • Various Domains: VisionLLM v2 shows strong perception across remote sensing, PCB, and medical images.These qualitative results extend evaluation beyond natural-image domains.
  • Captioning: VisionLLM v2 and VisionLLM v2-Chat achieve competitive zero-shot image-captioning performance on Flickr30K and NoCaps.The evaluation uses CIDEr on Flickr30K and NoCaps.
  • Object Detection: With a Swin-Tiny backbone, VisionLLM v2 is on par with GLEE-Plus using a Swin-Large backbone in OdinW13 APavg.The result is presented as evidence of robust general object detection from extensive joint training.
  • In-Context Evaluation: In-context segmentation and image captioning are evaluated with varying numbers of visual examples and a 4-shot captioning protocol.The segmentation benchmark uses 1 to 5 in-context examples based on COCO2017.

A.4 More Ablation Studies

The ablations examine in-context performance and the connection between the MLLM and task decoders, showing advantages for super-link queries on open-ended visual-prompt tasks and efficient multi-token inference.

  • In-Context Evaluation: The evaluation compares in-context segmentation and in-context image captioning performance.The supplied table caption identifies both tasks as the comparison axes.
  • Super-Link Queries: Super-link queries outperform token embeddings on more open-ended tasks such as interactive segmentation with visual prompts.The performance difference is negligible for text-prompt tasks such as instance segmentation, where category names are seen during training.
  • Zero-Shot Detection: Zero-shot object detection performance is compared on OdinW13.The table evaluates VisionLLM v2 on the OdinW13 dataset.
  • Super-Link Queries: Super-link queries require one forward pass, whereas token embeddings require sequential prediction of special tokens during inference.This makes super-link queries efficient when many tokens are needed, such as in image generation.

A.5 Qualitative Results

Qualitative results show VisionLLM v2 handling diverse perception, generation, editing, and multimodal in-context tasks across domains and prompt types.

  • Visual Perception: VisionLLM v2 supports perception from boxes and keypoints to pixels, including basic and novel classes across natural, industrial, and agricultural domains.The qualitative evaluations span Figures A2 through A6.
  • Visual Generation: VisionLLM v2 generates images that follow instructed concepts, relations, and styles, and edits images through multiple instruction types.The reported editing operations include style transfer, object replacement, object addition, and attribute change.
  • Visual Perception: The model demonstrates domain generalization on remote sensing, PCB, and medical images.Figure A1 visualizes results across these domains.
  • Visual Grounding: The model localizes prompted objects with bounding boxes and answers user queries.Figure A6 presents grounded-caption examples combining localization and responses.
  • In-Context Learning: VisionLLM v2 performs fine-grained recognition and in-context image captioning from examples.The captioning visualization describes text completion based on in-context examples.
  • In-Context Learning: In-context detection, segmentation, and regional perception use visual examples or masks to guide processing of later inputs.The demonstrations cover highlighted instances and constructed visual masks in input prompts.
  • Model Components: The region encoder represents points, scribbles, boxes, and other visual prompts as binary masks concatenated with the image.This produces a 4-channel input for the convolutional region encoder.
  • Model Components: Super-link queries connect the MLLM to object and image decoders through projected pooled features and a Q-Former.Object-decoder features are sent as text features, while image-decoder features are projected into Stable Diffusion’s feature space.

B.2 Task-specific Decoders

VisionLLM v2 connects its language model to task-specific decoders through super-link queries, enabling end-to-end optimization across downstream tasks. The framework covers object perception and image generation, with datasets organized across training stages.

  • Super-link queries connect the LLM with task-specific decoders and enable end-to-end optimization of the entire network.
  • Object Decoder: For object detection, Grounding DINO uses pooled hidden states from category-specific super-link queries to localize objects and classify their categories.
  • Image Decoder: For image generation, Stable Diffusion receives super-link features projected by an MLP and lightweight Q-Former after the [GEN] token.
  • Stage-2 datasets combine stage-1 and stage-3 datasets so the model learns multiple capacities without compromising conversational ability.
  • The visual-prompting and in-context datasets encode image, mask, and text modalities for diverse task formats.

C More Dataset Details

The dataset construction expands VisionLLM v2 beyond standard vision-language data by organizing diverse task examples, visual marks, and in-context demonstrations. The resulting multimodal corpus contains approximately 862K question-answer pairs.

  • The authors curate datasets across standard vision tasks, visual prompting, and in-context learning to enhance the model with varied capacities.
  • The in-context task suite includes object detection, segmentation, captioning, OCR, and visual question answering.
  • For in-context tasks, the model receives N samples with N ∈[2, 6], using the first N −1 as examples and the last as the target.
  • The dataset uses five visual marks—circle, hand-drawn circle, arrow, box, and mask—with solid or hollow variants.
  • ∼862K question&answer pairs form the constructed multimodal corpus.

D Training Details

VisionLLM v2 is trained through three consecutive stages: multimodal training, multi-capacity fine-tuning, and decoder-only fine-tuning. The staged procedure progressively adds task capabilities while preserving the model’s conversational behavior.

  • Each stage inherits the previous stage’s weights, while the image encoder remains frozen after the initial stage.
  • Stage-2: Stage-2 adds task-specific decoders and trains the LLM, region encoder, and decoders with separate peak learning rates.
  • Stage-3: Stage-3 freezes all components except task-specific decoders to maintain conversational ability.
  • In stages 2 and 3, the total loss is the direct sum of the LLM and decoder losses without reweighting.
  • Instruction templates for different tasks are used as LLM instruction-tuning data to support task-specific decoder invocation.
Loading 2406.08394v3…