PaperlayerView exact revision on arXiv

Source-linked AI summary

DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception

Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, Ling-Yu Duan

arXiv:2407.08303v2cs.CVcs.AI

TL;DR

Detailed image-text datasets are scarce, and existing caption engines can miss visual information needed for comprehensive MLLM perception. Perceptual Fusion integrates visual experts with an MLLM pivot and produces DenseFusion-1M; the trained model outperforms existing MLLMs across 10 vision-language benchmarks, especially for text recognition and high-resolution perception.

  • Problem

    High-quality image-text datasets with dense descriptions across visual elements are scarce, while caption engines often lack the perception needed for complete and accurate annotations.

    Source: §1 · p. 1

  • Method

    Perceptual Fusion combines diverse vision experts as image priors with an MLLM pivot, using GPT-4V-generated descriptions to train a low-budget caption engine that creates DenseFusion-1M.

    Source: §1 · p. 2, §3.2.2 · p. 6

  • Results

    The trained MLLM outperforms existing state-of-the-art MLLMs across 10 vision-language benchmarks, especially for detailed text recognition and high-resolution image perception.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that meticulous, accurate image descriptions are important for high-resolution vision perception, where low-resolution MLLMs can struggle to extract detailed clues.

    Source: §4.2 · p. 8

    Perceptual Fusion targets generalist MLLMs’ occasional object omissions and weak text recognition.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Existing Multimodal Large Language Models (MLLMs) increasingly emphasize complex understanding of various visual elements, including multiple objects, text information, and spatial relations. Their development for comprehensive visual perception hinges on the availability of high-quality image-text datasets that offer diverse visual elements and throughout image descriptions. However, the scarcity of such hyper-detailed datasets currently hinders progress within the MLLM community. The bottleneck stems from the limited perceptual capabilities of current caption engines, which fall short in providing complete and accurate annotations. To facilitate the cutting-edge research of MLLMs on comprehensive vision perception, we thereby propose Perceptual Fusion, using a low-budget but highly effective caption engine for complete and accurate image descriptions. Specifically, Perceptual Fusion integrates diverse perception experts as image priors to provide explicit information on visual elements and adopts an efficient MLLM as a centric pivot to mimic advanced MLLMs' perception abilities. We carefully select 1M highly representative images from uncurated LAION dataset and generate dense descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments validate that our engine outperforms its counterparts, where the resulting dataset significantly improves the perception and cognition abilities of existing MLLMs across diverse vision-language benchmarks, especially with high-resolution images as inputs. The dataset and code are publicly available at https://github.com/baaivision/DenseFusion.

1 Introduction

DenseFusion-1M addresses the shortage of detailed image-text data with a captioning pipeline that fuses visual experts and an MLLM pivot. The paper reports that training on the dataset improves MLLM performance across vision-language benchmarks, especially for text recognition and high-resolution perception.

1 Introduction

  • Current caption engines often miss important visual clues, including text, object attributes, localization, and external knowledge, while human- or GPT-4V-based captioning is expensive to scale.Source: §1 · pp. 1–2
  • Perceptual Fusion combines diverse visual experts as image priors with a low-budget MLLM pivot to integrate information and generate dense descriptions.Source: §1 · pp. 2–3
  • DenseFusion-1M provides dense descriptions of multiple objects, attributes, spatial relations, text, and world knowledge for diverse images.Source: §1 · p. 3
  • The trained MLLM outperforms existing state-of-the-art MLLMs across 10 vision-language benchmarks, especially on detailed text recognition and high-resolution image perception.Source: §1 · p. 3

2 Related Work

Related work covers MLLM architectures and high-resolution perception, large-scale image-text datasets, and efforts to create more detailed annotations. The paper positions its approach as a low-budget route to large-scale, hyper-detailed captions.

Large Multi-Modality Models

  • High-resolution MLLMs use patching, multiple encoders, dynamic crops, or multi-scale features to capture small visual clues, which can benefit from meticulous image descriptions.Source: §2 · p. 3

Image-Text Datasets

  • Image-text datasets range from synthetic web-scale captions and GPT-generated conversations to detailed GPT-4V or human-annotated captions, while region-level projects provide localized recognition and tags.Source: §2 · p. 4
  • Human-in-the-loop datasets offer detailed annotations but incur labor costs and restrict scale; DenseFusion instead proposes an automated, low-budget engine using diverse vision experts.Source: §2 · p. 4

3 Methodology

The methodology constructs DenseFusion-1M by filtering LAION images, combining specialist visual information, and training a caption engine with GPT-4V-generated guidance. The resulting dataset contains a million dense captions across diverse image categories.

3 Methodology

  • The methodology covers image-source filtering, perceptual fusion from vision experts, and construction of a caption engine for DenseFusion-1M.Source: §3 · p. 4

3.1 Data Processing

  • LAION-2B supplies varied web images; processing prioritizes high resolution and representative content, then filters images below 448 short-edge pixels and selects 20 images from each of 50,000 semantic clusters.Source: §3.1 · p. 4

3.2 Perceptual Fusion

  • Generalist MLLMs and specialist caption engines struggle to produce complete descriptions of diverse visual elements, particularly in high-resolution images.Source: §3.2 · p. 5

3.2.1 Mixture of Visual Experts

  • Perceptual Fusion combines image tagging, object detection, text recognition, and world knowledge to supply complementary perceptual information to generalist MLLMs.Source: §3.2.1 · pp. 5–6

3.2.2 Perceptual Fusion Engine

  • GPT-4V generates 100K expert-enriched descriptions as guidance for a LLaVA-1.6 (7B) caption engine, which uses high-resolution images and offline specialist context.Source: §3.2.2 · p. 6

3.3 Dataset Description

  • DenseFusion-1M scales to 1 million captions averaging 190 words and 11 sentences, with categories including photos, visual art, commercial design, and infographics.Source: §3.3 · p. 6, Figure 4 · p. 7

4 Experiments

Experiments evaluate DenseFusion-1M pretraining across vision-language benchmarks and examine perceptual fusion, resolution, and vision-encoder tuning. The reported gains span standard and high-resolution settings, with qualitative analyses of expert cues.

4.1 Implementation Details

  • Evaluation followed the LLaVA-1.5 setup across VQA and multimodal-understanding benchmarks, with DenseFusion-1M used for pretraining alignment.The evaluated model configuration used a CLIP vision encoder, a Vicuna language model, and S2 high-resolution training.Source: §4.1 · p. 7

4.2 Main Results

  • DenseFusion-1M pretraining consistently improved baseline models across vision-language benchmarks and outperformed ShareGPT4V; LLaVA-S2 also surpassed LLaVA-1.6 with one-fifth the visual tokens.The authors report that high-resolution training with their data further improves on the low-resolution version.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 Ablation Study

  • Vision-encoder fine-tuning improved performance, but full fine-tuning did not significantly outperform tuning the last 12 layers; the authors used the latter for fair comparisons.The study compared frozen, half-fine-tuned, and fully fine-tuned encoders.Source: §4.3 · p. 9, Table 4 · p. 9
  • On a 100K-example ablation subset, perceptual fusion improved benchmarks, especially OCR-rich TextVQA, with larger relative gains for high-resolution MLLMs.Table 3 compares perceptual fusion across MLLMs with different resolutions.Source: Table 3 · p. 9, §4.3 · p. 9
  • Detected objects help the caption engine focus on individual objects and add details and attributes; it still produces comprehensive captions when some expert information is absent.The visual analysis also examines contributions from world knowledge, text recognition, and detection.Source: §4.3 · p. 9, Figure 6 · p. 10

5 Conclusion

The conclusion presents DenseFusion-1M as a response to limited high-quality image-text data for detailed visual perception. It describes a perceptual-fusion pipeline that uses LAION-2B and vision experts to produce one million descriptions.

5 Conclusion

  • The paper addresses limited high-quality image-text data by developing a low-budget caption engine for high-resolution images and hyper-detailed captions.The conclusion frames the dataset as a resource for MLLM visual-text alignment.Source: §5 · p. 10
  • DenseFusion-1M is constructed from LAION-2B using perceptual fusion to guide a multimodal model in integrating information from vision experts.Source: §5 · p. 10
  • The pipeline yields one million descriptions that the authors expect will support more effective alignment between visual and textual data in MLLMs.Source: §5 · p. 10

A Overview

The overview directs readers to appendix material on DenseFusion-1M, framework implementation, and additional examples and results.

A Overview

B Dataset

DenseFusion-1M is a million-description image dataset built from a LAION subset and annotated with a caption engine that integrates diverse vision experts.

B Dataset

  • DenseFusion-1M contains 1 million hyper-detailed image descriptions derived from a subset of LAION.Source: B Dataset · p. 16
  • The dataset is designed to enhance the perceptual abilities of multimodal large language models.Source: B Dataset · p. 16
  • Its descriptions were annotated using a caption engine with perceptual fusion that integrates diverse vision experts.Source: B Dataset · p. 16

C.1 Training Details.

Training proceeds through pre-alignment, pre-training, and instruction tuning, with different parameter groups trained at each stage. The section also identifies Table 6 as the source of hyper-parameter settings.

C.1 Training Details.

  • Pre-alignment trains only the projector, producing more stable and slightly enhanced performance.Source: §C.1 · p. 16
  • Pre-training unfreezes the last 12 Vision Encoder layers, the Language Model, and the projector.Source: §C.1 · p. 16
  • Instruction tuning uses the original LLaVA-1.5 data and LLaVA-mix-665K to fine-tune the projector and language model.Source: §C.1 · p. 16
  • Table 6 is identified as the location of the training hyper-parameter settings.Source: Table 6 · p. 16

C.2 Prompt Engineering

The prompt-engineering process guides detailed caption generation by combining vision-expert information with caption context. The prompts specify broad visual coverage while requiring factual, concise descriptions.

C.2 Prompt Engineering

  • The construction pipeline uses carefully crafted prompts to guide GPT-4V toward comprehensive and accurate annotations informed by vision experts.Source: §C.2 · p. 16
  • The GPT-4V prompt asks for foreground and background elements, object attributes and positions, composition, and visible text in its original language.Source: §C.2 · p. 16
  • The prompt requires factual descriptions without speculation, mood, or reasoning commentary, and limits captions to 192 words.Source: §C.2 · p. 17
  • The prompt allows short captions to contribute relevant world knowledge but instructs the model to ignore inaccurate or irrelevant external details.Source: §C.2 · p. 17
  • The caption-engine prompt supplies world-knowledge captions, detection boxes, and OCR as external information, while treating boxes as references rather than caption content.Source: §C.2 · p. 17

D Visualizations on DenseFusion-1M

Caption reconstructions are reported as more consistent with original images than those from other caption engines. The visualizations show detailed descriptions spanning product, people, bookstore, and artwork images.

D Visualizations on DenseFusion-1M

Loading 2407.08303v2…