PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception

Yanpeng Sun, Jing Hao, Ke Zhu, Jiang-Jiang Liu, Yuxiang Zhao, Xiaofan Li, Na Zhao, Zechao Li, Jingdong Wang

arXiv:2412.14233v3cs.CV

TL;DR

Human annotations can be costly to scale, while generated captions may lack comprehensiveness and accuracy, limiting detailed visual descriptions. EDC combines off-the-shelf visual specialists with LLMs to generate captions from object and relational attributes. The paper reports improved performance across visual understanding and reasoning tasks, especially those involving precise attribute and relationship recognition.

  • Problem

    Human-annotated captions are costly to scale, and LMM-generated captions can fall short in comprehensiveness and accuracy, including omissions of fine-grained details and relationships.

    Source: §1 · p. 1

  • Method

    EDC uses off-the-shelf visual specialists to extract instance-level and relational attributes, then prompts LLMs to integrate them into region and image captions.

    Source: §1 · p. 2

  • Results

    Experiments report improved LMM performance across visual understanding and reasoning tasks, especially those requiring precise attribute and relationship recognition.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper presents EDC as a flexible framework for integrating additional visual expertise into captioning.

    Source: §5 · p. 9

Abstract

from arXiv · show

Training Large Multimodality Models (LMMs) relies on descriptive image caption that connects image and language. Existing methods for generating such captions often rely on distilling the captions from pretrained LMMs, constructing them from publicly available internet images, or even generating them through human annotation. However, these strategies can fall short in terms of precision and granularity, particularly when dealing with complex visual reasoning tasks. In this paper, we propose to leverage off-the-shelf visual specialists, which were trained from annotated images initially not for image captioning, for enhancing the image caption. Our approach, named EDC, explores object low-level and fine-grained attributes (e.g., depth, emotion and fine-grained categories) and object relations (e.g., relative location and human-object-interaction (HOI)), and combine the attributes into the descriptive caption. By systematically integrating these rich attributes into the generated captions, EDC significantly improves the descriptive quality of the captions, providing a deeper and more nuanced understanding of the visual content. Experiments demonstrate that such visual specialists are able to improve the performance for visual understanding tasks as well as reasoning that benefits from more accurate visual understanding. The complete source code of EDC pipeline and datasets will be available at https://github.com/syp2ysy/DCE.

1. Introduction

Human and LMM-generated captions can be costly or miss fine-grained visual details and relationships. EDC combines visual-specialist attributes with LLM captioning, and its captions improve LMM performance across 14 benchmarks.

1. Introduction

  • Human captions often omit fine-grained details and contextual relationships, while LMM-generated captions remain prone to overlooking objects and attributes, including 3D spatial relations.The paper notes that missing 3D relationships are problematic for tasks requiring comprehensive scene-structure understanding.Source: §1 · pp. 1–2
  • EDC uses off-the-shelf visual specialists for instance-level and relational attributes, then prompts LLMs to combine them into region captions and a comprehensive image caption.Instance-level attributes include depth, emotion, and fine-grained categories; relational attributes include relative location and human-object interaction.Source: §1 · p. 2
  • EDC captions were reported as the most comprehensive among compared methods, and EDC was applied to annotate 1.1 million images.The dataset comprises 1 million diverse images (EDC-1M) and 118K real-world scene images (EDC-118K).Source: §1 · p. 2, Figure 1 · p. 2
  • EDC captions improved perceptual capabilities and visual-language alignment across 14 benchmarks for LLaVA-v1.5 and LLaVA-NeXT.The paper reports that both models showed outstanding performance across the benchmarks.Source: §1 · p. 2

2. Related Work

Related work covers multimodal-model architectures and efforts to improve their data and visual representations. Captioning datasets and LMM-generated captions offer scale and detail, but retain limitations in granularity, accuracy, or reliability.

2.1. Large multi-modality models

  • LMM research addresses modality bridging, visual encoding efficiency, and high-quality pretraining data, while systematic large-scale curation remains an open research direction.Examples include differing vision-language bridging mechanisms and image-block approaches intended to extract fine-grained features efficiently.Source: §2.1 · pp. 2–3, Figure 3 · p. 3

2.2. Descriptive captions

  • Caption sources include web-harvested pairs, human annotations, and LMM-generated descriptions, but captions may lack fine-grained context or suffer hallucination and misinterpretation.EDC is presented as integrating visual specialists for object recognition, scene parsing, and fine-grained attribute detection to improve caption accuracy and contextual relevance.Source: §2.2 · pp. 3–4

3. Approach

EDC uses visual specialists to extract object attributes and relationships, then combines them with captions through prompted language models. Its annotations capture a wider diversity of visual attributes and improve performance on the evaluated LLaVA models.

3. Approach

  • EDC extracts visual properties through object localization, specialist-based attribute extraction, and object-relation extraction.Object localization combines in-domain and open-world detectors, then uses non-maximum suppression to remove overlapping boxes.Source: §3 · p. 4, §3.1 · p. 4

3.1. Object Attributes

  • Instance-level attributes include fine-grained categories, low-level properties such as depth and size, emotion, and OCR.Fine-grained specialists cover animals, plants, food, logos, aircraft, landmarks, and celebrities; the animal and plant categories include 891k and 427k species, respectively.Source: §3.1 · pp. 4–5

3.2. Object Relation

  • EDC extracts human-object interactions, 2D and 3D object positions, and object counts to represent relations among image objects.2D relations use bounding boxes, while 3D relative positions are determined from depth differences.Source: §3.2 · p. 5

3.3. Captioning with Attributes

  • Prompted language models integrate object attributes with reference captions and region locations to produce detailed, coherent image captions.The prompts guide inclusion of fine-grained categories and spatial and interaction information.Source: §3.3 · p. 5

3.4. Analysis

  • EDC-annotated captions significantly improve LLaVA-v1.5 and LLaVA-NeXT performance and capture a wider diversity of visual attributes than other models and human annotations.Table 3 compares image-caption annotation methods, while the reported performance comparison is on both LLaVA-v1.5 and LLaVA-NeXT.Source: §3.4 · pp. 5–6, Table 3 · p. 6

4. Experiments

Experiments evaluate EDC captions on LLaVA-v1.5 and LLaVA-NeXT, comparing them with existing captioning methods and multimodal models. EDC-trained models perform strongly across visual question answering and multimodal benchmarks, with some task-specific limitations.

4.1. implementation details

  • Experiments use CLIP-L with Vicuna-v1.5 in two-stage training and evaluate seven VQA tasks plus five multimodal benchmarks.Instruction tuning uses LLaVA-mix-665K for LLaVA-v1.5 and LLaVA-NeXT-data for LLaVA-NeXT.Source: §4.1 · p. 6

4.2. Comparison with the State-of-the-art

  • Models trained with EDC captions outperform models using other captioning methods on most evaluation metrics, with gains attributed to fine-grained details, spatial relations, and grounded region information.The comparisons include ShareGPT4V, DenseFusion, ReCap, DCI, DOCCI, and IT.Source: §4.2 · p. 6, Table 6 · p. 8

4.3. Main Results

  • EDC-1M-trained LLaVA models achieve state-of-the-art VQA performance and outperform LLaVA-v1.5 across all VQA benchmarks; both variants are competitive on complex multimodal benchmarks.The reported gains over LLaVA-NeXT are also consistent, while EDC-1M outperforms both LLaVA baselines across all LMM benchmarks.Source: §4.3 · pp. 6–7, Table 4 · p. 7, Table 5 · p. 7
  • Performance is weaker on MMBench-CN without Chinese data, POPE amid detection noise, and TextVQA when OCR limitations and high thresholds impede fine textual detail.The paper identifies multilingual captioning and noise reduction as future improvement areas.Source: §4.3 · p. 7, Figure 4 · p. 8

4.4. Ablation Study

  • EDC’s 118K-image comparison finds its captions improve downstream performance over human and LMM annotation, with object attributes helping OKVQA and TextVQA and relational attributes helping GQA.Case examples show OCR, fine-grained aircraft identification, and spatial relationships enriching captions.Source: §4.4 · p. 8

5. Conclusion

EDC uses off-the-shelf visual specialists and LLMs to produce detailed, contextually precise captions. Its appendices describe the specialist choices, relation filtering, and structured prompts that connect visual attributes to fluent descriptions.

5. Conclusion

  • EDC combines off-the-shelf visual specialists for perception with LLMs for cognitive reasoning, producing captions with detailed visual attributes and contextual precision.The conclusion highlights improved performance on visual understanding and reasoning tasks requiring precise attribute and relationship recognition, and presents EDC as a flexible framework for additional visual expertise.Source: §5 · pp. 8–9

A.1. The Specific Visual Expert Model

  • EDC uses Baidu APIs for OCR and some logo, celebrity, and landmark recognition because open-source OCR performance and labeled-data availability are limited.The framework lets users select visual specialists according to their resources.Source: §A.1 · p. 13

A.2. The Detail of Relation Attributes

  • EDC retains all HOI and 2D absolute-location relations, but keeps only one dominant 2D/3D relative-location relation per image to limit redundancy.The selected relative relation is the one with the greatest relative distance, prioritized for accuracy.Source: §A.2 · p. 13

A.3. The Prompt Templates of EDC

  • Two structured prompts first enrich object-level descriptions and then combine object captions with relation-level information into a grounded image caption.Object prompts add missing attributes and prioritize specialist values in conflicts; relation prompts use detection and depth information to describe 2D and 3D spatial relationships.Source: §A.3 · p. 13

B. Analysis of EDC datasets

The EDC dataset comprises two image-text collections, and the appendix points to additional materials describing the prompts and visual specialists used in the pipeline.

  • EDC-1M contains one million diverse image-text pairs sampled from Laion, while EDC-118K contains 118,000 real image-text pairs from COCO.The appendix says it analyzes EDC-118K captions against human and generic LMM captions.Source: B. Analysis of EDC datasets · pp. 13–14
  • Figure 5 is identified as the prompt for generating region captions from object attributes and reference captions.Source: Figure 5 · p. 14
  • Table 7 is identified as a listing of the specific visual expert models used in EDC.Source: Table 7 · p. 14

B.1. The Caption Length

The appendix compares EDC-118K caption lengths with human annotations and two generalist multimodal models.

  • InternVL2-26B captions average approximately 106 tokens, while human-generated captions are the shortest among the compared sources.Source: §B.1 · p. 14
  • LLaVA-NeXT-34B captions average around 228 tokens, while EDC-118K captions average 218 tokens, about 10 fewer.Source: §B.1 · p. 14
  • EDC-118K captions are about 10 tokens shorter than LLaVA-NeXT-34B captions, whose average is around 228 tokens.Source: §B.1 · p. 14

B.2. The Lexical Composition

EDC-118k captions contain the highest average number of lexical elements per sentence in the analysis. Word-cloud comparisons also show more frequent spatial-relation vocabulary in EDC captions than in captions from other methods.

B.2. The Lexical Composition

  • Figure 6 shows an LLM caption-generation prompt that considers relation attributes, region locations, and captions.Source: Figure 6 · p. 15
  • EDC-118k had the highest average number of lexical elements per sentence, indicating a more diverse and complex linguistic structure.The analysis examined the variety, frequency, and distribution of word categories.Source: §B.2 · pp. 14–15
  • Figure 7 compares the average frequency of lexical classes per sentence across datasets.Table 8 compares caption datasets using average token length (ATL).Source: Figure 7 · p. 15, Table 8 · p. 15

B.3. The Word Clouds

  • EDC captions used spatial-relation words more frequently than captions from other methods, including terms for 2D and 3D positions.Figure 8 presents word clouds for InternVL2-26B, LLaVA-Next-34B, and EDC.Source: §B.3 · p. 16

C. Training Details

Training uses three stages: pre-alignment, pre-training, and instruction tuning, with different model components trained at each stage.

  • During pre-alignment, only the projector is trained to establish a more stable and consistent vision-language connection.Source: C. Training Details · p. 16
  • During pre-training, the last 12 vision-encoder layers, language model, and projector are unfrozen.Source: C. Training Details · p. 16
  • During instruction tuning, the projector and language model are fine-tuned using LLaVA-mix-665K and LLaVA-NeXT data for their respective models.Source: C. Training Details · p. 16

D. Detalied Evaluation Results

EDC-generated descriptions improve MMBench reasoning performance and achieve competitive results on fine-grained and coarse perception tasks.

E. Visualizations on EDC

The visual comparisons present EDC captions as more detailed than generic LMM captions, with richer object attributes, relations, and contextual details.

  • EDC captions consistently capture more nuanced object attributes, relationships, and contextual details than InternVL2-26B and LLaVA-NeXT-34B.Figure 9 highlights object attributes, OCR, HOI, and 2D and 3D spatial relations.Source: E. Visualizations on EDC · p. 16, Figure 9 · p. 18
  • Figure 8 presents word clouds for captions from InternVL2, LLaVA-Next, and EDC.Source: Figure 8 · p. 17
  • The EDC example identifies the elephants as African Bush Elephants and adds a wooden fence and camera-relative positioning details.The InternVL2 and LLaVA-NeXT examples describe two elephants without those details.Source: E. Visualizations on EDC · p. 18
Loading 2412.14233v3…