PaperlayerView exact revision on arXiv

Source-linked AI summary

MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs

Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, Jiaqi Wang

arXiv:2406.11833v2cs.CVcs.AIcs.LG

TL;DR

Open-source LVLMs remain focused on simplified single-turn, single-image inputs, while real-world interaction calls for multi-turn, multi-image conversations over long histories. MMDU benchmarks these capabilities and MMDU-45k provides instruction-tuning data; evaluation finds a substantial gap between proprietary and open-source models, while fine-tuning improves benchmark performance.

  • Problem

    Open-source LVLMs primarily focus on single-turn, single-image inputs, while real-world interaction requires multi-turn conversations with multiple images and long-context histories.

    Source: §1 · p. 1

  • Method

    MMDU constructs multi-turn dialogues from clustered Wikipedia images and descriptions with GPT-4o-assisted questions and human-refined answers, and MMDU-45k supplies instruction-tuning data.

    Source: §1 · pp. 2–3

  • Results

    GPT-4o scored 70.2% on MMDU versus 42.8% for the best open-source model, and fine-tuning on MMDU-45k improves performance across benchmarks.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    MMDU evaluates multi-turn, multi-image dialog understanding, and MMDU-45k fine-tuning is associated with improved performance on MMDU and existing benchmarks.

    Source: §1 · p. 3

    MMDU primarily covers English and daily scenarios rather than multilingual abilities or specialized domain expertise.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models(LVLMs). While current open-source LVLMs demonstrate promising performance in simplified scenarios such as single-turn single-image input, they fall short in real-world conversation scenarios such as following instructions in a long context history with multi-turn and multi-images. Existing LVLM benchmarks primarily focus on single-choice questions or short-form responses, which do not adequately assess the capabilities of LVLMs in real-world human-AI interaction applications. Therefore, we introduce MMDU, a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and multi-image conversations. We employ the clustering algorithm to ffnd the relevant images and textual descriptions from the open-source Wikipedia and construct the question-answer pairs by human annotators with the assistance of the GPT-4o model. MMDU has a maximum of 18k image+text tokens, 20 images, and 27 turns, which is at least 5x longer than previous benchmarks and poses challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs using MMDU reveals that open-source LVLMs lag behind closed-source counterparts due to limited conversational instruction tuning data. We demonstrate that ffne-tuning open-source LVLMs on MMDU-45k signiffcantly address this gap, generating longer and more accurate conversations, and improving scores on MMDU and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA:+1.2%). Our contributions pave the way for bridging the gap between current LVLM models and real-world application demands. This project is available at https://github.com/Liuziyu77/MMDU.

1 Introduction

MMDU addresses the gap between open-source LVLMs’ single-turn, single-image focus and the demands of multi-turn, multi-image, long-context interaction. It combines a benchmark for these conversations with MMDU-45k instruction-tuning data.

1 Introduction

  • Open-source LVLMs focus primarily on single-turn, single-image inputs, while real-world interaction requires multi-image conversations and comprehension of long-context histories.Existing benchmarks partially assess these abilities but have limited total tokens and do not provide a complete picture of interaction capabilities.Source: §1 · pp. 1–2
  • MMDU builds multi-turn dialogues from relevant Wikipedia images and descriptions selected using clustering, with GPT-4o-assisted question generation and human-refined ground-truth answers.The collection process uses human annotators to assess and refine GPT-4o’s responses.Source: §1 · p. 2
  • MMDU combines up to 20 images, 17 turns, and 18k text-plus-image tokens with open-ended, free-form multi-turn evaluation.Its outputs are intended to prioritize scalability and explainability over close-ended questions and concise answers.Source: §1 · p. 2
  • 42.8% for the best open-source model versus 70.2% for GPT-4o shows a substantial performance disparity on MMDU.The authors identify multi-turn and multi-image instruction-tuning data as a practical need for open-source LVLMs.Source: §1 · p. 2
  • Fine-tuning InternLM-XC2 on MMDU-45k improves its performance by 14.5% on MMDU, 1.1% on MMStar, 1.5% on MathVista, and 1.2% on ChartQA.MMDU-45k contains 45k instruction-tuning examples, with random sampling of human verification rather than exhaustive human evaluation.Source: §1 · p. 3

2 MMDU Benchmark

MMDU is designed to test multi-image, multi-turn dialogue understanding in long-context settings. Its construction combines clustered Wikipedia image-text data, GPT-4o-generated dialogue, and expert review, with evaluation by GPT-4o across six dimensions.

2.1 Benchmark Overview

  • MMDU contains 110 dialogues and more than 1,600 questions, spanning 2–20 images and up to 18K image-text tokens.The authors note that dialogue quality and image recognition can deteriorate as image count or context length increases.Source: §2.1 · p. 3

2.2 Benchmark Construction

  • Wikipedia entries and image captions are clustered into related sets of 2–20 images, then GPT-4o drafts multi-turn questions that experts review and correct.The benchmark combines single-image and multi-image questions in its dialogues.Source: §2.2 · pp. 3–4
  • The interleaving format lets the authors concatenate dialogue components to create conversations with dozens or even hundreds of images.The authors describe the design as supporting dialogues of theoretically unlimited length.Source: §2.2 · p. 4
  • Each Q&A receives at least two manual review rounds, with experts correcting hallucinations and errors.Review combines preliminary checks by regular reviewers with in-depth expert examination and modification.Source: §2.2 · pp. 4–5

2.3 Evaluation

  • GPT-4o compares each response with a reference answer across six dimensions, then aggregates turn-level and sample-level scores.The dimensions are Creativity, Richness, Visual Perception, Logical Coherence, Answer Accuracy, and Image Relationship Understanding.Source: §2.3 · pp. 5–6

3 MMDU-45k for Instruct Tuning

MMDU-45k is built from clustered Wikipedia material into multi-image, multi-round instruction-tuning conversations. The dataset has long contexts and many turns; its manual quality inspection sampled 5% of the data.

3.1 Dataset Construction

  • Wikipedia entries are clustered using encoded text and a cosine-similarity threshold of τ = 0.75, then filtered information guides GPT-4o dialogue generation.InternLM-chat-20B performs information extraction and filtering before GPT-4o generates multi-image, multi-round dialogues.Source: §3.1 · p. 6
  • MMDU-45k spans categories including geography, history, culture, mathematics, physics, chemistry, animals, plants, and food.The authors state that this category range is intended to cover varied aspects of real life.Source: §3.1 · p. 6
  • Inspection sampled 5% of MMDU-45k, and the estimated probability of hallucinations and errors in that subset was below 5%.The authors say reviewing the full dataset was infeasible because of its size.Source: §3.1 · p. 6

3.2 Dataset Statistics

  • MMDU-45k contains 45k conversations averaging 5k image-text tokens and 9 Q&A turns, with maxima of 17k tokens and 27 turns.Each example includes content from 2–5 images, and the authors report that its context length and turn count exceed existing instruction-tuning datasets.Source: §3.2 · pp. 6–7

4 Experiments

The experiments compare 15 LVLMs on MMDU, examine adding MMDU-45k during supervised fine-tuning, and test multi-image performance and context-length and SFT-strategy ablations.

4.1 Main Results on MMDU

  • GPT-4o averages 70.2% on MMDU, while the best open-source model reaches 42.8% or lower, revealing a substantial performance gap.The authors speculate that scarce open-source instruction-tuning data with multi-turn and multi-image abilities contributes to the gap.Source: §4.1 · p. 7
  • InternLM-Xcomposer2 benefits more from MMDU than LLaVA1.5; the authors point to its different LLM backbone and broader training and architecture advantages.They suggest these advantages may support stronger generalization.Source: §4.1 · pp. 7–8

4.2 Fine-tuning Results using MMDU-45k

  • MMDU-45k improves MMDU performance and benefits existing multi-image and short-form QA benchmarks; examples show longer, more accurate dialogues.The comparison includes LLaVA 1.5 and InternLM-XC2 in supervised fine-tuning.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Multi-Image Benchmark Results

  • LLaVA1.5+MMDU-45k improves across five multi-image benchmarks, with the largest reported gain, 6.9%, on Mantis (sequence).MMMU results include only multi-image questions; Mantis is tested with merge and sequence methods.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Ablation Study on Token Length and SFT Strategies

  • Increasing context length improves performance, and fine-tuning on MMDU-45k can increase LLaVA’s context-window size.The tested fine-tuned model extends its context window to 8k using RoPE interpolation, compared with the baseline’s 2–4k window.Source: §4.4 · p. 9, Table 5 · p. 9
  • “Continue training” and “Add to the existing pool” yield essentially the same final outcomes.The comparison is reported in the token-length and SFT-strategy ablation.Source: §4.4 · p. 9, Table 5 · p. 9

5 Related Work

Related work covers LVLM benchmarks and visual instruction-tuning datasets, noting that existing evaluations do not combine long-context, multi-turn, and multi-image conversation capabilities.

LVLM Evaluation Benchmarks

  • LVLM benchmarks assess capabilities across tasks and domains, while specialized benchmarks target areas such as science, mathematics, OCR, and diagram analysis.The passage presents these benchmarks as standardized ways to measure performance and track progress.Source: §5 · p. 9
  • Prior benchmarks cover multi-turn chats of up to three turns or multi-image comparisons with up to four images, but not both alongside long conversational context.The authors identify this combination as a gap for conversation-oriented evaluation.Source: §5 · p. 9

LVLM Instruction-Tuning Datasets

  • Visual instruction-tuning datasets address general instruction following and specialized skills, while MMDU-45k is presented as the first open-source dataset combining multi-turn, multi-image, and long-context tuning.Examples of specialized datasets include caption generation, document understanding, and video comprehension.Source: §5 · p. 9

6 Conclusion

The paper introduces MMDU and MMDU-45k to evaluate and improve LVLMs in multi-image, multi-turn, long-context conversations. It reports a performance disparity between closed- and open-source models and acknowledges limitations in benchmark scope and potential fine-tuning risks.

6 Conclusion

  • MMDU evaluates multi-image, multi-turn, long-context dialog understanding, and MMDU-45k contains 45k instruction-tuning examples.The paper reports that fine-tuning on MMDU-45k improves performance across LVLM benchmarks.Source: §6 · pp. 9–10
  • The authors report that closed-source models such as GPT-4o outperform open-source LVLMs, and suggest scarce suitable open-source instruction-tuning data may contribute to the disparity.The dataset is presented as a response to this limitation.Source: §6 · p. 9
  • MMDU focuses on English and daily scenarios, not multilingual ability or specialized domain expertise.The authors specifically distinguish this daily-scenario focus from mathematical problem-solving in MathVista.Source: §6 · p. 10
  • Because MMDU-45k is built on Wikipedia, fine-tuned models may perpetuate biases and English-language preferences, and may produce factuality or hallucination issues.The authors frame these as potential societal risks.Source: §6 · p. 10

Appendices

The appendices provide further MMDU and MMDU-45k details, additional experimental discussion, and dataset datasheets.

Appendices

  • Appendix A gives project and benchmark download URLs, while Appendices B and C detail MMDU and MMDU-45k, respectively.Appendix D provides evaluation analysis and Appendix E provides datasheets for both datasets.Source: Appendices · p. 14

B More Details of MMDU

  • The MMDU details include the data-clustering pipeline, dialogue-generation prompt, generated-example visualizations, and comparison with existing benchmarks.Source: B More Details of MMDU · p. 14

B.1 Data preparation

MMDU preparation clusters related Wikipedia entries and uses their images and text to generate multi-image, multi-turn dialogues. GPT-4o drafts questions and image-grounded answers using prompts designed for breadth and generality.

B.1 Data preparation

  • Wikipedia images, captions, main content, and categories are collected; captions and categories support clustering, while images, captions, and content inform dialogue generation.The entry content is summarized by InternLM-chat-20B before the summary, captions, and images are passed to GPT-4o.Source: §B.1 · p. 14

B.2 Prompt of Dialogue generation

  • GPT-4o generates questions using image and textual context, but answers them based solely on the images and their interrelations.The authors avoid textual cues in answer generation so the data can be used across contexts without reliance on highly specific textual knowledge.Source: §B.2 · p. 14
  • Question prompts span descriptors, comparisons, social and cultural contexts, historical significance, emotions, symbolism, and relationships.The prompt casts GPT-4o as a question-design expert.Source: §B.2 · p. 14
  • Clustering groups Wikipedia entries with high relevance, using captions and categories as tags.Figure 7 illustrates the clustering pipeline.Source: §B.1 · p. 14, Figure 7 · p. 15
  • The placeholder “<image-i>” marks image positions in questions and answers, allowing image positions to be rearranged.The authors say this format theoretically permits conversations with more images and longer question-answer sequences.Source: §B.2 · p. 15

B.3 Example Visualization of MMDU

The examples illustrate multi-image, multi-round questions that revisit images, add new ones, and require detailed descriptions and cross-image understanding.

B.3 Example Visualization of MMDU

  • The examples use multi-round Q&A to show how questions can progress from image-specific content to relationships across images.Example (b) also asks about the subjects’ applications and environments.Source: §B.3 · pp. 15–16
  • Example (a) includes questions about each of two images and a later question that revisits both, testing long-text comprehension and memory.The figure displays three rounds of the multi-image Q&A dialogue.Source: §B.3 · p. 16, Figure 9 · p. 17
  • Question 1 in Example (b) asks for descriptions of five images; Questions 2 and 3 each add three images and ask about applications and environments.The answers provide detailed descriptions of image content and relations.Source: §B.3 · p. 16, Figure 8 · p. 16

B.4 Related Work with Existing Benchmarks

The appendix contrasts MMDU with existing benchmarks and describes annotation and MMDU-45k dataset details, including its scalable format and broad category coverage.

B.4 Related Work with Existing Benchmarks

  • MMDU contains 421 images and 1,645 Q&A pairs, with around 15 turns for 2–20 images, whereas MMVet’s 218 questions each concern one image.MMBench uses multiple-choice questions, which the authors say do not adequately assess generative and conversational capabilities; ConvBench has 577 multi-turn conversations aligned with single images.Source: §B.4 · pp. 17, 19
  • The other cited benchmarks cover distinct tasks, including balanced multimodal abilities, visual mathematics, diagram reasoning, image-context hallucination, and charts.These include MMStar, MathVista, AI2D, HallusionBench, and Chart QA.Source: §B.4 · pp. 18–19

B.5 More Details about Human Annotators

C. More Details of MMDU-45k

  • MMDU-45k uses the same construction method as MMDU, and its image-position format lets users stack dialogues to create longer contexts and more images.Users can modify the sequence identifier “i” in “<image-i>” to combine dialogues from different batches.Source: C More Details of MMDU-45k · p. 20, §C.1 · p. 20

C.1 The powerful scalability of MMDU-45k

  • MMDU-45k is presented as a building block for training data or test questions tailored to users’ needs, without collecting images and text from scratch.The authors describe it as enabling dialogues of any desired length.Source: §C.1 · p. 20

C.2 The Richness and Diversity of MMDU-45k

  • Wikipedia clustering yields categories spanning geography, history, culture, nature, vehicles, and sciences, which the authors say broadens coverage across daily life.The category word cloud is shown in Figure 12.Source: Figure 12 · p. 19, §C.2 · p. 20

D More Details

  • The appendix provides further evaluation details, including judgment prompts and quantitative and qualitative results.The section introduces the material that follows.Source: D More Details · p. 20

D.1 Judgment prompt

The appendix details GPT-4o’s scoring rubric and reports comparisons across judgment models, human-score concordance, example evaluations, finetuning settings, clustering checks, and reasoning dimensions.

D.1 Judgment prompt

  • GPT-4o scores responses on six dimensions from 0 to 10 and assigns an overall score; the total is normalized by question count and multiplied by 10.The six dimensions are creativity, richness, visual perception, logical coherence, answer accuracy, and image relationship understanding.Source: §D.1 · p. 20, Figure 13 · p. 21
  • GPT-4o and GPT-4-turbo show similar scoring trends, while Claude3-Opus scores generally slightly higher but is more conservative on image relationship understanding.The comparison is shown in Table 6 and Figure 14.Source: §D.1 · pp. 20–21, Table 6 · p. 22, Figure 14 · p. 22
  • GPT-4o’s Pearson, Spearman, and Kendall similarities with human scores are 97.5%, 97.3%, and 89.0%, respectively.Claude3-Opus and GPT-4-turbo also closely align with human scores, with similarities of (91.4%, 92.7%, 89.0%) and (97.2%, 97.0%, 88.5%), respectively.Source: §D.1 · pp. 21–22

D.2 More cases

  • The appendix presents individual question-answer examples with images, ground truth, model responses, and GPT-4o scoring rationales.Examples compare InternLM-Xcomposer2 with and without MMDU-45k.Source: §D.2 · pp. 22–23

D.3 Finetune details

  • Finetuning LLaVa1.5-7B and InternLM-Xcomposer2 mixed MMDU-45k with llava-665k for one epoch at a learning rate of 2e-4.Source: §D.3 · p. 23

D.4 Cluster Accuracy

  • Clustering uses Wikipedia tags and image captions; five random 100-entry samples received GPT-4o related-image accuracy scores of 94%, 90%, 92%, 89%, and 91%.The authors interpret these results as indicating high clustering accuracy.Source: §D.4 · p. 23

D.5 Reasoning in MMDU

  • MMDU’s stated reasoning dimensions include visual perception, image relationships, high-level semantics, and cross-modal reasoning.The appendix says these abilities are considered when constructing QA pairs and points to cases illustrating them.Source: §D.5 · p. 23

E Datasheet for Datasets

The datasheet identifies MMDU and MMDU-45k as resources for evaluating and improving LVLMs’ multi-turn, multi-image conversation abilities, and includes illustrative Q&A cases.

E Datasheet for Datasets

E.1 Motivation

  • MMDU and MMDU-45k were created to evaluate and improve LVLMs’ abilities in multi-turn and multi-image conversations.Source: §E.1 · p. 23
  • The paper’s authors created the datasets, and the work was supported by Shanghai AI Lab.Source: §E.1 · pp. 23–24

E.2 Composition

This section describes the composition and documentation of MMDU and MMDU-45k, including their scale, dialogue length, collection, preparation, distribution, and maintenance. It also summarizes checklist disclosures on ethics, societal impact, access safeguards, licensing, human-subject research, reproducibility, and experiment time.

E.2 Composition

  • MMDU contains 110 dialogues, 1,645 Q&A pairs, and 421 images; MMDU-45k contains 45,000 dialogues and 410k Q&A pairs.MMDU dialogues average 15 Q&A rounds and contain 2–20 images; MMDU-45k dialogues average 9 rounds and 3 images, with 2–5 images each. MMDU-45k is the training set; MMDU is the test set. MMDU is partly sampled from MMDU-45k and partly manually generated.Source: §E.2 · pp. 25–27

E.3 Collection Process

  • The authors used Wikipedia text and images, clustering methods, and GPT-4 to construct the dataset; paper co-authors collected, verified, and modified MMDU in May 2024.The authors state that the dataset does not contain individual-specific information.Source: §E.3 · pp. 28–29

E.4 Preprocessing/cleaning/labeling

  • Preprocessing selected more coherent clustered data, removed low-resolution images, and manually checked and modified MMDU for quality.The raw data was saved, and Python was used for preprocessing, cleaning, and labeling.Source: §E.4 · p. 29

E.5 Uses

  • The documentation reports no prior dataset use or linked repository, and specifies no additional tasks or tasks for which the dataset should not be used.It also reports no stated impacts on future uses from the dataset’s composition, collection, or preprocessing.Source: §E.5 · p. 29

E.6 Distribution

  • The dataset was slated for release on Hugging Face in mid-June 2024 under CC BY-NC 4.0, with no third-party IP or export restrictions reported.The documentation says it would not be distributed to third parties outside the creating entity.Source: §E.6 · p. 30

E.7 Maintenance

  • The authors will maintain the dataset, upload a new version if corrections are needed, and continue hosting older versions.Users wishing to contact the maintainers or contribute extensions are directed to the paper’s authors.Source: §E.7 · p. 30

NeurIPS Paper Checklist

  • The checklist says the paper’s claims match its results, limitations are discussed in Section 6, and reproducibility instructions are in Appendices A and D; error bars are omitted for computational cost.The authors also state that the code, model checkpoint, and data will be publicly released.Source: NeurIPS Paper Checklist · pp. 31–33
  • The authors state that experiments did not require much time and that the research adheres to the NeurIPS Code of Ethics.The checklist also asks whether compute resources are sufficiently documented, but the supplied passages do not provide those resource details.Source: NeurIPS Paper Checklist · p. 34
  • The checklist says potential societal impacts are discussed in Section 6 and that users will have to follow usage guidelines to access the model and datasets.The authors say these guidelines are intended to support responsible use and mitigate misuse risk.Source: NeurIPS Paper Checklist · pp. 34–35
  • The authors say existing assets are credited and used under CC-BY 4.0, and that new assets will be documented upon release.Source: NeurIPS Paper Checklist · pp. 35–36
  • The authors mark crowdsourcing and research with human subjects as not applicable, stating that the paper involves neither.Source: NeurIPS Paper Checklist · p. 36
Loading 2406.11833v2…