PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models

Young-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang, Byungsoo Ko, Han-Gyu Kim, Dongyu Yao, Xuankun Rong, Eojin Joo, Seung-Ho Han, Bowon Ko, Ho-Jin Choi

arXiv:2510.16641v1cs.CV

TL;DR

Single-turn VLM benchmarks and the few existing multi-turn datasets leave gaps in coverage of realistic, diverse conversations. MULTIVERSE builds a broad multi-turn benchmark and evaluates it with checklists; even strong models remain below 50% performance, while ground-truth dialogue context improves reasoning.

  • Problem

    Most VLM benchmarks assess simple single-turn interactions, while comparatively few multi-turn benchmarks cover the breadth and depth of users’ conversational scenarios.

    Source: §1 · p. 2, §4 · p. 8

  • Method

    MULTIVERSE builds persona- and goal-based multi-turn conversations from images drawn from 12 VLM benchmarks and evaluates responses with GPT-4o using instance-specific checklists.

    Source: §1 · pp. 2–3

  • Results

    Even the strongest model, GPT-4o, achieves less than 50% performance, while ground-truth dialogue history significantly enhances reasoning, including for smaller or weaker VLMs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings underscore the challenge of sustained dialogue reasoning and the importance of contextual understanding in long-form VLM interactions.

    Source: §1 · p. 3

    In pilot experiments, generating conversations solely from an image led to less diverse and lower-quality outcomes.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Vision-and-Language Models (VLMs) have shown impressive capabilities on single-turn benchmarks, yet real-world applications often demand more intricate multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only partially capture the breadth and depth of conversational scenarios encountered by users. In this work, we introduce MultiVerse, a novel multi-turn conversation benchmark featuring 647 dialogues - each averaging four turns - derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484 tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from factual knowledge and perception to advanced reasoning tasks such as mathematics and coding. To facilitate robust assessment, we propose a checklist-based evaluation method that leverages GPT-4o as the automated evaluator, measuring performance across 37 key aspects, including perceptual accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve only a 50% success rate in complex multi-turn conversations, highlighting the dataset's challenging nature. Notably, we find that providing full dialogue context significantly enhances performance for smaller or weaker models, emphasizing the importance of in-context learning. We believe MultiVerse is a landscape of evaluating multi-turn interaction abilities for VLMs.

1. Introduction

Existing VLM benchmarks largely test single-turn interactions, while the few multi-turn benchmarks leave gaps in task breadth and reasoning depth. MULTIVERSE introduces a diverse, checklist-evaluated benchmark, and evaluations show that even strong models struggle with multi-turn dialogue.

1. Introduction

  • Most existing VLM benchmarks use simple single-turn interactions, leaving open whether models that excel there can meet users’ needs in iterative, multi-turn problem-solving.Only a few benchmarks, including MMDU and ConvBench, address multi-turn VLM conversations.Source: §1 · p. 2
  • MULTIVERSE comprises 647 conversations averaging around four turns, spanning 484 tasks and 484 interaction goals drawn from images in 12 VLM evaluation benchmarks.Its task range includes advanced reasoning such as mathematics and coding, with varied user backgrounds and query styles.Source: §1 · p. 2
  • Its checklist-based evaluation uses GPT-4o to assess responses across 37 aspects, including perception and factual correctness.The checklists consist of binary questions tailored to each query and turn.Source: §1 · p. 3
  • Even the strongest evaluated model, GPT-4o, achieves less than 50% performance; ground-truth dialogue history significantly improves reasoning, including for smaller or weaker VLMs.The evaluation covers 18 VLMs, and the authors also report that scaling model size consistently improves multi-turn interactivity.Source: §1 · p. 3

2. MULTIVERSE

MULTIVERSE is constructed from images across existing benchmarks, then shaped into persona- and goal-based dialogues that undergo manual review. Its task, interaction-goal, and image-domain distributions span varied content, while its evaluation combines checklist completion with quality assessment.

2.1. Dataset Construction

  • The dataset pipeline samples images from 12 benchmarks, creates fictional user backgrounds and goals, generates four-turn conversations, then manually filters for naturalness, correctness, and image dependence.The initial 49,700 images are deduplicated, quality-rated, categorized, and weighted-sampled; manual review retains 647 conversations.Source: §2.1 · pp. 3–5

2.2. Analysis of MULTIVERSE

  • MULTIVERSE contains 647 conversations averaging 3.91 turns, with 8 primary interaction goals, 484 sub-interaction goals, nine main tasks, and 484 subtasks.The authors report high lexical diversity in both queries and responses.Source: §2.2 · p. 5, Table 1 · p. 5
  • MULTIVERSE emphasizes reasoning tasks, followed by knowledge tasks; analysis and verification are the leading interaction goals, and charts and graphs are the most common image domain.The paper also notes more advanced reasoning tasks, including mathematics, charts and diagrams, and coding, than MMDU and ConvBench.Source: Figure 4 · p. 4, §2.2 · p. 5

2.3. Automatic Evaluation Process

  • Each query-turn receives a reviewed, instance-specific checklist; GPT-4o assesses responses using checklist completion and a 1–10 quality score scaled to 10–100.Checklist completion ratio and quality assessment are positively correlated (R2 = 0.44), and their product forms the final score.Source: §2.3 · pp. 5–6, Figure 5 · p. 5

3. Experiments

The experiments compare 18 VLMs under Oracle and Self-Prediction settings, then analyze overall performance, the effects of dialogue history, interaction goals, model scaling, and response length. All models remain below 50% in Oracle, while performance patterns vary with context and task.

3.2. Experimental Results and Analysis

  • Oracle performance improves steadily across turns; smaller or weaker models gain most, while Qwen2.5-VL-72B and Claude-3.5-Sonnet do not improve.The authors attribute the negative slopes to linguistic differences from GPT-4o, which supplies the reference dialogue history.Source: Figure 6 · p. 6, §3.2 · p. 7
  • -44.64% is the maximum performance drop from Oracle to Self-Prediction, with Qwen2.5-VL-7B also showing a -30.44 gap.The authors suggest distinct linguistic styles or response patterns between Qwen2.5-VL and GPT-4o may contribute to the larger gaps.Source: Figure 7 · p. 6, §3.2 · p. 7

3.1. Experimental Setup

  • The evaluation covers 18 proprietary and open-source VLMs, using ground-truth dialogue history by default in Oracle and additionally testing Self-Prediction.Self-Prediction uses model-generated dialogue history; Oracle uses MULTIVERSE's ground-truth history.Source: §3.1 · p. 6

3.2. Experimental Results and Analysis

  • Under Oracle, every VLM scores below 50%, GPT-4o has the highest average, and proprietary models generally lead except Qwen2.5-VL.Qwen2.5-VL surpasses Claude-3.5-Sonnet, Claude-3.7-Sonnet, and Gemini-2.0-Flash; Claude-3.7-Sonnet scores 2.7% lower than Claude-3.5-Sonnet.Source: §3.2 · p. 6
  • Among open-source models, LLaMA-3.2-11B-Vision performs worst, below LLaVA-1.5-7B.Source: §3.2 · p. 7
  • Qwen2.5-VL shows relatively stronger interactivity across goals, while weaker or smaller models struggle particularly with verifiable tasks, including optimization.Source: §3.2 · p. 7
  • Larger models generally perform better across most tasks, but Qwen2.5-VL scaling is task-dependent: 72B excels in verifiable reasoning and 7B in creative thinking.Source: §3.2 · p. 7, Figure 8 · p. 8
  • Through turn 3, response-length/performance correlation weakens, with p-values rising and R-values falling, suggesting reduced verbosity bias.Source: §3.2 · p. 7, Figure 9 · p. 8

4. Related Work

Most VLM benchmarks assess single-turn responses, while the small number of multi-turn VLM benchmarks leave advanced reasoning and varied response formats undercovered. The related work therefore frames a need for richer multi-turn evaluation of real-world interactions.

Single-Turn Evaluation Benchmarks

  • Most existing VLM benchmarks use single-turn binary, multiple-choice, or free-form responses, leaving multi-turn interaction unassessed.Source: §4 · p. 8

Multi-Turn Evaluation Benchmarks

  • Numerous multi-turn conversation benchmarks have recently been introduced in NLP to assess LLMs’ interactive capabilities [4] [17] [26] [41] [59] [66] [78], whereas only a few benchmarks evaluate VLMs’ multi-turn interaction capabilities.Source: §4 · p. 8
  • Existing VLM multi-turn benchmarks omit advanced reasoning such as equation verification and often favor straightforward statements over code or mathematical expressions.Source: §4 · p. 8

5. Conclusion

The paper presents MULTIVERSE as a diverse, challenging benchmark for VLM multi-turn interaction and introduces a checklist-based evaluation framework. Its reported conclusion is that even recent state-of-the-art VLMs struggle with these interactions.

  • MULTIVERSE combines diverse interaction goals and tasks with an instance-specific checklist framework, and the paper concludes that recent state-of-the-art VLMs still struggle with multi-turn interaction.Source: §5 · p. 8

A. Additional Related Works

Related work describes architectural and modality extensions to vision-and-language models, alongside efforts to narrow performance gaps with closed-source systems.

B. Additional Details of MULTIVERSE

The appendix reports source-dataset details for MULTIVERSE and notes that its image collection can be redistributed under the applicable non-commercial licenses.

B. Additional Details of MULTIVERSE

C.1. Interaction Goal

MULTIVERSE organizes its interaction goals into varied categories, spanning verification, creation, analysis, exploration, optimization, calculation, understanding, and research.

C.1. Interaction Goal

  • Verification contains 87 distinct goals, including answer checking, data validation, calculation verification, and design feedback.Source: §C.1 · p. 15
  • Analysis is the largest listed category, with 123 goals covering comparisons, pattern detection, performance, diagnosis, and decision-making.Source: §C.1 · p. 15
  • Exploration has 30 goals, including information gathering, idea generation, cultural exploration, species identification, and design improvement.Source: §C.1 · pp. 15–16
  • Optimization covers 48 goals, including code and process efficiency, resource allocation, readability, and design improvement.Source: §C.1 · p. 16
  • Calculation comprises 32 goals, including financial, geometric, structural, statistical, and trigonometric problems.Source: §C.1 · p. 16
  • Understanding contains 86 goals, spanning data visualization, scientific and technical concepts, code, mathematics, and historical context.Source: §C.1 · p. 16
  • Research includes 72 goals, spanning information gathering, historical and cultural research, product information, and ecological study.Source: §C.1 · pp. 16–17
  • Creation comprises 43 goals, from design and educational materials to recipes, visualizations, games, and creative ideation.Source: §C.1 · p. 17

D. More examples on MULTIVERSE

The appendix points to additional MULTIVERSE examples and supplies prompt templates for image assessment, image categorization, and constructing a fictional user persona.

D. More examples on MULTIVERSE

E. Prompt Templates used for MULTIVERSE

2. Scenario Context Creation

The templates establish a realistic image-based scenario, character, and goal, then generate progressively challenging dialogue and evaluate assistant responses with checklists and scoring prompts.

2. Scenario Context Creation

  • The character, scenario, and goal are framed as a realistic user context, with the goal expressed as a concise phrase aligned with that scenario.Source: §2 · p. 24
  • The scenario-generation prompt asks for a plausible, practical multi-turn conversation grounded in an image, a fictional character, a scenario, and a goal.Source: §2 · p. 24
  • Conversation-generation guidance requires four turns, paired user and assistant utterances, and increasingly complex user requests with detailed assistant responses.Source: §2 · pp. 24–25
  • User utterances must refer to the image by pronouns and must not name its key entity; conversations that do not require image viewing are excluded.Source: §2 · p. 25
  • Each user query receives an instance-specific checklist of Yes/No questions tied to a criterion and designed to minimize subjective judgment.Source: §2 · p. 25
  • Checklist criteria cover conversational, contextual, linguistic, factual, reasoning, perceptual, and multimodal capabilities.Source: §2 · pp. 25–26
  • The quality-assessment template scores responses from 1 to 10 against dialogue history, the image, a reference answer, and the checklist.Source: §2 · p. 27
  • A separate checklist-completion prompt asks for a Yes or No response to each checklist question.Source: §2 · p. 28

Related papers

Loading 2510.16641v1…