PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models

Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo, Wenqi Shao, Kaipeng Zhang

arXiv:2403.20194v2cs.MM

TL;DR

Existing LVLM benchmarks typically evaluate capabilities independently in single-turn exchanges, leaving multi-turn capability interactions less directly assessed. ConvBench evaluates perception, reasoning, and creativity progressively in multi-turn conversations. Its results report a model–human performance gap and show that weak perception undermines reasoning and creativity.

  • Problem

    Existing benchmarks assess capability dimensions independently in single-turn conversations, limiting error attribution and evaluation of multi-turn interaction.

    Source: §1 · pp. 2–3

  • Method

    ConvBench evaluates perception, reasoning, and creativity progressively in multi-turn conversations, using annotated references and recursive evaluation settings for error attribution.

    Source: §1 · p. 3, §4.1 · p. 8

  • Results

    GPT-4V scores 39.51% overall in pairwise evaluation, and the reported results identify a performance gap between LVLMs and humans in multi-turn conversations.

    Source: §1 · p. 4, §5.2 · p. 11

  • Takeaways & Limitations

    The reported analysis identifies weak perception, particularly in fine-grained visual tasks, as undermining reasoning and creativity.

    Source: §1 · p. 4

Abstract

from arXiv · show

This paper presents ConvBench, a novel multi-turn conversation evaluation benchmark tailored for Large Vision-Language Models (LVLMs). Unlike existing benchmarks that assess individual capabilities in single-turn dialogues, ConvBench adopts a three-level multimodal capability hierarchy, mimicking human cognitive processes by stacking up perception, reasoning, and creativity. Each level focuses on a distinct capability, mirroring the cognitive progression from basic perception to logical reasoning and ultimately to advanced creativity. ConvBench comprises 577 meticulously curated multi-turn conversations encompassing 215 tasks reflective of real-world demands. Automatic evaluations quantify response performance at each turn and overall conversation level. Leveraging the capability hierarchy, ConvBench enables precise attribution of conversation mistakes to specific levels. Experimental results reveal a performance gap between multi-modal models, including GPT4-V, and human performance in multi-turn conversations. Additionally, weak fine-grained perception in multi-modal models contributes to reasoning and creation failures. ConvBench serves as a catalyst for further research aimed at enhancing visual dialogues.

1 Introduction

Existing LVLM benchmarks assess capabilities separately in single-turn exchanges, limiting error attribution and coverage of multi-turn assistant interactions. ConvBench evaluates perception, reasoning, and creativity progressively, and its results show substantial challenges for current models.

1 Introduction

  • Prior benchmarks assess capability dimensions independently in single-turn exchanges, making it difficult to determine whether a reasoning error stems from perception or reasoning.The paper also notes that multi-turn dialogue and instruction following matter for human preference and general-purpose assistants.Source: §1 · p. 3
  • ConvBench assesses perception, reasoning, and creativity progressively in multi-turn dialogue, enabling error attribution across capability levels.Each sample’s reasoning and creativity instructions respond to the instructions and answers at preceding levels.Source: §1 · p. 3
  • The benchmark contains 577 curated multi-turn QA samples across 71 perception, 65 reasoning, and 79 creation task types, and evaluates 19 publicly available LVLMs.Evaluation uses direct grading and pairwise comparison, with ablations for error attribution.Source: §1 · p. 3
  • Weak perception undermines LVLMs’ reasoning and creativity, while models also struggle with fine-grained recognition, object detection, and detailed descriptions.Extensive ablative evaluation supports the reported relationship between perception, reasoning, and creativity.Source: §1 · p. 4
  • GPT-4V scores 39.51% overall, 38.47% for perception, 39.34% for reasoning, and 37.61% for creativity in pairwise evaluation.The results also report a gap between model performance and human performance in multi-turn conversations.Source: §1 · p. 4

2 Related Work

Prior LVLM benchmarks range from basic perception tests to expert-level reasoning assessments, but generally treat capability types separately and emphasize single-turn interaction. ConvBench instead connects perception, reasoning, and creation in progressive multi-turn evaluation.

2 Related Work

  • LVLMs combine visual knowledge with language-model processing, and existing approaches include Q-Former alignment, caption-data training, instructional data, and high-resolution image fine-tuning.Examples include BLIP-2, MiniGPT4, LLaVA, and QWen-VL [2].Source: Large Vision-Language Models · p. 4
  • Existing benchmarks cover basic perception, broad real-world tasks, and expert-level reasoning, but often assess perception, reasoning, and creation in isolation and emphasize single-turn exchanges.Examples include VisIT-Bench for a broad task spectrum, and MathVista, MMMU, and MMT-Bench for expert-level knowledge and reasoning.Source: Large Vision-Language Models Benchmarks · pp. 4–5
  • ConvBench addresses these gaps by evaluating capabilities progressively from perception through reasoning to creation in multi-turn conversations.The benchmark’s 215 tasks are organized in a perception, reasoning, and creation hierarchy.Source: Large Vision-Language Models Benchmarks · p. 5, Figure 2 · p. 5

3 ConvBench

ConvBench organizes 577 image-instruction pairs into sequential perception, reasoning, and creation tasks across 215 task types. Its curation process develops instructions and image-conditioned references, adding focus-point criteria for open-ended creation tasks.

3.2 Data Curation Process

  • The data-curation pipeline develops three multi-turn instructions for each image and annotates referenced answers for automatic evaluation and error analysis.Source: Figure 3 · p. 6

3.1 Overview of ConvBench

  • ConvBench contains 577 image-instruction pairs across 215 task types, with sequential instructions for perception, reasoning, and creation.The benchmark includes 71 perception, 65 reasoning, and 79 creation tasks; manually annotated referenced answers support error attribution and automatic evaluation.Source: §3.1 · pp. 5–6

3.2 Data Curation Process

  • Curation proceeds through five stages, including task-category induction, instruction-conditioned caption annotation, and GPT-4V-generated reference responses that are refined as reference answers.The instructions align with perception, reasoning, and creation capabilities.Source: §3.2 · pp. 6–7
  • For open-ended creation instructions without standard answers, annotators provide focus points as criteria for assessing whether responses are instructive.Source: §3.2 · p. 7

4 Multi-turn Conversation Evaluation

ConvEval uses an LLM to assess multi-turn responses at capability levels and conversation level, with recursive evaluations designed to attribute downstream effects to perception and reasoning errors. It offers direct and pairwise grading schemes.

4 Multi-turn Conversation Evaluation

  • ConvEval uses an LLM evaluator for cost-effective, efficient assessment and provides direct and pairwise grading schemes.Direct grading assigns a 0–10 score under specific prompts, while pairwise grading reports a preference between the test response and reference.Source: §4 · pp. 7–8

4.1 Pipeline of ConvEval

  • Recursive ConvEval evaluates perception, reasoning, creation, and the overall conversation, comparing runs with perfect upstream references to examine how perception and reasoning errors affect later scores.Replacing perception responses with references supports comparisons for reasoning, creation, and overall conversation; perfect perception and reasoning references support comparisons for creation and overall conversation.Source: §4.1 · pp. 8–9
  • For ordinary model responses, ConvEval scores each capability level and then evaluates the overall conversation using instructions, responses, focal points, and level judgments.The pipeline applies a level-specific prompt and an overall-conversation prompt.Source: §4.1 · p. 8

4.2 Pairwise Grading

  • In pairwise grading, the LLM compares anonymously ordered response sets and reports the percentage of cases preferring the model; ties are disallowed and resolved arbitrarily.The evaluation input also includes instructions, responses, references, focal points, and instruction-conditioned captions.Source: §4.2 · p. 9

4.3 Direct Grading

  • Direct grading identifies the tested model and human reference and assigns 0–10 scores to each turn and the overall conversation.The prompt presents both full conversations to support assessment in context.Source: §4.3 · p. 9

5 Experiment and Analysis

The evaluation compares 19 LVLMs using progressive conversation prompts and finds substantial model–human performance gaps. Idealized perception and reasoning conditions yield higher downstream scores in several comparisons.

5.1 Evaluation Settings

  • 19 representative LVLMs were evaluated with images and progressive instructions; ConvBenchEval compared responses with human-verified answers using nine distinct scores.S1-S3 score perception, reasoning, and creation; ˆS2 and ˆS3 assume perfect perception, while ˜S3 assumes perfect perception and reasoning.Source: §5 · p. 10, §5.1 · p. 10, §5.2 · p. 10

5.2 Progressive Evaluation Comparisons and Analysis

  • 38.47, 39.34, and 37.61 (Pairwise Grading) and 7.30, 7.48, and 7.12 (Direct Grading) are GPT-4V's perception, reasoning, and creation scores, with a stark model-human discrepancy.Claude is reported as infinitely close to GPT-4V, while Claude has greater potential for improvement under progressive evaluation.Source: §5.2 · pp. 10–11
  • 11.21 and 5.31 are average reasoning and creation gains across 19 LVLMs under perfect perception with Pairwise Grading; Direct Grading gains 1.25 for each.With ideal perception and reasoning, creation scores change by +6.96 under Pairwise Grading and −0.39 under Direct Grading; the paper notes the latter may reflect less stable results.Source: §5.2 · pp. 11–12
  • Perception weaknesses concentrate in fine-grained recognition, object detection, and detailed descriptions; even with perfect perception, abstract reasoning remains challenging and creation gaps persist across applied domains.Source: §5.2 · pp. 12–13
  • Both grading strategies rank GPT-4V first and Reka second and broadly agree; pairwise grading detects fine distinctions but risks position bias, while direct grading can miss subtle pairwise differences.Table 1 reports pairwise win-rate vs human; Table 2 reports Direct Grading average scores. Figure 5 maps tasks across columns and models across rows.Source: Table 1 · p. 11, Table 2 · p. 12, §5.2 · pp. 13–14

5.3 Multi-Turn Conversation Comparisons and Analysis

  • 40.55 (Pairwise Grading) and 6.88 (Direct Grading) are GPT-4V's multi-turn conversation accuracy scores, underscoring a gap between model and human capabilities.Source: §5.3 · p. 14
  • 12.29 and 2.75 are the average multi-turn performance gains across 19 LVLMs under perfect perception and reasoning with Pairwise and Direct Grading, respectively.Multi-turn scores are generally below the average of the three turn scores, with mean differences of 0.44 and 0.29, respectively; the paper says the judge also evaluates instruction-following ability.Source: §5.3 · p. 14

6 Conclusion

ConvBench evaluates LVLM perception, reasoning, and creation through progressive multi-turn conversation assessment. The paper reports a gap from human performance and links inadequate perception to reasoning and creative failures.

6 Conclusion

  • ConvBench sequences perception, reasoning, and creation in a progressive multi-turn evaluation pipeline.The authors report a discrepancy between model and human performance and say inadequate perception can result in failures in reasoning and creative tasks.Source: §6 · p. 15
  • The benchmark is intended to identify shortcomings in multimodal AI systems.Source: §6 · p. 15

A LVLMs Configuration

C.1 Chain of Thoughts

ConvBench can examine whether decomposed questions aid reasoning, and its multi-turn approach yields better results than directly requesting reasoning or creation in a single turn.

C.1 Chain of Thoughts

  • Single-turn requests for reasoning or creation yield inferior results to the multi-turn method in Figure 2.The comparison uses GPT-4V and both Pairwise Grading and Direct Grading methods.Source: §C.1 · p. 22, Figure 2 · p. 22
  • ConvBench annotates decomposed sub-questions to test whether they contribute to solving the main question.Source: §C.1 · p. 22

Are Focus Demands neccessary?

Because creative prompts lack standard answers, ConvBench uses focal points to guide their evaluation. In a 50-instance experiment, evaluations with focal points were judged more successful than those without.

Are Focus Demands neccessary?

  • Creative instructions are open-ended and lack standard answers, so specific focal points are annotated to guide evaluation.Source: Are Focus Demands neccessary? · p. 22
  • In experiments comparing groups with all other variables held constant, evaluations using focal points had a 56% success rate versus 44% without them, manually assessed for identical responses.Source: Are Focus Demands neccessary? · p. 23
  • Figure 4 presents an example where the evaluation incorporating focus points is deemed more appropriate than the one without them.Source: Figure 4 · p. 23

Are the three turns’ judgements helpful for the overall conversation

The experiment compares overall conversation evaluations with and without judgments from the three turns. Including those judgments produced more successful evaluations and more consistently yielded an ultimate rating.

Are the three turns’ judgements helpful for the overall conversation

D Case Study

The case studies examine turn-level and overall conversation performance, illustrating perception and reasoning errors alongside creative and scientific-domain challenges. The prompt templates describe pairwise and direct grading, including evaluation of individual turns and whole conversations.

D Case Study

  • Fig. 6 tests turn-level and conversation-level performance by checking whether the third-turn response selects the second example from earlier responses.Source: Figure 6 · p. 25
  • Fig. 7 pairs accurate door-text interpretation with deficient creativity and assesses overall performance across all three turns.Source: Figure 7 · p. 26
  • Fig. 8 gives third-turn reference values—75W push power, 9.33% wheelchair mechanical power efficiency, and 136N friction—and highlights scientific-domain challenges for LVLMs.Source: Figure 8 · p. 27
  • Fig. 9 links unclear perception of countries around the Arabian Sea with impaired later reasoning; with clear perception, reasoning is accurate.Source: Figure 9 · p. 28
  • Fig. 10 attributes tangram reasoning errors to failures in perceiving the pieces and their positions, while detailed image descriptions still leave reasoning errors.Source: Figure 10 · p. 29

E Prompt Templates

Related papers

Loading 2403.20194v2…