Source-linked AI summary
TouchStone: Evaluating Vision-Language Models by Language Models
Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, Jingren Zhou
TL;DR
LVLM evaluations have largely emphasized recognition and reasoning rather than conversational and storytelling abilities. TouchStone combines a broad visual-dialogue dataset with detailed textual image annotations for LLM judging; GPT-4's dialogue-quality judgments align with human preferences.
Problem
LVLM assessments have focused mainly on recognition and reasoning, with limited evaluation of conversation and storytelling and challenges in scaling human ratings.
Source: §1 · pp. 1–2Method
TouchStone pairs an open-world visual-dialogue dataset spanning five categories and 27 subtasks with detailed textual image annotations for LLM-based judging.
Source: §1 · p. 3Results
GPT-4 can score dialogue quality using textual capabilities alone, with judgments highly consistent with human preferences.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
The authors conclude that current LVLMs still have ample room for improvement.
Source: §5 · p. 10The paper notes that LVLM abilities remain limited in spatial understanding, dense text recognition, and mathematical tasks.
Source: §4.4 · p. 10
Abstract
from arXiv · showhide
Large vision-language models (LVLMs) have recently witnessed rapid advancements, exhibiting a remarkable capacity for perceiving, understanding, and processing visual information by connecting visual receptor with large language models (LLMs). However, current assessments mainly focus on recognizing and reasoning abilities, lacking direct evaluation of conversational skills and neglecting visual storytelling abilities. In this paper, we propose an evaluation method that uses strong LLMs as judges to comprehensively evaluate the various abilities of LVLMs. Firstly, we construct a comprehensive visual dialogue dataset TouchStone, consisting of open-world images and questions, covering five major categories of abilities and 27 subtasks. This dataset not only covers fundamental recognition and comprehension but also extends to literary creation. Secondly, by integrating detailed image annotations we effectively transform the multimodal input content into a form understandable by LLMs. This enables us to employ advanced LLMs for directly evaluating the quality of the multimodal dialogue without requiring human intervention. Through validation, we demonstrate that powerful LVLMs, such as GPT-4, can effectively score dialogue quality by leveraging their textual capabilities alone, aligning with human preferences. We hope our work can serve as a touchstone for LVLMs' evaluation and pave the way for building stronger LVLMs. The evaluation code is available at https://github.com/OFA-Sys/TouchStone.
1 Introduction
The introduction presents TouchStone as a response to narrow and costly LVLM evaluation, proposing broad open-world visual dialogue coverage and LLM-based judging. It reports that GPT-4's judgments are highly consistent with human preferences.
1 Introduction
- Existing LVLM evaluations use limited question sets or traditional task metrics, while human ratings are difficult to scale and may not reflect human preferences.Binary judgments and answer-choice accuracy may also fail to capture open-ended dialogue complexity.Source: §1 · pp. 1–3
- TouchStone introduces an open-world visual dialogue dataset spanning five ability categories and 27 subtasks, from basic recognition and comprehension to literary creation.Images and questions are manually annotated and verified.Source: §1 · p. 3
- Detailed image annotations convert visual inputs into text so advanced LLMs can judge dialogue quality without human evaluators or visual-augmented LLMs.The method compares LVLM responses with GPT-4-generated answers and uses position balancing to address positional bias.Source: §1 · p. 3
- GPT-4 shows highly consistent judgments with human preferences when evaluating LVLM response quality.TouchStone uses text-based LLM judging to score dialogue quality and discern hallucinations.Source: §1 · p. 3
2 Related Work
The related-work section traces language and vision-language model development, then argues that evaluations need to better reflect real-world interactions. TouchStone aims to assess dialogue broadly and account for hallucinations.
2.1 Large Language Models
- Language models advanced from pretrained systems to instruction-following and conversational models, including InstructGPT and ChatGPT.The discussion highlights their language comprehension, generation, and interaction capabilities.Source: §2.1 · pp. 3–4
2.2 Vision-Language Models
- Vision-language research has connected visual encoders and language models through varied pretraining and alignment approaches, including Flamingo, BLIP-2, and LLaVA.The surveyed work also expands capabilities such as OCR, visual localization, and visual instruction following.Source: §2.2 · p. 4
2.3 Vision-Language Model Evaluation
- Existing LVLM evaluations use diverse task metrics and formats that may not capture human preferences, and hallucination remains a concern.TouchStone's stated contrast with VisIT-Bench is its broader assessment of model performance alongside hallucinations, rather than an instruction-following focus.Source: §2.3 · p. 5
3 Approach
TouchStone combines a five-dimension visual-dialogue dataset with LLM-based evaluation of LVLM answers. The dataset spans 908 questions, and the evaluation uses image annotations and GPT-4 scoring to compare answers and rank models.
3.1 Data Collection and Statistics
- The five dataset dimensions range from basic description and visual recognition to comprehension, storytelling, and multi-image analysis.Storytelling covers literary creation based on visual content; multi-image tasks include comparison and summarization.Source: §3.1 · pp. 5–6
3.2 Evaluation
- GPT-4 generates reference answers from manually annotated image descriptions and questions, while LVLMs answer using images and questions; GPT-4 then scores the answers for usefulness, relevance, and accuracy.Scores are averaged across questions, and a second scoring round swaps answer positions to mitigate position bias.Source: Figure 3 · p. 6, §3.2 · pp. 6–7
3.1 Data Collection and Statistics
- Recognition questions make up 44.1% of the 908 questions, followed by comprehension at 29.6%, basic description at 15.3%, storytelling at 7.4%, and multi-image analysis at 3.6%.The dataset emphasizes basic abilities, with recognition the largest category.Source: §3.1 · p. 6
3.2 Evaluation
- The authors compare their evaluation results with human judgments and report that the comparison supports using fine-grained annotations to let an LLM judge multimodal content.They describe open-world dialogue evaluation as still lacking a definitive solution.Source: §3.2 · p. 7
4 Results and Analysis
The results section validates the evaluator against human judgments, compares LVLM abilities, analyzes hallucinations, and discusses limitations and improvement directions.
4.1 Consistency evaluation
- 72.2% model2human consistency versus 78.4% human-generated consistency, a 6.2% difference; consistency declines with task difficulty, motivating more powerful judges as model ability rises.The authors report higher consistency for basic recognition and lower consistency for higher-scoring models.Source: §4.1 · p. 8
4.2 Performance Comparison
- Literary creation separates models most: MiniGPT4 (Zhu et al., 2023), InstructBLIP (Dai et al., 2023), and PandaGPT (Su et al., 2023) often describe rather than create, while LLaVA (Liu et al., 2023a) and mPLUG-Owl excel.The authors also report room for improvement in recognition, description, and understanding analysis; the stronger storytelling models typically undergo supervised fine-tuning involving the LLM.Source: §4.2 · p. 8
- Recognition varies by model and task; text recognition remains weak except for Qwen-VL, math and chart reasoning have gaps, and multi-image comparison and continuity summarization are weak.Attribute and emotion recognition differences are relatively small, but general recognition varies more; Qwen-VL has an advantage in text recognition and chart analysis, while multi-image inputs can reduce recognition accuracy.Source: §4.2 · pp. 8–9
4.3 Analysis of Model Hallucinations
- Descriptive scores suffer from hallucinations: PandaGPT has the highest degree, while InstructBLIP and Qwen-VL have the lowest scores, partly by favoring shorter answers.GPT-4 assesses hallucination by comparing model descriptions with fine-grained human annotations; the authors also identify hallucination as a contributor to poor descriptive scores.Source: §4.3 · p. 9, §4.2 · p. 9
4.4 Limitations and Potential Areas for Improvement
- Complex positional and structural relationships remain difficult; the authors suggest location-rich data such as detection, segmentation, and scene graphs may help spatial understanding.They frame spatial understanding as an improvement direction and connect it to layout understanding and spatial planning.Source: §4.4 · p. 10
- Single-image pre-training has limited utility for comparing and summarizing multiple images, prompting a call for more interleaved image-text data.Suggested sources include webpages, articles, and news.Source: §4.4 · p. 10
- Aligning vision encoders to LLMs quickly builds LVLMs, but abilities in spatial understanding, dense text recognition, and mathematics remain limited.The authors call for further exploration of improving LLM ability through multimodal content.Source: §4.4 · p. 10
- Insufficient visual input can contribute to hallucinations, while higher input resolution could improve recognition of small objects, dense text, and fine-grained details.The authors note that most LVLMs use 224×224 images and propose stronger judgment of nonexistent content and greater answer-image consistency as possible hallucination-reduction directions.Source: §4.4 · p. 10
5 Conclusion
TouchStone evaluates LVLM abilities with strong LLMs as judges and a dataset spanning five categories and 27 subtasks. Validation shows that GPT-4 can score dialogue quality using textual capabilities alone, aligning with human preferences.
5 Conclusion
- GPT-4 can score dialogue quality using textual capabilities alone, with scores aligning with human preferences.The paper reports this through validation of its evaluation method.Source: §5 · p. 10
- TouchStone uses strong LLMs as judges to comprehensively evaluate LVLM abilities.Detailed image annotations and descriptions transform multimodal input into a form language models can understand.Source: §5 · p. 10
- The TouchStone dataset covers five ability categories and 27 subtasks, from fundamental recognition and comprehension to literary creation.Source: §5 · p. 10