Source-linked AI summary
Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
Fan Lu, Wei Wu, Kecheng Zheng, Shuailei Ma, Biao Gong, Jiawei Liu, Wei Zhai, Yang Cao, Yujun Shen, Zheng-Jun Zha
TL;DR
Short-caption benchmarks and isolated matching provide limited support for evaluating comprehensive image captions. CompreCap uses human-annotated directed scene graphs to assess object coverage, attributes, and relations. Its evaluation aligns closely with human scores across LVLMs.
Problem
Short-caption annotations and isolated matching of objects, attributes, and relations limit evaluation of comprehensive captions.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
CompreCap provides a benchmark for more in-depth assessment of LVLMs’ comprehensive image captioning.
Source: §5 · p. 8The paper does not quantitatively evaluate hallucinations in generated captions and plans to assess hallucination error rates in future work.
Source: G. Limitations and Broader Impact · p. 21
Abstract
from arXiv · showhide
Generating detailed captions comprehending text-rich visual content in images has received growing attention for Large Vision-Language Models (LVLMs). However, few studies have developed benchmarks specifically tailored for detailed captions to measure their accuracy and comprehensiveness. In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view. Concretely, we first manually segment the image into semantically meaningful regions (i.e., semantic segmentation mask) according to common-object vocabulary, while also distinguishing attributes of objects within all those regions. Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image. Based on our directed scene graph, we develop a pipeline to assess the generated detailed captions from LVLMs on multiple levels, including the object-level coverage, the accuracy of attribute descriptions, the score of key relationships, etc. Experimental results on the CompreCap dataset confirm that our evaluation method aligns closely with human evaluation scores across LVLMs.
1. Introduction
Existing short-caption benchmarks and isolated scene-graph matching do not adequately assess comprehensive captions. CompreCap uses bound attributes and directional relations to evaluate detailed captions, and its scores align closely with human evaluations across LVLMs.
1. Introduction
- Short-caption annotations and isolated matching of objects, attributes, and relations limit assessment of comprehensive captions.Isolated matching can still score a caption when color attributes are assigned to the wrong objects.Source: §1 · p. 2
- CompreCap provides object annotations with attributes bound to objects and directional relationships between them.It uses manually refined semantic segmentation and annotates salient relationships to form a directed scene graph.Source: §1 · p. 2
- The evaluation parses detailed captions and assesses object coverage, associated attributes, and relationships against CompreCap annotations.It also includes a tiny-object VQA task to investigate the correlation between fine-grained perception and detailed-caption quality.Source: §1 · p. 2
- CompreCap evaluation metrics achieve strong consistency with human evaluation scores across LVLMs.The benchmark compares 10 LVLMs and reports human performance among the best on comprehensive captioning and fine-grained-object VQA.Source: §1 · p. 2
2. Related work
Related work covers the development of large vision-language models and approaches to image-caption evaluation. CompreCap’s directed scene graphs preserve object–attribute bindings and directional relations that isolated matching can disrupt.
2.1. Large Vision-Language Models
- LVLM development progressed from QFormer-based systems to models using MLPs or high-resolution image processing.Examples include InstructBLIP and MiniGPT4-v2 [12] [55], LLaVA-1.5, LLaVA-Next, and miniGemini-HD [25].Source: §2.1 · pp. 2–3
- Table 1 compares CompreCap with prior caption-evaluation datasets by annotation components.Its caption states that all compared datasets are human-annotated; the supplied passage does not report the table’s individual entries.Source: Table 1 · p. 3
2.2. Image Caption Evaluation
- Traditional caption metrics and CLIP-based metrics have limitations for long captions, while DetailCaps individually matches extracted scene-graph elements.Individual matching can disrupt the binding of attributes to objects and the direction of relations; CompreCap instead uses human-annotated integrated scene graphs.Source: Figure 2 · p. 3, §2.2 · p. 4
3. CompreCap
CompreCap combines manually annotated scene graphs with fine-grained-object VQA and structured evaluation of detailed captions. Its pipeline measures object coverage, attribute descriptions, and directional relations against the annotations.
3. CompreCap
- CompreCap is a manually annotated scene-graph benchmark for evaluating comprehensive image captions, with a separate fine-grained-object VQA task.The benchmark’s design and the evaluation pipeline are organized around objects, attributes, and relationships.Source: §3 · p. 4
3.1. Scene Graph Annotation
- CompreCap re-annotates common-object categories and segmentation maps, then binds object attributes and directional subject–verb–object relations.It contains 560 images, 412 object categories, and an average mask coverage of 95.83%.Source: §3.1 · p. 4
3.2. Question-Answering Annotation
- CompreQA-P tests detection of tiny objects, while CompreQA-Cap tests selection of their correct captions.Presence questions include absent objects; caption options include one correct and two incorrect descriptions, with errors manually checked against image content.Source: §3.2 · p. 4
3.3. Structured Evaluation on Detailed Caption
- The evaluation decomposes captions into sub-captions and matches extracted nouns to annotated object categories using Sentence BERT similarities.It also measures pixel coverage for objects accurately mentioned in captions.Source: §3.3 · pp. 4–5
- For mentioned objects, an LLM scores bound attribute descriptions from 0 to 5, and the relation score evaluates descriptions containing both objects in an annotated relation.The unified score weights scaled object, attribute, and relation scores at 25%, 35%, and 40%, respectively.Source: §3.3 · p. 5
4. Experiments
The experiments compare 10 LVLMs on CompreCap, assess caption quality and fine-grained object perception, and test whether scene-graph-based evaluation aligns with human judgment. LLaVA-Next-34B and GPT-4o perform best overall among LVLMs, while human performance exceeds all LVLMs and the proposed scores show strong consistency with human evaluations.
4.1. Experimental Setups
- The experiments evaluate 10 LVLMs by generating detailed captions with a shared prompt and scoring object, attribute, and relation levels, alongside fine-grained object VQA.Caption generation and evaluation are repeated three times, with mean and standard deviation reported.Source: §4.1 · pp. 5–6
4.2. Comprehensive Image Caption Evaluation
- Humans outperform all LVLMs across every CompreCap metric, and human caption and fine-grained VQA performance supports the benchmark’s rationale.The Table 2 caption states that human performance surpasses all LVLMs across every metric.Source: Table 2 · p. 6, §4.2 · p. 6
- LLaVA-Next-34B slightly outperforms GPT-4o in object coverage, attribute scores, and pixel coverage, while GPT-4o’s relation score is lower by 0.14 on a five-point scale.GPT-4o generates about 60% of LLaVA-Next-34B’s output length, and caption quality does not correlate with length.Source: §4.2 · p. 6
- The proposed evaluation maintains strong consistency with human scoring, whereas n-gram metrics and CLIPScore are limited in evaluating detailed captions.The cited limitations are difficulty modeling long contextual dependencies for n-gram methods and a 77-token constraint for CLIPScore.Source: §4.2 · p. 7
- Scene-graph-based evaluation aligns more faithfully with human scores than directly scoring captions with Llama3, under which human captions score below most LVLMs.The proposed method’s human performance is significantly superior, as illustrated in Figure 5.Source: §4.2 · pp. 7–8, Figure 5 · p. 7
4.3. Fine-grained Object VQA
- Most omitted objects are tiny, occupying less than 5% of the image’s segmentation area; fine-grained object VQA rankings generally align with caption-quality rankings.Humans exceed all LVLMs on both VQA tasks, with accuracy over 96%; GPT-4V is not included because its API was deprecated after the final VQA annotations.Source: §4.3 · p. 8, Figure 6 · p. 8
5. Conclusion
The conclusion presents CompreCap as a human-annotated benchmark for evaluating LVLM captions through objects, their attributes, and directional relationships. Its scene-graph-based evaluation aligns with human scores across LVLMs.
5. Conclusion
- Human performance on CompreQA-P and CompreQA-Cap surpasses all LVLMs, as reported in the accuracy table.The table caption states that human performance surpasses all LVLMs but provides no specific values in the supplied passage.Source: Table 4 · p. 8
- CompreCap integrates a wide array of image objects with bound attribute descriptions and directional relationships between corresponding objects.The benchmark uses comprehensive scene-graph annotations to match generated captions at object, attribute, and relationship levels.Source: §5 · p. 8
- The proposed evaluation metrics faithfully align with human evaluation scores across various LVLMs.The conclusion characterizes CompreCap as practical and credible for more in-depth LVLM assessment.Source: §5 · p. 8
Supplementary Material
The supplementary analyses examine LVLM components, evaluator and parser choices, metric validity, caption stability, and prompt effects. They report strong human-score alignment for the proposed evaluation method, while showing that direct scoring and traditional caption metrics are less reliable.
A.1. Analysis on the effect of components of LVLMs
- Increasing image resolution can help LVLMs recognize more objects and describe key relations more precisely, but LLava-Next-13B still falls short on CompreQA-Cap, showing resolution alone is insufficient.Compared with LLaVA-1.5-13B, LLava-Next-13B performs better on CompreQA-P but worse on CompreQA-Cap; upgrading the base LLM to Yi-34B [48] widens its captioning advantage and gives LLava-Next-34B a significant advantage on both fine-grained VQA metrics.Source: §A.1 · p. 12
A.2. Consistency with human evaluation scores
- Our CompreCap metrics show strong consistency with human evaluation scores across 10 LVLMs and human performance.The supplementary results describe this alignment as supporting the credibility of CompreCap and its evaluation method.Source: §A.2 · p. 12
A.3. Compare with directly scoring with Llama3; A.4. Ability of Llama3 to Distinguish Caption Quality
- Directly scoring dense captions with Llama3 can rank human captions behind most LVLMs, whereas scene-graph-based matching aligns more faithfully with human assessments.In a separate test, Llama3 distinguishes good from corrupted captions when attributes are shuffled or relation subjects and objects are swapped.Source: §A.3 · p. 12, §A.4 · p. 13
A.5. Analysis on the influence of LLM evaluator
- GPT-4 evaluator results also align strongly with human scores, while Llama3 is selected for its speed and stability advantages in non-API evaluation.Direct scoring of dense text with Llama3 is reported to fail to yield solid evaluation results; the authors emphasize decoupling captions and matching object, attribute, and relation annotations.Source: §A.5 · p. 13
A.6. Analysis on the effect of parsers
- Using NLTK instead of spaCy preserves high alignment with human evaluation, indicating that the metrics are not sensitive to the parser choice.The method parses nouns and decouples generated captions into a hierarchical structure.Source: §A.6 · p. 13
A.7. Results of traditional caption metrics on CompreCap
- Traditional caption metrics fail to align with human judgments on comprehensive captions, and human-generated captions do not outperform all LVLMs on every metric.The tested metrics include BLEU-4, METEOR, ROUGE-L, CIDER, and CLIPScore.Source: §A.7 · pp. 13–14, Figure S4 · p. 15
A.8. Analysis on the stability of generated detailed caption
- Across repeated generations by GPT-4o and LLaVA-Next-34B, evaluation scores remain relatively consistent despite variation in caption length.The models were each prompted three times.Source: §A.8 · p. 16
A.9. Analysis of prompts for caption generation
- Prompts that emphasize visible objects improve caption quality across metrics, and adding emphasis on relationships further improves comprehensiveness.The comparison uses three prompts, progressing from general image description to objects and then objects plus their relationships.Source: §A.9 · p. 16
A.10. Analysis on the failure cases in the fine-grained object VQA
The fine-grained object VQA failure analysis identifies tiny objects as a recurring challenge for LVLMs, with additional examples of GPT-4o misreading background details.
A.10. Analysis on the failure cases in the fine-grained object VQA
- Across nine LVLMs, error captions tend to involve tiny objects with segmentation-map coverage below 2%, and smaller objects pose greater description challenges.Examples show GPT-4o overlooking a television in a carriage and mischaracterizing telephone and bicycle attributes.Source: §A.10 · pp. 16–17
B. Comparison with Similar Benchmarks
CompreCap is distinguished from related benchmarks by its object, attribute, relation, and pixel-level mask annotations, as well as longer captions and wider coverage. The comparison also identifies limitations in existing benchmarks for comprehensive long-caption evaluation.
B. Comparison with Similar Benchmarks
- Related benchmarks lack object-bound attributes and directional relations, while Visual Genome’s shorter annotations and lower coverage can omit main objects and it lacks a method for evaluating coherent long captions.Visual Genome instead generates captions sequentially for image regions and uses traditional n-gram metrics.Source: B. Comparison with Similar Benchmarks · p. 17
- CompreCap provides object, attribute, and relation annotations with pixel-level masks, averaging 172 words per caption versus Visual Genome’s 92.Its bounding-box coverage is 99.23%, compared with 85.26% for Visual Genome.Source: B. Comparison with Similar Benchmarks · p. 17
C. Data Comparison with MSCOCO
CompreCap re-annotates MSCOCO Panoptic Segmentation images because its labels and maps are not precise enough and some visible objects are missing or overlapped. The resulting dataset adds object types, longer text, and inter-object relations to the same images.
C. Data Comparison with MSCOCO
- The authors re-annotate 560 MSCOCO Panoptic Segmentation images, using a common-category vocabulary and more precise labels and masks for common objects.The source annotations could omit visible objects outside its common-object categories and could overlap other visible objects.Source: C. Data Comparison with MSCOCO · p. 17
- Across nine LVLMs, inaccurate captions tend to concern tiny objects whose segmentation-map coverage is below 2%.The distribution is reported for error objects in CompreQA-for-Caption.Source: Figure S7 · p. 18
- CompreCap adds more object types, longer texts, and relations between objects while retaining the same images as the MSCOCOsub subset.Table S9 presents data statistics for the two datasets.Source: C. Data Comparison with MSCOCO · p. 18, Table S9 · p. 18
D. Data Samples from CompreCap dataset
CompreCap samples organize image content through object, attribute, and relation annotations. The supplementary examples also show segmentation refinements and a tiny-object description failure by GPT-4o [35].
D. Data Samples from CompreCap dataset
- CompreCap’s scene-graph samples combine human annotations of objects, attributes, and relations.The annotations organize the global scene description.Source: D. Data Samples from CompreCap dataset · p. 18
- CompreQA-for-Presence samples include equal numbers of present and absent objects.Source: D. Data Samples from CompreCap dataset · p. 18
- GPT-4o [35] fails to comprehend the visual content of tiny objects in the shown examples.The figure uses green for the answer and red for error options; magnification is for display only.Source: Figure S8 · p. 19
- CompreCap refines common-object class labels and segmentation maps relative to MSCOCO panoptic segmentation.Examples include adding or re-annotating labels and improving pixel-level segmentation accuracy.Source: Figure S9 · p. 20
- CompreQA-for-Caption samples pair one accurate object description with two inaccurate descriptions.Source: D. Data Samples from CompreCap dataset · p. 20
E. Discussion about More Long Caption Datasets
CompreCap’s annotations distinguish it from long-caption datasets that omit object-level structure or object relations. The paper also reports the human-participant groups and their roles in captioning and scoring.
E. Discussion about More Long Caption Datasets
- Unlike IIW [16] and DOCCI [34], which provide one long text per image, CompreCap annotates objects, attributes, relations, and segmentation maps; DCI [42] omits object relationships.Source: E. Discussion about More Long Caption… · p. 20
F. Information of Human Participants
- The study involved 50 participants, split into groups of 20, 10, and 20 for annotation, manual captioning, and scoring, with no personnel overlap across stages.Source: F. Information of Human Participants · p. 20
- Ten experts each described 56 images without being instructed to focus on objects, attributes, or relations; 20 experts independently scored 6,160 model- and human-generated captions.The average score was used as the final score for models and human performance.Source: F. Information of Human Participants · p. 20
G. Limitations and Broader Impact
CompreCap is proposed for comparing LVLMs’ descriptions of text-rich visual content, with tiny-object VQA as an additional perception task. The authors note that they do not quantitatively evaluate hallucinations.
G. Limitations and Broader Impact
- CompreCap supports identifying which LVLM describes text-rich visual content more accurately and adds VQA based on tiny objects to assess fine-grained perception.Source: G. Limitations and Broader Impact · p. 20
- Hallucinations—incorrect descriptions and nonexistent visual information—are not quantitatively evaluated; the authors plan to assess them with error rates in future work.Source: G. Limitations and Broader Impact · p. 21
- CompreQA-for-Presence examples contain equal numbers of present and absent objects.Source: Figure S12 · p. 23