CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era

1 Introduction

• CapArena rankings place leading models such as GPT-4o at or above human performance, while most open-source models lag, with InternVL2-26B a strong exception.The rankings are shown in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
• Detailed captions lack reliable evaluation benchmarks because they have no explicit answers, leaving VLM captioning ability and model improvement difficult to assess.Source: §1 · p. 1
• CapArena uses over 6,000 human annotations and pairwise caption battles to evaluate 14 advanced VLMs and humans.The authors report traditional scoring methods were unsuitable for the fine-grained intricacy and diversity of detailed captions.Source: §1 · p. 2
• VLM-as-a-Judge with reference captions aligns most strongly with human judgments at both caption and model levels, while other metrics show failures or ranking biases.CLIPScore and other short-caption metrics fail in detailed captioning; METEOR has decent caption-level agreement but systematic biases across VLMs.Source: §1 · p. 2
• CapArena-Auto uses VLM-as-a-Judge to evaluate detailed captioning, reporting 94.3% correlation with human rankings at $4 per test.The automated benchmark is presented as a fast and robust evaluation pipeline.Source: §1 · p. 2

2.1 Vision-Language Models

• VLMs typically combine visual encoders with LLMs, using large-scale caption pretraining followed by supervised instruction tuning, while evaluations emphasize question answering and reasoning.Source: §2.1 · p. 2

2.2 Image Captioning and Evaluation Metrics

• Caption metrics compare generated descriptions with reference sentences; CLIP-based metrics show higher human consistency on short captions, but traditional metrics struggle with longer, detailed VLM outputs.Source: §2.2 · pp. 2–3
• Unlike recent CAPTURE and scene-graph metrics, this work combines a large-scale human-centered VLM study with metric-preference analysis using annotations 50 times larger than prior work.Source: §2.2 · p. 3

3.1 Evaluation Protocol

• Because detailed-caption grading was complex and subjective, annotators struggled to assign precise grades and showed low inter-annotator consistency, so the study shifted to pairwise comparison, validated in a preliminary study.The protocol evaluates precision and informativeness and imposes strict penalties for hallucinations. For similarly good captions, noticeably excessive length is less favorable.Source: §3.1 · pp. 3–4

3.2 CapArena: Pairwise Battle Platform

• CapArena uses anonymous pairwise battles on DOCCI images, with long human-annotated descriptions as the human baseline.The platform samples model pairs in proportion to expected confidence-interval reduction and uses Bradley–Terry coefficients to rank models.Source: §3.2 · p. 4
• The study collected 6,522 annotation instances by February 2025, with inter-annotator agreement of 0.782 on a validation sample.The validation involved 400 samples; the authors report that remaining disagreement reflected subjectivity about which image aspects to emphasize.Source: §3.2 · p. 5

3.3 VLMs Performance Analysis

• Leading models such as GPT-4o reach or surpass human-level performance in detailed captioning.Human annotators occasionally overlook image details that GPT-4o descriptions cover more comprehensively.Source: §3.3 · p. 5
• Most open-source models lag behind commercial models, including large models such as Llama-3.2-90B, while InternVL2-26B is an exception.The authors attribute InternVL2-26B’s performance to its large-scale vision encoder, InternViT-6B.Source: §3.3 · p. 5
• Strong performance on general vision-language benchmarks does not always translate to detailed-captioning performance.InternVL2-26B excels in captioning despite a relatively low MMMU score, and POPE lacks discriminative capacity for detailed captioning.Source: Figure 4 · p. 5, §3.3 · p. 6
• VLMs commonly miss unusual scenes and subtle details, struggle with scenes requiring knowledge association, and misread clock times.The paper identifies these as recurring failure cases in detailed captioning.Source: §3.3 · p. 6

4.1 Captioning Metrics

• The section surveys rule-based, CLIP-based, detailed-caption metrics, and VLM-as-a-Judge, which compares two descriptions given an image.The surveyed approaches include BLEU, METEOR, SPICE, CIDEr, CLIPScore, LongCLIPScore, Polos, FLEUR, CAPTURE, and VDC-Score; a reference-enhanced judge also uses human descriptions.Source: §4.1 · p. 6

4.2 Experiment Settings

• The evaluation compares metric decisions with human judgments on CapArena battles, measuring agreement per battle and agreement between resulting model rankings.Caption-level agreement counts matching wins, losses, or ties; model-level agreement compares rankings derived with the same ELO mechanism using Spearman’s rank coefficient and Kendall’s τ.Source: §4.2 · pp. 6–7

4.3 Results Analysis

• Traditional metrics show low caption- and model-level agreement on detailed captions, with CLIP-based representations failing to align fine details in long descriptions.Flexible descriptions also complicate n-gram matching for rule-based metrics such as CIDEr.Source: §4.3 · pp. 7–8
• GPT-4o as a judge shows the strongest consistency with human preferences, and human reference descriptions further improve its model-level agreement.The table caption identifies GPT-4o as the best-performing evaluator; the results also say reference descriptions help clarify uncertain image details.Source: Table 2 · p. 7, §4.3 · p. 8, Figure 5 · p. 8
• METEOR and Output Length have decent caption-level agreement but notably lower model-level agreement; GPT-4o-as-a-Judge has 4.4% average bias versus METEOR’s 8.2%.All metrics exhibit systematic biases, and Output Length shows particularly strong bias; the figure encodes bias direction and magnitude.Source: §4.3 · p. 8, Figure 5 · p. 8
• On the hardest Level 3/4 battles, VLM-as-a-Judge remains below inter-annotator agreement, indicating difficulty distinguishing subtle caption differences.The reported comparisons are 0.650/0.620 vs. 0.572/0.560.Source: §4.3 · p. 8

5 CapArena-Auto: An Automated Benchmark for Detailed Captioning

• Test images come from DOCCI’s test split and are sampled uniformly from 149 clusters, then filtered using CLIP features to remove overly similar samples.The paper notes the images were released after the timestamp of most existing VLMs’ training, suggesting minimal contamination risk.Source: §5 · pp. 8–9
• The benchmark evaluates 600 images through pairwise caption battles against three baselines, using GPT-4o as judge with human reference captions for support.The baselines are GPT-4o, CogVLM-19B, and MiniCPM-8B.Source: §5 · pp. 8–9
• A test model’s score sums +1 for each win, -1 for each loss, and 0 for each draw across the 600 samples.Source: §5 · p. 9
• CapArena-Auto has the highest alignment with CapArena’s golden ranking among the compared benchmarks and costs $4 per evaluation.The table caption identifies its ranking correlation as the highest; the results describe a large margin over existing benchmarks.Source: Table 3 · p. 9, §5 · p. 9

6 Conclusion

• Leading models such as GPT-4o achieve or surpass human-level performance, while open-source models lag behind.The paper reports this as a result of its large-scale human-centered empirical study.Source: §6 · p. 9
• VLM-as-a-Judge sets a new standard, while systematic biases in existing metrics affect their alignment with human preferences.The paper's analysis identifies these biases as the key factor affecting alignment.Source: §6 · p. 9
• CapArena-Auto is released as an efficient, cost-effective automated benchmark that closely aligns with human preferences.The paper presents the benchmark as a tool for detailed captioning evaluation.Source: §6 · p. 9

Limitations

• The evaluation covers 14 representative VLMs, a selection constrained by available annotation resources.Consequently, several models released after dataset compilation are not included.Source: Limitations · p. 9
• The images mostly depict everyday-life scenarios, leaving artwork and medical images unrepresented.The paper notes that adding these domains could provide a more complete picture of VLM performance.Source: Limitations · p. 9
• Several recent models released after the dataset was compiled are absent from the evaluation.The paper attributes this boundary to the available annotation resources.Source: Limitations · p. 9

A DOCCI Dataset

• DOCCI contains long, human-annotated English image descriptions, with most images showing natural indoor or outdoor scenes.The dataset targets challenges including spatial relations, counting, text rendering, and world knowledge.Source: A DOCCI Dataset · p. 12
• DOCCI captions describe objects, attributes, and actions in detailed, accurate, grammatically clear language.The annotations are intended to provide high quality and consistency for evaluating captioning models.Source: A DOCCI Dataset · p. 12

B Prompts for generating detailed caption

• Table 4 lists prompts for detailed captioning, and Figure 6 shows captions produced by one VLM using different prompts.The passages identify these resources but do not report a preferred prompt or its outcome.Source: B Prompts for generating detailed… · p. 12, Table 4 · p. 12

C Model List uesd in annotation

• The commercial models include GPT-4o-0806, GPT-4o-mini-0718, Gemini-2.0-flash-exp, Gemini-1.5-pro-002, and Claude-3.5-Sonnet-0620.These are the commercial models selected for the experiments.Source: C Model List uesd in annotation · p. 12
• The open-source models include LLama-3.2-90B-Vision-Instruct, Qwen2-VL-72B-Instruct, Qwen2-VL-7B-Instruct, Qwen2-VL-2B-Instruct, and LLaVA-v1.6-34B.These are part of the open-source model set selected for the experiments.Source: C Model List uesd in annotation · p. 12
• The remaining open-source models are LLaVA-v1.5-7B, InternVL2-26B, CogVLM2llama3-chat-19B, and MiniCPM-V2.6-8B.The experiments aim to provide a comprehensive comparison of commercial and open-source VLMs for detailed captioning.Source: C Model List uesd in annotation · p. 12

• GPT-4o described the surrounding environment in greater detail and specified that a red leash was attached to the collar and trailed across the grass.Source: D GPT-4o vs Human Example · p. 12
• Figure 6 presents detailed captions generated with different prompts.Source: Figure 6 · p. 13
• GPT-4o described the yacht in greater detail and noticed a poster on the window that human annotators did not mention.Source: D GPT-4o vs Human Example · p. 13

E Quality Control

• Experts intermittently checked annotations and gave feedback; the platform flagged consistently fast annotation times for closer review and discarded submissions under five seconds.Source: E Quality Control · p. 13

F Protocol Example

• Annotators compare paired captions against the image and may mark a tie when the difference is difficult to distinguish.Source: F Protocol Example · p. 13

G Failure Cases

• Table 7 presents four representative types of persistent model mistakes, including typical errors from weaker models in its last row.Source: G Failure Cases · p. 13, Table 7 · p. 16

H Guidelines for Annotation

• The caption-judging guidelines provide a framework, but annotators are encouraged to use informed judgment for cases they do not cover.Source: H Guidelines for Annotation · p. 13, Figure 7 · p. 17

I Prompt Template

• The authors designed a VLM-as-a-Judge prompt for pairwise judgments, building on Chen et al. (2024a) and Li et al. (2024b).Source: I Prompt Template · p. 13, Figure 8 · p. 18

J CapArena-Auto Leaderboard

• The current CapArena-Auto leaderboard is provided in Table 8.Source: J CapArena-Auto Leaderboard · p. 13, Table 8 · p. 17