PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Zhipeng Liu, Haochen Wang, Zhaoxiang Zhang

arXiv:2608.02589v1cs.CV

TL;DR

Existing caption evaluations conflate visual coverage with factual reliability, leaving unclear which property best supports multimodal understanding versus text-to-image generation. CAPEval separates these dimensions and tests them through controlled captioner comparisons, finding that Coverage aligns more with understanding while Precision dominates generation.

  • Problem

    Existing benchmarks conflate caption coverage and factual reliability, leaving unclear which property better supports understanding and generation.

  • Method

    CAPEval uses human-written captions and atomic semantic checklists to decompose caption quality into Coverage and Precision, then varies caption sources in controlled training experiments.

  • Results

    Coverage is more closely associated with understanding, whereas Precision is the only statistically significant predictor of text-to-image generation quality.

  • Takeaways & Limitations

    Caption selection should be task-aware: understanding favors broad semantic Coverage, while generation benefits from high factual Precision.

Abstract

from arXiv · show

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

1 INTRODUCTION

CAPEval addresses the conflation of caption coverage and factual reliability by evaluating them separately and testing their downstream effects under controlled captioner comparisons. It finds that broad Coverage better supports understanding, whereas factual Precision is the stronger driver of generation quality.

  • Benchmark design: CAPEval decomposes caption quality into Coverage, which measures visual-content coverage, and Precision, which measures the factual reliability of expressed claims.The benchmark uses dense atomic checklists derived from high-resolution images and long-form human-annotated ground-truth captions, with a judge model computing both scores.
  • Benchmark design: The benchmark evaluates captioners end to end by varying only their captions during controlled VLM pretraining and text-to-image fine-tuning.These experiments produce aggregate understanding and generation scores for each captioner, linking caption properties to downstream performance.
  • Key findings: Caption quality profile can matter more than model scale: InternVL3.5 1B reaches avg. U = 58.5 versus 57.3 for 8B, alongside C = 48.3 versus 46.5.Within the same model family, the smaller captioner has both higher understanding performance and higher Coverage than the larger model.
  • Implications: CAPEval supports task-aware caption curation: understanding models benefit from broad semantic Coverage, while generation models benefit from high factual Precision.The paper frames the relevant caption-quality dimension as determined by the downstream objective rather than captioner scale or aggregate quality score.
  • Key findings: Precision is the only statistically significant predictor of generation quality across ten captioners, with β̂P = +0.189 on SD3.5M and β̂P = +0.235 on Qwen-Image.Both effects have p-value < 0.001, whereas Coverage is not statistically significant in either pipeline, with both p-values ≥0.171.

2 RELATED WORK

Captions are established as foundational supervision for multimodal understanding and text-to-image generation, with prior work linking improved caption quality to downstream gains. However, existing benchmarks do not explicitly disentangle coverage from factual reliability or evaluate captions through end-to-end training utility.

  • Caption supervision: Captions provide foundational supervision for multimodal understanding and text-to-image generation, and improving caption quality has been shown to improve downstream performance.In understanding, captions support cross-modal alignment and capability building; in generation, they provide semantic control over visual outputs.
  • Multimodal understanding: Multimodal understanding work uses captions to align images, text, and regions while developing image description, visual grounding, and text reading abilities.Qwen-VL is described as aligning image–caption–box tuples, while ShareGPT4V links detailed captions to downstream gains.
  • Text-to-image generation: Text-to-image systems treat captions as the primary conditioning interface, with recent systems using synthetic captions or recaptioning to improve prompt following and visual control.The passage cites Stable Diffusion 3 and DALL-E 3 as examples of caption-quality-driven generation systems.
  • Evaluation paradigms: Existing evaluations range from lexical matching to checklist, rubric, and grounded-error methods, but they do not provide a fully actionable decomposition of caption quality.Prior methods target dense visual coverage, structured scoring, grounded error localization, or joint understanding-generation consistency.
  • Research gap: No existing benchmark separates caption quality into coverage and reliability or tests caption utility as training supervision through end-to-end downstream experiments.Existing frameworks either conflate these dimensions into one scalar score or treat captions as standalone generation outputs.

3 CAPEVAL DESIGN

CAPEval evaluates caption quality through separate Coverage and Precision axes grounded in human-authored captions and verified atomic checklists. It links these intrinsic measures to downstream vision-language understanding and text-to-image generation using controlled experiments in which caption source is the sole variable.

  • CAPEval design: CAPEval pairs each image with a human-written ground-truth caption decomposed into manually verified atomic checklist items spanning eight semantic dimensions.Captions are required to describe objectively visible content comprehensively and undergo second-annotator review.
  • CAPEval design: Precision P measures the fraction of mentioned checklist items that are factually correct, capturing factual reliability.The evaluation distinguishes correctly described, incorrectly described, and omitted checklist items.
  • Data construction: The benchmark uses 300 images from four visually diverse domains, with substantial human captions and dense eight-dimensional semantic checklists.Images come from public web sources and real-world photographs with varied resolutions and aspect ratios.
  • Downstream experiments: Controlled experiments compare ten captioning models from four families across two VLM pipelines and two diffusion-based T2I pipelines, varying only the caption source.The VLMs are evaluated across 16 benchmarks, while T2I models are tested on compositional generation, object-level accuracy, and dense prompt following.

4 EXPERIMENTS

Experiments show that CAPEval’s Coverage and Precision capture distinct captioner profiles and predict downstream utility differently. Coverage aligns more strongly with understanding, whereas Precision is the dominant predictor for generation.

  • Captioner profiles: Coverage and Precision vary substantially and show no monotonic relationship across captioners, including broad but less reliable and conservative but more precise profiles.Among open-source captioners, Coverage ranges from 41.2 to 68.5 and Precision from 45.1 to 86.1.
  • Controlled comparison: Within InternVL3.5, understanding performance ranks 1B (58.5) > 4B (57.5) > 8B (57.3), while 1B has the highest Coverage (C = 48.3) and 4B the highest Precision (P = 73.5).The 1B captioner achieves the strongest understanding performance, whereas 4B achieves the strongest generation performance, showing that scale alone does not determine utility.
  • Regression analysis: For understanding, Coverage is significant for SigLIP-Qwen3 (β̂C = +0.118, p-value = 0.026) and CLIP-Vicuna (β̂C = +0.215, p-value = 0.047).Precision is not statistically significant for SigLIP-Qwen3 (p-value = 0.274) or CLIP-Vicuna (p-value = 0.445).
  • Regression analysis: For generation, Precision predicts SD3.5M (β̂P = +0.189, p-value < 0.001) and Qwen-Image (β̂P = +0.235, p-value < 0.001), while Coverage is not significant.Coverage has p-value = 0.322 for SD3.5M and p-value = 0.171 for Qwen-Image after Precision is controlled for.
  • Overall finding: Across the controlled comparison and cross-captioner regressions, understanding favors broader semantic Coverage, whereas generation depends more strongly on factual Precision.These results support evaluating caption quality as two decoupled properties rather than a single scalar.

5 CONCLUSION · APPENDIX

CAPEval decomposes caption quality into Coverage and Precision, separately measuring visual-content coverage and factual correctness. It also links these axes to downstream multimodal performance through controlled experiments that vary only the caption source.

  • 5 CONCLUSION: CAPEval introduces a decoupled framework for evaluating caption quality along two distinct axes.The framework separates properties that a single scalar objective would conflate.
  • 5 CONCLUSION: Coverage measures how much visual content a caption attempts to describe.It focuses on the breadth of the caption’s described visual content.
  • 5 CONCLUSION: Precision measures how reliably the facts described by a caption are correct.It evaluates the correctness of the caption’s expressed factual claims.
  • 5 CONCLUSION: CAPEval connects both evaluation axes to downstream multimodal performance.The framework is designed not only to diagnose captions but also to relate their properties to task outcomes.
  • 5 CONCLUSION: The connection is tested through controlled training experiments.These experiments examine downstream performance under controlled conditions.
  • 5 CONCLUSION: The experiments keep the image pool, model architecture, and training configuration fixed while varying only the caption source.This design isolates caption source as the experimental variable.

A ADDITIONAL CAPEVAL INFORMATION · A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE · A.2 IMAGE RESOLUTION DISTRIBUTION

The example supplements CAPEval with a detailed ground-truth caption and atomic checklists spanning visual content, attributes, spatial relations, composition, atmosphere, and human details. The benchmark images also cover diverse real-world aspect ratios and resolutions, while this example has no text, UI, or world-knowledge checklist items.

  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: The ground-truth caption depicts a vibrant amusement park scene with attractions, artificial palm trees, a purple roller coaster, fences, and other foreground and mid-ground elements.It also describes relationships such as occlusion, spatial placement, and background structures.
  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: The atomic checklist verifies object presence for attractions and scene elements, including palm trees, fences, bucket-style ride seats, tubular slides, pools, and inflatable inner tubes.These items are organized as semantic checks over captioned visual content.
  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: The checklist evaluates fine-grained visual attributes such as color, shape, and material, including the roller coaster’s purple color and curved form and structures made from orange metal beams.It also checks specific fence-panel colors and palm-frond color.
  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: The checklist covers scene-level information, including a vibrant amusement-park setting, a skyline of tall buildings, and a calm atmosphere.Additional checks assess warm pinkish-orange dawn-or-dusk lighting, soft diffuse light, and low contrast.
  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: The checklist also evaluates picture composition and human-related details, including element placement across depth, clothing, positions, actions, roles, and mural participation.Examples include a person near the pool’s edge, a person facing away, and people standing on a platform.
  • A.1 GT CAPTION AND CHECKLIST OF THE EXAMPLE: This example provides no text checklist items, UI checklist items, or world-knowledge checklist items.All three checklist categories are explicitly marked as absent.
  • A.2 IMAGE RESOLUTION DISTRIBUTION: The CAPEval benchmark spans a wide range of image aspect ratios and resolutions, reflecting diverse real-world visual content.Figure 8 presents the distribution of these image resolutions.

A.3 CHECKLIST DIMENSION

CAPEval uses dense, verified checklist annotations to evaluate caption quality across multiple visual dimensions. Its 300 images contain 14,965 atomic facts, averaging 49.88 checklist items per image.

  • Checklist coverage: 14,965 verified atomic facts annotate 300 CAPEval images, averaging 49.88 checklist items per image.The annotations provide a dense factual basis for caption evaluation.
  • Checklist coverage: Instance, attribute, relation, and image-level dimensions are covered across all images.Additional dimensions capture specialized visual information when applicable.
  • Evaluation scope: The diverse annotation structure enables comprehensive caption-quality evaluation beyond object recognition.Its checklist design supports assessment of varied visual content rather than object recognition alone.

A.4 GT CAPTION AND CHECKLIST ANNOTATION

CAPEval establishes human-annotated ground-truth captions and a structured checklist to represent image content comprehensively and verify claims objectively. Captions undergo secondary review, while checklist items are atomic, fixed-schema, image-grounded statements reviewed for correctness.

  • Ground-Truth Caption Annotation: Trained annotators write comprehensive English captions covering visible subjects, text, aesthetic and photographic attributes, portrait attributes, and special image properties.Descriptions address each dimension only when the corresponding image elements are present.
  • Ground-Truth Caption Annotation: Captions describe only objectively visible elements, avoid fabrication and speculation, state each fact once, and use precise rather than vague phrasing.These constraints prevent unsupported or redundant descriptions.
  • Ground-Truth Caption Annotation: A second annotator reviews each completed caption for factual accuracy and completeness, resolving disagreements through discussion before finalization.This review provides a human verification step for the ground-truth captions.
  • Checklist Annotation: Human annotators decompose every ground-truth caption into atomic, verifiable checklist statements containing a concept tag and a binary image-checkable question.The fixed schema requires eight mandatory dimensions; items must express one fact, be grounded in the caption and image, and receive secondary review.

A.5 SCORES OF DOWNSTREAM UNDERSTANDING AND GENERATION BENCHMARKS · B CAPEVAL SCORES AND PER-CAPTIONER DOWNSTREAM SCORES · B.1 CAPEVAL SCORES

The paper standardizes heterogeneous downstream benchmark scores on a common 0–100 scale and reports normalized understanding and generation results. CAPEval evaluates 14 captioners using Coverage and Precision across four semantic domains, alongside aggregate downstream scores and training configurations.

  • A.5 SCORES OF DOWNSTREAM UNDERSTANDING AND GENERATION BENCHMARKS: Raw benchmark scores are mapped to a common 0–100 scale before aggregation.This normalization makes heterogeneous benchmarks comparable.
  • A.5 SCORES OF DOWNSTREAM UNDERSTANDING AND GENERATION BENCHMARKS: Normalized understanding and generation benchmark scores are presented in Table 4.
  • B.1 CAPEVAL SCORES: Table 3 reports Coverage and Precision scores for all 14 captioners.
  • B.1 CAPEVAL SCORES: The 14 captioners are evaluated across four semantic domains.
  • B CAPEVAL SCORES AND PER-CAPTIONER DOWNSTREAM SCORES: The downstream score tables report CAPEval scores together with aggregate understanding and generation scores.Table 4 identifies Coverage (C), Precision (P), aggregate understanding (U), and aggregate generation (G).
  • B.1 CAPEVAL SCORES: Table 4 highlights CAPEval scores and aggregate downstream scores in gray.The highlighted metrics are C, P, U, and G.
  • B.1 CAPEVAL SCORES: The paper separately documents VLM training configuration in Table 5.
  • B.1 CAPEVAL SCORES: The paper separately documents T2I training configuration in Table 6.

C TRAINING CONFIGURATION · C.1 VISION-LANGUAGE UNDERSTANDING · C.2 TEXT-TO-IMAGE GENERATION

The training configuration uses standardized optimization and hardware settings while varying model components across vision-language understanding and text-to-image generation pipelines. Understanding uses distinct vision-language backbones and projector initialization, whereas generation trains only diffusion-transformer parameters with frozen text encoders and VAE components.

  • C.1 VISION-LANGUAGE UNDERSTANDING: Both vision-language pipelines use 8 GPUs, DeepSpeed ZeRO-3, bf16 precision, AdamW with lr=2×10−5, cosine scheduling, warmup ratio 0.03, checkpointing, and max length 2048.Both share LLaVA 558K pretraining data, while supervised fine-tuning combines a 665K instruction mixture with 1.24M captioner-specific captions filtered by token length.
  • C.1 VISION-LANGUAGE UNDERSTANDING: CLIP pairs ViT-L/336px with Vicuna-7B, while SigLIP pairs SigLIP-SO400M/384px with Qwen3-4B.During alignment, ViT layers ≥12 are unfrozen.
  • C.1 VISION-LANGUAGE UNDERSTANDING: CLIP warm-starts its projector from the pretrained LLaVA MLP adapter, whereas SigLIP trains its projector from scratch.This projector difference is part of the vision-language model configuration.
  • C.2 TEXT-TO-IMAGE GENERATION: Both text-to-image suites use bf16, AdamW with β1=0.9, β2=0.999, weight decay 10−4, cosine scheduling, gradient clipping at max norm 1.0, resolution 1024, sequence length 512, and global batch size 64.They use DeepSpeed ZeRO-2 for SD3.5M and ZeRO-3 with optimizer offload for QwenImage.
  • C.2 TEXT-TO-IMAGE GENERATION: Only diffusion-transformer parameters are trainable in both text-to-image suites, while text encoders and the VAE remain frozen.Embeddings or latents are precomputed rather than trained end-to-end.
  • C.2 TEXT-TO-IMAGE GENERATION: SD3.5M precomputes T5 and CLIP text embeddings, while QwenImage precomputes both VAE latents and text embeddings.The preprocessing pipelines therefore differ in whether VAE latents are precomputed.
Loading 2608.02589v1…