PaperlayerView exact revision on arXiv

Source-linked AI summary

DOCCI: Descriptions of Connected and Contrasting Images

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, Jason Baldridge

arXiv:2404.19753v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Fine-grained visual descriptions and reference images are limited in existing resources for evaluating text-to-image models. DOCCI pairs 15k curated images with detailed human descriptions, and its evaluations show image-to-text gains alongside persistent text-to-image and evaluation challenges.

  • Problem

    Existing test prompts often lack fine-grained visual details and ground-truth images, while common image-caption datasets typically provide brief descriptions.

    Source: §1 · p. 2

  • Method

    DOCCI is a dataset of 15k curated images with detailed human descriptions, including related images intended to distinguish fine-grained visual differences.

    Source: §1 · pp. 2–3

  • Results

    DOCCI improves image-to-text generation with PaLI 5B, while text-to-image models omit details from long prompts and automatic metrics do not align with human evaluations.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors encourage improved model architectures and evaluation metrics suited to detailed visual descriptions.

    Source: §7 · p. 15

    DOCCI’s images are a biased sample in content and geographical extent because of their collection.

    Source: §2.1 · p. 5

Abstract

from arXiv · show

Vision-language datasets are vital for both text-to-image (T2I) and image-to-text (I2T) research. However, current datasets lack descriptions with fine-grained detail that would allow for richer associations to be learned by models. To fill the gap, we introduce Descriptions of Connected and Contrasting Images (DOCCI), a dataset with long, human-annotated English descriptions for 15k images that were taken, curated and donated by a single researcher intent on capturing key challenges such as spatial relations, counting, text rendering, world knowledge, and more. We instruct human annotators to create comprehensive descriptions for each image; these average 136 words in length and are crafted to clearly distinguish each image from those that are related or similar. Each description is highly compositional and typically encompasses multiple challenges. Through both quantitative and qualitative analyses, we demonstrate that DOCCI serves as an effective training resource for image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or superior results compared to highly-performant larger models like LLaVA-1.5 7B and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for text-to-image generation, highlighting the limitations of current text-to-image models in capturing long descriptions and fine details.

1 Introduction

DOCCI addresses the lack of detailed, reference-grounded descriptions for evaluating visual models by pairing curated images with comprehensive human annotations. Evaluations use the dataset to show gains for image-to-text generation and persistent text-to-image and evaluation challenges.

1 Introduction

  • Existing text-to-image prompt sets often omit fine-grained visual details and lack ground-truth images, while common caption datasets offer brief descriptions.DCI descriptions exceed 1,000 words but concatenate segment captions without rich linguistic structure or coherence.Source: §1 · p. 2
  • DOCCI contains 15k human-described images intentionally collected to test visual properties such as spatial relations, counting, and text rendering.The images include related or similar examples, enabling descriptions to distinguish between them.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Annotators create each description in three stages: identify salient aspects, combine them into a coherent draft, then add details and refine it.Quality-control steps are used to ensure descriptions meet annotation standards.Source: §1 · p. 3
  • DOCCI shows improved image-to-text generation with PaLI 5B, while text-to-image models omit details from long prompts and automatic metrics do not align with human evaluations.The reported text-to-image error modes include spatial relationships, counting, and text rendering.Source: §1 · p. 4

2 Dataset Construction

DOCCI’s construction combines a deliberately curated set of images with a staged process for producing detailed, coherent descriptions. Its related-image groups and annotation goals emphasize distinctions in visual content and relationships.

2.1 Images

  • The 15k images were taken by one author and family, largely in the United States, with some scenes arranged or framed to test known model limitations.The collection reflects opportunity-driven subjects and includes both everyday scenes and images selected to probe properties such as counting and spatial relations.Source: §2.1 · p. 4
  • DOCCI’s image content and geographical extent are necessarily biased by how the collection was assembled.The authors hope others will donate images to expand its visual diversity.Source: §2.1 · p. 5
  • Related, substantially similar images vary in details such as cats’ orientations and poses, apple counts and arrangements, or text formats and surfaces.These groups are intended to challenge models to reflect image details in both text-to-image and image-to-text tasks.Source: §2.1 · p. 5

2.2 Text Descriptions

  • Descriptions are intended to prioritize key visual details and organize secondary information later, covering objects, attributes, spatial relations, text, counting, scenes, and optical effects.The annotation goal also includes world knowledge and views.Source: §2.2 · pp. 5–6
  • The annotation protocol separates extracting key aspects, combining concise descriptions into a draft, and adding detail and refinement.The stages distribute the skills and workload because composing detailed descriptions from scratch is time-consuming, even for expert annotators.Source: §2.2 · p. 6

3 Dataset Analysis

DOCCI pairs long, dense descriptions with broad coverage of visual challenges, and its descriptions distinguish similar images while remaining complex but readable. Its statistics and quality analyses position it as a detailed dataset for evaluating visual understanding.

3.1 Dataset Statistics

  • DOCCI descriptions are longer on average than Stanford Visual Paragraphs, similar in length to DCI, and denser than DCI descriptions.Compared with larger datasets such as Localized Narratives and COCO, they are also more detailed.Source: §3.1 · p. 6
  • DOCCI has the highest median description length, while DCI has the highest mean because of outlier descriptions exceeding 1,000 words.The length distribution is shown in Fig. 4, with quartiles marked by vertical dotted lines.Source: §3.1 · p. 6, Figure 4 · p. 7

3.2 Challenge Types

  • Descriptions cover multiple challenge types: spatial relationships appear in 99.9%, color and state in 97%, count in 54.6%, and text rendering in 23.3%.The challenge-type analysis is an automatically generated proxy and may contain errors; descriptions average 17.7 objects.Source: §3.2 · pp. 7–8, Table 2 · p. 7

3.3 Description Quality

  • DOCCI has the highest syntactic complexity, while DCI has the highest semantic complexity; DOCCI descriptions are generally plain English, yet not overly simplistic.A spelling and grammar checker suggested an average of 0.3 errors per DOCCI description, compared with 1.2 for DCI.Source: §3.3 · p. 8, Table 3 · p. 8

4 Evaluating I2T Generation Models with DOCCI

The authors evaluate DOCCI as training data for image-to-text generation using automatic metrics and human comparisons. PaLI 5B finetuned on DOCCI outperforms larger instruction-tuned models on reference-based metrics, while human comparisons show a detail–accuracy trade-off against GPT-4v.

Setup

  • The evaluation uses PaLI 5B, LLaVA-1.5 7B, and InstructBLIP, with PaLI finetuned on 9,647 DOCCI training examples.The comparison also includes GPT-4v in the side-by-side human evaluation.Source: §4 · p. 9

Quantitative Metrics

  • 121.8 words on average: PaLI 5B finetuned on DOCCI substantially improves all reported reference-based metrics and outperforms larger instruction-tuned models.The comparison is reported on the qualification-test set, with the result summarized in Table 4.Source: §4 · p. 9, Table 4 · p. 10

Human Evaluation Results

  • PaLI provides more details than GPT-4v, but GPT-4v is more accurate; PaLI's added detail carries a risk of inaccurate information.The human evaluation focuses on precision and recall rather than writing quality.Source: Figure 6 · p. 10, §4 · p. 10

DOCCI Descriptions vs GPT-4v

  • Annotators preferred DOCCI descriptions over GPT-4v's for recall 89% of the time, although DOCCI descriptions were shorter and more detailed.For precision, annotators selected “Neutral” 59% of the time; outside that response, they judged DOCCI descriptions more accurate.Source: §4 · p. 11, Figure 7 · p. 11

5 Evaluating T2I Generation Models with DOCCI

The authors evaluate text-to-image models on DOCCI using automatic metrics, human preference, and reconstruction against reference images. Results expose disagreements among metrics and reveal model weaknesses on detailed prompts and complex imagery.

Setup

  • The evaluation generates images from DOCCI descriptions with Imagen, DALL-E 3, and SDXL, and assesses image quality, text-image alignment, and human preference.The setup includes automatic metrics and side-by-side ranking of three generated images for each description.Source: §5 · pp. 11–12

Automatic Metrics and User Preference

  • DALL-E 3 had the best mean user-preference rank at 1.42, followed by Imagen at 1.84 and SDXL at 2.38, despite automatic image-quality metrics favoring SDXL.All three models substantially underperformed the Random and Text Ret. baselines on the reported automatic image-quality metrics.Source: §5 · p. 12, Table 5 · p. 12

Image-Text Alignment

  • SDXL achieves the highest CLIPScore, but DSGVQA and DSGHuman align more closely with human evaluation and user preference, with DALL-E 3 topping DSG scores.CLIPScore truncates long descriptions at 77 tokens, whereas DSG extracts atomic validation questions from the full description.Source: §5 · pp. 12–13

Text-to-Image Reconstruction

  • Reconstruction scores suggest comparable high-level performance, but conceal nuanced deficits in recreating complex imagery; the authors call for better automatic metrics for DOCCI's level of detail.The benchmark compares generated images with original references, enabling an analysis unavailable in prompt-only evaluation sets.Source: §5 · p. 13
  • Comprehensive descriptions tend to accompany high-similarity reconstructions, while underspecified visual features are associated with low-similarity cases.The analysis also uses mixed-similarity cases to expose models' relative strengths and weaknesses.Source: Figure 8 · p. 13, §5 · p. 14

6 Related Work

Prior image-text datasets span captioning, dense descriptions, multilingual annotation, and large-scale web collection. DOCCI emphasizes description density and quality, and its human annotations retain detail advantages over machine-generated or elaborated dense descriptions.

Related Work

  • Earlier datasets developed from everyday-scene captions toward denser, multimodal, multilingual, and much larger web-collected image-text resources.Examples include Stanford Visual Paragraphs for dense captions and web datasets ranging from millions to billions of image-text pairs.Source: §6 · p. 14
  • DOCCI focuses on description density and quality, comparable in balance of size and density to Stanford Visual Paragraphs and DCI; human annotations retain advantages in detail and lack of hallucinations over machine-generated dense descriptions.DAC improves description quality with an LLM and achieves higher downstream-task performance, while the authors' human evaluation favors human annotations on detail and hallucinations.Source: §6 · p. 14

7 Conclusion

DOCCI is a 15k-image dataset with human-annotated detailed descriptions, used to expose problems in text-to-image models and evaluation. The authors identify limited input length and unreliable automatic metrics as outstanding issues.

7 Conclusion

  • DOCCI comprises 15k newly curated images with detailed human-annotated descriptions.The conclusion presents the dataset as the paper's central contribution.Source: §7 · p. 14
  • The paper uses DOCCI to showcase limited input length and unreliable automatic metrics as outstanding problems in text-to-image models and evaluation.The authors encourage future work on architectures and metrics better suited to detailed visual descriptions.Source: §7 · pp. 14–15

Appendix

  • The appendix covers image collection and curation, visual features, annotation, object statistics, text-to-image error analysis, and a dataset datasheet.Source: Appendix · p. 21

A Image Collection and Curation

The collection was built to probe challenging visual properties and includes related images that differ in details. The section also describes the visual features, annotation stages, quality controls, and annotator qualification used to create DOCCI.

A Image Collection and Curation

  • The 15,000-plus images grew from a collection of deliberately challenging scenes, including controlled examples of text, spatial relations, counting, attribute binding, and mixed media.The curator selected and cropped images to reduce visual clutter and retain distinctive examples.Source: A Image Collection and Curation · pp. 21–23
  • The collection reflects the curator’s opportunities and interests, with biases toward tennis, cars, Texas wildlife, pets, graffiti, and Austin views.The photographs were generally quick snapshots rather than polished work by a trained photographer.Source: A Image Collection and Curation · pp. 21–22
  • The authors also release DOCCI-AAR, a further set of images gathered on later trips and curated with greater freedom in cropping and composition.These images were released alongside DOCCI for research use.Source: A Image Collection and Curation · p. 23

B Key Visual Features

  • DOCCI’s visual-feature scope includes object attributes, spatial relations, text, counts, world knowledge, scenes, views, and optical effects.Counts focus on approximately twenty objects or fewer because tracking attributes for larger numbers is difficult.Source: B Key Visual Features · pp. 23–24

C.1 Annotation Pipeline

C.2 Quality Control

  • Quality control combined pilot guidelines, clustered batches with daily feedback, image privacy and safety review, and targeted checks for description errors and omissions.Low-confidence VQA checks flagged possible precision errors, while unusually short descriptions were reviewed for recall errors.Source: §C.2 · pp. 24–25

C.3 Annotator Qualification Tests

  • Stage 3 annotators first qualified by describing ten images and scoring at least 4 out of 5, with up to three attempts.Candidates who passed were invited to full-scale annotation.Source: §C.3 · p. 26

C.4 Annotation UI

The section describes interfaces for elaborating descriptions and conducting human evaluations, alongside object statistics showing diverse subject coverage in everyday scenes.

C.4 Annotation UI

  • In Stage 3, annotators see an image and two Stage 2 descriptions, then expand and refine the descriptions while reporting PII or inappropriate content.The interface also encourages Google Image Search to identify unfamiliar objects.Source: Annotating Text Description · p. 26
  • The pivot-image evaluation uses 1,000 groups of five similar images, each judged by three annotators through elimination rather than direct selection.The elimination interface reduces the need to read the full description and brings average response time below one minute.Source: Image Elimination · pp. 26–27
  • The side-by-side human evaluation collects five-point precision and recall ratings, highlighted error and omission spans, and written justifications.Source: Side-by-Side Human Evaluation · p. 27

D Object Statistics in DOCCI Images

  • Detected popular object types span a wide range of subjects from indoor and outdoor everyday scenes.Figure 12 plots counts from an object detection tool run on the images.Source: D Object Statistics in DOCCI Images · p. 27

E Error Analysis for Text-to-Image Generation models

The error analysis shows that current text-to-image models miss fine-grained content in different ways, with spatial relations and directions posing a common challenge. Limited prompt lengths may also prevent models from using all description details.

E Error Analysis for Text-to-Image Generation models

  • Prompts exceeding model input limits are truncated, potentially limiting how fully Imagen and SDXL incorporate long descriptions.The authors conclude that these examples point to difficulty adhering precisely to detailed descriptions, citing textural input understanding and architectural input-length constraints.Source: E Error Analysis for Text-to-Image… · pp. 28–29
  • All three models struggle with spatial relations and object directions, while each also makes distinct errors in style, media blending, counting, or physical plausibility.Imagen turns a mural into a photorealistic scene; DALL-E 3 blends a real car with drawings; SDXL miscounts cats and places a dog impossibly.Source: E Error Analysis for Text-to-Image… · p. 28
  • DALL-E 3 captures the statue’s serving pose toward the sun but uses the wrong viewpoint and style and adds two tennis balls; Imagen and SDXL miss the prompt’s key idea.The authors suggest the latter misses may reflect the models’ limited input lengths.Source: E Error Analysis for Text-to-Image… · p. 29
  • For the toy clock, Imagen misspells “watch” and depicts a standard clock, DALL-E 3 confuses blocks on and beneath the clock, and SDXL produces an inaccurate clock.The example combines an uncommon familiar-object variant with text rendering.Source: E Error Analysis for Text-to-Image… · p. 29
  • All three models struggle to count the toy horses; DALL-E 3 gets their relative sizes and right-facing orientation right but misplaces the tile wall, while Imagen and SDXL miss orientation.The authors suggest the latter errors may occur because orientation appears later in the description and may be truncated.Source: E Error Analysis for Text-to-Image… · p. 29

F Datasheet for DOCCI

The datasheet describes DOCCI’s intended uses, composition, collection and preprocessing, and its distribution and maintenance. It records both dataset features and boundaries relevant to its use.

F Datasheet for DOCCI

F.1 Motivation For Datasheet Creation

  • DOCCI is directly usable for text-to-image and image-to-text generation and can also support image-to-text and text-to-image retrieval.The dataset creation was funded by Google Research.Source: What tasks could the dataset be used… · p. 31

F.2 Datasheet Composition

Are there any errors, sources of noise, or redundancies in the dataset?

F.3 Collection Process

F.4 Data Preprocessing

  • The team manually reviewed images for PII, removing some and blurring detected faces, phone numbers, and URLs; descriptions were screened for PII, and raw data were not saved.Automatic tools were used to scan descriptions after annotation.Source: Was any… · p. 32

Is the software used to preprocess/clean/label the instances available?

F.5 Dataset Distribution

The datasheet states the dataset’s release terms, maintenance expectations, and legal and privacy-related handling.

F.5 Dataset Distribution

  • DOCCI was released in March 2024 under CC-BY 4.0, is available on the dataset website, and has no fees or access/export restrictions.Source: §F.5 · p. 33

F.6 Dataset Maintenance

  • The authors will maintain DOCCI and do not anticipate updates because it is designed for evaluation, though significant errors may prompt modifications.Updates will be posted on the dataset website.Source: Will the dataset be updated? · p. 33

F.7 Legal and Ethical Considerations

  • An ethical review process was conducted for the dataset.Source: §F.7 · p. 33

Does the dataset contain data that might be considered confidential?

  • Very few images as taken contained PII; detected faces, phone numbers, and URLs were removed or blurred, and the authors report that individuals cannot be identified.Text descriptions were collected from hired human annotators.Source: Does the dataset identify any… · p. 34
Loading 2404.19753v1…