PaperlayerView exact revision on arXiv

Source-linked AI summary

A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributions

Giyeong Oh, Junghun Park, Yuhan Bae, Youngjae Yu

arXiv:2610.00952v1cs.CVcs.AI

TL;DR

Recaptioned corpora are difficult to audit because length proxies miss register artifacts and downstream T2I benchmarks entangle corpus and training choices. The paper introduces a reusable matched-budget, five-axis audit using CBUs; across cross-corpus pairs, the released surface has higher supported claim yield under both Judges.

  • Problem

    Length-based proxies miss caption-register artifacts, while downstream T2I benchmarks entangle the corpus with training choices, complicating corpus-scale audits.

    Source: §1 · p. 1

  • Method

    A reusable matched-budget audit profiles recaptioned supervision across five axes and uses CBUs as a shared claim set for caption counts and image-conditioned judgments.

    Source: §1 · p. 2, §3.1 · p. 4, §4 · p. 5

  • Results

    Across four cross-corpus pairs at B = 64, supported CBU per caption rises by +3.4 to +6.4 under both Qwen and Gemma Judges, with lower unsupported risk in all eight cells.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    With the captioner fixed, the policy alone raises supported CBU per caption by +3.0 to +3.6 over a plain detailed-caption instruction.

    Source: §1 · p. 2

    Image-output comparisons combine corpus effects with text-encoder, length-sampling, dropout, persona-augmentation, generator, and training-budget choices.

    Source: §7 · p. 9

Abstract

from arXiv · show

Recaptioned image-text corpora are now standard for text-to-image (T2I) training, with vision--language model (VLM) captioners replacing sparse alt-text by dense descriptions. A recaptioned corpus is a supervision distribution induced by a documented captioning policy ($\pi$), captioner ($V_c$), and source corpus ($C$). Length-correlated proxies miss caption-register artifacts and downstream T2I benchmarks entangle the corpus with training choices, so this distribution is hard to audit at corpus scale. We introduce a reusable matched-budget audit framework for recaptioned supervision distributions $D_{\pi,V_c,C}$: at a fixed text budget of $B = 64$ it reports a five-axis profile spanning prompt-side coverage, image-conditioned faithfulness, and caption-surface health, with claimed controllable basic units (CBU) as the common claim unit. We instantiate the framework on seven paired comparisons over five public source corpora. Across the four cross-corpus pairs, the released surface raises supported CBU per caption by $+3.39$ to $+6.36$ under both Qwen and Gemma Judges, and on CC12M the same framework exposes a long-vs-dense frontier that is consistent across both judges and four budgets. We release the audited multi-source recap corpus ($\approx$ 490M) together with the audit-artifact bundle.

1 Introduction

VLM captions bring describe-style artifacts into T2I supervision, while captioning policy determines which biases and how much controllable content enter the corpus. The paper formalizes this corpus as an audit target, proposes a matched-budget framework, and reports stronger supported claim yield for the released surface across paired comparisons.

1 Introduction

  • VLM captions inherit describe-style frames that sit outside user-prompt register, and recaptioning policy determines which biases enter the corpus and how much text carries controllable content.T2I models train on caption-image pairs but respond to user prompts.Source: §1 · p. 1
  • Public releases are often read through length or downstream T2I benchmarks, so the paper audits the captioning distribution directly with a fixed-budget, five-axis profile.The audit covers prompt-side coverage, image-conditioned faithfulness, and caption-surface health.Source: §1 · pp. 1–2
  • Across four cross-corpus pairs, the released surface raises supported CBU per caption by +3.39 to +6.36 under both Judges and exposes a CC12M long-vs-dense frontier across four budgets.The framework is instantiated on seven paired comparisons over five public source corpora.Source: §1 · p. 2
  • Controllable basic units (CBUs) provide a shared claim set for caption-only claim counts and image-conditioned judgments of supported yield and unsupported risk.The same claims are counted in captions and re-asked against paired images.Source: §1 · p. 2
  • The paper releases an audited multi-source recap corpus of about 490M image identities with artifacts for rerunning the audit on future targets.The artifact bundle includes the tokenizer, prompt pools, request templates, CBU and Judge schemas, caption hashes, and metadata.Source: §1 · p. 2

2 Related Work

Related work spans dense VLM recaptioning, prompt adaptation, image-grounded faithfulness checks, and corpus-level probes. This paper assembles those audit elements around matched text budgets, user-prompt register, and claim-level image verification.

VLM recaptioning

  • Dense VLM captioning scaled from DALL-E 3 to open corpus releases, while structured caption schemas and principled prompt design extend that lineage.The cited releases include Recap-DataComp-1B (32), PixelProse (49), and ShareGPT4V (11).Source: §2 · p. 3

Prompt-side adaptation and the caption-prompt gap

  • Inference-side prompt rewriting, expansion, and optimization adapt user prompts toward caption-style training text, while caption length and claim density remain distinct quantities.Most public recap pipelines use describe-style frames; Re-LAION-Caption (35) uses an enumerated four-part template.Source: §2 · p. 3

Image-grounded faithfulness and corpus-level auditability

  • Prior work offers claim-level VQA, global alignment scores, and downstream image-output benchmarks, but recap releases rarely combine caption-only and image-caption audits.The corpus-level framework assembles audit inputs and adds a matched text budget and user-prompt register to claim-level image verification.Source: §2 · p. 3

3 Audit Targets and the Diagnostic Gap

The paper defines the recaptioned corpus as a policy- and captioner-induced supervision distribution, then compares surfaces over fixed source rows. Length and downstream scores alone miss surface patterns that corpus-level diagnostics reveal.

3.1 Notation

  • A captioner Vc run under policy π on source corpus C induces a paired caption-image supervision distribution; CBUs are extracted from captions and judged against paired images.The audit target is distinct from the downstream T2I model trained on that distribution.Source: §3.1 · pp. 3–4

3.2 Target Source Families and Audited Surfaces

  • The release spans nine public source families, with five paired audits and four release-only families; comparisons fix source rows while varying captioner and policy.Each released surface uses a versioned, domain-branched policy that writes in the order of a user prompt.Source: §3.2 · p. 4, Table 2 · p. 4

3.3 The Diagnostic Gap

  • Corpus diagnostics measure opener rate, frequent-prefix mass, and distinct-3; Ours keeps opener rates at 1.3–2.3% and improves prefix mass and Distinct-3 against long-form references.High content-prefix mass and low distinct-3 on references expose repetition that caption length leaves unreported.Source: §3.3 · pp. 4–5, Table 3 · p. 5

4 Matched-Budget Audit Framework

The framework audits a recaptioned supervision distribution at a fixed text budget before generator training, combining prompt-side coverage, image-conditioned faithfulness, and caption-surface health. Its five-axis profile uses controllable basic units as a shared claim set for measuring density, supported yield, and risk.

4 Matched-Budget Audit Framework

  • The audit cuts each caption to budget B, computes text statistics, extracts stated CBUs, and re-asks each claim against its paired image.It reads the supervision signal before any generator is trained.Source: §4 · p. 5

4.2 Axes and Metrics

  • The five axes cover budget eligibility, prompt-pool support and n-gram JSD, claimed CBU density, surface concentration, and image-conditioned support and risk.The audit also specifies better directions and a cap on paired rows per slice.Source: §4 · p. 5, §4.2 · p. 6, Table 4 · p. 6

4.1 Caption ↔Prompt Structural Properties

  • Coverage measures whether caption components span user-prompt distributions through prompt-pool support and claimed CBU density.Prompt-mass support counts pool n-gram mass carried by captions; n-gram JSD compares the distributions across prompt-reference pools.Source: §4.1 · p. 5, §4.2 · p. 6
  • Faithfulness checks claims against paired images: supported yield counts claims judged supported, while risk is the share answered unsupported.Unsupported claims bind text to image-absent content, creating inconsistent training pairs.Source: §4.1 · p. 5
  • Surface health treats prefix concentration and repetition as text properties, favoring prefixes that spend their space on visual content.Describe-style frames can concentrate shared form in the caption prefix.Source: §4.1 · p. 6

5 Framework Instantiation

Across four cross-corpus pairs, Ours raises supported CBU yield and lowers unsupported risk under both Judges. The framework also tests policy effects, a CC12M length-density frontier, human agreement, and encoder-sensitive probes.

5 Framework Instantiation

  • The instantiation uses seven paired comparisons, with text statistics on up to 1M rows per slice and claim extraction and judging on about 5,000 rows per surface.CC12M uses 4,494 aligned images; prompt-pool support draws on seven sampled reference pools per slice.Source: §5 · p. 6

5.1 Cross-Corpus Results

  • +3.4 to +6.4 supported CBU per caption and a 0.05 to 0.16 risk drop across the four cross-corpus pairs under both Judges.Ours raises supported yield and lowers unsupported risk in all eight pair-by-Judge cells; gains persist for six claim types excluding count and relation.Source: §5.1 · p. 6, Table 5 · p. 7

5.2 Captioner Control: One Captioner, Two Policies

  • With captioner and decoding fixed, the released policy raises claimed CBU by +3.7 to +3.9 and supported CBU by +3.0 to +3.6 per caption, with risk 0.01 to 0.02 higher.On CC12M, the naive surface is 1.8× as long and has content-prefix mass 0.33, versus 0.07–0.16 for released surfaces.Source: §5.2 · p. 7, Table 6 · p. 7

5.3 CC12M: A Long-vs-Dense Frontier Case Study

  • On CC12M, Ours leads long-form surfaces in supported yield and risk, while the ≈12-lex Qwen tag surface leads CBU per 100 lex (55.84 against 23.16).Across B = 16, 32, 48, and 64, Ours leads both readings at every budget; at B = 64, its window carries 15.21 claims versus 6.44 for a complete tag caption.Source: §5.3 · pp. 7–8, Table 8 · p. 8, Figure 2 · p. 9

5.4 Human Verification Matches the Judges

  • Human explicit-no rates order the long-form surfaces as Ours (0.0%), LLaVA-NeXT (4.3%), and PixelProse (15.4%), matching both Judges’ risk ordering.Annotators were blind to surface identity and machine labels; none of Ours’ 111 judgments was an explicit no.Source: §5.4 · p. 8

5.5 Auxiliary Embedding-Space Probes

  • Embedding and retrieval rankings depend on encoder and truncation: PRDC favors Ours under Qwen3-Embedding-4B but Ref under BGE-M3, while pre-truncation moves LLaVA-NeXT to the top.Vendi and eRank can reverse Ours-versus-Ref ordering on the same encoder.Source: §5.5 · pp. 8–9

6 Dataset and Audit-Artifact Release

The release pairs a multi-source recaption corpus with an audit-artifact bundle for re-running the five-axis profile on future corpora. The corpus is linked to source releases by row-level keys, while the bundle provides reusable audit components.

6 Dataset and Audit-Artifact Release

  • The corpus contains generated captions for nine source families and is joined by row-level keys to source releases holding images and reference recaptions.Captions are released as generated and caption-only; each training pipeline applies its own personal-information, safety, NSFW, and deduplication filters.Source: §6 · p. 9
  • The reusable bundle supplies the interface for re-running the five-axis profile, and third parties can audit other surfaces using caption text, source-row identifiers, and paired image access for image-conditioned measures.Text-side axes require captions alone; the bundle reuses the tokenizer, prompt pools, CBU schema, VQA template, label schema, and manifest.Source: §6 · p. 9

7 Limitations

The audit measures caption–image supervision before T2I training, rather than image-output quality. Its scope and risk estimates are bounded by training choices, claim coverage, and the sampled sources and human judgments.

7 Limitations

  • Image-output quality also depends on encoders, caption-length sampling, dropout, persona augmentation, the generator, and training budget, so output comparisons combine corpus and training effects.99.89–99.99% of Ours captions exceed CLIP’s 77-token context; image-side safety, NSFW, and deduplication filters belong to training pipelines.Source: §7 · p. 9, Table 10 · p. 10
  • Risk aggregates captioner, extractor, and Judge errors, and the audit scores stated claims rather than omitted image content.The Judges agree least on count and text-rendering claims, at 85.1% and 87.6% exact agreement, respectively.Source: §7 · p. 9
  • Human judgments anchor the image-conditioned axis on sampled CC12M claims, while the paired audit spans five source families and prompt pools cover natural-language T2I prompts.The authors identify additional source families, booru-style prompt pools, and larger human-anchored samples as future targets.Source: §7 · p. 9, Table 9 · p. 10

8 Conclusion

The conclusion presents the audit as a reusable, matched-budget profile of recaptioned supervision, with the released surface favored across cross-corpus pairs and a consistent CC12M frontier. The appendices document its terminology, compute, and protocol details.

8 Conclusion

  • Across four cross-corpus pairs, the released surface has higher supported yield and lower risk under both Judges; the CC12M long-vs-dense frontier persists across two Judges and four budgets.A same-captioner policy ablation locates the cross-corpus gap on the policy axis; the release supports reproducing the profile and auditing future corpora.Source: §8 · p. 10

A Notation and Audit Vocabulary

B Computational Resources

  • The two image-conditioned audit stages dominate the sampled VLM request and compute scales, while text-only extraction is one to two orders smaller.The encoder probes and LongCLIP retrieval add a smaller combined contribution on the order of low tens of H200 GPU-hours.Source: Table 12 · p. 18, B Computational Resources · p. 18
  • The reported total compute is approximately 12.7k H200 GPU-hours, dominated by caption generation; sampled audits and encoder probes contribute about 102 H200 GPU-hours.Caption generation covers 514.6M rows at approximately 90 accepted captions per second on eight H200 GPUs; the stated throughput range gives approximately 9.9–14.3k GPU-hours.Source: B Computational Resources · pp. 17–18

C Protocol Details

  • Appendix H provides request templates for caption generation, the naive control, claim extraction, and image-conditioned judging.Claim extraction uses the CBU type taxonomy, while image verification uses supported, unsupported, and uncertain replies.Source: C Protocol Details · p. 18
  • Each slice uses seven public prompt-reference pools, each contributing up to one million prompts; an eight-pool variant is released for sensitivity comparison.The pools include four FLUX-style sets, one Stable Diffusion set, one SDXL-refiner set, and Pick-a-Pic rankings; DiffusionDB-derived deduplications are omitted to avoid double-counting upstream content.Source: C Protocol Details · p. 18, Table 13 · p. 18
  • Text statistics use the first 64 regex-tokenized lexical units, while claim extraction takes the first 64 whitespace-delimited words, so punctuation can make the windows differ.Using a 64-lexical-unit window preserves the CC12M surface ordering, changes claimed CBU per caption by at most 3.1%, and changes CBU/100 lex by less than 0.15.Source: C Protocol Details · pp. 18–19
  • Figure 3 compares Ours-minus-Reference prompt-mass support and direction-aligned JSD across seven pools and paired comparisons at B = 64.Positive values indicate more pool n-gram coverage in the left panel and closer alignment to the pool distribution in the right; the separator distinguishes four cross-corpus comparisons from three CC12M references.Source: Figure 3 · p. 19

D Per-Slice Audit Results

The per-slice audit reports higher supported yield and lower risk across six claim types, while also detailing denominators, surface descriptors, and uncertainty. Human verification further examines image support by surface and claim type.

D Per-Slice Audit Results

  • Table 14’s VQA denominators count audited captions as responses and CBU questions issued at B = 64; B-eligibility records the share reaching that lexical-unit budget.Short captions in three named surfaces fill part of the window, and Q/cap is average CBU per audited caption.Source: D Per-Slice Audit Results · p. 19, Table 14 · p. 20
  • The CC12M denominator accounting tracks images shared by four surfaces through valid extractions, zero-claim captions, VQA responses, and identical Judge question sets.Of 17,976 extraction requests, 99 (0.55%) hit the output cap without a parsable record; the budget sweep uses two extraction runs, and on 800 repeated requests claimed CBU per caption agrees within 1.8% for every surface.Source: D Per-Slice Audit Results · pp. 19–20, Table 15 · p. 20
  • Four surface descriptors measure opener rate, raw-prefix concentration, content-prefix concentration after boilerplate removal, and distinct 3-gram rate.The distinct 3-gram rate is the fraction of unique 3-grams over all corpus 3-grams, with lower values indicating more repetition.Source: D Per-Slice Audit Results · p. 20, Table 16 · p. 21
  • Attribute is the most consistent driver of CBU growth on Ours; style and text-rendering decrease relative to several comparisons, and PD12M Ours states slightly fewer object CBUs than its reference.Excluding count and relation preserves the supported-yield and risk ordering described above.Source: D Per-Slice Audit Results · p. 20
  • Excluding count and relation, Ours raises supported CBU per caption by +2.66 to +5.22 and lowers risk in all eight pair-and-Judge cells.Across claim types, count and text-rendering have the highest risk under both Judges and the lowest Qwen–Gemma exact agreement on CC12M; Ours has lower risk than pooled references in every type under both Judges.Source: D Per-Slice Audit Results · p. 20, Table 17 · p. 21, Table 18 · p. 22
  • Table 19 reports VQA cells as means and standard deviations over 2,000 caption-level bootstrap resamples.It includes supported CBU per caption and risk per surface under both Judges.Source: D Per-Slice Audit Results · p. 21, Table 19 · p. 22

D.1 Human Verification Detail

  • Human verification reports the full census of caption-side screens and image responses, with the broadest Ours margins in object and text-rendering claims.The interface asks annotators first whether the caption licenses a claim and type, then whether the image supports that claim with the caption hidden; other annotators’ responses and surface identity remain hidden.Source: §D.1 · p. 23, Figure 4 · p. 23, Table 21 · p. 23

E Extended Captioner-Control Results

The naive CC12M and DataComp surfaces show recurring descriptive frames and concentrated raw prefixes, alongside reported claimed-CBU counts for both controls.

E Extended Captioner-Control Results

  • Greedy decoding of the naive prompt yields 11.43 claimed CBU per caption on CC12M and 10.84 on DataComp.The captioner-control denominators are reported at B = 64; the DataComp control runs on 4,775 images and the cross-corpus pair on 4,922.Source: E Extended Captioner-Control Results · p. 24, Table 22 · p. 24
  • 0.60 top-100 raw-prefix mass on the naive surface is more than 4× Ours (0.14) and close to PixelProse (0.71).The five most frequent five-token prefixes all begin with “this is a”; “this is a close up” occurs in 238/4,494 captions.Source: E Extended Captioner-Control Results · p. 24

F Sensitivity Diagnostics

The sensitivity diagnostics show that retrieval comparisons change with caption truncation, while encoder-token caps affect how much of a caption is evaluated.

F Sensitivity Diagnostics

  • DataComp text-space rankings vary by encoder and metric across 50k paired rows.The table reports Vendi, eRank, prompt-to-caption PRDC, and directional caption-to-caption kNN cosine measures.Source: Table 23 · p. 24
  • In full-caption LongCLIP retrieval, Ours leads R@1 in both directions, while 32.4% of its captions exceed the 248-token cap.The naive surface has 99.2% over-cap captions and scores 0.906 I2T R@1 in full mode, versus 0.812 in input64 mode, the lowest of five surfaces.Source: F Sensitivity Diagnostics · p. 24, Table 24 · p. 25
  • With captions pre-truncated to B = 64, CC12M-LLaVA-NeXT has the highest R@1 in both retrieval directions; Ours ranks third on T2I (0.850).Ours is within 0.002 of CC12M-Qwen3-VL† on I2T (0.859 against 0.861), reversing its full-mode lead.Source: F Sensitivity Diagnostics · pp. 24–25, Table 25 · p. 25
  • Long Ours captions exceed the CLIP-77 and SigLIP2-64 caps on about 100% of captions, and the LongCLIP-248 cap on 22–40%.Short CC12M-Qwen3-VL† and PD12M released surfaces fit most encoders; rates use 100k-row samples per surface, except 4,494 naive-control captions.Source: F Sensitivity Diagnostics · p. 25, Table 26 · p. 25

G Release Artifacts, Hosting, and Croissant/RAI Coverage

The release packages caption records and audit artifacts across source families, documents personal-information handling, and specifies the VLM prompts and schemas used in the audit.

G Release Artifacts, Hosting, and Croissant/RAI Coverage

  • Each source-family caption row records source keys, image location, caption text and hash, record identifier, captioner artifact, and token counts.DataComp and Megalith-CC0 rows also include source-specific identifiers; Table 27 groups the public release fields.Source: G Release Artifacts, Hosting, and… · p. 26, Table 27 · p. 26
  • Captions are released as generated and may include visible names, contact details, or identifiers; training pipelines apply their own safety and policy filters.Dataset cards document this and provide a removal-request channel keyed to the row.Source: G Release Artifacts, Hosting, and… · p. 26
  • Each collection dataset has a Croissant 1.1 record with core and Responsible-AI fields, and every record passes the mlcroissant validator.The Hub serves a core record at each dataset’s Croissant endpoint; code and result summaries are also released.Source: G Release Artifacts, Hosting, and… · p. 26

H VLM Request Templates

  • The released VLM templates include distinct photorealistic and anime caption-generation prompts, with the anime prompt grounded by booru metadata.Both prompts direct captions to begin with the dominant subject or action and expand through scene details; the anime instructions specify confirmed-character handling.Source: H VLM Request Templates · pp. 27–28
  • The audit templates use a text-only extractor to return caption-claimed atomic visual units in eight categories, with a span and target for each unit.The claim categories are object, attribute, relation, style, camera, lighting, count, and text_rendering; the output is constrained by a JSON schema.Source: H VLM Request Templates · pp. 28–29
  • The image-conditioned Judge answers deterministic questions about extracted claims using visible image evidence, with yes, no, or uncertain responses.The audit aggregates support as yes/questions, uncertainty as uncertain/questions, and risk as no/questions; deterministic question construction fixes the denominator.Source: H VLM Request Templates · pp. 29–30

I DataComp Crawl and Survival Snapshot

The DataComp crawl snapshot reports survival at two processing stages and identifies the minimum-image-edge gate as the main post-download loss; the checklist describes release and evaluation documentation.

I DataComp Crawl and Survival Snapshot

  • The DataComp forward-partition snapshot is distinct from the deployed corpus, which combines forward and backward partitions into 325.6M caption rows over 325.5M unique normalized URLs.The crawl continued after the snapshot cutoff, and the released totals exceed the snapshot.Source: I DataComp Crawl and Survival Snapshot · p. 30, Table 29 · p. 30
  • The DataComp crawl tracks download-stage survival and end-to-end usable survival, and the dominant post-download loss is image_too_small under the min_short_edge = 256 gate.The snapshot’s processed share is calculated over target URLs, while other shares use processed URLs as their denominator.Source: Table 29 · p. 30, I DataComp Crawl and Survival Snapshot · p. 30

NeurIPS Paper Checklist

  • The checklist reports public access to the audited corpus, audit pipeline, and result summaries, with Appendix G documenting hosting, record schema, and reproduction workflow.The released collection includes Croissant 1.1 and Responsible-AI metadata.Source: §2 · p. 33
  • Supported yield and risk are reported with caption-level bootstrap standard deviations, while Judge–human agreement uses image-cluster bootstrap standard deviations.The checklist points to Appendix Table 19 and Table 9, respectively.Source: §2 · p. 33

2. Limitations

  • Generated captions can contain personal information shown in source images; the release documentation covers training-use filtering and a removal channel.The checklist points to Section 6 and Appendix G for these release considerations.Source: §2 · p. 34
  • Human input consisted of volunteer verification of machine-extracted claims against captions and images, and the task collected no annotator data.The checklist reports seven volunteers and no compensation.Source: §2 · p. 36
Loading 2610.00952v1…