PaperlayerView exact revision on arXiv

Source-linked AI summary

DataEvolver: Self-Evolving Multi-Agent Data Construction for Text-Rich Image Generation

Siyu Yan, Yizhen Gao, Yilin Wang, Dongxing Mao, Alex Jinpeng Wang

arXiv:2606.31537v1cs.CVcs.MA

TL;DR

Text-rich image data construction must support visual fidelity, legible and aligned text, and coherent layouts, but existing pipelines typically discard rejected samples. DataEvolver uses rejection feedback in a closed-loop multi-agent construction process, improving OCR-oriented performance across benchmarks and downstream generators, including an OCR-F1 increase from 4.56 to 8.45 on TextScenesHQ with PixArt-α.

  • Problem

    Existing text-rich image data pipelines typically crawl, filter once, and freeze accepted samples, leaving feedback from rejected samples underused.

  • Method

    DataEvolver uses Retriever, Verifier, Critic, and Generator agents in a closed loop that converts construction failures into feedback for subsequent rounds.

  • Results

    DataEvolver consistently improves OCR-oriented performance across benchmarks and downstream generators, raising OCR-F1 from 4.56 to 8.45 on TextScenesHQ with PixArt-α.

  • Takeaways & Limitations

    The results support treating text-rich image data construction as an adaptive process rather than a fixed preprocessing step.

  • Takeaways & Limitations

    Scaling results indicate a trend rather than a strict scaling law, the framework depends on verifier reliability, and its demonstrated scope is text-rich image generation.

Abstract

from arXiv · show

Text-rich image generation is one of the most challenging settings in image generation, since models must simultaneously produce visually realistic images and render legible, semantically aligned, and layout-consistent text. Existing data pipelines usually follow a static crawl-filter-freeze paradigm. They collect candidate samples, filter them once, and freeze the accepted data for training. However, rejected samples are usually discarded, although they often contain useful failure signals such as OCR errors and semantic mismatches. As a result, later construction rounds may repeat the same failure modes. To address these limitations, we propose DataEvolver, a self-evolving multi-agent framework for text-rich image data construction. DataEvolver treats data construction as feedback-driven construction policy evolution. A Retriever collects candidate samples, a Verifier assigns quality scores and rejection causes, a Critic summarizes round-level feedback into semantic feedback, and a Generator completes under-covered regions through targeted synthesis. The updated feedback memory then guides the next construction round. Experiments on text-rich image generation benchmarks show that DataEvolver produces more useful training data than fixed-dataset baselines under matched data budgets. At the 0.75M scale on PixArt-alpha, DataEvolver improves OCR-F1 over the strongest baseline by 85.3 percent on TextScenesHQ and 35.3 percent on LongTextBench. The improvements are consistent across both evaluated benchmarks and also transfer to Show-o2, indicating that the benefit of DataEvolver is not tied to a single downstream generator. These results suggest that rejected samples can provide actionable feedback for improving text-rich image data construction.

1 Introduction

Text-rich image generation is increasingly limited by static data-construction pipelines that discard failure signals from rejected samples. DataEvolver addresses this limitation with a closed-loop, self-evolving multi-agent framework that converts verification outcomes into feedback for targeted data construction.

  • Motivation: Text-rich image data must jointly achieve visual fidelity, text legibility, and semantic alignment, making training-data construction a key constraint on model progress.
  • Problem: Static crawl–filter–freeze pipelines collect candidates, apply fixed filters, and freeze accepted data while discarding feedback from rejected samples.Rejected samples can reveal unreliable retrieval queries, duplicates, and under-covered semantic regions.
  • Failure-aware construction: DataEvolver reframes construction as policy evolution, treating rejected samples and failure cases as reusable diagnostic feedback rather than filtering byproducts.The framework uses feedback to improve the construction process across rounds.
  • Framework: DataEvolver combines a Retriever, Verifier, Critic, and Generator in a closed loop for candidate discovery, rejection analysis, semantic feedback, and targeted completion.The Generator focuses on under-covered regions, while the Critic distills verification feedback and rejection patterns into natural-language semantic feedback.
  • Evaluation: Under matched data budgets and a shared downstream generator, DataEvolver consistently improves OCR-oriented metrics while maintaining competitive semantic alignment and visual quality.It is evaluated against strong public data sources and ablated variants on TextScenesHQ and LongTextBench.

2 Related Work

Prior work has established that multimodal dataset construction choices affect downstream performance, while feedback-driven methods improve language-model and agent behavior using human, AI-generated, or rule-based feedback.

  • Multimodal Data Construction: Multimodal datasets commonly use web crawling, offline filtering, and deduplication, with data sources, filtering criteria, and composition influencing downstream model performance.Representative datasets include CC12M, LAION-5B, COYO-700M, DataComp, MMC4, and OBELICS.
  • Feedback-Driven Self-Improvement: Feedback-driven learning methods optimize language-model and agent behavior with human feedback, preference data, AI-generated feedback, or rule-based principles.Examples include RLHF, PPO, DPO, Constitutional AI, and RLAIF.

3 Preliminaries and Problem Setup

Data construction is formulated as an iterative policy-refinement process rather than a one-shot crawl, filter, and storage pipeline. Each round uses multidimensional feedback on accepted quality and rejected failure modes to revise retrieval, generation, and memory policies.

  • Data construction targets a high-quality dataset for a domain of topics and subtopics that supports downstream text-rich image generation models.
  • Each round executes a construction policy that governs retrieval and generation, then uses observed feedback to revise the next-round policy.The policy includes retrieval and generation strategies plus an experience library storing high-value feedback and policy traces.
  • The Verifier configuration and verification criteria remain fixed in the reported experiments and are not optimized policy variables.
  • The round-level feedback records pass rate, mean OCR quality, mean semantic consistency, mean image quality, and distributions of rejection causes.Recorded failure types include blur, OCR failure, semantic mismatch, and layout corruption.
  • These feedback signals measure current policy effectiveness and explain failures, enabling updates to retrieval queries, generation prompts, and experience memory.The refinement updates controllable pipeline variables in natural-language policy space rather than through gradient-based optimization or model-parameter training.

4 DataEvolver

DataEvolver constructs text-rich image data through a closed loop in which retrieval, verification, critic-guided policy updates, and targeted synthesis iteratively improve coverage and quality. Rejected samples become semantic feedback that guides later retrieval and generation decisions.

  • Retrieval: The Retriever uses a policy-conditioned query space that adapts structured topic and subtopic templates according to previous feedback.Templates can incorporate keywords, descriptors, semantic expansions, and source-level constraints before producing a topic-aware candidate pool.
  • Verification: The Verifier applies OCR, perceptual-hash deduplication, and assessments of image quality, text recognition, and semantic consistency before admitting samples.It converts raw candidates into a pass set while also attributing structured failure causes for feedback.
  • Feedback and policy update: The Critic converts round-level rejection statistics into natural-language guidance for query diversification, prompt refinement, and experience-memory updates.Semantic feedback captures dominant failures, under-covered topics, duplicate-heavy queries, and ineffective templates; in one logged case, deduplication failures fell from 278 to 148.
  • Targeted synthesis: The Generator targets topic–subtopic regions with coverage below distributional averages or minimum counts, then sends synthesized samples through verification and uses repeated failures to refine prompts.This complements retrieval for rare topics, complex layouts, long-tail text patterns, and sparsely covered semantic regions.

5 Experiments

Experiments show that DataEvolver produces stronger text-rich image training data than fixed baselines across downstream models, benchmarks, budgets, and category coverage. Ablations and construction diagnostics attribute the gains to Critic-driven feedback and Generator-assisted coverage expansion.

  • Main results: DataEvolver achieves the best OCR-F1 across both benchmarks on PixArt-α and Show-o2 under the 0.75M matched data budget.On PixArt-α, OCR-F1 improves from 4.56 to 8.45 on TextScenesHQ and from 6.71 to 9.08 on LongTextBench; on Show-o2, it improves from 0.19 to 0.45 and from 0.27 to 0.44, respectively.
  • Main results: Across all four benchmark-model settings, DataEvolver obtains the highest OCR accuracy, recall, and F1, improving text coverage while maintaining competitive precision.MARIO achieves slightly higher OCR precision on PixArt-α, but its lower recall and F1 indicate fewer correctly matched text instances overall.
  • Ablations: Removing the Critic causes the larger ablation drop, reducing F1 from 1.78 to 1.01 on TextScenesHQ and from 2.16 to 0.90 on LongTextBench.The result indicates that verification signals alone are insufficient without policy-level feedback that helps later rounds avoid recurring failure modes.
  • Construction analysis: The full framework achieves the highest pass rate, OCR confidence, and topic coverage, while enabling the Critic raises mean OCR confidence from 0.861 to 0.938.The proportion of high-confidence samples above 0.90 rises from 29.1% to 81.1%, and DataEvolver also achieves the highest category and tail coverage among compared datasets.
  • Scaling: DataEvolver’s OCR-F1 advantage persists when scaling the data budget from 0.1M to 0.75M on both PixArt-α and Show-o2.On PixArt-α, TextScenesHQ increases from 1.78 to 4.17, 6.67, and 8.45, while LongTextBench increases from 2.16 to 4.04, 8.35, and 9.08 across the completed scales.

6 Conclusion and Limitations

DataEvolver replaces static dataset construction with a feedback-driven multi-agent loop that turns construction failures into semantic feedback for later rounds. Experiments support its usefulness under matched data budgets, while limitations concern scaling-law evidence, Verifier reliability, and scope beyond text-rich image generation.

  • Framework: DataEvolver uses Retriever, Verifier, Critic, and Generator agents in a closed loop to refine retrieval queries, generation prompts, and experience memory.The framework converts construction-time failures into semantic feedback for subsequent rounds.
  • Findings: Under matched data budgets, feedback-driven construction produces more useful training data than fixed-dataset baselines.Ablations indicate that the Critic and Generator have different but complementary roles.
  • Findings: The Critic makes rejection feedback actionable for policy revision, while the Generator improves coverage by completing under-represented regions.These roles explain how feedback supports policy evolution and coverage improvement during construction.
  • Limitations: Scaling results indicate a trend rather than a strict scaling law because formal analysis would require more data scales and repeated runs.The paper also notes dependence on Verifier reliability and focuses on text-rich image generation rather than broader extensions.

A Implementation Details · A.1 Verifier Configuration

The verifier extracts OCR signals and filters candidate images through deduplication, quality assessment, semantic relevance checking, and OCR-text consistency checking. Rejection statistics are aggregated into round-level feedback for the Critic’s subsequent policy updates.

  • A.1 Verifier Configuration: OCR is first applied to extract text signals from each candidate image.
  • A.1 Verifier Configuration: Perceptual-hash deduplication removes duplicate candidate images.
  • A.1 Verifier Configuration: OCR/image-quality assessment evaluates candidate image quality.
  • A.1 Verifier Configuration: CLIP-based semantic relevance checking uses CLIP ViT-B/32.
  • A.1 Verifier Configuration: OCR-text consistency checking uses Sentence-Transformer (all-MiniLM-L6-v2).
  • A.1 Verifier Configuration: Rejection statistics are aggregated into round-level feedback for the Critic’s subsequent policy updates.

A.2 Downstream Training Details

Downstream training fine-tunes PixArt-α and Show-o2 under matched data budgets, generally resizing inputs to 512 × 512. The implementations specify distinct architectures, optimization settings, precision configurations, and sampling parameters for each model.

  • Shared training setup: Both models are fine-tuned under matched data budgets, with input images resized to 512 × 512 unless otherwise specified.Unset options follow the default settings of the corresponding training code.
  • PixArt-α: PixArt-α uses PixArt_XL_2 initialized from the official PixArt-XL-2 512×512 checkpoint with the SD VAE sd-vae-ft-ema.Window attention and relative positional encoding are disabled, attention is computed in FP32, and training uses batch size 32 with gradient checkpointing and gradient clipping set to 0.01.
  • Show-o2: Show-o2 uses AdamW with module-specific learning rates of 2×10−6 for the visual encoder and 1 × 10−5 for both the projection module and Show-o backbone.Training uses β1 = 0.9, β2 = 0.999, zero weight decay, ϵ = 1 × 10−8, cosine decay with warmup ratio 0.03, BF16 mixed precision, TF32, gradient accumulation of 16, and batch size 1.
  • Sampling configuration: Flow-matching sampling uses a linear path with velocity prediction, log-normal SNR weighting, Euler sampling, guidance scale 5.0, and 50 inference steps.Absolute and relative tolerances are 10−6 and 10−3, respectively, with time shifting enabled at shifting factor 3.0.

A.3 Semantic Diversity Evaluation Details · B Prompts

The appendix specifies a controlled semantic-diversity evaluation and documents prompt templates for coverage planning, retrieval, generation, critique, refinement, and experience-library updates. These prompts enforce structured outputs and use verification feedback to guide subsequent data construction.

  • A.3 Semantic Diversity Evaluation Details: Semantic diversity is classified from image captions with the same zero-shot DeBERTa classifier across datasets.The classifier predicts one label from a fixed taxonomy, assigning low-confidence samples to an additional other category when confidence is below 0.15.
  • A.3 Semantic Diversity Evaluation Details: Category coverage measures predefined categories exceeding 0.1% frequency, while tail coverage sums the ten least frequent categories.Both statistics use the 33 predefined candidate categories after excluding samples assigned to other, with all datasets evaluated at the 0.5M scale.
  • B Prompts: The appendix introduces prompt templates for DataEvolver’s retrieval, generation, critique, and experience-library components.The templates are organized under retrieval and generation prompts and critic and experience-library prompts.
  • B.1 Retrieval and Generation Prompts: Coverage planning allocates additional images only to underrepresented subtopics and returns nonnegative integer quantities as a JSON array.Search prompts generate up to 3 Bing queries containing text-readability and quality keywords, optionally using reliable source-site filters.
  • B.1 Retrieval and Generation Prompts: Query diversification generates up to 5 distinct searches that explicitly request text-rich images with clean, high-contrast, detailed layouts.The prompt forbids simple or minimalist images and requires output without explanations.
  • B.1 Retrieval and Generation Prompts: Generation prompt planning combines theme, subtheme, textual elements, visual content, OCR-readable text, and quality constraints in concise JSON-listed prompts.Requirements include clear high-contrast text, watermark avoidance, natural layouts, and simpler layouts for complex scenes when needed.
  • B.2 Critic and Experience-Library Prompts: Semantic-feedback prompts compare current and previous rejection counts with keyword performance to produce concise recommendations for keyword adjustment.They identify rejection-associated keywords to weaken or remove and accepted-item keywords to strengthen, using only current data when previous data is unavailable.

C Examples of Rejected Cases

Rejected samples are excluded from downstream training but analyzed to summarize rejection patterns and guide subsequent construction rounds. Representative failures include unreadable, tiny, distorted, sparse, and cluttered text conditions.

  • Rejected samples are not used for downstream training; their rejection patterns are summarized as feedback for subsequent construction rounds.
  • Heavily distorted overlaid text can be too difficult to recognize reliably.
  • Tiny screen text occupies little image area and provides weak OCR supervision.
  • Perspective distortion and background clutter can impair verification of scene text.
  • Sparse handwriting and cluttered scenes contain limited, difficult-to-verify text regions.
Loading 2606.31537v1…