Source-linked AI summary
From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation
Xingjian Wang, Zhao Wang, Taihang Hu, Jun Zheng, Qing Jin, Qinye Zhou, Zhengtao Wu, Yongchao Du, Zuan Gao, Chao Lin, Yefeng Shen, Xiaoli Xu, Zhengze Xu, Hao Yan, Yuhang Yu, Mingzhou Zhang, Mengting Chen
TL;DR
Generalist image-generation data pipelines often curate task-specific datasets separately, leaving organization around capability dependencies underexplored. This paper introduces a capability-driven infrastructure combining interoperable supervision engines with dependency-aligned curriculum scheduling, enabling transfer across tasks and training stages at large scale.
Problem
Conventional pipelines treat datasets as task-specific units, leaving underexplored how heterogeneous supervision should be organized around dependencies among generative capabilities.
Method
The framework combines three interoperable capability-specific data engines, shared captioning, and a multi-stage curriculum that jointly evolves task composition, concepts, quality, and resolution.
Results
The infrastructure promotes transfer across tasks and training stages while supporting a 440M-image T2I corpus, over 120M editing pairs, and approximately 27M image-entity pairs.
Takeaways & Limitations
Capability-driven organization turns otherwise isolated datasets into an adaptive data infrastructure for generalist image generation.
Abstract
from arXiv · showhide
Large-scale image generation has benefited from advances in data scale, quality, rebalancing, and recaptioning, yet conventional pipelines typically optimize task-specific datasets in isolation. A central challenge is not only how to curate each task-specific corpus, but also how to organize heterogeneous supervision according to the dependencies among generative capabilities. We present a \textbf{capability-driven data infrastructure} that couples capability-specific supervision construction with capability-aligned curriculum scheduling. Its three specialized yet interoperable data engines build complementary relational supervision for text-image grounding, inter-image transformation, and image-knowledge association, while caption experts align T2I and editing supervision across tasks and granularities. A multi-stage curriculum jointly evolves task composition, visual-concept distribution, data quality, and image resolution along the dependency order of capability acquisition, with capability-aware evaluation closing the loop through targeted retrieval, expert construction, and gap-aware resampling. At scale, the framework curates a 440M-image T2I corpus, 120M editing pairs, and over 27M image-entity pairs. With this infrastructure, we train multimodal diffusion models at two scales from scratch, with 3B and 6B sizes respectively. We conduct quantitative evaluation on CPI-Bench, along with qualitative evaluations across diverse text-to-image and editing scenarios. Experimental results present broad visual coverage, versatile rendering, and effective transfer across generative capabilities.
1 Introduction
The paper introduces a capability-driven data infrastructure that combines interoperable, capability-specific supervision with dependency-aware curriculum scheduling for generalist image generation. It organizes heterogeneous data and training progression to support transfer across text-to-image, editing, and knowledge-grounded generation.
- Motivation: Generalist capabilities emerge in a dependency order, motivating curricula that progress from semantic alignment and simple content toward structured generation, editing, higher resolution, and complexity.T2I semantic alignment provides reusable concepts for editing, while coarse content supports higher-resolution and more complex generation.
- Framework: The framework couples capability-specific data pipelines with capability-aligned curriculum scheduling into a unified infrastructure.Specialized engines tailor supervision to individual capabilities and dependencies, while the curriculum dynamically composes their outputs as capabilities evolve.
- Data pipeline: Three interoperable engines construct relational supervision for text-image grounding, image editing, and knowledge-grounded generation.The T2I engine expands concepts, rebalances long-tailed distributions, and aligns captions at multiple granularities; the other engines provide editing and knowledge associations.
- Data pipeline: Shared wrangling, annotation conventions, and caption experts transfer visual concepts across pathways while aligning editing instructions with T2I vocabulary and structure.Dense captions provide precise supervision for text-rich and structurally complex images, and the captioning framework spans entity tags through long-form descriptions.
- Curriculum scheduling: The five-stage curriculum jointly evolves task composition, visual concepts, data quality, and resolution according to capability acquisition dependencies.Training starts with large-scale T2I data and uses capability-aware evaluation to direct targeted retrieval, expert construction, and adaptive resampling.
- Contributions: The pipeline curates a 440M-image T2I corpus and over 120M high-quality image-editing pairs while preserving transfer through shared preprocessing and captioning.These specialized engines expand long-tail and defect-aware T2I coverage, mine natural visual associations for editing, and ground generation in structured knowledge.
2 Related Work
Prior work established web-scale corpora, data quality, diversity, filtering, and curriculum design as foundations for image generation. This work shifts data design from corpus-level curation to capability-organized supervision spanning T2I, editing, and knowledge grounding.
- Web-scale corpora such as LAION-5B, COYO-700M, and MMC4 provide foundational data for visual representation learning and image generation.
- Data design has become a central scaling dimension, with studies examining candidate-pool filtering, selection, data quality, semantic diversity, and text-conditioning density.
- The proposed framework shifts data design from corpora to capabilities through interoperable engines for T2I, image editing, and knowledge-grounded supervision.Shared semantic metadata and annotation interfaces allow concepts learned in one pathway to support another while exposing capability-specific coverage gaps.
- Instruction-based editing commonly uses source-image, instruction, and target-image triplets, with scarce natural pairs motivating synthesized transformations, generated targets, and automatic filtering.AnyEdit adds a fine-grained editing taxonomy and task-adaptive construction pipelines.
- Generalist models increasingly unify T2I and instruction-based editing, motivating operation-specific construction and mined editing relations.The framework also uses naturally associated images and expert-generated examples for sparsely covered tasks, while reusing T2I caption vocabulary so editing supervision emphasizes preservation and transformation.
- Curriculum learning and data-mixture optimization organize examples and domain weights to support progressively more complex large-scale pretraining.DoReMi uses a proxy model and distributionally robust optimization to estimate domain weights.
3 Capability-specific Data Pipeline
The pipeline organizes heterogeneous supervision into three specialized but interoperable engines for T2I generation, image editing, and knowledge-grounded generation. Shared processing and caption interfaces enable concept transfer across tasks and granularities.
- Pipeline organization: Three interoperable data engines target T2I generation, image editing, and knowledge-grounded generation while preserving concept transfer through shared processing and supervision interfaces.Each engine uses construction and annotation mechanisms tailored to its target capability.
- T2I data engine: The T2I engine converts a billion-scale image pool into 440 million images while jointly optimizing quality, semantic diversity, text-image alignment, and long-tail concept coverage.Its coverage expansion targets entity-level concepts and both desirable and undesirable visual patterns.
- Image-editing data engine: The editing engine combines operation-specific construction with naturally associated images to address scarce supervision and improve realism beyond purely synthetic transformations.It also uses VLMs, object masks, expert-generated candidates, and filtering for instruction alignment, identity preservation, and AIGC likelihood.
- Image-editing data engine: Editing supervision includes controlled geometric, photographic, color, and text changes plus bidirectional RGB-to-structure pairs for spatially and numerically precise editing.Structural representations include depth, edge, normal, and human-pose maps.
- Knowledge-grounded data engine: Knowledge-grounded visual data combines image-centric fine-grained labeling with entity-centric retrieval guided by structured knowledge graphs, yielding over 27 million high-quality image-entity pairs.The entity pipeline begins with over 100 million Wikidata entities and produces approximately 3 million high-salience entity names before image retrieval and filtering.
- Shared caption supervision: T2I captions and editing instructions serve as shared supervision interfaces, linking tasks through common visual vocabulary and organizing annotations from entity concepts and concise prompts to dense descriptions.Multi-granularity recaptioning addresses inadequate original metadata and supports fine-grained language–visual correspondences.
4 Capability-aligned Curriculum Scheduling
The curriculum schedules data in capability-dependency order while jointly evolving task composition, visual concepts, quality, and resolution. Capability-aware evaluation closes the loop by targeting retrieval, expert construction, and resampling toward persistent gaps.
- Curriculum design: The multi-stage curriculum replaces a fixed mixture with capability-ordered changes in task composition, visual-concept distribution, data quality, and image resolution.It spans foundational pre-training, continual training, and supervised fine-tuning.
- Curriculum stages: Stage 1 uses inclusive 256px T2I pre-training for broad semantic coverage and long-tail visual content, avoiding aggressive aesthetic filtering.Filtering relies primarily on image metadata and heuristic rules, preserving semantically useful imperfect images.
- Curriculum stages: Stage 2 raises T2I resolution from 256px to 512px while adding dense text, layout-sensitive, and knowledge-grounded content for greater structural fidelity.Resolution and content complexity increase together.
- Curriculum stages: Stage 3 jointly pre-trains 512px T2I and editing data, reusing T2I concepts while learning reference preservation and controlled transformation.Editing pairs include natural and synthetic samples balanced across instruction categories.
- Curriculum stages: Stage 4 progressively increases resolution from 512px to 1024px and shifts toward cleaner, higher-fidelity sources, while Stage 5 uses curated 1024px data with VLM and human review.The shift aims to improve visual quality while preserving world knowledge and strengthening instruction alignment.
- Active feedback loop: Capability gaps from stratified evaluation drive targeted retrieval, expert construction, and gap-aware resampling, increasing weights for persistent failures and down-weighting resolved gaps.Failure cases are annotated by task type, semantic tags, and failure dimensions before informing subsequent data updates.
5 Experiments
Experiments show broad and complex text-to-image generation, while qualitative editing results demonstrate transformation, restoration, multi-image composition, and reference-aware instruction binding. Quantitative evaluation is conducted on CPI-General-Bench and CPI-Practical-Bench, with capability-aligned supervision supporting these behaviors.
- Qualitative T2I Evaluation: Qualitative T2I results cover designed illustrations, multi-panel images, knowledge-structure visualizations, and photographic style control.The data engine supplies text-rich images and structured layouts for posters, interfaces, diagrams, and multi-panel composition.
- Qualitative T2I Evaluation: The curriculum introduces complex structures and higher-resolution supervision after broad visual grounding, improving diversity, compositional accuracy, and rendering quality together.
- Quantitative Evaluation: Image editing is quantitatively evaluated on CPI-General-Bench and CPI-Practical-Bench, complementary CPI-Bench subsets covering fundamental capabilities and practical application scenarios.CPI-Bench is described as a comprehensive, practical, and intelligent real-world image-editing benchmark.
- Qualitative Evaluation: Natural editing pairs support challenging hybrid transformation and reasoning-editing cases that require inferring intended visual states while preserving content beyond appearance matching.The supervision is mined from natural sources whose transformations reflect relationships occurring in everyday settings.
- Qualitative Evaluation: Controlled degraded-image supervision enables restoration of old photographs, low-clarity images, and motion-blurred images while distinguishing defects from valid content.The training retains a small controlled portion of degraded images with explicit descriptions of their degradation states.
- Qualitative Evaluation: Multi-image editing requires explicit text–image correspondences so the model can bind reference-specific attributes to instruction-defined visual elements.Evaluated capabilities include viewpoint transfer, reference relation understanding, text rendering, and composition.
6 Conclusion
The paper presents a capability-driven data infrastructure for generalist image generation and editing. It combines interoperable supervision engines, shared captioning, and capability-aligned curriculum design to coordinate learning across capabilities and granularities.
- Infrastructure: Three specialized yet interoperable engines construct complementary supervision for visual-text grounding, inter-image transformation, and image-knowledge association.The engines target distinct but related generative capabilities within a unified infrastructure.
- Cross-task alignment: A shared captioning interface enables concept transfer across tasks and granularities.The interface aligns supervision across the infrastructure’s different task settings.
- Curriculum: A capability-aligned curriculum jointly evolves task composition, visual-concept distribution, and data quality.The curriculum organizes data development around capability acquisition rather than isolated task-specific pipelines.
Appendix · A Shared Data Wrangling
The shared data-wrangling pipeline supports all capability-specific data engines by standardizing validity and quality, extracting comparable semantic metadata, and enabling curriculum-stage filtering and rebalancing.
- A Shared Data Wrangling: The shared pipeline standardizes data validity and quality across capability-specific data engines.It also extracts comparable semantic metadata and supplies attributes for stage-specific curriculum filtering and rebalancing.
A.1 Filtering and Quality Control
The pipeline combines rule-based, technical, perceptual, deduplication, watermark/text, and safety filters to control image validity, quality, redundancy, and content. Filtering thresholds vary by training stage to preserve early semantic diversity while emphasizing later visual quality.
- Basic Rule Filtering: Basic rules remove undecodable images, enforce at least 256² total pixels, and filter extreme aspect ratios according to stage-specific requirements.All images are converted to RGB and profiled using basic metadata.
- Technical Quality Filtering: Technical filters detect blur, compression artifacts, solid-color images, nearly blank images, and problematic white-background samples using specialized image statistics and quality scores.Blur combines Laplacian variance with BRISQUE scores; compression checks use file entropy and JPEG-quality estimation.
- Perceptual Quality Filtering: Perceptual-quality predictors assess clarity and aesthetics with stage-dependent thresholds that preserve semantic diversity early and progressively emphasize visual quality later.The thresholds change across training stages rather than remaining fixed.
- Watermark, AIGC, and Safety Detection: Specialized detectors separately identify watermarks, logos, subtitles, overlaid text, synthetic images, and unsafe content for filtering, captioning, or safety control.High-confidence AIGC samples are removed from pre-training, while NSFW and unsafe-keyword filters provide additional content safety.
A.2 Hierarchical Metadata Extraction and Rebalancing
The pipeline extracts hierarchical semantic metadata from the filtered web-scale pool and uses it to rebalance long-tailed concepts across multiple granularities. Fine-grained tags, adaptive assignment, and hierarchical resampling provide stage-tailored control over semantic coverage and distribution.
- A.2 Hierarchical Metadata Extraction and Rebalancing: Hierarchical semantic metadata and stage-specific rebalancing address the filtered pool’s long-tailed distribution dominated by frequent concepts.The schedules are tailored to different training stages.
- Taxonomy Construction: The taxonomy uses over 280K fine-grained tags across four levels containing 15, 74, and 331 upper-level categories.Each fine-grained tag maps to a leaf node, enabling coarse- and fine-grained distribution control.
- Tag Assignment: Caption-to-tag cosine similarity retrieves the top-1000 candidates, after which an adaptive hierarchical filter retains up to 15 diverse representative tags per image.The resulting compact metadata spans multiple conceptual dimensions.
- Data Rebalancing: Hierarchical resampling represents all semantic tags, emphasizes rare concepts, and approximately balances samples across first-level categories and child categories through the third level.This reduces frequent-concept dominance while preserving semantic diversity at multiple granularities.