Source-linked AI summary
Fine-T2I: An Open, Large-Scale, and Diverse Dataset for High-Quality T2I Fine-Tuning
Xu Ma, Yitian Zhang, Qihua Dong, Yun Fu
TL;DR
High-quality open fine-tuning data remains scarce for text-to-image generation. Fine-T2I constructs and rigorously filters a large open dataset combining synthetic and curated real images, and fine-tuning consistently improves visual quality and text-image alignment across model families.
Problem
High-quality instruction-aligned fine-tuning data is scarce, costly, and often inaccessible or legally restricted for the open community.
Method
Fine-T2I combines synthetic data and curated real images through a pipeline of prompt design, generation, de-duplication, safety checks, alignment verification, and aesthetic filtering.
Results
Fine-tuning on Fine-T2I consistently improves generation quality and instruction adherence across diverse model architectures.
Takeaways & Limitations
Fine-T2I provides an open dataset and construction pipeline intended as a foundation for future text-to-image generation research.
Takeaways & Limitations
Automatic benchmarks have limited prompt coverage and may misalign with human preference when evaluating aesthetics, style fidelity, and nuanced instruction following.
Abstract
from arXiv · showhide
High-quality and open datasets remain a major bottleneck for text-to-image (T2I) fine-tuning. Despite rapid progress in model architectures and training pipelines, most publicly available fine-tuning datasets suffer from low resolution, poor text-image alignment, or limited diversity, resulting in a clear performance gap between open research models and enterprise-grade models. In this work, we present Fine-T2I, a large-scale, high-quality, and fully open dataset for T2I fine-tuning. Fine-T2I spans 10 task combinations, 32 prompt categories, 11 visual styles, and 5 prompt templates, and combines synthetic images generated by strong modern models with carefully curated real images from professional photographers. All samples are rigorously filtered for text-image alignment, visual fidelity, and prompt quality, with over 95% of initial candidates removed. The final dataset contains over 6 million text-image pairs, around 2 TB on disk, approaching the scale of pretraining datasets while maintaining fine-tuning-level quality. Across a diverse set of pretrained diffusion and autoregressive models, fine-tuning on Fine-T2I consistently improves both generation quality and instruction adherence, as validated by human evaluation, visual comparison, and automatic metrics. We release Fine-T2I under an open license to help close the data gap in T2I fine-tuning in the open community.
1. Introduction
Open T2I fine-tuning is constrained by scarce, costly, and legally restricted high-quality data, despite accessible model-centric advances. Fine-T2I addresses this gap with an open dataset and construction pipeline, reporting consistent gains across models.
- High-quality instruction-aligned fine-tuning data is often private, leaving open teams with a persistent gap despite access to model architectures and training recipes.
- $10+ per image and restrictive licenses make high-quality fine-tuning images expensive and difficult to redistribute legally.
- Fine-T2I combines synthetic images from strong diffusion models with curated real images shared by photographers under an open license.
- Over 95% of initially collected data is removed through de-duplication, alignment, safety, and aesthetic-quality filtering.
- Human, qualitative, and quantitative evaluations show consistent improvements after fine-tuning across diverse text-to-image architectures.
2. Related Work
Text-to-image systems commonly progress from broad pretraining to curated fine-tuning and preference alignment. Open fine-tuning datasets support this ecosystem but remain limited relative to modern production-grade quality requirements.
- Autoregressive and diffusion models are the two prevalent architectural designs for text-to-image generation.
- Progressive training typically moves from large-scale text-image pretraining toward curated, high-aesthetic fine-tuning.
- Preference-alignment methods such as DPO and reinforcement learning are incorporated to better match human preferences.
- Open fine-tuning datasets such as JourneyDB, Pick-a-Pic, T2I-2M, LAION variants, and BLIP3o-60k still exhibit gaps relative to production-grade models.
3. Fine-T2I Dataset
Fine-T2I is introduced as a new fine-tuning dataset for modern high-quality image generation in the open community, with a pipeline covering synthetic and curated real-image data.
- Fine-T2I is designed as a new fine-tuning dataset for modern high-quality image generation in the open community.
- Its construction pipeline covers both synthetic sets and a curated real-image set.
3.1. Generating Synthetic Set
The synthetic set combines controlled, enhanced prompts with multiple generation formats, then applies staged filtering to improve diversity, safety, visual quality, and text-image alignment. The resulting design yields four synthetic subsets and 6,145,693 high-quality synthetic samples.
- Controlled prompts are filtered, de-duplicated, and enhanced into more descriptive counterparts before image generation.
- Both original and enhanced prompts are rendered with strong diffusion models using square and randomized-resolution settings, producing four synthetic subsets.
- 6,145,693 high-quality synthetic samples remain after generation and additional filtering.
- Prompt cleaning includes semantic de-duplication with a cosine-similarity threshold of 0.8, safety filtering, and attribute-alignment verification.
- 1-3 candidate images are sampled per prompt, and the candidate with the highest Aesthetic Predictor V2.5 score is retained.
- Aesthetic filtering keeps images only when the Aesthetic Predictor V2.5 score exceeds 5.5, while VLM reasoning checks text-image alignment and artifacts.
3.2. Curating Real-Image Set
The curated real-image set combines creator-sourced photography with stricter quality filtering and paired short and detailed captions for text-conditioned training.
- Final subset: The curated real-image set is described as containing about 168k extremely high-quality samples from open-sourced platforms.
- Data sourcing: Fine-T2I curates real images from creator-driven platforms, using photographers’ and creators’ open-term uploads as an initial aesthetic prior.The sources provide broad coverage of everyday photographic content.
- Quality filtering: Images are filtered with Aesthetic Predictor V2.5, retaining only scores ≥6.5, while extremely large images are discarded for training stability.
- Captioning: Each retained image receives a short initial prompt and a more detailed enhanced prompt generated with a fine-tuned Qwen2.5-VL-7B model.This supports both concise user inputs and more explicit instructions for fine-tuning.
- Final subset: 168,424 images remain in the real-image subset after filtering and captioning.
4. Fine-T2I Dataset Specifications
Fine-T2I’s synthetic subset emphasizes broad category, style, and task coverage, while its real and synthetic subsets provide complementary aesthetic-quality profiles.
- Attribute coverage: The synthetic subset contains mostly People (37.9%) and Nature (27.8%), with additional Text Rendering (17.4%), Design (10.6%), and Rare cases (6.3%).
- Attribute coverage: General & Photorealistic is the largest style group at 20%, alongside diverse artistic and design-oriented styles.
- Attribute coverage: The final distributions remain close to intended real-world demand despite non-uniform rejection during safety, alignment, and quality filtering.
- Aesthetic quality: 63.21% of curated real images fall in the [6.5, 7.0) aesthetic-score range, while the synthetic subset is broader, with 36.64% in [5.5, 6.0) and 35.70% in [6.0, 6.5).The real-image set supplies consistently high aesthetic scores, whereas synthetic data contributes scale and diverse controlled attributes.
5. Experiments with Fine-T2I
Experiments test Fine-T2I across diffusion and autoregressive backbones using human-centered evaluation, finding consistent gains in visual quality and instruction adherence.
- Experimental setting: The evaluation spans SD-XL, a diffusion model, and LlamaGen, an autoregressive model, rather than a single modeling family.
- Evaluation: Human evaluation uses 500 diverse public prompts and measures text-image alignment and overall human-preferred visual quality.GenEval is also reported as a reference, but the paper cautions against relying heavily on automated metrics.
- Human evaluation: 80.7% visual-quality win rate and 65.3% text-image-alignment win rate are achieved by fine-tuned LlamaGen versus its non-fine-tuned counterpart.
- Human evaluation: Fine-tuning on Fine-T2I produces clear preference gains for both LlamaGen and SD-XL, with the largest improvement observed in visual quality.The authors associate this emphasis with the dataset’s extremely high aesthetic quality.
- Qualitative and automatic results: Fine-tuned generations appear more natural and coherent, with cleaner local details, fewer distracting artifacts, and better reflection of prompt content.GenEval also improves for both models, providing an additional signal for controllability.
- Dataset comparison: Models fine-tuned on Fine-T2I generate better text alignment and visual quality than models fine-tuned on T2I-2M or BLIP3o-60k.The comparison uses LlamaGen, identical settings, and human evaluation over 500 prompts.
6. Conclusion
Fine-T2I addresses the scarcity of open, high-quality fine-tuning data with a large, licensed dataset and systematic construction pipeline. Fine-tuning consistently improves instruction adherence and visual quality across model families.
- Conclusion: Fine-T2I contains over 6M samples combining diverse synthetic text-image pairs with a rigorously curated real-image set.
- Conclusion: Its construction pipeline prioritizes diversity, alignment, and visual quality at scale.
- Conclusion: Fine-tuning on Fine-T2I consistently improves instruction adherence and visual quality across different model families.
- Conclusion: Fine-T2I is released under an open license as a foundation for future open-community text-to-image research.
A. Detailed Distribution analysis
Fine-T2I analyzes prompt and style distributions across its synthetic subsets, using five templates to broaden prompt structure. Enhanced prompts are longer and more varied than original prompts.
- Prompt length: Enhanced prompts are substantially longer and span a wider range of lengths than original prompts, which are typically under 50 words.This pattern appears in both the synthetic sets and the curated real-image set.
- Prompt structure: Five prompt templates diversify structures ranging from simple descriptions and user requests to sentences, mixed attributes, and word combinations.The templates include formats such as “a photo of xxx,” request-like prompts, multi-sentence prompts, attribute combinations, and simple word lists.
- Style distribution: Table 5 reports detailed style statistics for the four synthetic subsets, distinguishing enhanced versus original prompts and randomized versus square aspect ratios.“PE” denotes enhanced prompts, “PO” original prompts, “AR” randomized aspect ratios, and “AS” square aspect ratios.
B. Problems, Ambiguity, Expectation
The dataset construction faces persistent redundancy, imperfect metadata alignment, shifting aesthetic standards, and the absence of a robust image-quality metric. These issues motivate aggressive filtering and cautious interpretation of attributes and aesthetic scores.
- Prompt diversity: Nearly 90% of generated candidates were removed during deduplication because prompts remained highly similar or identical across batches and GPUs.Higher sampling temperature and attribute conditioning reduced but did not eliminate duplication, while temperature also produced malformed prompts.
- Attribute alignment: Category, style, and task attributes are soft metadata because contradictory or complex combinations may not be faithfully expressed in generated prompts.An additional LLM-based reasoning filter can improve adherence but causes substantially more aggressive filtering.
- Aesthetic evaluation: Aesthetic scores may disagree with human preference because standards change and natural-looking images can be preferred over higher-scoring stylized images.The authors therefore do not treat any aesthetic metric as ground truth.
- Evaluation: No strong, robust, widely accepted indicator currently assesses image or image–text-pair quality, so Fine-T2I relies on many filtering steps.The authors expect stronger evaluation metrics, potentially aided by VLM reasoning, to simplify future curation.
C. Detailed studies
Detailed studies use system prompts, random qualitative samples, fine-tuning comparisons, and human evaluation to examine dataset quality and downstream effects. Fine-T2I is reported to provide stronger alignment and visual quality than comparison datasets, despite not minimizing training loss.
- Filtering: System prompts guide final filtering for text–image alignment and aesthetic quality, while the Qwen-VL prompt explicitly targets high-quality alignment and image generation.These prompts are shown in Figure 12.
- Dataset comparison: Randomly sampled comparisons show some alternative datasets contain low-resolution, low-quality, poorly aligned pairs that are unsuitable for fine-tuning.Figure 13 avoids cherry-picking by selecting one tar file and displaying its first several samples.
- Fine-tuning comparison: Fine-T2I delivers the best fine-tuning results even though BLIP3o-60k achieves the lowest training loss.The authors attribute this contrast to BLIP3o-60k’s structurally simple images, which may enable optimization shortcuts without indicating better generation.
- Human evaluation: Human evaluation compares fine-tuning results across datasets using A/B judgments, requiring annotators to select one best sample and averaging results across participants.The protocol also discourages labeling samples as ties.