Source-linked AI summary
Alchemist: Unlocking Efficiency in Text-to-Image Model Training via Meta-Gradient Data Selection
Kaixin Ding, Yang Zhou, Xi Chen, Miao Yang, Jiarong Ou, Rui Chen, Xin Tao, Hengshuang Zhao
TL;DR
Training-data quality limits text-to-image models, while existing selection approaches rely on costly manual curation or narrow heuristics. Alchemist uses meta-gradient rating and pruning to select informative text-image pairs. Experiments report improved efficiency and performance, including cases where a selected 50% subset outperforms full-data training.
Problem
T2I data selection lacks scalable methods that integrate complementary metrics and optimize for downstream model performance rather than relying on costly manual curation or surface-form heuristics.
Source: §1 · p. 2Method
Alchemist learns sample influence from proxy-model gradient information with a lightweight, multi-granularity rater, then prunes data using Shift-GSample.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
Alchemist's selected subsets can reduce the data needed for T2I training while improving performance, with a 15M subset matching random sampling performance at 5× faster training.
Source: §1 · p. 2The rater-gradient simplification assumes the training-only reference model approximates optimal parameters, giving near-zero loss on each training sample.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Recent advances in Text-to-Image (T2I) generative models, such as Imagen, Stable Diffusion, and FLUX, have led to remarkable improvements in visual quality. However, their performance is fundamentally limited by the quality of training data. Web-crawled and synthetic image datasets often contain low-quality or redundant samples, which lead to degraded visual fidelity, unstable training, and inefficient computation. Hence, effective data selection is crucial for improving data efficiency. Existing approaches rely on costly manual curation or heuristic scoring based on single-dimensional features in Text-to-Image data filtering. Although meta-learning based method has been explored in LLM, there is no adaptation for image modalities. To this end, we propose **Alchemist**, a meta-gradient-based framework to select a suitable subset from large-scale text-image data pairs. Our approach automatically learns to assess the influence of each sample by iteratively optimizing the model from a data-centric perspective. Alchemist consists of two key stages: data rating and data pruning. We train a lightweight rater to estimate each sample's influence based on gradient information, enhanced with multi-granularity perception. We then use the Shift-Gsampling strategy to select informative subsets for efficient model training. Alchemist is the first automatic, scalable, meta-gradient-based data selection framework for Text-to-Image model training. Experiments on both synthetic and web-crawled datasets demonstrate that Alchemist consistently improves visual quality and downstream performance. Training on an Alchemist-selected 50% of the data can outperform training on the full dataset.
1. Introduction
Text-to-image models depend on training-data quality, but current curation is costly or relies on narrow heuristics rather than downstream performance. Alchemist uses meta-gradients to rate and prune data, and selected subsets can improve training outcomes while reducing data use.
1. Introduction
- Low-quality or redundant web-crawled and synthetic data can degrade visual fidelity, destabilize training, and waste computation.Examples include blurry images, plain-background advertisements, and redundant content.Source: §1 · p. 2
- Existing curation is prohibitively expensive or relies on single-dimensional heuristics, while current T2I selection does not optimize for downstream model performance.Meta-learning had been explored for LLM training, but not adapted to image modalities.Source: §1 · p. 2
- Alchemist rates samples using gradient information from a T2I proxy model and multi-granularity perception, then prunes the rated data with Shift-GSample.The rater estimates each sample's influence and its contextual contribution within a group; Shift-GSample retains informative middle-to-late samples that are detailed yet learnable.Source: §1 · p. 2
- A 50% Alchemist-selected subset often outperforms full-data training, while a 15M selected subset matches random sampling performance with 5× faster training.The selected subsets also filter images with plain backgrounds or highly chaotic content.Source: §1 · p. 2
2. Related Work
Prior coreset methods often use static proxy scores, while gradient-driven selection offers a link between sample value and the learning objective. Alchemist adapts meta-learning to T2I and uses continuous scoring with penalty relaxation to make bilevel-style selection more efficient.
2. Related Work
- Static heuristics and pretrained proxy scores can remove outliers, but are not directly optimized for the final model's downstream performance.Gradient-driven selection instead uses training-process signals to identify high-impact data [20] [41].Source: §2 · p. 2
- Alchemist uses continuous sample scoring based on alignment between each sample's gradient and the validation gradient, enabling efficient adaptation to downstream tasks.It is most similar to Shen et al. [35].Source: §2 · p. 3
- Exact validation-gradient bilevel optimization can be computationally infeasible for T2I, motivating Alchemist's use of penalty relaxation methods [15] [34] [36] [42].Second-order approaches seek precise solutions but are computationally expensive [4] [8] [24].Source: §2 · pp. 2–3
3. Method
Alchemist rates text-image samples through a proxy-model meta-learning process, then prunes the rated data to retain an efficient subset. Its rating uses validation performance, while Shift-Gsample targets informative middle-to-late samples without restricting selection to a narrow segment.
3.1. Data Rating via Meta-Learning
- The rater learns continuous weights for training samples by minimizing the proxy model’s loss on a held-out validation set.This bilevel formulation treats each sample’s influence through its effect on validation performance.Source: §3.1 · p. 3
- Because solving the bilevel objective to convergence for every rater update is prohibitive, Alchemist unrolls a few training steps to approximate the meta-gradient.The reference model trained on training data approximates near-zero-loss parameters, simplifying gradient estimation; minibatches make the optimization practical.Source: §3.1 · pp. 3–4
- A Group MLP derives a batch weight from pooled-feature mean and variance, while an Instance MLP produces normalized per-sample weights; their product combines batch coherence with instance distinctiveness.The sample weights are normalized within each batch for training stability.Source: §3.1 · pp. 3–4
3.2. Data Pruning Strategy
- Top-ranked samples have low loss but little gradient change, while middle-to-late samples show more active gradients; rapid convergence can make Top-K selection overfit.Tail samples’ gradients hardly descend, and the highest-scoring region is therefore pruned.Source: §3.2 · pp. 4–5
- Shift-Gsample discards the top n% and samples the remainder with a shifted Gaussian, balancing informativeness and diversity.A narrow mid-to-late Block segment captures informative data but reduces diversity.Source: §3.2 · p. 5
- With equal 20% subsets (6M samples), Shift-Gsample achieves the best performance improvement among five pruning methods.The comparison includes Random, Top-K, Block, and Gsample.Source: §3.2 · p. 5
- Selected data focus on detailed yet learnable middle-to-late samples, excluding many plain head-region and chaotic tail-region images.The reported examples align with human intuition about which samples are informative.Source: §3.2 · p. 5, Figure 4 · p. 5
4. Experiments
Experiments compare Alchemist-selected subsets across selection baselines, model scales, data domains, and ablations. The reported results cover data efficiency, transfer, and training behavior.
4.1. Experimental Setup
- Experiments use LAION-30M, Flux-reason-6M, and HPDv3-2M, evaluate FID, CLIP-Score, and GenEval, and test STAR models and LoRA-tuned FLUX-mini.STAR models are trained from scratch; FLUX-mini is fine-tuned with LoRA.Source: §4.1 · pp. 5–6
4.2. Main Results
- 50% Alchemist-selected data achieves performance comparable to full-data training, while 20% matches 50% random sampling.Table 1 compares methods on MJHQ-30K and GenEval; the models are trained from scratch for three epochs.Source: §4.2 · p. 6, Table 1 · p. 7
- Across model scales and model families, Alchemist-selected data consistently outperforms random selection and transfers to STAR training and FLUX-mini LoRA fine-tuning.The comparison covers STAR-T2I-0.3B, STAR-T2I-0.9B, and FLUX-mini-3B.Source: §4.2 · p. 7, Table 2 · p. 7
- On HPDv3-2M and Flux-reason-6M, Alchemist outperforms the baseline at both 20% and 50% data retention.These datasets represent mixed real–synthetic preference pairs and synthetic reasoning-oriented data, respectively.Source: Table 3 · p. 7, §4.2 · p. 7
- 2.33× and 5× faster training time is reported for STAR-0.3B at 20% and 50% retention, respectively, while reaching performance comparable to random selection.The comparison tracks performance over training time on MJHQ-30K.Source: §4.2 · p. 7, Figure 6 · p. 8
4.3. Ablation Study
- Alchemist selection produces steadier, faster improvement than random sampling, with performance fluctuating near convergence at approximately 9–10 epochs.Figure 6 tracks models trained on 6M and 15M data.Source: Figure 6 · p. 8, §4.3 · p. 8
- Adding group-MLP collective context improves performance over evaluating samples individually, mitigating bias from minibatch gradient updates.The ablation compares the group perspective with individual sample evaluation.Source: §4.3 · pp. 7–8, Table 4 · p. 8
- Shift-Gsample performs best among the three pruning strategies; middle-to-late ranked samples show the largest gains and fastest gradient changes.The strategy is guided by the data rating.Source: §4.3 · p. 8, Table 4 · p. 8
5. Analysis
Alchemist’s ranking distinguishes visually clean but semantically limited images from informative mid-ranked samples and noisy tail samples. The analysis explains why rating and pruning must balance convergence speed with final performance.
5. Analysis
- The top-ranked 0–20% is often clean and plain-backgrounded but semantically limited, while 40–80% contains informative images with clear subjects and actions.The 90–100% tail is predominantly noisy, cluttered, or visually chaotic.Source: §5 · p. 8
- Images can contain up to 70% redundancy, so lower-loss, visually clean samples may provide little learning information despite accelerating convergence.The analysis reports that models focused on these samples can overfit and produce lower-quality outputs.Source: §5 · p. 8
- Alchemist uses ratings to estimate each sample’s influence on convergence speed, then prunes to balance convergence speed and final performance while retaining some easy or noisy examples for diversity.Source: §5 · p. 8
- The autoregressive proxy is chosen for single-step generation, enabling efficient gradient propagation and stable meta-optimization; the analysis says the modeling paradigm itself is not crucial.Diffusion intermediate-step losses can be high-noise and misleading even when associated with good visual quality.Source: §5 · p. 8
6. Conclusion
Alchemist is presented as an automatic, scalable, gradient-based data-selection method for real-world Text-to-Image training. Experiments show improvements in training efficiency and model performance.
6. Conclusion
- Alchemist is presented as the first automatic, scalable, gradient-based data-selection method specifically designed for real-world Text-to-Image training.Source: §6 · pp. 8–9
- Extensive experiments show that Alchemist consistently improves both training efficiency and model performance.The conclusion also says the method opens opportunities for larger text–image datasets.Source: §6 · p. 9
- The authors state that Alchemist establishes a new paradigm for meta-learning-based data curation in Text-to-Image development and opens opportunities for larger text–image datasets.Source: §6 · p. 9